Cloud desktop multi-screen display method and device, medium and equipment

Abstract

The embodiment of the invention provides a cloud desktop multi-screen display method and device, a medium and equipment, and the method comprises the steps: loading a virtual display card and a drive program in a virtual machine of a cloud server; when the virtual machine is initialized, declaring that the virtual graphics card supports N displays by means of a driving program, and triggering an operating system to create a corresponding number of logic displays; receiving physical display display mode information reported by the cloud desktop client and matching a logic display display mode according to the physical display display mode information; a frame data storage area is distributed for each logic display, data is written into the frame data storage area and an event notification is triggered when picture updating is monitored, picture data is read in response to the event notification, and a video stream is generated through coding and sent to a client, so that the video stream is transmitted to the client in a cloud desktop environment. The plurality of physical displays of the cloud desktop client can synchronously present the cloud picture matched with the local display mode, so that the requirement on multi-screen display in a cloud desktop scene is met, and the method does not depend on the version of an operating system and is high in universality.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cloud desktop, and particularly relates to a cloud desktop multi-screen display method, device, medium and equipment. BACKGROUND

[0002] With the popularity of remote office and hybrid office mode, the demand of users for cloud desktop multi-screen display is increasing. Especially in the financial, design, home entertainment and other scenarios, users need to handle multiple window tasks at the same time, and the single-screen cloud desktop is difficult to meet the demand of efficient parallel processing due to repeated stacking and frequent switching of windows, and high degree of operation complexity. Therefore, cloud desktop multi-screen display is a key link to improve the efficiency of multi-task parallel processing.

[0003] At present, the cloud desktop multi-screen display is generally realized by relying on the indirect display driver (IDD) mechanism of the Windows desktop bottom layer. That is, a virtual display is created through the IDD mechanism on the cloud desktop protocol server, and the virtual display and a virtual primary display natively supported by a system graphics card are configured as a dual-screen combination. Then, only the image data of the two virtual displays needs to be obtained and encoded into two independent video streams for transmission to the cloud desktop client, so as to realize the dual-screen display function. However, the IDD technology has strong dependence on the version of the Windows operating system, and has poor universality. SUMMARY

[0004] The embodiments of the application provide a cloud desktop multi-screen display method, device, medium and equipment, which can improve the universality of cloud desktop multi-screen display.

[0005] In a first aspect, the embodiments of the application provide a cloud desktop multi-screen display method, which comprises: loading a virtual graphics card and a driver program of the virtual graphics card in a virtual machine; In the process of initializing the virtual machine, the driver program is used to modify the driver interface parameter to N, so as to declare to the operating system that the virtual graphics card supports N displays, and trigger the operating system to create N logical displays, wherein N is a positive integer; receiving target display information reported by a cloud desktop client, and setting the display mode of N logical displays to be consistent with the display mode of physical displays according to the target display information, wherein the target display information comprises display mode information of N physical displays configured locally by the cloud desktop client; allocating frame data storage areas for each logical display by using the driver program, wherein the frame data storage area is used to store picture data of the corresponding logical display; When a picture update of the logical display is monitored, the updated target picture data is written into the corresponding frame data storage area by the driver, and an event notification of picture update is triggered; In response to the event notification, the target picture data is read from the frame data storage area, and a target video stream is generated by encoding; The target video stream is sent to the cloud desktop client, so that the cloud desktop client decodes and outputs the target video stream to the physical display.

[0006] In an implementable embodiment, target display information reported by the cloud desktop client is received, and display modes of N logical displays are set to be consistent with a display mode of a physical display according to the target display information, specifically including: In the process of initializing the driver, the display mode information supported by the virtual graphics card is obtained, and N adjustable mode information corresponding to the logical displays is added in the display mode information; Target display information reported by the cloud desktop client is received; In response to the target display information, the adjustable mode information of each logical display is modified to be consistent with the target display information by the driver, so that the display modes of the logical displays are consistent with the display configuration of the physical display.

[0007] In an implementable embodiment, the operating system is triggered to create N logical displays, specifically including: A virtual hot plug event is sent to the operating system by calling a hot plug event notification interface, to simulate the plug-and-play behavior of a physical multi-interface graphics card; When the operating system detects the virtual hot plug event and identifies the declaration of the virtual graphics card, a display adapter sub-device enumeration interface is called by the operating system to query the number of display output interfaces of the virtual graphics card; Based on the number of display output interfaces, N logical displays are created by the operating system.

[0008] In an implementable embodiment, when a picture update of the logical display is monitored, the updated target picture data is written into the corresponding frame data storage area by the driver, and an event notification of picture update is triggered, specifically including: An event object is allocated for each logical display, and the event object is used to represent the picture update state of the corresponding logical display; When a picture update of any of the logical displays is monitored, the updated target picture data is written into the frame data storage area corresponding to the logical display by a driver; after the writing of the target picture data is completed, the event object of the logical display is updated by the driver to trigger the event notification of the picture update of the logical display.

[0009] In an implementable embodiment, in response to the event notification, the target picture data is read from the frame data storage area and a target video stream is generated by encoding, specifically comprising: An encoding thread is started for the frame data storage area of each of the logical displays, and the encoding thread is blocked by an event waiting interface to wait for a picture update event signal of the corresponding logical display; When the encoding thread monitors that the picture update event signal of the corresponding logical display is triggered, the frame data storage area of the corresponding logical display is locked; The target picture data is read from the locked frame data storage area to an encoding buffer; The target picture data in the encoding buffer is encoded to generate the target video stream of the corresponding logical display.

[0010] In an implementable embodiment, the target video stream is sent to the cloud desktop client, specifically comprising: A sending thread is started for the target video stream of each of the logical displays, and the sending thread is bound to a picture transmission channel of the corresponding logical display; The target video stream is sent to the cloud desktop client by the sending threads.

[0011] In an implementable embodiment, in response to a display switching request initiated by the cloud desktop client between the physical displays, the association between each of the logical displays and each of the frame data storage areas is modified by the driver.

[0012] In a second aspect, an embodiment of the present application provides a cloud desktop multi-screen display device, and the device comprises: A loading module is configured to load a virtual graphics card and a driver of the virtual graphics card in a virtual machine; An initialization module is configured to modify a driver interface parameter to N by the driver during initialization of the virtual machine to declare to an operating system that the virtual graphics card supports N displays, and trigger the operating system to create N logical displays, wherein N is a positive integer; The display configuration module is used to receive target display information reported by the cloud desktop client, and set the display mode of the N logical displays to be consistent with the display mode of the physical displays according to the target display information. The target display information includes the display mode information of the N physical displays configured locally by the cloud desktop client. The memory allocation module is used to allocate a frame data storage area for each of the logical displays through the driver program. The frame data storage area is used to store the screen data of the corresponding logical display. The screen update module is used to, when a screen update of the logical display is detected, write the updated target screen data into the corresponding frame data storage area through the driver, and trigger a screen update event notification. The encoding input module is used to, in response to the event notification, read the target image data from the frame data storage area and encode it to generate a target video stream; The decoding output module is used to send the target video stream to the cloud desktop client, so that the cloud desktop client can decode the target video stream and output it to the physical display.

[0013] Thirdly, embodiments of this application provide a cloud desktop multi-screen display device, the device including: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement any of the above-mentioned cloud desktop multi-screen display methods.

[0014] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement any of the above-described cloud desktop multi-screen display methods.

[0015] Fifthly, the embodiments of this application provide a computer program product in which the instructions are executed by the processor of an electronic device, causing the electronic device to execute any of the above-mentioned cloud desktop multi-screen display methods.

[0016] This application discloses a cloud desktop multi-screen display method, apparatus, medium, and device. It loads a virtual graphics card and driver in a virtual machine on a cloud server. During virtual machine initialization, the driver declares that the virtual graphics card supports N displays and triggers the operating system to create the corresponding number of logical displays. It receives physical display mode information reported by the cloud desktop client and matches the logical display modes accordingly. It allocates frame data storage areas to each logical display, writes data to the frame data storage area when a screen update is detected, triggers an event notification, and then reads the screen data in response to the event notification, encodes it to generate a video stream, and sends it to the client. Thus, in a cloud desktop environment, multiple physical displays of the cloud desktop client can synchronously present cloud-based images adapted to local display modes, meeting the multi-screen display requirements in cloud desktop scenarios and exhibiting high universality regardless of operating system version.

[0017] In addition, during driver initialization, the driver obtains information on the display modes supported by the virtual graphics card and adds information on N adjustable modes corresponding to the logical displays. Then, when the target display information reported by the cloud desktop client is received, the driver can modify the adjustable mode information of each logical display to make it consistent with the target display information, thereby enabling support for non-standard resolutions in cloud desktop multi-screen display and improving the display flexibility of cloud desktop. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a cloud desktop multi-screen display method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the cloud desktop client reporting display mode information provided in this embodiment of the application; Figure 3 This is a schematic diagram of the target data stream being sent by the cloud server according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a cloud desktop multi-screen display device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a cloud desktop multi-screen display device provided in an embodiment of this application. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] With the increasing prevalence of remote and hybrid work models, users' demand for multi-screen display on cloud desktops is growing. Especially in scenarios such as finance, design, and home entertainment, users need to handle multiple window tasks simultaneously. Single-screen cloud desktops, due to the repeated stacking and frequent switching of windows, are cumbersome and cannot meet the needs of efficient parallel processing. Therefore, multi-screen display on cloud desktops is a key element in improving the efficiency of multi-tasking parallel processing.

[0023] Currently, multi-screen display on cloud desktops generally relies on the Indirect Display Driver (IDD) mechanism at the Windows desktop level. This involves creating a virtual monitor on the cloud desktop protocol server using the IDD mechanism, then configuring this virtual monitor and the system's natively supported virtual master monitor as a dual-screen setup. Afterward, the image data from both virtual monitors is acquired separately, encoded into two independent video streams, and transmitted to the cloud desktop client to achieve dual-screen display functionality. However, IDD technology is highly dependent on the Windows operating system version, resulting in poor versatility.

[0024] To address the problems in the prior art, this application provides a method, apparatus, device, and computer storage medium for multi-screen display of cloud desktops.

[0025] In practical applications, the execution subject of the cloud desktop multi-screen display method in this application embodiment can be a terminal device, such as a desktop computer or laptop computer, or a remote device similar to a server. Of course, this application embodiment can also adopt an execution subject in the form of software, such as a cloud desktop client or software program installed on a terminal device. The specific type of execution subject corresponding to the technical solution provided in this application embodiment is not strictly limited here, and can be flexibly selected according to the actual application scenario and actual needs.

[0026] The following describes specific embodiments of a cloud desktop multi-screen display method, apparatus, medium, and device provided in this application. First, a cloud desktop multi-screen display method is described.

[0027] Figure 1 This illustration shows a flowchart of a cloud desktop multi-screen display method according to an embodiment of this application. Figure 1 As shown, the method includes steps S100 to S106: S100: Load the virtual graphics card and its driver in the virtual machine.

[0028] In one or more embodiments of this application, in order to build a basic driver environment in a virtual machine that supports multi-screen display of cloud desktops, so as to enable the declaration of multi-monitor support and the creation of logical displays in subsequent steps, this application needs to load a virtual graphics card and its driver in the virtual machine.

[0029] Specifically, after the virtual machine starts and completes kernel initialization, it loads a preset virtual graphics card. Then, after the operating system detects the virtual graphics card, it automatically loads the corresponding driver to complete the registration in the operating system kernel, forming an interaction interface between the virtual graphics card and the operating system, and providing a communication foundation for the driver and the operating system in subsequent steps.

[0030] It should be noted that this application does not limit the specific type of virtual graphics card and its corresponding driver. The type can be set according to actual needs. For example, the virtual graphics card can be a Video Graphics Array (VGA) graphics card, and its corresponding driver can be a VGA driver. In this case, the driver will follow the VGA standard to complete initialization, providing driver-level support for subsequent multi-screen display configuration operations. This application does not limit the type of virtual machine, such as the Quick Emulator (QEMU), i.e., the QEMU virtual machine.

[0031] S101: During the initialization of the virtual machine, the driver interface parameter is modified to N through the driver program to declare to the operating system that the virtual graphics card supports N displays and to trigger the operating system to create N logical displays, where N is a positive integer.

[0032] In one or more embodiments of this application, in order for the virtual machine's operating system to recognize that the virtual graphics card has multi-monitor support capabilities, and thus achieve the matching of display modes between the logical display and the physical display in subsequent steps, this application needs to declare the virtual graphics card's support for multiple displays and create the logical display during the virtual machine initialization process.

[0033] Specifically, when the virtual machine finishes loading the virtual graphics card and driver and enters the initialization process, the driver will process the DisplayMiniport Driver Start Device Function (DxgkDdiStartDevice) in the operating system's display driver architecture. The driver interface parameter (Parameter of Number of Views, pNumberOfViews) used to indicate the number of supported displays in this function will be changed from its default value to N (e.g., if the cloud desktop client has two physical displays, this parameter can be set to 2). This declares to the operating system that the virtual graphics card supports N displays, thereby triggering the operating system to create N logical displays, where N is a positive integer.

[0034] It should be noted that, in this application, modifying the driver interface parameters simulates the hot-plugging behavior of a physical multi-interface graphics card, triggering the operating system's device enumeration mechanism. That is, the operating system calls the Display Miniport Driver Query Child Relations Function (DxgkDdiQueryChildRelations) to enumerate / query display adapter sub-devices. At this time, the driver adjusts the native logic of this interface: changing the logic from returning only one display adapter sub-device to returning N display adapter sub-devices, enabling the operating system to recognize N display adapter sub-devices and automatically generate N corresponding logical displays. Specifically, in one or more embodiments of this application, this application first sends a virtual hot-plug event to the operating system by calling the hot-plug event notification interface to simulate the plug-and-play behavior of a physical multi-interface graphics card. Secondly, when the operating system detects the virtual hot-plug event and recognizes the virtual graphics card declaration, it calls the display adapter sub-device enumeration interface to query the number of display output interfaces of the virtual graphics card. Finally, the operating system creates N logical displays based on the number of display output interfaces.

[0035] The display miniport driver's device startup function is a core interface in the Windows display driver model, used to complete device startup and capability declaration during virtual graphics card initialization. Driver interface parameters are the input and output parameters of the display miniport driver's device startup function. Modifications to the driver interface parameters and adjustments to the sub-device enumeration logic in this application are all completed at the virtual graphics card driver level, without relying on additional configuration of the virtual machine hardware emulation layer or a specific operating system version, thus exhibiting strong versatility. The created logical displays must meet the number requirements of the physical displays configured in the cloud desktop client, so that a screen transmission relationship can be established between the logical and physical displays in subsequent steps, satisfying the needs of multi-screen display in the cloud desktop. Furthermore, the number of logical displays and physical displays can be consistent to ensure a one-to-one correspondence between them, thereby improving the accuracy of multi-screen display.

[0036] S102: Receive target display information reported by the cloud desktop client, and set the display mode of the N logical displays to be consistent with the display mode of the physical displays according to the target display information, wherein the target display information includes the display mode information of the N physical displays configured locally by the cloud desktop client.

[0037] In one or more embodiments of this application, in order to generate the logical display screen data on the cloud server and display the screen on the physical display of the cloud desktop client in subsequent steps, this application needs to ensure that the logical display of the cloud server and the physical display of the cloud desktop client are matched in display mode in this step.

[0038] Specifically, when the cloud desktop client starts and detects N locally connected physical monitors, or when the display mode of the cloud desktop window changes on the physical monitors, the cloud desktop client collects the display mode information (such as resolution, color depth, etc.) of these N physical monitors. This display mode information is then reported as target display information to the cloud desktop service deployed on the cloud server via the network. Upon receiving the target display information, the cloud desktop service calls the display mode configuration interface provided by the virtual graphics card driver to adjust the display modes of the N logical monitors to match the display modes of the N physical monitors.

[0039] It should be noted that this application does not limit the timing of the cloud desktop client reporting target display information. It can be set according to actual needs, such as during the initial connection phase of the cloud desktop client, when a new physical display is connected during user use, or when the user adjusts the display mode of the physical display / cloud desktop window. In one or more embodiments of this application, this application can add dynamically adjustable mode information to support non-standard display modes (such as resolution), thereby improving the versatility of multi-screen cloud desktop display, as detailed below: First, this application can obtain all display mode information supported by the virtual graphics card during the driver initialization process, and add N adjustable mode information corresponding to the logical display to all display mode information; Secondly, receive the target display information reported by the cloud desktop client; Finally, in response to the target display information, the driver modifies the adjustable mode information of each logical display to match the target display information, so that the display mode of the logical display is consistent with the display configuration of the physical display.

[0040] When the cloud desktop client connects to the cloud server, it first detects and collects the display mode information (such as resolution) of all currently connected physical monitors and reports it to the cloud desktop service over the network. Upon receiving this information, the cloud desktop server calls the logical monitor display mode (such as resolution) setting interface to configure the display modes of the two logical monitors in the cloud to match the display modes of the physical monitors configured locally on the cloud desktop client. Taking a virtual VGA graphics card as an example, its VGA driver has added N dynamically configurable VESA BIOS Extension (VBE) display modes during initialization. Responding to the cloud desktop server's configuration commands, the cloud desktop client application modifies the adjustable mode information of the corresponding logical monitor using escape functions, ensuring a perfect match between the logical monitor's display mode and the physical monitor. Simultaneously, when the operating system calls back the Commit Video Present Network Function (CommitVidPn), the driver simulates writing the modified adjustable mode information into the virtual VGA's processing logic, ensuring the adjustable mode settings take effect. Among them, the data transmission between the cloud desktop client and the cloud server must comply with the cloud desktop protocol, and the drawing application is a graphics program such as a video browser configured on the cloud desktop.

[0041] Figure 2 This is a schematic diagram illustrating the reporting of display mode information by the cloud desktop client provided in an embodiment of this application. For example... Figure 2 As shown, the cloud desktop client 201 first collects the hardware-supported resolutions of each local physical monitor 202 (or responds to user manual adjustment operations), generates a configuration request containing display mode information of each physical monitor, and transmits it to the cloud desktop service 203 of the cloud virtual machine via the cloud desktop protocol. After receiving the configuration request, the cloud desktop service transmits the display mode information corresponding to each physical monitor to the driver 205 of the virtual graphics card 204 via the cloud desktop protocol. The driver adjusts the display mode (resolution attribute) of the corresponding logical monitor based on these parameters, so that the resolution of the logical monitor is precisely matched with the target resolution of the client's physical monitor. Subsequently, the kernel 206 of the Windows operating system controls the size of the image rendering performed by the drawing application 207 according to the resolution of the logical monitor, ensuring that the generated image data meets the target resolution requirements. The target video stream for subsequent encoding and transmission is also generated based on this resolution. Finally, after the cloud desktop client receives the target video stream, it completes the decoding and display according to the resolution matched to the local physical monitor, realizing real-time synchronization of multi-screen resolutions.

[0042] S103: The driver program allocates a frame data storage area to each of the logical displays, and the frame data storage area is used to store the screen data of the corresponding logical display.

[0043] In one or more embodiments of this application, in order to capture the updated screen data of each logical display in real time in subsequent steps, an independent data storage medium needs to be provided for each logical display. That is, in this step, this application needs to allocate an independent frame data storage area for each logical display through a driver program. This frame data storage area is used to store the screen data of the corresponding logical display.

[0044] Specifically, after the creation of logical displays and the matching of display modes between each logical display and the physical display are completed, the virtual graphics card driver will allocate an independent buffer in the virtual machine's memory space equal to the number of logical displays, as the frame data storage area for each logical display. For example, in a dual-screen scenario, two independent buffers are allocated, which serve as the frame data storage areas for the main and secondary logical displays, respectively.

[0045] It should be noted that this application sets up a shared memory and event notification mechanism (the shared memory is N frame data storage areas allocated by the driver, corresponding to N logical displays / indoor displays). That is, the virtual graphics card driver allocates N independent buffers in memory, each representing a frame data storage area for one of the N logical displays (physical displays). For example, in a dual-screen scenario, the EVENT_DISPLAY1_UPDATE event is used to monitor changes in the main logical display's screen, and the EVENT_DISPLAY2_UPDATE event is used to monitor changes in the secondary logical display's screen. This application does not limit the capacity of the frame data storage areas; it can be set according to actual needs, such as dynamically adapting to the display mode of the logical displays to ensure complete storage of single-frame image data and improve the effective utilization of memory resources.

[0046] S104: When a screen update of the logical display is detected, the updated target screen data is written to the corresponding frame data storage area through the driver, and a screen update event notification is triggered.

[0047] In one or more embodiments of this application, in order to obtain and encode the updated screen data of each logical display in a timely manner in subsequent steps, this application triggers an event notification after data storage upon detecting a screen update.

[0048] Specifically, the operating system monitors the screen changes of each logical display in real time. When the screen corresponding to any logical display is updated (such as when a window moves or content is refreshed), the virtual graphics card driver captures this update and writes the updated target screen data (including color data, pixel information, etc.) into the corresponding frame data storage area of ​​the logical display according to its display mode. After completing the writing of the target screen data, the driver triggers a screen update event notification bound to that logical display.

[0049] It should be noted that when the screen of any logical display is updated, the driver writes the updated target screen data to its corresponding frame data storage area and triggers the corresponding global event signal to notify the cloud desktop service of the screen update status. For example, in a dual-screen scenario, an update to the main screen triggers EVENT_DISPLAY1_UPDATE, and an update to the secondary screen triggers EVENT_DISPLAY2_UPDATE. That is, in one or more embodiments of this application, this application can assign an event object to each logical display, and the event object is used to represent the screen update status of its corresponding logical display; when a screen update of any logical display is detected, the driver writes the updated target screen data to the frame data storage area corresponding to that logical display; after the target screen data is written, the driver updates the event object of that logical display to trigger the event notification of the screen update of that logical display.

[0050] S105: In response to the event notification, read the target image data from the frame data storage area and encode it to generate a target video stream.

[0051] In one or more embodiments of this application, in order to transmit the screen data of each logical display to the cloud desktop client in the form of a video stream in subsequent steps, and to realize remote synchronous display of multiple screens, this application needs to respond to screen update event notifications, complete the reading and encoding processing of target screen data, and construct the key link from data storage to video stream generation.

[0052] Specifically, when the encoding module (cloud desktop service) receives an event notification of a screen update, it will locate the corresponding frame data storage area based on the logical display identifier corresponding to the event notification; read the updated target screen data from the frame data storage area; and then use an adapted encoding algorithm to compress the target screen data to generate a target video stream that can be used for transmission.

[0053] It should be noted that this application does not limit the specific encoding processing method, and can be set according to actual needs, such as NVIDIA Encoder (NVENC), Intel QuickSync Video (QSV), Advanced Video Coding (H.264), High Efficiency Video Coding (HEVC), etc. The monitoring of screen updates is completed collaboratively by the driver and the operating system to ensure real-time response to screen changes. In this application, a locking mechanism is used when writing target screen data to the frame data storage area to prevent incomplete content caused by the encoding module / program (cloud desktop service) reading the data during the writing process. Triggered event notifications are direct signals for the encoding module / program (cloud desktop service) to initiate data reading. Each event notification corresponds one-to-one with a specific logical display, ensuring that the encoding module can accurately locate the frame data storage area to be processed, ensuring the accuracy and efficiency of data reading. In one or more embodiments of this application, an independent encoding thread can be set for each logical display to ensure parallel encoding processing of the logical displays, specifically including: First, this application initiates an encoding thread for the frame data storage area of ​​each logical display. The encoding thread blocks and waits for the screen update event signal of its corresponding logical display through an event waiting interface. Second, when the encoding thread detects that the screen update event signal of its corresponding logical display has been triggered, it locks the frame data storage area of ​​its corresponding logical display; from the locked frame data storage area, it reads the target screen data into the encoding buffer. Finally, it encodes the target screen data in the encoding buffer to generate the target video stream of its corresponding logical display.

[0054] S106: Send the target video stream to the cloud desktop client so that the cloud desktop client can decode the target video stream and output it to the physical display.

[0055] In one or more embodiments of this application, in order for the cloud desktop client to receive and restore the updated screen of the cloud logical display and finally achieve synchronous display on the local physical display, it is necessary to accurately transmit the encoded target video stream to the cloud desktop client to complete the delivery of screen data from the cloud to the local device (cloud desktop multi-screen display).

[0056] Specifically, after the cloud server (encoding module) generates the target video stream corresponding to each logical display, it sends the target video stream to the cloud desktop client through the network transmission channel between the cloud desktop service and the cloud desktop client. After receiving the target video stream, the cloud desktop client starts the corresponding decoding process to decode the video stream into the original target screen data, and outputs the decoded screen data to the locally matched physical display according to the logical display identifier corresponding to the target video stream (e.g., the main screen video stream corresponds to the client's first physical display, and the secondary screen video stream corresponds to the second physical display).

[0057] It should be noted that the transmission of the target video stream must maintain a one-to-one correspondence with the logical display to ensure that the client can accurately identify the physical display corresponding to each video stream. In one or more embodiments of this application, the target video stream can be sent to the cloud desktop client, specifically including: starting a sending thread for the target video stream of each logical display, binding the sending thread to the screen transmission channel of its corresponding logical display; and sending the target video stream to the cloud desktop client through each sending thread. In this embodiment, this multi-threaded independent transmission design can avoid mutual blocking during multi-screen video stream transmission, improving transmission efficiency and screen synchronization. At the same time, the cloud desktop client receives and sends the target data streams separately (multi-channel) into a (multi-channel) decoding queue. The decoder (such as DXVA2 / VDPAU) decodes in parallel and outputs RGB frames to the corresponding physical video memory area (the display area of ​​the cloud desktop on the physical display). The rendering engine outputs the decoded image to the physical display corresponding to the logical display according to the identifier (screen ID) of the logical display, and maintains a layout consistent with the cloud desktop in the logical display (virtual machine) (such as extension, copy). In other words, the virtual machine is configured with a default virtual graphics card, and then a driver that internally implements the function of N displays is installed. This driver then manages the layout (such as setting the main and secondary screens, relative positions) and display modes of the N (physical / logical displays) screens. The cloud desktop service then obtains the screen data (such as cursor data) through the driver, compresses and encodes the screen data of the N screens into N video streams, and sends them to the cloud desktop client. After receiving the N video streams, the cloud desktop client decodes them and finally draws them onto the N physical displays of the terminal.

[0058] Figure 3 This is a schematic diagram illustrating the distribution of target data streams by a cloud server, as provided in an embodiment of this application. Figure 3As shown, in the virtual machine's operating system, after the graphics application 207 generates a screen update request, it passes the relevant instructions to the kernel 206. The kernel processes the instructions and then sends the screen update task to the driver 205 of the virtual graphics card 204. The driver generates updated target screen data for each of the N logical displays based on the virtual graphics card. Then, the cloud desktop service 203 performs parallel encoding processing through the cloud desktop protocol based on the frame data storage area corresponding to the N logical displays, and then transmits the encoded target video stream to the cloud desktop client via the network. After receiving the data, the cloud desktop client 201 passes it to the application, and finally the application outputs the screen content to the N physical displays 202.

[0059] In the aforementioned cloud desktop multi-screen display method, this application modifies the virtual graphics card driver interface parameters during the virtual machine initialization phase to declare multi-monitor support capabilities, triggering the operating system to create logical monitors with the same number of physical monitors as the cloud desktop client. By receiving display mode information reported by the cloud desktop client, the display modes of the logical monitors and physical monitors are precisely matched; by allocating independent frame data storage areas to each logical monitor and combining shared memory and event notification mechanisms, real-time capture and efficient transmission of screen update data are achieved. By configuring independent encoding and sending threads for each logical monitor, parallel encoding and independent transmission of multi-screen screen data are achieved, ultimately enabling the cloud desktop client to decode the received target video stream and output it to the corresponding local physical monitor. In other words, through collaborative design at the driver and application layers, this method breaks through the limitations of traditional single-screen display with virtual graphics cards, achieving a one-to-one mapping between cloud-based logical monitors and client-side physical monitors, and real-time synchronized display of multi-screen images. This effectively improves screen transmission efficiency and display consistency in cloud desktop multi-screen scenarios, reduces coupling with the virtualization platform and operating system, and offers strong versatility, low transmission latency, high efficiency, and strong compatibility and scalability.

[0060] Furthermore, in one or more embodiments of this application, the application can support flexible switching of physical displays by the cloud desktop client to improve the flexibility of multi-screen display and user experience, specifically as follows: When the cloud desktop client detects a user-initiated display switching request between physical displays (such as swapping or specifying the main screen and secondary screen), it will synchronize the display switching request to the cloud server. In response to the display switching request, the cloud server dynamically modifies the association between each logical display and each frame data storage area through the virtual graphics card driver (e.g., the frame data storage area originally bound to the main screen logical display is rebound to the secondary screen logical display, and vice versa). After this association is updated, the screen update data is written to the frame data storage area according to the new association, and subsequent encoding and transmission processes are also adapted and adjusted accordingly, ultimately enabling the client's physical display to present the switched screen content. This switching mechanism enables seamless switching across the entire display chain (e.g., in a dual-screen scenario, new main screen data is written to the secondary screen frame data storage area, and new secondary screen data is written to the main screen frame data storage area, while the rest of the chain remains unchanged. After receiving the data, the cloud desktop client draws the main screen data onto the secondary screen and the secondary screen data onto the main screen, thus achieving a display swap without needing to rebuild the connection). This enables dynamic adjustment of the multi-screen display layout, meeting diverse user operation needs.

[0061] Based on the aforementioned method for multi-screen display of a cloud desktop, this application also provides a specific embodiment of a cloud desktop multi-screen display device.

[0062] like Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a cloud desktop multi-screen display device provided in an embodiment of this application. The device 400 includes a loading module 401, an initialization module 402, a display configuration module 403, a memory allocation module 404, a screen update module 405, an encoding input module 406, and a decoding output module 407.

[0063] The loading module 401 is used to load a virtual graphics card and its driver in a virtual machine. The initialization module 402 is used to modify the driver interface parameter to N through the driver program during the virtual machine initialization process, so as to declare to the operating system that the virtual graphics card supports N displays and trigger the operating system to create N logical displays, where N is a positive integer; The display configuration module 403 is used to receive target display information reported by the cloud desktop client, and set the display mode of the N logical displays to be consistent with the display mode of the physical displays according to the target display information, wherein the target display information includes the display mode information of the N physical displays configured locally by the cloud desktop client; The memory allocation module 404 is used to allocate a frame data storage area for each of the logical displays through the driver program. The frame data storage area is used to store the screen data of the corresponding logical display. The screen update module 405 is used to, when a screen update of the logical display is detected, write the updated target screen data into the corresponding frame data storage area through the driver program, and trigger a screen update event notification. The encoding input module 406 is used to, in response to the event notification, read the target image data from the frame data storage area and encode it to generate a target video stream; The decoding output module 407 is used to send the target video stream to the cloud desktop client, so that the cloud desktop client decodes the target video stream and outputs it to the physical display.

[0064] In one feasible implementation, the display configuration module is specifically used to: obtain the display mode information supported by the virtual graphics card during the driver initialization process; add N adjustable mode information corresponding to the logical display to the display mode information; receive target display information reported by the cloud desktop client; and, in response to the target display information, modify the adjustable mode information of each logical display to be consistent with the target display information through the driver, so that the display mode of the logical display is consistent with the display configuration of the physical display.

[0065] In one feasible implementation, the initialization module is specifically used to send a virtual hot-plug event to the operating system by calling the hot-plug event notification interface to simulate the plug-and-play behavior of a physical multi-interface graphics card; when the operating system detects the virtual hot-plug event and recognizes the declaration of the virtual graphics card, the operating system calls the display adapter sub-device enumeration interface to query the number of display output interfaces of the virtual graphics card; and the operating system creates N logical displays based on the number of display output interfaces.

[0066] In one feasible implementation, the screen update module is specifically used to assign an event object to each of the logical displays, the event object representing the screen update status of the corresponding logical display; when a screen update of any logical display is detected, the updated target screen data is written to the frame data storage area corresponding to the logical display through the driver; after the target screen data is written, the event object of the logical display is updated through the driver to trigger an event notification of the screen update of the logical display.

[0067] In one feasible implementation, the encoding input module is specifically used to start an encoding thread for the frame data storage area of ​​each logical display. The encoding thread blocks and waits for the screen update event signal of its corresponding logical display through an event waiting interface. When the encoding thread detects that the screen update event signal of its corresponding logical display has been triggered, it locks the frame data storage area of ​​its corresponding logical display. It reads the target screen data from the locked frame data storage area into the encoding buffer. It encodes the target screen data in the encoding buffer to generate the target video stream corresponding to the logical display.

[0068] In one feasible implementation, the decoding output module is specifically used to start a sending thread for the target video stream of each of the logical displays, and the sending thread is bound to the screen transmission channel of its corresponding logical display; the target video stream is sent to the cloud desktop client through each of the sending threads.

[0069] In one feasible implementation, the decoding output module is specifically used to respond to the display switching request initiated by the cloud desktop client between the physical displays by modifying the association between the logical displays and the frame data storage areas through the driver.

[0070] Figure 5 This illustration shows a hardware structure diagram of a cloud desktop multi-screen display device provided in an embodiment of this application.

[0071] A cloud desktop multi-screen display device may include a processor 501 and a memory 502 storing computer program instructions.

[0072] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0073] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one embodiment, memory 502 may include removable or non-removable (or fixed) media, or memory 502 may be non-volatile solid-state memory. Memory 502 may be internal or external to the integrated gateway disaster recovery device.

[0074] In one instance, memory 502 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0075] Memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0076] The processor 501 reads and executes computer program instructions stored in the memory 502 to achieve... Figure 1 The cloud desktop multi-screen display method in the illustrated embodiment.

[0077] In one example, a cloud desktop multi-screen display device may further include a communication interface 503 and a bus 504. Wherein, as Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 504 and complete communication with each other.

[0078] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0079] Bus 504 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 504 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0080] Furthermore, in conjunction with the cloud desktop multi-screen display method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the cloud desktop multi-screen display methods in the above embodiments.

[0081] This application also provides a computer program product, including a computer program, which, when executed, implements any of the cloud desktop multi-screen display methods described in the above embodiments.

[0082] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0083] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0084] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0085] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0086] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for multi-screen display of a cloud desktop, characterized in that, Applications in cloud servers, including: Load the virtual graphics card and its driver into the virtual machine; During the virtual machine initialization process, the driver interface parameter is modified to N through the driver program to declare to the operating system that the virtual graphics card supports N displays and to trigger the operating system to create N logical displays, where N is a positive integer; Receive target display information reported by the cloud desktop client, and set the display mode of the N logical displays to be consistent with the display mode of the physical displays according to the target display information, wherein the target display information includes the display mode information of the N physical displays configured locally by the cloud desktop client; The driver program allocates a frame data storage area to each of the logical displays, and the frame data storage area is used to store the screen data of the corresponding logical display. When an update to the screen of the logical display is detected, the updated target screen data is written to the corresponding frame data storage area through the driver, and a screen update event notification is triggered. In response to the event notification, the target frame data is read from the frame data storage area and encoded to generate a target video stream; The target video stream is sent to the cloud desktop client so that the cloud desktop client can decode the target video stream and output it to the physical display.

2. The method according to claim 1, characterized in that, Receive target display information reported by the cloud desktop client, and set the display mode of the N logical displays to be consistent with the display mode of the physical displays according to the target display information, specifically including: During the driver initialization process, the display mode information supported by the virtual graphics card is obtained, and N adjustable mode information corresponding to the logical display is added to the display mode information. Receive target display information reported by the cloud desktop client; In response to the target display information, the driver modifies the adjustable mode information of each logical display to match the target display information, so that the display mode of the logical display matches the display configuration of the physical display.

3. The method according to claim 1, characterized in that, Triggering the operating system to create N logical displays specifically includes: By calling the hot-plug event notification interface, a virtual hot-plug event is sent to the operating system to simulate the plug-and-play behavior of a physical multi-interface graphics card. When the operating system detects a virtual hot-plug event and recognizes the declaration of the virtual graphics card, it calls the display adapter sub-device enumeration interface to query the number of display output interfaces of the virtual graphics card. Using the operating system, N logical displays are created based on the number of display output interfaces.

4. The method according to claim 1, characterized in that, When a screen update is detected on the logical display, the updated target screen data is written to the corresponding frame data storage area via the driver, and a screen update event notification is triggered, specifically including: Each of the logical displays is assigned an event object, which is used to represent the screen update status of the corresponding logical display; When an update to the screen of any of the logical displays is detected, the updated target screen data is written to the frame data storage area corresponding to the logical display through the driver. After the target screen data is written, the event object of the logical display is updated through the driver to trigger an event notification of the screen update of the logical display.

5. The method according to claim 1, characterized in that, In response to the event notification, the target frame data is read from the frame data storage area and encoded to generate a target video stream, specifically including: An encoding thread is started in the frame data storage area of ​​each logical display. The encoding thread blocks and waits for the screen update event signal of its corresponding logical display through the event waiting interface. When the encoding thread detects that the screen update event signal of its corresponding logical display is triggered, it locks the frame data storage area of ​​its corresponding logical display. Read the target image data from the locked frame data storage area into the encoding buffer; The target image data in the encoding buffer is encoded to generate the target video stream corresponding to the logical display.

6. The method according to claim 5, characterized in that, Sending the target video stream to the cloud desktop client specifically includes: A sending thread is started for the target video stream of each of the logical displays, and the sending thread is bound to the screen transmission channel of the corresponding logical display. The target video stream is sent to the cloud desktop client via each of the aforementioned sending threads.

7. The method according to claim 1, characterized in that, The method further includes: In response to the cloud desktop client initiating a display switching request between the physical displays, the driver modifies the association between the logical displays and the frame data storage areas.

8. A cloud desktop multi-screen display device, characterized in that, include: The loading module is used to load the virtual graphics card and its driver into the virtual machine. The initialization module is used to modify the driver interface parameter to N through the driver program during the virtual machine initialization process, so as to declare to the operating system that the virtual graphics card supports N displays and trigger the operating system to create N logical displays, where N is a positive integer; The display configuration module is used to receive target display information reported by the cloud desktop client, and set the display mode of the N logical displays to be consistent with the display mode of the physical displays according to the target display information. The target display information includes the display mode information of the N physical displays configured locally by the cloud desktop client. The memory allocation module is used to allocate a frame data storage area for each of the logical displays through the driver program. The frame data storage area is used to store the screen data of the corresponding logical display. The screen update module is used to, when a screen update of the logical display is detected, write the updated target screen data into the corresponding frame data storage area through the driver, and trigger a screen update event notification. The encoding input module is used to, in response to the event notification, read the target image data from the frame data storage area and encode it to generate a target video stream; The decoding output module is used to send the target video stream to the cloud desktop client, so that the cloud desktop client can decode the target video stream and output it to the physical display.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the cloud desktop multi-screen display method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the cloud desktop multi-screen display method as described in any one of claims 1-7.

Images