A 3D multi-screen display method, system, device and medium for cloud desktops

Abstract

The present application provides a cloud desktop 3D multi-screen display method, system, device and medium. The method includes: creating a virtual graphics card in a server physical machine and configuring the GPU driver type of the server virtual machine, creating a general port and a plurality of streaming ports; the server virtual machine creates a plurality of streaming processing threads according to the number of streaming ports, and the streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing thread; the streaming processing thread sends the encoded bitstream to the dynamic library of the server physical machine through the streaming port, and sends it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display. The embodiments of the present application can broaden the selection range of the CPU and GPU and improve security.

Description

Technical Field

[0001] This application relates to the field of computer applications, and particularly to a cloud desktop 3D multi-screen display method, system, device, and medium. Background Art

[0002] The remote connection method of the cloud desktop of the linux server generally uses the SPICE (Simple Protocol for Independent Computing Environment) protocol. The multi-screen display method of the SPICE protocol generally uses the QXL virtual graphics card device, and the QXL virtual graphics card device is only applicable to the 2D scenario. Currently, all existing methods in the 3D scenario require GPU support, such as using the virtio-gpu method or the vhost-user-gpu method, all of which require GPU support. However, some GPUs have limited support for this technology or do not support it at all.

[0003] For cloud desktop scenarios that require 3D rendering, such as GIS applications, etc., physical GPUs generally need to be used in the virtual machine to process 3D rendering. If the cloud desktop has only one screen, the SPICE protocol can be used, and in the virtual machine, the spice-stream-agent is used to push the stream + GPU hard encoding method, and in the client, the spice-gtk + virt-viewer is used for decoding and display. The virtual machine uses the virtio virtual graphics card. However, the spice-stream-agent only supports single-screen streaming and does not support multi-screen streaming. Some currently published patents also involve cloud desktop screen display solutions, but most of them are only applicable to multi-screen display in the 2D scenario or single-screen display in the 3D scenario, and still cannot solve the multi-screen display problem in the 3D scenario. Summary of the Invention

[0004] To solve one of the above technical defects, a cloud desktop 3D multi-screen display method, system, device, and medium are provided in the embodiments of this application.

[0005] In the first aspect of the embodiments of this application, a cloud desktop 3D multi-screen display method is provided, and the method includes:

[0006] Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine;

[0007] When the server virtual machine starts, create a general port in the server physical machine, and create multiple stream ports according to the number of screens of the streaming media device connected by the client;

[0008] The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads.

[0009] The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the stream ports, and the dynamic library sends it to the client. The client processes the encoded bitstream and then performs 3D multi-screen display.

[0010] In the second aspect of the embodiments of the present application, a cloud desktop 3D multi-screen display system is provided. The system includes:

[0011] A client, which is used to connect to a streaming media device, output the display output size of the screen of the streaming media device, and perform 3D multi-screen display;

[0012] A server, including a server physical machine and a server virtual machine;

[0013] The server physical machine is used to create a virtual graphics card and configure the GPU driver type of the server virtual machine, and when the server virtual machine is started, create a general port and create multiple stream ports according to the number of screens of the streaming media device connected by the client;

[0014] The server virtual machine is used to create multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads. The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the stream ports, so that the dynamic library of the server physical machine sends the encoded bitstream to the client.

[0015] In the third aspect of the embodiments of the present application, an electronic device is provided, including: a processor and a memory;

[0016] Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device executes the method described in the first aspect of the embodiments of the present application.

[0017] In the fourth aspect of the embodiments of the present application, a computer storage medium is provided, including computer instructions. When the computer instructions run on an electronic device, the electronic device executes the method described in the first aspect of the embodiments of the present application.

[0018] By adopting the cloud desktop 3D multi-screen display method provided in the embodiments of the present application, through modifying and optimizing the streaming method in the server virtual machine, it can support GPU multi-screen splitting. Only one client needs to connect to the physical screens of multiple streaming devices to achieve multi-screen display of the server virtual machine in a 3D rendering scenario, reducing the number of clients and simplifying the cumbersome process of system wiring. It also has good practical application prospects in the in-vehicle mobile scenario. At the same time, the embodiments of the present application can broaden the selection of CPU and GPU types, and the security can be well satisfied. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions thereof are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0020] Figure 1 It is a flowchart of a cloud desktop 3D multi-screen display method described in Embodiment 1 of the present application;

[0021] Figure 2 It is a flowchart of another cloud desktop 3D multi-screen display method described in Embodiment 1 of the present application;

[0022] Figure 3 It is a schematic diagram of the principle of a cloud desktop 3D multi-screen display system described in Embodiment 1 of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] In order to make the technical solutions and advantages in the embodiments of the present application clearer, the following further details the exemplary embodiments of the present application with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than an exhaustive list of all embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. Embodiment 1

[0024] As Figure 1 shown, this embodiment proposes a cloud desktop 3D multi-screen display method, which specifically includes:

[0025] S101. Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine;

[0026] S102. When the server virtual machine starts, create a general port in the server physical machine and create multiple stream ports according to the number of screens of the streaming devices connected by the client.

[0027] Specifically, in this embodiment, the server includes two parts: a server physical machine and a server virtual machine. Among them, the SPICE protocol server libspice runs in the server physical machine. The SPICE protocol server libspice can be used as a dynamic library of QEMU (Quick EMUlator), which is collectively referred to as the dynamic library of the server physical machine hereinafter.

[0028] In this embodiment, the dynamic library of the server physical machine is modified to support creating connections for multiple streaming media devices. First, a virtual graphics card is created and the GPU driver type of the server virtual machine is configured. The virtual graphics card can be selected according to the actual situation. In this embodiment, the virtio type is taken as an example. The GPU driver type of the virtual machine can be selected as a passthrough GPU graphics card (GPU passthrough) or a sliced GPU (vGPU). This embodiment does not make special limitations.

[0029] When the server virtual machine starts, a general port spicevmc is created in the server physical machine, and multiple stream ports are also created. The number of stream ports can be determined according to the number of screens of the streaming media devices connected by the client. For example, if the number of screens of the streaming media devices connected by the client is two, that is, dual-screen display, then two stream ports need to be created.

[0030] S103. The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screens of the streaming media devices connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads.

[0031] Specifically, in this embodiment, two streaming components are installed in the server virtual machine. One streaming component is spice-stream-agent, and the other streaming component is spice-vdagent. These two streaming components can be configured to start automatically when the server virtual machine boots. Among them, the spice-stream-agent streaming component needs to be set to restart immediately after abnormal termination (i.e., Restart=always). In this embodiment, the spice-stream-agent streaming component is mainly used for multi-screen screen capture and GPU encoding. The spice-vdagent streaming component is mainly used for handling mouse events, shared clipboard, and changes in the physical screen resolution of the client.

[0032] After the spice-stream-agent streaming component is started, it will create multiple streaming processing threads according to the number of stream ports. In this embodiment, each stream port corresponds to a unique index number. Each streaming processing thread will obtain the display output size of the streaming media device screen connected by the client from the server virtual machine according to the index number of the corresponding stream port. The display output size includes, but is not limited to, data information related to the screen size such as the starting point coordinates, width, and height of the split screen. After obtaining the display output size, the streaming processing thread will send it to the GPU driver. The GPU driver will obtain the frame buffer data of the corresponding split screen, that is, the split screen image data, from the DRM (Direct Rendering Manager) according to the received display output size. After compressing and encoding the split screen image data, the GPU driver will return the encoded bitstream to the streaming processing thread. The streaming processing thread will then send the encoded bitstream out through the stream port.

[0033] It should be noted that in this embodiment, before the client sends a start message, the spice-stream-agent streaming component will only start the streaming processing thread, but will not actually start the streaming processing. When it is found that the size or position of the streaming media device screen connected by the client has changed, the spice-stream-agent streaming component will exit the current streaming processing and start a new streaming. This aspect will be described in detail in the subsequent description.

[0034] S104. The streaming processing thread sends the encoded bitstream to the dynamic library of the server physical machine through the stream port, and sends it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display.

[0035] Specifically, in this embodiment, after the streaming processing thread obtains the encoded bitstream, it will send the encoded bitstream to the dynamic library libspice of the server physical machine through the stream port. The dynamic library libspice of the server physical machine will then send the encoded bitstream to the client through the TCP protocol. The display images of different stream ports will carry different identification information. After processing, the client will display the corresponding desktop 3D images on different screens according to this identification information.

[0036] In this embodiment, by modifying and optimizing the streaming method in the server virtual machine, it can support GPU multi-screen splitting. Only one client needs to connect to the physical screens of multiple streaming media devices to achieve multi-screen display of the server virtual machine in a 3D rendering scenario, reducing the number of clients and simplifying the cumbersome process of system wiring. It also has good practical application prospects in in-vehicle mobile scenarios.

[0037] In some alternative embodiments, such as Figure 2 shown, the cloud desktop 3D multi-screen display method further includes:

[0038] S105. Obtain the mouse operation of the user on the streaming media device;

[0039] S106. Determine the mouse position according to the mouse operation;

[0040] S107. Respond to the mouse operation on the corresponding split screen according to the mouse position.

[0041] Specifically, when the user performs a mouse operation on the streaming media device connected to the client, the client responds to the mouse operation and generates mouse position information. This mouse position information is relative mouse position information. The client will encapsulate the information carrying the screen identification information and the mouse position information into a TCP message through the inputs-channel channel of the SPICE protocol and send it to the dynamic library libspice of the server physical machine. After being processed by the dynamic library libspice of the server physical machine, it is sent to the spice-vdagent streaming component in the server virtual machine through a common port for processing.

[0042] In the spice-vdagent streaming component, the starting position information of the split screen corresponding to the screen of the streaming media device operated by the user is obtained by querying according to the screen identification information. Then, the absolute position information of the mouse on the entire screen is obtained by converting according to the starting position information and the mouse coordinate information, and this event is written. In this way, the mouse on one of the streaming media device screens will be processed on the correct split screen, otherwise it will only be operated on the first screen displayed, thereby realizing that the mouse operation of multiple screens will be processed on the correct split screen.

[0043] In some alternative embodiments, the cloud desktop 3D multi-screen display method further includes: when the client is connected to the server virtual machine, the server virtual machine obtains and records the display output sizes of the multiple streaming media device screens connected by the client.

[0044] Specifically, when the client is started, the client will connect to the server virtual machine. At this time, the client will send a message containing the display output sizes of the multiple streaming media device screens connected to it to the spice-vdagent streaming component in the server virtual machine.

[0045] In some alternative embodiments, the cloud desktop 3D multi-screen display method further includes: when a new, deleted, or replaced streaming media device connected to the client occurs, modify the display output sizes recorded in the server virtual machine.

[0046] Specifically, when there are changes in the streaming media devices connected to the client, such changes include but are not limited to addition, deletion, or replacement. At this time, the message sent by the client to the spice-vdagent streaming component in the server virtual machine, which contains the display output sizes of the screens of multiple streaming media devices connected to it, will change. Once the server virtual machine detects such a change in the message, it will modify the recorded display output sizes.

[0047] In some optional embodiments, the cloud desktop 3D multi-screen display method further includes: the server virtual machine periodically obtains and records the resolutions of the screens of multiple streaming media devices connected to the client. If there are differences between the resolutions of the screens of multiple streaming media devices connected to the client obtained in the current acquisition period and those obtained in the previous acquisition period, the server virtual machine exits the current streaming processing thread and creates a new streaming processing thread.

[0048] Specifically, before each streaming by the spice-stream-agent streaming component in the server virtual machine, it checks whether the display output size of the screen of the streaming media device connected to the client has changed. Among them, when the resolution of the streaming media device screen changes, the spice-stream-agent streaming component will immediately stop and exit the current streaming, and then create a new streaming processing thread to start a new streaming with the new resolution. Embodiment 2

[0049] As Figure 3 shown, this embodiment proposes a cloud desktop 3D multi-screen display system, which includes:

[0050] A client, used to connect to a streaming media device, output the display output size of the streaming media device screen, and perform 3D multi-screen display;

[0051] A server, including a server physical machine and a server virtual machine;

[0052] The server physical machine is used to create a virtual graphics card and configure the GPU driver type of the server virtual machine, and when the server virtual machine starts, create a general port and create multiple stream ports according to the number of screens of the streaming media devices connected to the client;

[0053] A server virtual machine is used to create multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing thread. The streaming processing thread sends the encoded bitstream to the dynamic library of the server physical machine through the stream port, so that the dynamic library of the server physical machine sends the encoded bitstream to the client.

[0054] Specifically, the cloud desktop 3D multi-screen display system proposed in this embodiment mainly includes three parts: a client, a server physical machine, and a server virtual machine. The SPICE protocol server libspice runs on the server physical machine, and two streaming components run on the server virtual machine. One streaming component is spice-stream-agent, and the other streaming component is spice-vdagent. The spice-stream-agent streaming component is mainly used for multi-screen screen capture and GPU encoding. The spice-vdagent streaming component is mainly used for handling mouse events, shared clipboard, and changes in the physical screen resolution of the client. Among them, the specific working process can refer to the content recorded in Embodiment 1, and this embodiment will not be elaborated here. Embodiment 3

[0055] This embodiment proposes an electronic device, including: a processor and a memory;

[0056] Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device is caused to execute the following method:

[0057] Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine;

[0058] When the server virtual machine starts, create a general port in the server physical machine and create multiple stream ports according to the number of screens of the streaming media device connected by the client;

[0059] The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing thread;

[0060] The streaming processing thread sends the encoded bitstream to the dynamic library of the server physical machine through the stream port, and sends it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display. Embodiment 4

[0061] This embodiment provides a computer storage medium, including computer instructions. When the computer instructions run on an electronic device, the electronic device executes the method described as follows:

[0062] Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine;

[0063] When the server virtual machine starts, create a general port in the server physical machine, and create multiple stream ports according to the number of screens of the streaming media device connected by the client;

[0064] The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing thread sends the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing thread;

[0065] The streaming processing thread sends the encoded bitstream to the dynamic library of the server physical machine through the stream port, and sends it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display.

[0066] In this application, unless otherwise clearly defined and limited, terms such as "installation", "connection", "connection", "fixation" and other terms should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection, an electrical connection or can communicate with each other; it can be directly connected, or indirectly connected through an intermediate medium, and can be the connection inside two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific situations.

[0067] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0068] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A cloud desktop 3D multi-screen display method, characterized in that, The method includes: Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine; When the server virtual machine starts, create a general port in the server physical machine, and create multiple stream ports according to the number of screens of the streaming media device connected by the client; The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads; The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the stream ports, and send it to the client through the dynamic library. The client processes the encoded bitstream and performs 3D multi-screen display.

2. The method according to claim 1, characterized in that, The process of creating multiple streaming processing threads according to the number of stream ports, where the streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver, the GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads includes: Create multiple streaming processing threads according to the number of stream ports, and each stream port corresponds to an index number; Each streaming processing thread obtains the display output size of the screen of the streaming media device connected by the client from the server virtual machine according to the index number of the corresponding stream port, and sends the display output size to the GPU driver; The GPU driver obtains split-screen image data from the DRM according to the display output size and encodes the split-screen image data; The GPU returns the bitstream after encoding the split-screen image data to the streaming processing threads.

3. The method according to claim 1, wherein The method further includes: Obtain the mouse operation of the user on the streaming media device; Determine the mouse position according to the mouse operation; Respond to the mouse operation on the corresponding split screen according to the mouse position.

4. The method according to claim 3, characterized in that, The process of obtaining the mouse operation of the user on the streaming media device and determining the mouse position according to the mouse operation includes: The client responds to the mouse operation of the user on the streaming media device and generates mouse coordinate information; Send the screen identification information and mouse coordinate information of the split screen where the user operates to the dynamic library of the server physical machine; The dynamic library of the server physical machine sends the screen identification information and mouse coordinate information to the server virtual machine through the general port; The server virtual machine queries and obtains the starting position information of the split screen where the user operates through the screen identification information; The server virtual machine obtains the absolute position information of the mouse in the entire screen according to the starting position information and the mouse coordinate information.

5. The method according to claim 1, characterized in that The method further includes: when the client is connected to the server virtual machine, the server virtual machine obtains and records the display output sizes of multiple screens of the streaming media device connected by the client.

6. The method according to claim 5, wherein The method further includes: when there is an addition, deletion, or replacement of the streaming media device connected to the client, modify the display output sizes recorded in the server virtual machine.

7. The method according to claim 1, characterized in that The method further includes: The server virtual machine periodically obtains and records the resolutions of multiple screens of the streaming media device connected by the client; When there are differences between the resolutions of the screens of multiple streaming media devices connected by the client obtained in the current acquisition cycle and the resolutions of the screens of multiple streaming media devices connected by the client obtained in the previous acquisition cycle, the server virtual machine exits the current streaming processing thread and creates a new streaming processing thread.

8. A cloud desktop 3D multi-screen display system, characterized in that, It includes: A client, which is used to connect to a streaming media device, output the display output size of the screen of the streaming media device, and perform 3D multi-screen display; A server, which includes a server physical machine and a server virtual machine; The server physical machine is used to create a virtual graphics card and configure the GPU driver type of the server virtual machine, and when the server virtual machine starts, create a general port and create multiple streaming ports according to the number of screens of the streaming media devices connected by the client; The server virtual machine is used to create multiple streaming processing threads according to the number of streaming ports. The streaming processing threads send the display output size of the screens of the streaming media devices connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads. The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the streaming ports, so that the dynamic library of the server physical machine sends the encoded bitstream to the client.

9. An electronic device, characterized in that, It includes: A processor and a memory; Among them, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, It includes computer instructions. When the computer instructions run on an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images