Cloud desktop 3D multi-screen display method, system and device and medium

Abstract

The invention provides a cloud desktop 3D multi-screen display method, system and device and a medium. The method comprises the steps that a virtual graphics card is created in a server physical machine, a server virtual machine GPU drive type is configured, and a universal port and a plurality of stream ports are created; the server virtual machine creates a plurality of stream pushing processing threads according to the number of stream ports, the stream pushing processing threads send the display output size of a streaming media equipment screen connected with the client to the GPU driver, and the GPU driver obtains image data according to the display output size, encodes the image data and returns encoded code streams to the stream pushing processing threads; and the stream pushing processing thread sends the coded code stream to a dynamic library of a server physical machine through a stream port and sends the code stream to a client through the dynamic library, and the client processes the coded code stream and then performs 3D multi-screen display. According to the embodiment of the invention, the model selection range of the CPU and the GPU can be widened, and the safety is improved.

Description

Technical Field

[0001] This application relates to the field of computer applications, and particularly to a 3D multi-screen display method, system, device and medium for cloud desktops. Background Art

[0002] The remote connection method of the cloud desktop of the linux server generally uses the SPICE (Simple Protocol for Independent Computing Environment) protocol. The multi-screen display method of the SPICE protocol generally uses the QXL virtual graphics card device, and the QXL virtual graphics card device is only applicable to 2D scenarios. In 3D scenarios, the existing methods all require GPU support, such as using the virtio-gpu method or the vhost-user-gpu method, and these all require GPU support. However, some GPUs have limited support for this technology or do not support it at all.

[0003] For cloud desktop scenarios that require 3D rendering, such as GIS applications, etc., it is generally necessary to use a physical GPU in the virtual machine to process 3D rendering. If the cloud desktop has only one screen, the SPICE protocol can be used, and the spice-stream-agent push stream + GPU hard encoding method can be used in the virtual machine, and the spice-gtk + virt-viewer decoding display can be used on the client. The virtual machine uses the virtio virtual graphics card. However, spice-stream-agent only supports single-screen push stream and does not support multi-screen push stream. Some of the current public patents also involve cloud desktop screen display solutions, but most of them are only applicable to multi-screen display in 2D scenarios or single-screen display in 3D scenarios, and still cannot solve the multi-screen display problem in 3D scenarios. Summary of the Invention

[0004] To solve one of the above technical defects, the embodiments of this application provide a 3D multi-screen display method, system, device and medium for cloud desktops.

[0005] The first aspect of the embodiments of this application provides a 3D multi-screen display method for cloud desktops, and the method includes: Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine; When the server virtual machine starts, create a general port in the server physical machine, and create multiple stream ports according to the number of screens of the streaming media device connected by the client; The server virtual machine creates multiple streaming processing threads according to the number of streaming ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads. The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the streaming ports, and send it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display.

[0006] In the second aspect of the embodiments of the present application, a cloud desktop 3D multi-screen display system is provided. The system includes: A client, which is used to connect to a streaming media device, output the display output size of the screen of the streaming media device, and perform 3D multi-screen display. A server, including a server physical machine and a server virtual machine. The server physical machine is used to create a virtual graphics card and configure the GPU driver type of the server virtual machine, and when the server virtual machine is started, create a general port and create multiple streaming ports according to the number of screens of the streaming media device connected by the client. The server virtual machine is used to create multiple streaming processing threads according to the number of streaming ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads. The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the streaming ports, so that the dynamic library of the server physical machine sends the encoded bitstream to the client.

[0007] In the third aspect of the embodiments of the present application, an electronic device is provided, including: a processor and a memory; Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device executes the method described in the first aspect of the embodiments of the present application.

[0008] In the fourth aspect of the embodiments of the present application, a computer storage medium is provided, including computer instructions. When the computer instructions run on an electronic device, the electronic device executes the method described in the first aspect of the embodiments of the present application.

[0009] By adopting the cloud desktop 3D multi-screen display method provided in the embodiments of the present application, through modifying and optimizing the streaming method in the server virtual machine, it can support GPU multi-screen splitting. Just one client connecting to the physical screens of multiple streaming devices can achieve multi-screen display of the server virtual machine in a 3D rendering scenario, reducing the number of clients and simplifying the cumbersome process of system wiring. It also has good practical application prospects in in-vehicle mobile scenarios. At the same time, the embodiments of the present application can broaden the types of CPUs and GPUs selected, and the security can be well satisfied. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 It is a flowchart of a cloud desktop 3D multi-screen display method described in Embodiment 1 of the present application; Figure 2 It is a flowchart of another cloud desktop 3D multi-screen display method described in Embodiment 1 of the present application; Figure 3 It is a schematic diagram of the principle of a cloud desktop 3D multi-screen display system described in Embodiment 1 of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] In order to make the technical solutions and advantages in the embodiments of the present application clearer and more understandable, the following further details the exemplary embodiments of the present application with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than an exhaustive list of all embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. Embodiment 1

[0012] As Figure 1 shown, this embodiment proposes a cloud desktop 3D multi-screen display method, which specifically includes: S101. Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine; S102. When the server virtual machine starts, create a general port in the server physical machine and create multiple stream ports according to the number of screens of the streaming devices connected by the client.

[0013] Specifically, in this embodiment, the server includes two parts: a server physical machine and a server virtual machine. Among them, the SPICE protocol server libspice runs on the server physical machine. The SPICE protocol server libspice can be used as a dynamic library of QEMU (Quick EMUlator), which will be collectively referred to as the dynamic library of the server physical machine hereinafter.

[0014] In this embodiment, the dynamic library of the server physical machine is modified to support creating connections for multiple streaming media devices. First, a virtual graphics card is created and the GPU driver type of the server virtual machine is configured. The virtual graphics card can be selected according to the actual situation. In this embodiment, the virtio type is taken as an example. The GPU driver type of the virtual machine can be selected as a passthrough GPU graphics card (GPU passthrough) or using a sliced GPU (vGPU). This embodiment does not make special limitations.

[0015] When the server virtual machine starts, a general port spicevmc is created in the server physical machine, and multiple stream ports are also created. The number of the stream ports can be determined according to the number of screens of the streaming media devices connected by the client. For example, if the number of screens of the streaming media devices connected by the client is two, that is, dual-screen display, then two stream ports need to be created.

[0016] S103. The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screens of the streaming media devices connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads.

[0017] Specifically, in this embodiment, two streaming components are installed in the server virtual machine. One streaming component is spice-stream-agent, and the other streaming component is spice-vdagent. The two streaming components can be configured to start automatically when the server virtual machine boots. Among them, the spice-stream-agent streaming component needs to be set to restart immediately after abnormal stop (i.e., Restart=always). In this embodiment, the spice-stream-agent streaming component is mainly used for multi-screen screen capture and GPU encoding. The spice-vdagent streaming component is mainly used for handling mouse events, sharing the clipboard, and changes in the physical screen resolution of the client.

[0018] After the spice-stream-agent streaming component starts, it creates multiple streaming processing threads according to the number of stream ports. In this embodiment, each stream port corresponds to a unique index number. Each streaming processing thread obtains the display output size of the screen of the streaming media device connected by the client from the server virtual machine according to the index number of the corresponding stream port. The display output size includes, but is not limited to, data information related to the screen size such as the starting point coordinates, width, and height of the split screen. After obtaining the display output size, the streaming processing thread sends it to the GPU driver. The GPU driver obtains the frame buffer data of the corresponding split screen, that is, the split screen image data, from the DRM (Direct Rendering Manager) according to the received display output size. After compressing and encoding the split screen image data, the GPU driver returns the encoded bitstream to the streaming processing thread. The streaming processing thread then sends the encoded bitstream out through the stream port.

[0019] It should be noted that in this embodiment, before the client sends a start message, the spice-stream-agent streaming component only starts the streaming processing thread, but does not actually start the streaming processing. When it is found that the size or position of the screen of the streaming media device connected by the client changes, the spice-stream-agent streaming component will exit the current streaming processing and start a new streaming. This aspect will be described in detail in the subsequent description.

[0020] S104. The streaming processing thread sends the encoded bitstream to the dynamic library of the server physical machine through the stream port, and sends it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display.

[0021] Specifically, in this embodiment, after the streaming processing thread obtains the encoded bitstream, it sends the encoded bitstream to the dynamic library libspice of the server physical machine through the stream port. The dynamic library libspice of the server physical machine then sends the encoded bitstream to the client through the TCP protocol. The display images of different stream ports carry different identification information. After processing, the client will display the corresponding desktop 3D images on different screens according to this identification information.

[0022] In this embodiment, by modifying and optimizing the streaming method in the server virtual machine, it can support GPU multi-screen splitting. Only one client needs to connect to the physical screens of multiple streaming media devices to achieve multi-screen display of the server virtual machine in a 3D rendering scenario, reducing the number of clients and simplifying the cumbersome process of system wiring. It also has good practical application prospects in the in-vehicle mobile scenario.

[0023] In some alternative embodiments, such as Figure 2 shown, the 3D multi-screen display method for cloud desktops further includes: S105. Obtain the mouse operations of the user on the streaming media device; S106. Determine the mouse position according to the mouse operations; S107. Respond to the mouse operations on the corresponding split screen according to the mouse position.

[0024] Specifically, when the user performs mouse operations on the streaming media device connected to the client, the client responds to the mouse operations and generates mouse position information. This mouse position information is relative mouse position information. The client will encapsulate the information carrying the screen identification information and the mouse position information into a TCP message through the inputs-channel channel of the SPICE protocol and send it to the dynamic library libspice of the server physical machine. After being processed by the dynamic library libspice of the server physical machine, it is sent to the spice-vdagent streaming component in the server virtual machine through a general port for processing.

[0025] In the spice-vdagent streaming component, the starting position information of the split screen corresponding to the screen of the streaming media device operated by the user is obtained by querying according to the screen identification information. Then, the absolute position information of the mouse on the entire screen is obtained by converting according to the starting position information and the mouse coordinate information, and this event is written. In this way, the mouse on one of the streaming media device screens will be processed on the correct split screen, otherwise, the operation will only be performed on the first screen displayed, thereby realizing that the mouse operations on multiple screens will be processed on the correct split screens.

[0026] In some alternative embodiments, the 3D multi-screen display method for cloud desktops further includes: when the client is connected to the server virtual machine, the server virtual machine obtains and records the display output sizes of the multiple streaming media device screens connected by the client.

[0027] Specifically, when the client is started, the client will connect to the server virtual machine. At this time, the client will send a message containing the display output sizes of the multiple streaming media device screens connected to it to the spice-vdagent streaming component in the server virtual machine.

[0028] In some alternative embodiments, the 3D multi-screen display method for cloud desktops further includes: when there are additions, deletions, or replacements of the streaming media devices connected to the client, modify the display output sizes recorded in the server virtual machine.

[0029] Specifically, when there are changes to the streaming media devices connected to the client, such changes include, but are not limited to, addition, deletion, or replacement. At this time, the message sent by the client to the spice-vdagent streaming component in the server virtual machine, which contains the display output sizes of the screens of multiple streaming media devices connected to it, will change. Once the server virtual machine detects such a change in the message, it will modify the recorded display output sizes.

[0030] In some alternative embodiments, the cloud desktop 3D multi-screen display method further includes: the server virtual machine periodically obtains and records the resolutions of the screens of multiple streaming media devices connected to the client. If there are differences between the resolutions of the screens of multiple streaming media devices connected to the client obtained in the current acquisition period and those obtained in the previous acquisition period, the server virtual machine exits the current streaming processing thread and creates a new streaming processing thread.

[0031] Specifically, before each streaming by the spice-stream-agent streaming component in the server virtual machine, it checks whether there are changes in the display output size of the screen of the streaming media device connected to the client. Among them, when the resolution of the streaming media device screen changes, the spice-stream-agent streaming component immediately stops and exits the current streaming, and then creates a new streaming processing thread to start a new streaming with the new resolution. Embodiment 2

[0032] As Figure 3 shown, this embodiment proposes a cloud desktop 3D multi-screen display system, which includes: A client, used to connect to a streaming media device, output the display output size of the streaming media device screen, and perform 3D multi-screen display; A server, including a server physical machine and a server virtual machine; The server physical machine is used to create a virtual graphics card and configure the GPU driver type of the server virtual machine, and when the server virtual machine starts, create a general port and create multiple stream ports according to the number of screens of the streaming media devices connected to the client; The server virtual machine is used to create multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output sizes of the screens of the streaming media devices connected to the client to the GPU driver. The GPU driver obtains image data according to the display output sizes, encodes the image data, and returns the encoded bitstream to the streaming processing threads. The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the stream ports, so that the dynamic library of the server physical machine sends the encoded bitstream to the client.

[0033] Specifically, the cloud desktop 3D multi-screen display system proposed in this embodiment mainly includes three major parts: the client, the server physical machine, and the server virtual machine. The SPICE protocol server libspice runs on the server physical machine, and two streaming components run on the server virtual machine. One of the streaming components is spice-stream-agent, and the other streaming component is spice-vdagent. The spice-stream-agent streaming component is mainly used for multi-screen screen capture and GPU encoding. The spice-vdagent streaming component is mainly used for handling mouse events, shared clipboards, and changes in the physical screen resolution of the client. Among them, for the specific working process, reference can be made to the content recorded in Embodiment 1, and this embodiment will not be elaborated here. Embodiment 3

[0034] This embodiment provides an electronic device, including: a processor and a memory; Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device is caused to execute the following method: Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine; When the server virtual machine starts, create a general port in the server physical machine, and create multiple stream ports according to the number of screens of the streaming media device connected by the client; The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing thread; The streaming processing thread sends the encoded bitstream to the dynamic library of the server physical machine through the stream port, and sends it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display. Embodiment 4

[0035] This embodiment provides a computer storage medium, including computer instructions. When the computer instructions run on an electronic device, the electronic device executes the following method: Create a virtual graphics card in the server physical machine and configure the GPU driver type of the server virtual machine; When the server virtual machine starts, create a general port in the server physical machine, and create multiple stream ports according to the number of screens of the streaming media device connected by the client; The server virtual machine creates multiple streaming processing threads according to the number of stream ports. The streaming processing threads send the display output size of the screen of the streaming media device connected by the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded bitstream to the streaming processing threads. The streaming processing threads send the encoded bitstream to the dynamic library of the server physical machine through the stream ports, and send it to the client through the dynamic library. The client processes the encoded bitstream and then performs 3D multi-screen display.

[0036] In this application, unless otherwise clearly defined and limited, the terms "install", "connect", "connection", "fix" and other terms should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection, an electrical connection or can communicate with each other; it can be directly connected, or indirectly connected through an intermediate medium, and can be the communication inside two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific situations.

[0037] Although the preferred embodiments of this application have been described, once those skilled in the art learn the basic creative concept, additional changes and modifications can be made to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of this application.

[0038] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A cloud desktop 3D multi-screen display method, characterized in that: The method comprises: Create a virtual graphics card in the server physical machine and configure the server virtual machine GPU driver type; When the server virtual machine is started, a universal port is created in the server physical machine, and multiple streaming ports are created according to the number of streaming device screens connected to the client; The server virtual machine creates multiple streaming processing threads according to the number of streaming ports. The streaming processing threads send the display output size of the screen of the streaming media device connected to the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded code stream to the streaming processing thread. The push stream processing thread sends the encoded code stream to the dynamic library of the server physical machine through the stream port, and sends it to the client through the dynamic library. The client processes the encoded code stream and performs 3D multi-screen display.

2. The method according to claim 1, characterized in that The process of creating multiple streaming processing threads according to the number of stream ports, wherein the streaming processing thread sends the display output size of the screen of the streaming media device connected to the client to the GPU driver, and the GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded code stream to the streaming processing thread includes: Create multiple streaming processing threads according to the number of streaming ports, and each streaming port corresponds to an index number; Each streaming processing thread obtains the display output size of the screen of the streaming media device connected to the client from the server virtual machine according to the index number of the corresponding streaming port, and sends the display output size to the GPU driver; The GPU driver obtains the split-screen image data from the DRM according to the display output size, and encodes the split-screen image data; The GPU returns the encoded code stream of the split-screen image data to the streaming processing thread.

3. The method according to claim 1, characterized in that The method further comprises: Get the user's mouse operation on the streaming device; Determine the mouse position according to the mouse operation; The mouse operation is responded to on the corresponding split screen according to the mouse position.

4. The method according to claim 3, characterized in that The process of obtaining the mouse operation of the user on the streaming media device and determining the mouse position according to the mouse operation includes: The client responds to the user's mouse operation on the streaming device and generates mouse coordinate information; Sending the screen marking information and mouse coordinate information of the split screen operated by the user to the dynamic library of the server physical machine; The dynamic library of the server physical machine sends the screen marking information and mouse coordinate information to the server virtual machine through a universal port; The server virtual machine obtains the starting position information of the split screen operated by the user through the screen marking information query; The server virtual machine obtains the absolute position information of the mouse in the entire screen according to the starting position information and the mouse coordinate information.

5. The method according to claim 1, characterized in that The method further includes: when the client is connected to the server virtual machine, the server virtual machine acquires and records the display output sizes of the screens of the multiple streaming media devices connected to the client.

6. The method according to claim 5, characterized in that The method further comprises: when a streaming media device connected to the client is added, deleted or replaced, modifying the display output size recorded in the server virtual machine.

7. The method according to claim 1, characterized in that The method further comprises: The server virtual machine periodically obtains and records the screen resolutions of multiple streaming media devices connected to the client; If the resolutions of the screens of the multiple streaming media devices connected to the client acquired in the current acquisition cycle are different from the resolutions of the screens of the multiple streaming media devices connected to the client acquired in the previous acquisition cycle, the server virtual machine exits the current streaming processing thread and creates a new streaming processing thread.

8. A cloud desktop 3D multi-screen display system, characterized in that: include: The client is used to connect to the streaming device, output the display output size of the streaming device screen, and perform 3D multi-screen display; Servers, including physical servers and virtual servers; The server physical machine is used to create a virtual graphics card and configure the GPU driver type of the server virtual machine. When the server virtual machine is started, it creates a universal port and creates multiple streaming ports according to the number of streaming device screens connected to the client. The server virtual machine is used to create multiple streaming processing threads according to the number of stream ports. The streaming processing thread sends the display output size of the screen of the streaming media device connected to the client to the GPU driver. The GPU driver obtains image data according to the display output size, encodes the image data, and returns the encoded code stream to the streaming processing thread. The streaming processing thread sends the encoded code stream to the dynamic library of the server physical machine through the stream port, so that the dynamic library of the server physical machine sends the encoded code stream to the client.

9. An electronic device, characterized in that: include: Processor and memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device executes the method as claimed in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The method comprises computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images