Virtual desktop data transmission method and device, electronic equipment and storage medium
By obtaining the properties and screen content of the virtual desktop window object for video encoding, and adjusting the display accuracy according to the layer order, the problem of inaccurate judgment of static images and video modes in virtual desktop data transmission is solved, and bandwidth saving and user experience improvement is achieved .
Patent Information
- Application Number
- CN202310637310.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
The existing virtual desktop data transmission method is inaccurate in judgment between static images and video modes, resulting in waste of network bandwidth and reduced user experience, especially under weak network conditions.
By obtaining the attribute information and screen display content of each window object to be displayed on the virtual desktop, video encoding and target video stream are generated, and the display accuracy is adjusted according to the layer order of the window object to optimize network transmission.
Simplifies the virtual desktop data transmission logic, saves network bandwidth, and significantly improves user experience and system performance, especially under weak network conditions.
Smart Images

Figure CN119071534A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of virtual desktops, and in particular to a method, device, electronic device and storage medium for transmitting virtual desktop data. Background Art
[0002] At present, when transmitting data of virtual desktops, different transmission formats are provided based on different configurations of virtual desktop clients, such as QXL (which is a VDI (Virtual Desktop Infrastructure) interface) queue, or bitmap compression and video formats, etc. This is equivalent to using the virtual desktop client as a graphic terminal. The server transmits the data to be displayed on the virtual desktop in various formats to the virtual desktop client for display, and at the same time passes the input of the virtual desktop client to the server. The default display mode of the server system will encode the virtual desktop data to be displayed in two modes, static image and video, according to the frequency of change of the display content, and then transmit the video stream to the virtual desktop client for display, that is, the entire desktop display of the virtual desktop is processed as a single image or video frame.
[0003] There are many formats for virtual desktop input and output, including characters, graphics, audio, video, files, etc., corresponding to different transmission protocol logics, but the data contained in the entire screen of the virtual desktop has to be processed as a single image or video frame. However, the judgment of static images and dynamic videos in the screen display of the virtual desktop is not accurate. For example, when users quickly browse web pages or edit text, they will produce rapid screen display changes similar to video playback, which can easily be misjudged as video, thereby causing the server to select an inaccurate output encoding format.
[0004] When using static image mode for encoding output, the motion compensation mechanism in video encoding cannot be used, and it is difficult to eliminate temporal redundancy. For example, when editing text, 99% of the content of the two frames of images are the same, but the two frames of images need to be transmitted twice independently, resulting in a waste of bandwidth.
[0005] In addition, when encoding and outputting in video mode, if the virtual desktop currently has multiple windows, one of which is playing a video, and the other windows are static, such as documents and folders opened on the desktop, in this case, the data displayed on the entire virtual desktop screen must be encoded in video mode, resulting in a large bandwidth overhead, which will seriously affect the user experience, especially under weak network conditions. Summary of the invention
[0006] In order to solve the problem of wasting network bandwidth in the existing virtual desktop data transmission method, the embodiments of the present application provide a virtual desktop data transmission method, device, electronic device and storage medium.
[0007] In a first aspect, an embodiment of the present application provides a virtual desktop data transmission method, including:
[0008] Obtaining display content contained in a window object to be displayed of the virtual desktop, wherein the display content contained in the window object to be displayed is unobstructed content in the window object to be displayed;
[0009] Performing video encoding on the display content contained in the window object to be displayed to generate a first video stream;
[0010] Generate a target video stream based on the first video stream; and
[0011] The target video stream is sent to the virtual desktop client.
[0012] In a possible implementation, obtaining display content contained in a window object to be displayed on the virtual desktop specifically includes:
[0013] Acquire attribute information of each window object to be displayed on the virtual desktop, wherein the attribute information of the window object to be displayed includes position information of the window object to be displayed;
[0014] Obtain all data content and corresponding location information displayed on the virtual desktop screen; and
[0015] Based on the position information of each window object to be displayed and all data contents displayed on the virtual desktop screen and the corresponding position information, the display contents included in each window object to be displayed are determined.
[0016] In a possible implementation manner, performing video encoding on the display content included in the window object to be displayed to generate the first video stream specifically includes:
[0017] Performing video encoding on the display contents contained in each of the window objects to be displayed, respectively, to generate first video streams corresponding to each of the window objects to be displayed; and
[0018] Generating a target video stream based on the first video stream specifically includes:
[0019] A target video stream is generated based on each of the first video streams.
[0020] In a possible implementation manner, the attribute information of the window object to be displayed further includes layer sequence information of the window object to be displayed on the virtual desktop;
[0021] Before sending the target video stream to the virtual desktop client, the method further includes:
[0022] If it is determined that the sum of the encoding speeds for generating the first video streams is greater than the current network bandwidth for transmission to the virtual desktop client, or the network delay generated when generating the first video streams is greater than a preset threshold, the display accuracy corresponding to the first video streams is adjusted based on the layer order information of the window objects to be displayed on the virtual desktop.
[0023] In a possible implementation manner, adjusting the display accuracy corresponding to each first video stream based on the layer sequence information of each window object to be displayed on the virtual desktop specifically includes:
[0024] The display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop is set to be higher than the display accuracy of the first video streams corresponding to the window objects to be displayed at other layers.
[0025] In a possible implementation, setting the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers specifically includes:
[0026] The display accuracy of the first video stream corresponding to each window object to be displayed is set in descending order from the top layer to the bottom layer in the layer sequence of the virtual desktop.
[0027] In a possible implementation, setting the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers specifically includes:
[0028] Setting the display accuracy of the first video stream corresponding to the top-level window object to be displayed to a first preset display accuracy;
[0029] The display precision of the first video stream corresponding to the window objects to be displayed in the other layers is set to a second preset display precision, wherein the second preset display precision is lower than the first preset display precision.
[0030] In a possible implementation, setting the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers specifically includes:
[0031] Setting the display accuracy of the first video stream corresponding to the top-level window object to be displayed to a third preset display accuracy;
[0032] The display accuracy of the first video streams corresponding to the window objects to be displayed in the other layers is adjusted according to the display types of the window objects to be displayed in the other layers, wherein the display accuracy of the first video streams corresponding to the window objects to be displayed in the other layers is lower than the third preset display accuracy.
[0033] In a second aspect, an embodiment of the present application provides a virtual desktop data transmission device, including:
[0034] An acquisition unit, configured to acquire display content included in a window object to be displayed of a virtual desktop, wherein the display content included in the window object to be displayed is unobstructed content in the window object to be displayed;
[0035] An encoding unit, configured to perform video encoding on the display content contained in the window object to be displayed to generate a first video stream;
[0036] A generating unit, configured to generate a target video stream based on the first video stream;
[0037] The sending unit is used to send the target video stream to the virtual desktop client.
[0038] In a possible implementation, the acquisition unit is specifically used to acquire attribute information of each window object to be displayed on the virtual desktop, wherein the attribute information of the window object to be displayed includes position information of the window object to be displayed; acquire all data content displayed on the virtual desktop screen and the corresponding position information; and determine the display content contained in each window object to be displayed based on the position information of each window object to be displayed and all data content displayed on the virtual desktop screen and the corresponding position information.
[0039] In a possible implementation manner, the encoding unit is specifically configured to perform video encoding on display contents contained in each of the window objects to be displayed, respectively, to generate first video streams corresponding to each of the window objects to be displayed;
[0040] The generating unit is specifically configured to generate a target video stream based on each first video stream.
[0041] In a possible implementation manner, the attribute information of the window object to be displayed further includes layer sequence information of the window object to be displayed on the virtual desktop;
[0042] The device further comprises:
[0043] An adjustment unit is used to adjust the display accuracy corresponding to each of the first video streams based on the layer sequence information of each window object to be displayed on the virtual desktop before sending the target video stream to the virtual desktop client, if it is determined that the sum of the encoding speeds for generating the first video streams is greater than the current network bandwidth for transmission to the virtual desktop client, or the network delay generated when generating the first video streams is greater than a preset threshold.
[0044] In a possible implementation, the adjustment unit is specifically configured to set the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers.
[0045] In a possible implementation, the adjustment unit is specifically configured to set the display accuracy of the first video stream corresponding to each window object to be displayed in a decreasing order from the top layer to the bottom layer in the layer sequence of the virtual desktop.
[0046] In a possible implementation, the adjustment unit is specifically used to set the display accuracy of the first video stream corresponding to the window object to be displayed in the top layer to a first preset display accuracy; and set the display accuracy of the first video streams corresponding to the window objects to be displayed in other layers to a second preset display accuracy, wherein the second preset display accuracy is lower than the first preset display accuracy.
[0047] In a possible implementation, the adjustment unit is specifically used to set the display accuracy of the first video stream corresponding to the window object to be displayed in the top layer to a third preset display accuracy; and adjust the display accuracy of the first video stream corresponding to the window objects to be displayed in other layers according to the display types of the window objects to be displayed in other layers, wherein the display accuracy of the first video stream corresponding to the window objects to be displayed in other layers is lower than the third preset display accuracy.
[0048] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the virtual desktop data transmission method described in the present application is implemented.
[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the virtual desktop data transmission method described in the present application.
[0050] The beneficial effects of this application are as follows:
[0051] The embodiments of the present application provide a method, device, electronic device, and storage medium for transmitting virtual desktop data. The virtual desktop server obtains display content contained in a window object to be displayed of the virtual desktop. The display content contained in the window object to be displayed is the unobstructed content in the window object to be displayed. The display content contained in the window object to be displayed is video-encoded to generate a first video stream. A target video stream is generated based on the first video stream, and the target video stream is sent to the virtual desktop client. In this way, the data transmission of the virtual desktop is defined as a video stream composed of the display content of a group of window objects to be displayed on the virtual desktop. The display content of each window object to be displayed is encoded using a video format to obtain the first video stream corresponding to each of them. Then, the target video stream is generated according to each first video stream and transmitted to the virtual desktop client. Therefore, the virtual desktop server does not need to judge the static image mode and the video mode of the virtual desktop data to be displayed, and does not need to select different transmission protocol logics according to the format of the data. The virtual desktop data transmission logic is simplified, the network bandwidth is effectively saved, the user experience can be significantly improved under weak network conditions, and the overall system performance is significantly improved.
[0052] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0054] Figure 1 An example diagram of a virtual desktop screen display content provided in an embodiment of the present application;
[0055] Figure 2 A schematic diagram of an application scenario of the virtual desktop data transmission method provided in an embodiment of the present application;
[0056] Figure 3 A schematic diagram of the implementation process of the virtual desktop data transmission method provided in an embodiment of the present application;
[0057] Figure 4 A schematic diagram of an implementation flow of obtaining display content contained in a window object to be displayed on a virtual desktop provided in an embodiment of the present application;
[0058] Figure 5 A schematic diagram of the structure of a virtual desktop data transmission device provided in an embodiment of the present application;
[0059] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0060] At present, when transmitting data of virtual desktops, different transmission formats are provided based on different configurations of virtual desktop clients, such as QXL queues, or bitmap compression and video formats, etc. This is equivalent to using the virtual desktop client as a graphic terminal. The server transmits the data to be displayed on the virtual desktop in various formats to the virtual desktop client for display, and at the same time passes the input of the virtual desktop client to the server. The default display mode of the server system will encode the virtual desktop data to be displayed in two modes, static image and video, according to the frequency of change of the displayed content, and then transmit the video stream to the virtual desktop client for display, that is, the entire desktop display of the virtual desktop is processed as a single image or video frame.
[0061] There are many formats for virtual desktop input and output, including characters, graphics, audio, video, files, etc., corresponding to different transmission protocol logics, but the data contained in the entire screen of the virtual desktop has to be processed as a single image or video frame. However, the judgment of static images and dynamic videos in the screen display of the virtual desktop is not accurate. For example, when users quickly browse web pages or edit text, they will produce rapid screen display changes similar to video playback, which can easily be misjudged as video, thereby causing the server to select an inaccurate output encoding format.
[0062] When using static image mode for encoding output, the motion compensation mechanism in video encoding cannot be used, and it is difficult to eliminate temporal redundancy. For example, when editing text, 99% of the content of the two frames of images are the same, but the two frames of images need to be transmitted twice independently, resulting in a waste of bandwidth.
[0063] Furthermore, when encoding and outputting in video mode, if the virtual desktop currently has multiple windows, one of which is playing a video, and the other windows are static, such as documents and folders opened on the desktop, in this case, the data displayed on the entire virtual desktop screen must be encoded in video mode, resulting in a large bandwidth overhead. Figure 1As shown, it is an example diagram of the content displayed on a virtual desktop screen. The data in the current top-level window 101 of the virtual desktop 100 is the text 1 being edited, the data in the next-level window 102 is the video being played, the data in the third-level window 103 is the opened desktop folder, and the data in the bottom-level window 104 is the opened text 2. The data in window 102 is mostly blocked by the top-level window 101, and the data in window 103 is mostly blocked by windows 102 and 101. The virtual desktop 100 is not completely covered by the data of windows 101 to 104, but video encoding will also be performed on the data of the entire virtual desktop, which will result in a large waste of network bandwidth, which will seriously affect the user experience especially under weak network conditions.
[0064] In order to solve the above technical problems, embodiments of the present application provide a virtual desktop data transmission method, device, electronic device and storage medium.
[0065] The exemplary embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the exemplary embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.
[0066] First reference Figure 2 , which is a schematic diagram of an application scenario of the virtual desktop data transmission method provided in an embodiment of the present application, and may include a virtual desktop server 201 and a virtual desktop client 202. The virtual desktop is set on the virtual desktop client 202. The virtual desktop server 201 and the virtual desktop client 202 are connected through a network. The virtual desktop server 201 obtains the display content contained in the window object to be displayed of the virtual desktop from the virtual desktop client 202. The display content contained in the window object to be displayed is the unobstructed content in the window object to be displayed (for example, the display content not obstructed by other windows). The display content contained in the window object to be displayed is video-encoded to generate a first video stream, a target video stream is generated based on the first video stream, and the target video stream is sent to the virtual desktop client 202. The virtual desktop client 202 decodes the received target video stream and displays it.
[0067] Furthermore, in one embodiment of the present application, the data transmission of the virtual desktop is defined as a video stream composed of the display contents of a group of window objects to be currently displayed on the virtual desktop, the display contents of each window object to be displayed are encoded using a video format to obtain the first video streams corresponding to each of them, and then a target video stream is generated based on each first video stream and transmitted to the virtual desktop client. As a result, the virtual desktop server does not need to judge the static image mode and video mode of the virtual desktop data to be displayed, and does not need to select different transmission protocol logics according to the format of the data, which simplifies the virtual desktop data transmission logic, effectively saves network bandwidth, can significantly improve user experience under weak network conditions, and significantly improves the overall system performance.
[0068] Furthermore, in conventional technologies, when all data displayed on the entire virtual desktop screen is encoded in video mode, it becomes difficult to distinguish important display content and optimize it. Especially in weak network conditions, all display content is output with low precision, which seriously affects the user experience. Figure 1 At this time, the user of the virtual desktop client is concerned about the editing text window at the front end, that is, the top layer of the virtual desktop, and the display quality requirements for the videos, texts and folders in the following three windows are not so high. Based on this, the spatiotemporal hierarchical coding mechanism in the video standard can be combined with the network bandwidth and system performance conditions to adaptively adjust the output display accuracy of the video streams corresponding to each window object. Under the condition of limited network bandwidth, the display accuracy of the video stream corresponding to the window object that the user focuses on (for example, the top layer window of the virtual desktop) can be adjusted to meet the demand, and the display accuracy of the video streams corresponding to other layer windows behind the top layer window can be reduced. When the user switches to other windows, the display accuracy of the video stream corresponding to the window object is dynamically adjusted according to the object that the user focuses on. In this way, the important display content can be adaptively distinguished and optimized, thereby further saving the network bandwidth without causing too much impact on the user experience.
[0069] It should be noted that the virtual desktop in the embodiment of the present application can be but is not limited to a cloud desktop, and the embodiment of the present application is not limited to this.
[0070] Based on the above application scenarios, the following will refer to the attached Figures 3-4 The exemplary embodiments of the present application are described in more detail. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in any way. On the contrary, the implementation of the present application can be applied to any applicable scenario.
[0071] like Figure 3 As shown, it is a schematic diagram of the implementation process of the virtual desktop data transmission method provided in the embodiment of the present application, which can be applied to Figure 2The virtual desktop server 201 shown may include the following steps:
[0072] S31. Obtain display content included in the window object to be displayed of the virtual desktop, where the display content included in the window object to be displayed is unobstructed content in the window object to be displayed.
[0073] When implementing it specifically, you can follow the following steps: Figure 4 The process shown in the figure obtains the display content contained in the window object to be displayed on the virtual desktop, and may include the following steps:
[0074] S41: Acquire attribute information of each window object to be displayed on the virtual desktop, where the attribute information of the window object to be displayed includes position information of the window object to be displayed.
[0075] In specific implementation, the virtual desktop server obtains the attribute information of each window object to be displayed on the virtual desktop. The attribute information of the window object to be displayed includes the position information of the window object to be displayed. The position information of the window object to be displayed may include the position coordinate information of the window object to be displayed in the virtual desktop. In the embodiment of the present application, any vertex of the four vertices of the virtual desktop can be used as the origin, and the embodiment of the present application is not limited to this. The position coordinates of the window object to be displayed are the position coordinates relative to the origin. For example, in Figure 1 In the example in, the vertex of the lower left corner of the virtual desktop 100 can be used as the origin, and its coordinates are (0, 0). The window objects to be displayed include: window object 101, window object 102, window object 103 and window object 104. Then, the coordinates of the four vertices of window object 101 with the vertex of the lower left corner of the virtual desktop 100 as the origin can be obtained to determine the position of window object 101. Similarly, the positions of window objects 102, window objects 103 and window objects 104 can be obtained.
[0076] In another embodiment, the attribute information of the window object to be displayed may further include layer sequence information of the window object to be displayed on the virtual desktop.
[0077] Specifically, the virtual desktop server can call a preset window enumeration function in the virtual machine operating system Guest OS (Operating System, operating system) running thereon to enumerate all window objects to be displayed on the virtual desktop, and obtain the position coordinates of each window object to be displayed in the virtual desktop and the layer order information of each window object to be displayed in the virtual desktop. For example, when the virtual machine operating system uses the Windows operating system, the EnumWindows function in the system API (Application Programming Interface, application program interface) can be used to enumerate all windows on the virtual desktop. The enumeration order can be from the top window to the bottom window, wherein the top window is the window on the virtual desktop that is currently closest to the user, that is, the window farthest from the virtual desktop, and the bottom window is the window on the virtual desktop that is currently farthest from the user, that is, the window closest to the virtual desktop. All enumerated windows are all window objects to be displayed on the virtual desktop, and the position coordinates from the top window object to the bottom window object on the virtual desktop are obtained. During implementation, the virtual machine operating system is not limited to the Windows operating system, and any other operating system, such as the Linux operating system, may be used. When obtaining the attribute information of the window object to be displayed on the virtual desktop, no matter which type of virtual machine operating system is used, the virtual desktop server can obtain the window on the virtual desktop through the corresponding window enumeration function of the virtual machine operating system.
[0078] S42: Acquire all data contents displayed on the virtual desktop screen and corresponding location information.
[0079] In specific implementation, the virtual desktop server obtains all data contents displayed on the virtual desktop screen and the position coordinate information corresponding to all data contents. All data contents displayed on the virtual desktop screen include both the display contents of each window object and the display contents of the part of the screen desktop that is not blocked by the window object.
[0080] Still Figure 1For example, the top-level window object is the window 101 where the text 1 being edited is located at the front layer of the virtual desktop, which can be recorded as the first-level window object, and its display content is not blocked by other window objects. The second-level window object is the video window 102 being played at a layer behind the top-level window object, and its display content in the lower left corner is blocked by the top-level window object. The third-level window object is the desktop folder window 103 opened at a layer behind the second-level window object, and its display content in the lower left corner is blocked by the top-level window object and the second-level window object. The bottom-level window object (i.e., the fourth-level window object) is the window 104 where the text 2 is opened at a layer behind the third-level window object, and most of its right-side content is blocked by the top-level window object, the second-level window object, and the third-level window object. Then, the virtual desktop server obtains all the data content currently to be displayed on the virtual desktop screen and its corresponding position coordinates, including: Figure 1 All data to be displayed in the four layers of window objects currently visible and the corresponding position coordinates, and the data and corresponding position coordinates of other areas on the virtual desktop screen except these four layers of window objects. All data to be displayed in the four layers of window objects currently visible include: data to be displayed in the top-level window object and data to be displayed in unobstructed areas of window objects in other layers except the top-level window object, that is, there is no need to obtain data to be displayed in obstructed areas of the second layer window objects, the third layer window objects and the fourth layer window objects.
[0081] S43: Determine the display content included in each window object to be displayed based on the position information of each window object to be displayed and all data contents displayed on the virtual desktop screen and the corresponding position information.
[0082] In the specific implementation, all data contents and corresponding position coordinates displayed on the virtual desktop screen are matched with the position coordinates of each window object to be displayed, and the data contents displayed on the virtual desktop screen contained in each window object to be displayed are extracted respectively, that is, the display contents contained in each window object to be displayed are obtained.
[0083] Continuing with the above example, Figure 1 In the figure, the data content displayed on the virtual desktop screen contained in the top-level window object of the virtual desktop is: the data content to be displayed contained in the entire top-level window object: the data content of text 1, the data content displayed on the virtual desktop screen contained in the second-level window object of the virtual desktop is: the data content to be displayed in the unobstructed area of the second-level window object, the data content displayed on the virtual desktop screen contained in the third-level window object of the virtual desktop is: the data content to be displayed in the unobstructed area of the third-level window object, and the data content displayed on the virtual desktop screen contained in the bottom-level window object of the virtual desktop is: the data content to be displayed in the unobstructed area of the bottom-level window object.
[0084] S32. Perform video encoding on the display content included in the window object to be displayed to generate a first video stream.
[0085] In a specific implementation, the virtual desktop server performs video encoding on the display contents contained in each window object to be displayed, and generates first video streams corresponding to each window object to be displayed.
[0086] Specifically, the virtual desktop server may use a preset video encoding method to perform video encoding on the display content contained in each window object to be displayed, taking each window object as a unit, to generate a video stream corresponding to each window object to be displayed.
[0087] Specifically, the display content contained in each window object to be displayed can be video encoded based on, but not limited to, encoding methods such as MPEG-4 or MPEG-7, and the display content contained in each window object to be displayed can also be video encoded using encoding methods such as H.265 or AVS3, which is not limited in the embodiment of the present application. Since the user is mainly concerned with the foreground object in the image, and the static background changes relatively little, the background objects that are not focused on in the desktop display can be synthesized into a panoramic image, which is encoded and transmitted once at the beginning of the entire video stream. Subsequently, in each frame of the displayed image, only a small number of objects with changes need to be dynamically updated in the panoramic image. There are few changes in each frame, and most of the data updates are mainly on the foreground object, thereby obtaining better display quality. In this way, the bandwidth can be mainly used to transmit the encoding of the foreground object, thereby greatly reducing the amount of data encoded, decoded and transmitted. In the embodiment of the present application, the top-level window object of the virtual desktop can be regarded as the foreground, and other window objects except the top-level window object can be regarded as the background. Since the shapes of the foreground and the background are both regular rectangular windows, the processing logic is simpler. The goal of video standards such as MPEG-4 or MPEG-7 is to target narrowband transmission, high-quality compression, and the expression of the integration of natural objects and synthetic objects. They also have wide adaptability and scalability. A major feature of these video standards is object-oriented and user interaction. Since video objects can be natural landscapes or virtual objects synthesized by computers, each window object of the virtual desktop can be understood as one or a group of synthetic objects. In this way, the service provided by the virtual desktop server is a process of dynamic video synthesis and playback. The virtual desktop client can be used as a video player that supports interactive functions. The user's input can be regarded as customizing the interactive information. In some application scenarios, the virtual desktop client can also be made into a browser plug-in for direct use without installing client software. Based on the above video standards, the data transmission of the virtual desktop can be defined as a video stream composed of a group of window objects. Therefore, the output of the virtual desktop can be simplified and unified, and the user experience can be improved under weak network and weak terminal conditions.
[0088] The coding structure of reference MPGE-4 may include the following levels, as shown in Table 1:
[0089] Video-object Sequence (VS) Video Object (VO) Video Object (VOL) Group of VOPs (GOV) Video Object Plane (VOP)
[0090] Table 1
[0091] Video Object Sequence (VS): contains one or more Video Objects (VO) to form a complete MPEG-4 visual scene.
[0092] Video object (VO): in the embodiment of the present application, it refers to the display content contained in the window object to be displayed, and can also be a background in the form of a panoramic image.
[0093] Video Object Layer (VOL): In scalable coding, a video object can have one or more video object layers.
[0094] Group of Video Object Planes (GOV): A group of Video Object Planes (VOPs). This level is optional.
[0095] Video object plane (VOP): is a snapshot of a video object at a specific moment, reflecting the shape, texture and motion parameters of the object at that time, and several VOPs may overlap. In the embodiment of the present application, the video object plane is a snapshot of the display content contained in the window object to be displayed at a specific moment.
[0096] MPEG-4 encodes each video object plane (VOP) instead of each frame. This object-oriented method is also called VOP-based coding. MPEG-4 uses a motion compensation mechanism to eliminate temporal redundancy. For example, an intra-frame-coded VOP is called an I-VOP. When only forward prediction is used, an inter-frame-coded VOP is called a P-VOP. When bidirectional prediction is used, it is called a B-VOP. The encoding of I-VOP is a static image encoding, and motion compensation is required when encoding P-VOP and B-VOP. Motion compensation can be divided into three steps: motion estimation, motion compensation prediction, and prediction error coding. Each VOP can be divided into multiple macroblocks. The luminance element macroblock size and the chrominance element macroblock size can be set according to actual needs. For example, the luminance element macroblock size can be set to 16×16, and the chrominance element macroblock size can be set to 8×8. The embodiments of the present application do not limit this.
[0097] The embodiment of the present application may adopt mature video encoding methods, such as wavelet transform encoding, context-based adaptive binary arithmetic coding (CABAC), etc. to perform video encoding on the display content contained in each window object to be displayed, and the embodiment of the present application is not limited to this.
[0098] S33. Generate a target video stream based on the first video stream.
[0099] During specific implementation, the virtual desktop server may generate a target video stream based on each first video stream.
[0100] Specifically, the first video streams corresponding to the window objects to be displayed generated in step S12 can be used as sub-video streams, and the sub-video streams are packaged and combined into a target video stream.
[0101] In the embodiment of the present application, in order to further save network bandwidth, considering that the focus of users of the virtual desktop client is generally on the data content displayed in the window object at the top level of the virtual desktop, the display quality requirements for the data content displayed by the window objects in the following layers are not so high. Based on this, the network bandwidth and system performance conditions can be combined to adaptively adjust the output display accuracy of the video streams corresponding to each window object to be displayed under conditions of limited network bandwidth, such as weak network conditions (or terminal conditions). Under conditions of limited network bandwidth, the display accuracy of the video stream corresponding to the window object to be displayed that the user focuses on can be set to be higher than the display accuracy of the video streams corresponding to other window objects.
[0102] Specifically, before sending the target video stream to the virtual desktop client, the method further includes:
[0103] If it is determined that the sum of the encoding speeds for generating each first video stream is greater than the current network bandwidth for transmission to the virtual desktop client, or the network delay generated when generating each first video stream is greater than a preset threshold, the display accuracy corresponding to each first video stream is adjusted based on the layer order information of each window object to be displayed on the virtual desktop.
[0104] Specifically, if the virtual desktop server determines that the encoding speed of generating each video stream is greater than the current network bandwidth transmitted to the virtual desktop client, or the network delay generated when generating each first video stream is greater than a preset threshold, the display accuracy of the first video stream corresponding to the top-level window object to be displayed on the virtual desktop can be set to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed in other layers. Among them, the display accuracy index may include the resolution of the video stream and / or the number of frames per second, etc., which is not limited in the embodiments of the present application.
[0105] In one implementation, the virtual desktop server may set the display accuracy of the first video stream corresponding to each window object to be displayed in a decreasing order from the top layer to the bottom layer in the virtual desktop layer order.
[0106] For example, the resolution of the first video stream corresponding to the window object to be displayed on the top layer of the virtual desktop can be set to "ultra-high definition", the resolution of the first video stream corresponding to the window object to be displayed on the second layer can be set to "high definition", and the resolution of the first video stream corresponding to the window objects to be displayed on the third layer and subsequent layers can be set to "standard definition".
[0107] In one embodiment, the virtual desktop server may also set the display accuracy of the first video stream corresponding to the top-level window object to be displayed to a first preset display accuracy, and set the display accuracy of the first video streams corresponding to the window objects to be displayed in other layers to a second preset display accuracy, wherein the second preset display accuracy is lower than the first preset display accuracy.
[0108] For example, the resolution of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop can be set to "ultra-high definition" (or "high definition"), and the resolution of the first video streams corresponding to the window objects to be displayed in subsequent layers can be set to "standard definition".
[0109] In one embodiment, the virtual desktop server may further set the display accuracy of the first video stream corresponding to the top-level window object to be displayed to a third preset display accuracy, and adjust the display accuracy of the first video stream corresponding to the window objects to be displayed in other layers according to the display types of the window objects to be displayed in other layers, wherein the display accuracy of the first video stream corresponding to the window objects to be displayed in other layers is lower than the third preset display accuracy.
[0110] For example, the resolution of the first video stream corresponding to the top-level window object to be displayed can be set to "ultra-high definition". For the subsequent layers of window objects to be displayed, if the display type of the window object is video, the resolution of the first video stream corresponding to the window object can be set to "high definition". If the display type of the window object is text, the resolution of the first video stream corresponding to the window object can be set to "standard definition".
[0111] During implementation, the embodiments of the present application are not limited to the above-mentioned settings, and the resolution of the first video stream corresponding to each layer of window objects to be displayed can also be flexibly set according to actual needs.
[0112] S34. Send the target video stream to the virtual desktop client.
[0113] In a specific implementation, the virtual desktop server sends the target video stream to the virtual desktop client, so that the virtual desktop client decodes the target video stream and displays it.
[0114] Specifically, the virtual desktop server sends the target video stream to the virtual desktop client, and the virtual desktop client decodes the received target video stream to obtain decoded first video streams (ie, sub-video streams), and displays them according to the display accuracy of each first video stream.
[0115] In the virtual desktop data transmission method provided by the embodiment of the present application, the virtual desktop server obtains the display content contained in the window object to be displayed of the virtual desktop, the display content contained in the window object to be displayed is the unobstructed content in the window object to be displayed, performs video encoding on the display content contained in the window object to be displayed, generates a first video stream, generates a target video stream based on the first video stream, and sends the target video stream to the virtual desktop client. In this way, the data transmission of the virtual desktop is defined as a video stream composed of the display content of a group of window objects to be displayed on the virtual desktop, encodes the display content of each window object to be displayed using a video format to obtain the first video stream corresponding to each, and then generates a target video stream based on each first video stream and transmits it to the virtual desktop client. Therefore, the virtual desktop server does not need to judge the static image mode and video mode of the virtual desktop data to be displayed, and does not need to select different transmission protocol logics according to the format of the data, which simplifies the virtual desktop data transmission logic, effectively saves network bandwidth, can significantly improve user experience under weak network conditions, and significantly improves the overall performance of the system. Furthermore, the output display accuracy of the video streams corresponding to the various window objects can be adaptively adjusted using the spatiotemporal hierarchical coding mechanism in the video standard in combination with network bandwidth and system performance conditions. Under the condition of limited network bandwidth, the display accuracy of the video streams corresponding to the window objects that the user focuses on can be adjusted to meet the needs, and the display accuracy of the video streams corresponding to other layer windows behind the top-level window can be reduced. When the user switches to other windows, the display accuracy of the video streams corresponding to the window objects can be dynamically adjusted according to the objects that the user focuses on. In this way, important display content can be adaptively distinguished and optimized, thereby further saving network bandwidth without causing too much impact on the user experience.
[0116] Based on the same inventive concept, an embodiment of the present application also provides a virtual desktop data transmission device. Since the principle of solving the problem by the above-mentioned virtual desktop data transmission device is similar to that of the above-mentioned virtual desktop data transmission method, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0117] like Figure 5 As shown, it is a schematic diagram of the structure of the virtual desktop data transmission device provided by the embodiment of the present application, which can be applied to Figure 2 The virtual desktop server 201 shown may include:
[0118] An acquisition unit 51 is used to acquire display content included in a window object to be displayed of a virtual desktop, wherein the display content included in the window object to be displayed is unobstructed content in the window object to be displayed;
[0119] The encoding unit 52 is used to perform video encoding on the display content contained in the window object to be displayed to generate a first video stream;
[0120] A generating unit 53, configured to generate a target video stream based on the first video stream;
[0121] The sending unit 54 is configured to send the target video stream to the virtual desktop client.
[0122] In a possible implementation, the acquisition unit 51 is specifically used to acquire attribute information of each window object to be displayed on the virtual desktop, wherein the attribute information of the window object to be displayed includes position information of the window object to be displayed; acquire all data content displayed on the virtual desktop screen and the corresponding position information; and determine the display content contained in each window object to be displayed based on the position information of each window object to be displayed and all data content displayed on the virtual desktop screen and the corresponding position information.
[0123] In a possible implementation manner, the encoding unit 52 is specifically configured to perform video encoding on the display contents contained in each of the window objects to be displayed, respectively, to generate first video streams corresponding to each of the window objects to be displayed;
[0124] The generating unit 53 is specifically configured to generate a target video stream based on each first video stream.
[0125] In a possible implementation manner, the attribute information of the window object to be displayed further includes layer sequence information of the window object to be displayed on the virtual desktop;
[0126] The device further comprises:
[0127] An adjustment unit is used to adjust the display accuracy corresponding to each of the first video streams based on the layer sequence information of each window object to be displayed on the virtual desktop before sending the target video stream to the virtual desktop client, if it is determined that the sum of the encoding speeds for generating the first video streams is greater than the current network bandwidth for transmission to the virtual desktop client, or the network delay generated when generating the first video streams is greater than a preset threshold.
[0128] In a possible implementation, the adjustment unit is specifically configured to set the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers.
[0129] In a possible implementation, the adjustment unit is specifically configured to set the display accuracy of the first video stream corresponding to each window object to be displayed in a decreasing order from the top layer to the bottom layer in the layer sequence of the virtual desktop.
[0130] In a possible implementation, the adjustment unit is specifically used to set the display accuracy of the first video stream corresponding to the window object to be displayed in the top layer to a first preset display accuracy; and set the display accuracy of the first video streams corresponding to the window objects to be displayed in other layers to a second preset display accuracy, wherein the second preset display accuracy is lower than the first preset display accuracy.
[0131] In a possible implementation, the adjustment unit is specifically used to set the display accuracy of the first video stream corresponding to the window object to be displayed in the top layer to a third preset display accuracy; and adjust the display accuracy of the first video stream corresponding to the window objects to be displayed in other layers according to the display types of the window objects to be displayed in other layers, wherein the display accuracy of the first video stream corresponding to the window objects to be displayed in other layers is lower than the third preset display accuracy.
[0132] Based on the same technical concept, the present application embodiment also provides an electronic device 600, referring to Figure 6 As shown, the electronic device 600 is used to implement the virtual desktop data transmission method described in the above method embodiment. The electronic device 600 of this embodiment may include: a memory 601, a processor 602, and a computer program stored in the memory and executable on the processor, such as a virtual desktop data transmission program. When the processor executes the computer program, the steps in each of the above virtual desktop data transmission method embodiments are implemented.
[0133] The specific connection medium between the memory 601 and the processor 602 is not limited in the embodiment of the present application. Figure 6 In the embodiment, the memory 601 and the processor 602 are connected via a bus 603. The bus 603 is Figure 6 The connection between other components is shown by bold lines, which is only for schematic illustration and is not intended to be limiting. The bus 603 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0134] The memory 601 may be a volatile memory, such as a random-access memory (RAM); the memory 601 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 601 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 601 may be a combination of the above memories.
[0135] The processor 602 is used to implement the virtual desktop data transmission method provided in the embodiment of the present application.
[0136] An embodiment of the present application also provides a computer-readable storage medium that stores computer-executable instructions required to execute the above-mentioned processor, which includes a program required to execute the above-mentioned processor.
[0137] In some possible implementations, various aspects of the virtual desktop data transmission method provided by the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the virtual desktop data transmission method according to various exemplary implementations of the present application described above in this specification.
[0138] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0142] Although the exemplary embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the exemplary embodiments and all changes and modifications falling within the scope of the present application.
[0143] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A virtual desktop data transmission method, characterized in that: include: Acquire display content included in the window object to be displayed of the virtual desktop, where the display content included in the window object to be displayed is unobstructed content in the window object to be displayed; Performing video encoding on the display content contained in the window object to be displayed to generate a first video stream; Generate a target video stream based on the first video stream; and The target video stream is sent to the virtual desktop client.
2. The method according to claim 1, characterized in that Obtain the display content contained in the window object to be displayed on the virtual desktop, including: Acquire attribute information of each window object to be displayed on the virtual desktop, wherein the attribute information of the window object to be displayed includes position information of the window object to be displayed; Obtain all data content and corresponding location information displayed on the virtual desktop screen; and Based on the position information of each window object to be displayed and all data contents displayed on the virtual desktop screen and the corresponding position information, the display contents included in each window object to be displayed are determined.
3. The method according to claim 2, characterized in that Performing video encoding on the display content contained in the window object to be displayed to generate the first video stream specifically includes: Performing video encoding on the display contents contained in each of the window objects to be displayed, respectively, to generate first video streams corresponding to each of the window objects to be displayed; and Generating a target video stream based on the first video stream specifically includes: A target video stream is generated based on each of the first video streams.
4. The method according to claim 3, characterized in that The attribute information of the window object to be displayed further includes layer sequence information of the window object to be displayed on the virtual desktop; Before sending the target video stream to the virtual desktop client, the method further includes: If it is determined that the sum of the encoding speeds for generating the first video streams is greater than the current network bandwidth for transmission to the virtual desktop client, or the network delay generated when generating the first video streams is greater than a preset threshold, the display accuracy corresponding to the first video streams is adjusted based on the layer order information of the window objects to be displayed on the virtual desktop.
5. The method according to claim 4, characterized in that The adjusting the display accuracy corresponding to each first video stream based on the layer sequence information of each window object to be displayed on the virtual desktop specifically includes: The display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop is set to be higher than the display accuracy of the first video streams corresponding to the window objects to be displayed at other layers.
6. The method according to claim 5, characterized in that The step of setting the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers specifically includes: The display accuracy of the first video stream corresponding to each window object to be displayed is set in descending order from the top layer to the bottom layer in the layer sequence of the virtual desktop.
7. The method according to claim 5, characterized in that The step of setting the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers specifically includes: Setting the display accuracy of the first video stream corresponding to the top-level window object to be displayed to a first preset display accuracy; The display precision of the first video stream corresponding to the window objects to be displayed in the other layers is set to a second preset display precision, wherein the second preset display precision is lower than the first preset display precision.
8. The method according to claim 5, characterized in that The step of setting the display accuracy of the first video stream corresponding to the window object to be displayed at the top layer of the virtual desktop to be higher than the display accuracy of the first video stream corresponding to the window objects to be displayed at other layers specifically includes: Setting the display accuracy of the first video stream corresponding to the top-level window object to be displayed to a third preset display accuracy; The display accuracy of the first video streams corresponding to the window objects to be displayed in the other layers is adjusted according to the display types of the window objects to be displayed in the other layers, wherein the display accuracy of the first video streams corresponding to the window objects to be displayed in the other layers is lower than the third preset display accuracy.
9. A virtual desktop data transmission device, characterized in that: include: An acquisition unit, configured to acquire display content included in a window object to be displayed of a virtual desktop, wherein the display content included in the window object to be displayed is unobstructed content in the window object to be displayed; An encoding unit, configured to perform video encoding on the display content contained in the window object to be displayed to generate a first video stream; A generating unit, configured to generate a target video stream based on the first video stream; The sending unit is used to send the target video stream to the virtual desktop client.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the virtual desktop data transmission method according to any one of claims 1 to 8 is implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the virtual desktop data transmission method according to any one of claims 1 to 8 are implemented.
