A desktop sharing and transmission method based on composite layer structure

Abstract

The present invention discloses a desktop sharing and transmission method based on a composite layer structure, which relates to the field of desktop sharing processing technology and aims to solve the problem that existing technical solutions always merge multiple layers in the operating system into a single layer for transmission, without taking into account the characteristics of the desktop sharing scenario and some characteristics of the graphics system. The key points of the technical solution are S1: images are distinguished by layers in the desktop; S2: the transformation relationship between layers needs to be synchronized in real time between the sharing end and the shared end; S3: before being hung in the display tree, the new tiles need to be compared one by one with the existing tiles on the display tree; S4: each layer also needs to maintain a larger display tree, and this tree structure needs to be synchronized in real time between the content sharing side and the shared side. The content sharing end and the content receiving end use the larger display tree to make judgments and obtain the final image for output. This reduces unnecessary drawing and significantly improves performance.

Description

Technical Field

[0001] The present invention relates to the technical field of desktop sharing processing, and in particular to a desktop sharing transmission method based on a composite layer structure. Background Art

[0002] Over the past decade, with the rapid development of cloud computing, the demand for real-time sharing and forwarding of operating system graphical interfaces has become increasingly strong. The rapid generation and distribution of screen content has become one of the popular transmission technologies. Screen content sharing has the characteristics of multiple distributions from a single source. How to share the screen content of content producers with the audience under low latency and low bandwidth is the key point of this technical research. Reducing the delay of distribution transmission can significantly improve the interactivity and ease of sharing, bringing a good user experience. Reducing the transmission bandwidth can exponentially reduce the network pressure of distribution, improve distribution efficiency, and reduce bandwidth costs. Finding an efficient and reliable screen sharing technology can greatly improve the user experience of multiple application scenarios such as desktop sharing / video conferencing / wireless screen projection protocol / remote assistance.

[0003] The current desktop sharing technology mainly focuses on the image content itself. The main approach is to input the image stream of the so-called source of the desktop screenshot into the encoder, determine the difference area through comparison to form the area to be encoded, encode the encoded area using a standard encoder, and then distribute the output code stream to each user through the network.

[0004] The above-mentioned existing technical solutions have the following defects: the existing technical solutions always merge multiple layers in the operating system into a single layer for transmission, without taking into account the characteristics of the desktop sharing scenario and some characteristics of the graphics system, such as some laws and rules of the graphics system, the human-computer interaction method, etc., resulting in some common and simple interactive scenarios having relatively large scene switches from the perspective of image content, a sharp increase in encoding costs, and a significant increase in transmission bandwidth. Summary of the Invention

[0005] The purpose of the present invention is to provide a desktop sharing and transmission method based on a composite layer structure that realizes a fundamental transformation from image data stream to instruction stream, improves the compression ratio by hundreds or thousands of times, achieves extreme bandwidth, and greatly enhances transmission efficiency.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A desktop sharing and transmission method based on a composite layer structure is as follows:

[0008] S1: The desktop image is divided into layers. The desktop itself is the base layer, and the chat window is the area layer. The movement of the chat window is converted into a change in the relative position of the layers, represented by a binary vector. Once this vector is sent to the terminal, the terminal will synchronize the layers to achieve the same display effect.

[0009] S2: The transformation relationship between layers needs to be synchronized in real time between the sharing end and the shared end. To maintain tile updates within the layer and minimize the number of memory copies, the content sharing end needs to maintain a display tree for each layer.

[0010] S3: Before being added to the display tree, the new tile needs to be compared one by one with the existing tiles in the display tree. The main purpose of comparing the relationships between different tiles is to confirm the occlusion relationship of the current tile in the layer and determine whether the tile is necessary for rendering.

[0011] S4: Each layer also needs to maintain a larger display tree. The main function of this tree is to manage the relationship between layer areas and decide the areas that need to be compressed and encoded. The tree structure needs to be synchronized in real time between the content sharing side and the shared side. The content sharing end and the content receiving end use the larger display tree to make judgments and obtain the final image for output.

[0012] Furthermore, the tiles belonging to layer n generated by the system in S2 are not directly rendered onto the surface of layer n. If they are directly rendered onto the surface of layer n, unnecessary losses will be generated due to continuous rendering. Instead, they are hung on the display tree under layer n.

[0013] Furthermore, if the occlusion relationship in S3 confirms that the existing tiles on the tree have been completely occluded by the upper layer and no area needs to be displayed, then the old tiles need to be removed from the display tree to avoid meaningless rendering. After the old tiles are removed, the new tiles are hung in the display tree.

[0014] Furthermore, if the new tile itself is completely obscured by the upper-layer tile in the confirmation of the obstruction relationship in S3, the tile will also be discarded. Ultimately, each layer maintains a display tree to reduce rendering pressure.

[0015] Furthermore, the content sharing end in S4 determines whether there is a sub-layer destruction process. If so, a new sub-layer needs to be peeled off from the base layer until the base layer cannot be separated. After the separation is completed, the corresponding sub-layer needs to be created, and the creation information is packaged into an independent information unit Unit1.

[0016] Furthermore, in S4, the content sharing end determines whether there is a sub-layer destruction process. If no sub-layer is created or destroyed, it checks whether there is a layer area change. If there is a change, the layer needs to be re-established and Unit1 needs to be encapsulated.

[0017] Furthermore, after determining whether there is a sub-layer destruction process in S4, it is determined whether the relationship between the layers has changed. If there is a change, the layer transformation unit Unit2 is encapsulated, and finally the content changes of the uncovered areas of each sub-layer are checked, and the real image data unit Unit3 is encapsulated and transmitted through the network.

[0018] Furthermore, in S4, the content receiving end first checks whether there is an information unit Unit1 of a newly added sublayer, and if so, creates a corresponding sublayer and establishes a related hierarchical relationship.

[0019] Furthermore, in S4, after the content receiving end detects the newly added sublayer, it then detects whether there is an existing layer transformation unit Unit2, adjusts the relationship between the sublayers according to the change, and finally checks whether there is a necessary image data transmission unit Unit3.

[0020] Furthermore, if the content receiving end in S4 detects that the image data transmission unit Unit3 exists, it performs corresponding decoding to update the image of the relevant area. Finally, the top-level output is performed at one time through the external display tree, and multiple sub-layers are stacked to obtain the final output.

[0021] In summary, the beneficial technical effects of the present invention are:

[0022] 1. The present invention utilizes sublayers and maintains layer hierarchical relationships. This method is the key to streaming basic operating instructions in the event operating system. Synchronizing layer relationships can achieve synchronized display effects.

[0023] 2. The layer-based desktop content sharing framework of the present invention can realize command streaming transmission. Window movement or layer changes do not require any data transmission. Unit 2 is used to synchronize layer motion information between the source and the terminal, directly converting data transmission into command transmission. Similar principles also apply to operations such as foreground and background switching.

[0024] 3. In the present invention, due to the difference between the system's image output speed and the encoding transmission speed (often the image output speed is much greater than the encoding speed), a buffer is required between the two. The buffer is repeatedly rendered and read and written due to the speed difference (the system draws the image in the buffer, and the encoding is transmitted to the buffer to read the image), resulting in a large performance loss. The double-layer display tree structure proposed in this article solves this problem well, reduces unnecessary drawing, and performs an overall synthesis only when encoding and taking the image, which significantly improves performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of a layer maintenance method of a content sharing terminal of the present invention;

[0026] Figure 2 Schematic diagram of the layer maintenance method of the content receiving end of the present invention. DETAILED DESCRIPTION

[0027] The method of the present invention is further described in detail below with reference to the accompanying drawings.

[0028] Example 1

[0029] A desktop sharing and transmission method based on a composite layer structure is as follows:

[0030] S1: The desktop image is divided into layers. The desktop itself is the base layer, and the chat window is the area layer. The movement of the chat window is converted into a change in the relative position of the layers, represented by a binary vector. Once this vector is sent to the terminal, the terminal will synchronize the layers to achieve the same display effect.

[0031] Display tree: It is used to manage the layer hierarchy and reduce the data structure of invalid repeated rendering. The whole tree is presented. Each time the image is displayed, it is the overall effect after stacking from top to bottom.

[0032] Desktop sharing: refers to distributing the content of a personal operating system to a designated audience via network transmission to achieve the purpose of simultaneous demonstration;

[0033] Layers: Generally, the display content in an operating system is composed of N different layers stacked together. A layer can be understood as a canvas on which the application can draw the required content. The operating system determines the foreground and background relationship of the displayed content based on the canvas hierarchy.

[0034] S2: The transformation relationship between layers needs to be synchronized in real time between the sharing end and the shared end. To maintain tile updates within the layer and minimize the number of memory copies, the content sharing end needs to maintain a display tree for each layer. Tiles generated by the system belonging to layer n are not directly rendered onto the surface of layer n. If they were rendered directly onto the surface of layer n, unnecessary rendering losses would occur due to continuous rendering. Instead, they are placed on the display tree under layer n.

[0035] S3: Before being hung in the display tree, the new block needs to be compared one by one with the existing blocks in the display tree. The comparison of the relationship between different blocks is mainly to confirm the occlusion relationship of the current block in the layer and determine whether the block rendering is necessary. If the occlusion relationship is confirmed and the existing block in the tree is completely occluded by the upper layer and no area needs to be displayed, then the old block needs to be removed from the display tree to avoid meaningless rendering. After the old block is removed, the new block is hung in the display tree. During the occlusion relationship confirmation, if the new block itself is completely occluded by the upper layer block, the block will also be discarded. Finally, each layer maintains a display tree to reduce rendering pressure;

[0036] Reference Figure 1 , S4: Each layer also needs to maintain a larger display tree. The main function of this tree is to manage the relationship between layer areas and decide the areas that need to be compressed and encoded. The tree structure needs to be synchronized in real time between the content sharing side and the shared side. The content sharing end and the content receiving end use the larger display tree to make judgments and obtain the final image for output. The content sharing end determines whether there is a sub-layer destruction process. If so, a new sub-layer needs to be peeled off from the base layer until the base layer cannot be separated. After the separation is completed, the corresponding sub-layer needs to be created and the creation information packaged into an independent information unit Unit1. If the sub-layer is not created or destroyed, check whether there is a change in the layer area. If there is a change, the layer needs to be re-established and encapsulated Unit1. After determining whether there is a sub-layer destruction process, determine whether the relationship between the layers has changed. If there is a change, encapsulate the layer transformation unit Unit2. Finally, check the content changes of the uncovered areas of each sub-layer, encapsulate them into the real image data unit Unit3, and transmit them through the network;

[0037] Reference Figure 2 The content receiving end first checks whether there is a new sub-layer information unit Unit1. If so, it creates the corresponding sub-layer and establishes the relevant hierarchical relationship. Then it checks whether there is an existing layer transformation unit Unit2, and adjusts the relationship between the sub-layers according to the changes. Finally, it checks whether there is a necessary image data transmission unit Unit3. If so, it performs corresponding decoding to update the image of the relevant area. Finally, it outputs the top level at one time through the external display tree, and stacks multiple sub-layers to obtain the final output.

[0038] Example 2

[0039] Existing methods are unable to reconstruct the relationships between layers on the terminal; in effect, multiple layers are merged into a single layer for transmission. While this transmission method is logically simple and easy to implement, it also introduces issues such as loss of layer hierarchical relationships and repeated transmission of redundant content. This drawback is exacerbated during basic operating system operations such as maximizing and minimizing, moving windows, switching between foreground and background windows, and switching between desktops.

[0040] The key idea behind the method proposed in this article is to separate the shared desktop content into its own layers by acquiring the layer information of the collection end in real time. Each layer is rendered independently and independently, preserving and reconstructing the shared layer information. During this process, the collection end and the receiving end must simultaneously maintain a layer relationship manager, referred to herein as the layer display tree. This manager converts the layer relationships between different levels of the tree into masking relationships between the top level and several lower levels, which is used to manage and decide the transmission area and content. The specific management and transmission methods are detailed below.

[0041] First, let's examine the basic characteristics of a typical graphical operating system. A typical user desktop consists of multiple windows, each representing a layer. By monitoring the relationships between these layers and any changes in those relationships, the display relationship between the foreground and background can be altered. If this relationship can be reconstructed on the shared end during screen sharing, some common operating system operations can be directly converted into command transmissions, replacing the image data transmission required for a single layer. This reduces data transmission at the source. For example, a simple chat window move operation, under the existing single-layer approach, directly converts the change between the previous and next frames into an image data change. The total area of the change is equal to the combined area of the window before and after the previous frame move, resulting in a significant amount of data transmission. If the concept of layers existed, images within the desktop could be separated into layers, with the desktop itself as the base layer and the chat window as area layer 1. Then, the movement of the chat window would become a change in the relative position of the layers, which could be represented by a simple binary vector. Simply sending this vector to the terminal synchronizes the layer changes to achieve the same display effect, with zero image data transmission required, truly achieving the transformation of data transmission into command stream transmission.

[0042] The above examples fully illustrate the unique advantages of layer-based desktop sharing. However, actual application scenarios are often much more complex than these examples. In actual application, we need to solve the following problems:

[0043] 1. In actual applications, except for the base layer, there is often not a single regional layer. As long as the memory of the sharing end operating system is sufficient, the number of stacked windows is theoretically unlimited. If each layer matches an independent channel, the number of channels will also be infinite, which is unrealistic. The number of layers needs to be controlled.

[0044] 2. In daily use, each window has its own independent content system, such as Figure 1 There may be video web pages, map web pages, chat windows, file managers, etc., which are stacked on each other in a complex hierarchical relationship. How to build, maintain and synchronize these complex relationships to the terminal is a problem that needs to be faced.

[0045] 3. The content in each window layer will also change in real time. How to handle the changes of the obscured underlying image and keep it synchronized on the terminal is also a prominent issue.

[0046] In the face of many problems in the above complex scenarios, general operating systems such as Windows / Linux have image graphics interfaces. Taking Windows as an example, its graphics interface has the basic function of managing and resolving layers. By making full use of this layer relationship, several layers on the current desktop are merged and divided into a limited number of transmission layers, such as Figure 1 As shown, the limited number of layers is set to 6 (or other numbers). These 6 layers are the 5 layers at the top and the merged layer of all layers after the 5 layers. This layer is called the bottom layer. The transformation relationship between the 6 layers needs to be synchronized in real time between the sharing end and the shared end. In order to maintain the update of the tiles in the layer and reduce the number of memory copies as much as possible, the content sharing end needs to maintain a display tree for each layer. The tiles generated by the system belonging to layer n (n is 0-5) are not directly rendered onto the surface of layer n. If that is done, unnecessary losses will be generated due to continuous rendering. Instead, Hang it on the display tree under layer n. Before hanging it in the display tree, the new block needs to be compared one by one with the existing blocks on the display tree. Comparing the relationship between different blocks is mainly to confirm the occlusion relationship of the current block in the layer. If the existing block on the tree has been completely obscured by the upper layer and there is no area that needs to be displayed, then the old block needs to be removed from the display tree to avoid meaningless rendering. After the old block is removed, the new block is hung in the display tree. Of course, if the new block itself is completely obscured by the upper layer block, the block will also be discarded. Finally, each layer maintains a display tree to reduce rendering pressure.

[0047] A large display tree needs to be maintained between each layer. This display tree has only six sublayers, corresponding to the six layers mentioned above. The main function of this tree is to manage the relationship between the layer areas and determine the areas that need compression encoding. This tree structure needs to be synchronized in real time between the content sharing side and the shared side. Let's expand on this in detail:

[0048] >>Content sharing terminal:

[0049] like Figure 1 As shown, the main steps of judgment are: whether there is a sub-layer destruction process, if so, a new sub-layer needs to be peeled off from the base layer again until the base layer cannot be separated. After the separation is completed, the corresponding sub-layer needs to be created, and the creation information (including size and position information, etc.) is packaged into an independent information unit Unit1. If the sub-layer is not created or destroyed, check whether there is a change in the layer area. If there is a change, the layer needs to be re-established and encapsulated Unit1, and then determine whether the relationship between the layers has changed. If there is a change, encapsulate the layer transformation unit Unit2, and finally check the content changes of the uncovered areas of each sub-layer, encapsulate them into the real image data unit Unit3, and transmit them through the network.

[0050] >>Content receiving end:

[0051] like Figure 2 As shown, first check whether there is a new sub-layer information unit Unit1. If so, create the corresponding sub-layer and establish the relevant hierarchical relationship. Then check whether there is an existing layer transformation unit Unit2, and adjust the relationship between the sub-layers according to the changes. Finally, check whether there is a necessary image data transmission unit Unit3. If so, perform corresponding decoding to update the image of the relevant area. Finally, perform the top-level output at one time through the external display tree, and stack multiple sub-layers to obtain the final output.

[0052] This layer-based desktop content sharing framework can realize command streaming transmission. There is no data transmission at all when the window moves or the level changes. It is only necessary to synchronize the layer motion information at the source and the terminal through the Unit2 unit, and directly convert the data transmission into command transmission. Similar principles also exist for operations such as foreground and background switching. At the same time, due to the difference between the system's image output speed and the encoding transmission speed (often the image output speed is much greater than the encoding speed), a buffer is required between the two. The buffer is repeatedly rendered and read and written due to the speed difference (the system outputs the image in the buffer, and the encoding is transmitted to the buffer to read the image), resulting in a large performance loss. The double-layer display tree structure proposed in this article solves this problem well, reduces unnecessary drawing, and performs an overall synthesis when encoding and taking the image, which significantly improves performance.

[0053] It is important to note that the construction and arrangement of the present application shown in a number of different exemplary embodiments are merely illustrative. Although only a few embodiments are described in detail in this disclosure, it should be readily understood by those reading this disclosure that many modifications are possible (e.g., the size, scale, structure, shape, and proportion of various elements, as well as parameter values (e.g., temperature, pressure, etc.), mounting arrangements, use of materials, color, orientation, etc.) without materially departing from the novel teachings and advantages of the subject matter described in this application. For example, elements shown as integrally formed may be composed of multiple parts or elements, the positions of elements may be inverted or otherwise changed, and the nature, number, or position of discrete elements may be altered or changed, and therefore, all such modifications are intended to be included within the scope of the present invention. The order or sequence of any process or method steps may be changed or reordered according to alternative embodiments. In the claims, any "means-plus-function" clause is intended to cover structures described herein that perform the function, and not only structural equivalence but also equivalent structures. Other substitutions, modifications, changes, and omissions may be made in the design, operating conditions, and arrangement of the exemplary embodiments without departing from the scope of the present invention. Therefore, the invention is not limited to the specific embodiments, but extends to various modifications that still fall within the scope of the appended claims.

[0054] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment (i.e., those features that are not relevant to the best mode presently contemplated for carrying out the invention or those that are not relevant to implementing the invention) may not be described.

[0055] It will be appreciated that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but will be a routine task of design, fabrication, and production for those of ordinary skill having the benefit of this disclosure without undue experimentation.

[0056] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A desktop sharing and transmission method based on a composite layer structure, characterized by: The method is as follows: S1: The desktop image is divided into layers. The desktop itself is the base layer, and the chat window is the area layer. The movement of the chat window is converted into a change in the relative position of the layers, represented by a binary vector. Once this vector is sent to the terminal, the terminal will synchronize the layers to achieve the same display effect. S2: The transformation relationship between layers is synchronized in real time between the sharing end and the shared end. To maintain tile updates within the layer and reduce the number of memory copies, the content sharing end maintains a primary display tree for each layer. S3: Before being placed in the first display tree, the new block is compared one by one with the existing blocks in the first display tree, and the relationship between different blocks is compared to confirm the occlusion relationship of the position of the new block in the layer, and determine whether to render the block; S4: Each layer also maintains a second display tree, which is used to manage the relationship between layer areas and decide on the compression encoding areas. The second display tree structure is synchronized in real time between the content sharing side and the shared side. The content sharing end and the content receiving end are judged through the second display tree to obtain the final image for output. The judgment includes whether there is a sub-layer destruction process, whether there is a layer area change, whether there is a change in the relationship between the layers after the sub-layer destruction process, and the content change of the uncovered area of each sub-layer.

2. The desktop sharing and transmission method based on a composite layer structure according to claim 1, characterized in that: The tiles belonging to layer n generated by the system in S2 are not directly rendered onto the surface of layer n. If they are directly rendered onto the surface of layer n, losses will occur due to continuous rendering. Instead, they are hung on the first display tree under layer n.

3. The desktop sharing and transmission method based on a composite layer structure according to claim 2, characterized in that: The occlusion relationship in S3 is confirmed to be that the existing tile on the tree has been completely occluded by the upper layer, and no area is displayed. Then the old tile needs to be removed from the first display tree to avoid a meaningless rendering. After the old tile is removed, the new tile is hung in the first display tree.

4. The desktop sharing and transmission method based on a composite layer structure according to claim 3, characterized in that: If the new tile itself is completely obscured by the upper layer tile during the confirmation of the obstruction relationship in S3, the new tile will also be discarded. Ultimately, each layer maintains a second display tree to reduce rendering pressure.

5. The desktop sharing and transmission method based on a composite layer structure according to claim 4, characterized in that: The content sharing end in S4 determines whether there is a sub-layer destruction process. If so, a new sub-layer is peeled off from the base layer until the base layer cannot be separated. After the separation is completed, the corresponding sub-layer is created and the creation information is packaged into an independent information unit Unit1. If the sub-layer is not created or destroyed, check whether there is a change in the layer area. If there is a change, re-establish the layer and encapsulate Unit1.

6. The desktop sharing and transmission method based on a composite layer structure according to claim 5, characterized in that: The content sharing end in S4 determines whether there is a sub-layer destruction process and then determines whether there is a change in the relationship between the layers. If there is a change, the layer transformation unit Unit2 is encapsulated, and finally the content changes of the uncovered areas of each sub-layer are checked, and encapsulated into the real image data transmission unit Unit3 for transmission through the network.

7. The desktop sharing and transmission method based on a composite layer structure according to claim 6, characterized in that: In S4, the content sharing end and the content receiving end first check whether there is an information unit Unit1 of the newly added sub-layer, and if so, create a corresponding sub-layer and establish a related hierarchical relationship.

8. The desktop sharing and transmission method based on a composite layer structure according to claim 7, characterized in that: After the content receiving end detects the newly added sub-layer in S4, it then detects whether there is an existing layer transformation unit Unit2, adjusts the relationship between the layers according to the changes, and finally checks whether there is an image data transmission unit Unit3.

9. The desktop sharing and transmission method based on a composite layer structure according to claim 8, characterized in that: In the S4, the content receiving end detects that if the image data transmission unit Unit3 exists, it performs corresponding decoding to update the image of the relevant area. Finally, the top-level output is performed at one time through the external display tree, and multiple sub-layers are stacked to obtain the final output.

Images