Cloud application screenshot method, system, and computer-readable storage medium

Abstract

The application discloses a cloud application screenshot method, system and storage medium, and the method comprises the following steps: when the cloud application is started, the local device divides a first screenshot obtained by screenshot into image data of a first local window and a first local desktop, and transmits the image data of the first local desktop to a virtual machine; when the local device receives a screenshot notification, the local device divides a second screenshot obtained by screenshot into image data of a second local window and a second local desktop; the image data of the first local desktop and the second local desktop are subjected to difference analysis, and the difference data of the desktop and the image data of the second local window are transmitted to the virtual machine; the virtual machine restores the image data of the second local desktop, and combines the image data of the second local window to obtain a combined image; the image data of a cloud application window is acquired, and the combined image is combined to obtain the second screenshot.

Description

TECHNICAL FIELD

[0001] The present application relates to the cloud application technical field, and particularly relates to a cloud application screenshot method, system and computer readable storage medium. BACKGROUND

[0002] The cloud application is also called remote application and virtual application, which is an application mode of cloud computing technology. In simple terms, the cloud application is to install a virtualization software in a server in a data center, to create multiple virtual machines by configuring the software, and to provide multiple users with the application installed in the virtual machine for independent use. Different users can use different virtual machines or share the same virtual machine, and share the CPU and memory resources of the server without conflict.

[0003] In the application scenario of the cloud application, the Windows operating system is generally installed in the virtual machine to meet the use habits of most users. The software used by the user is published to the local device of the user in the form of an application shortcut icon. When the user needs to use the application, the user can start the corresponding application in the remote virtual machine by double-clicking the shortcut icon with the mouse. The content of the application window is projected to the corresponding position in the screen of the local device of the user. At the same time, the keyboard and mouse operations for the cloud application are captured on the local device of the user and projected to the remote virtual machine to control the cloud application. In this way, the user can browse web pages, edit documents and perform other operations through the cloud application. The user can use the cloud application and the local application on the local device at the same time. The main cloud application technical solutions currently available include RemoteApp of Microsoft Corporation of the United States and XenApp of Citrix Corporation of the United States.

[0004] In the use of the cloud application, the user often has the need for screenshot, such as when using instant messaging software or when editing a document. Due to the special architecture of the cloud application, the application is not directly run on the local device of the user but directly run on the virtual machine. Therefore, the screenshot can only capture the image in the virtual machine and cannot capture the image on the local device of the user. Thus, the screenshot cannot be correctly obtained.

[0005] For this problem, some solutions have been proposed. The most common and effective solution is completed in the following steps:

[0006] 1. capturing and identifying the screenshot instruction in the virtual machine;

[0007] 2. When the screenshot instruction is identified, the user is notified of the relevant component on the local device, and the component performs a screenshot operation;

[0008] 3. The result of the screenshot is sent to the virtual machine;

[0009] 4. The screenshot on the user's local device is used to replace the result of the screenshot instruction in the virtual machine.

[0010] Through the above four steps, the screenshot in the virtual machine can capture the image on the user's local device.

[0011] However, this solution also has defects. Due to the development of technology, the size of the display used by the user's local device is getting larger and larger, and the resolution is getting higher and higher. The amount of image data to be transmitted in step 3 of the above solution will be relatively large. In some use scenarios of cloud applications, the network bandwidth is often not very good, such as limited dedicated line network and wireless network. At this time, the time spent in step 3 will be longer, and the user's entire operation needs to be completed in 1-4 steps. In the case of poor bandwidth and large image data, not only more bandwidth and more traffic will be consumed, but also the screenshot operation delay of the user in the cloud application will be higher, and it may even be stuck. SUMMARY

[0012] The technical problem to be solved by the present application is to provide a cloud application screenshot method, system and computer readable storage medium, which can reduce the operation delay of the cloud application screenshot and reduce the bandwidth and traffic usage of the cloud application screenshot.

[0013] To solve the above technical problems, the technical solution adopted by the present application is as follows: a cloud application screenshot method, comprising:

[0014] When the cloud application is started, the local device performs full-screen screenshot to obtain a first screenshot;

[0015] The local device divides the first screenshot to obtain image data of a first local window and a first local desktop, and transmits the image data of the first local desktop to a virtual machine where the cloud application is located, wherein the image data of the first local desktop includes image data of a first local desktop background and a first local desktop taskbar;

[0016] When the local device receives a screenshot notification, it performs full-screen screenshot to obtain a second screenshot;

[0017] The local device divides the second screenshot to obtain image data of a second local window and a second local desktop, wherein the image data of the second local desktop includes image data of a second local desktop background and a second local desktop taskbar;

[0018] The local device performs difference analysis on the image data of the first local desktop and the second local desktop to obtain desktop difference data, and sends the desktop difference data and image data of the second local window to the virtual machine;

[0019] The virtual machine restores image data of the second local desktop according to the image data of the first local desktop and the desktop difference data, and merges the image data of the second local window and the second local desktop to obtain a merged image.

[0020] The virtual machine obtains image data of a cloud application window, and synthesizes a second screenshot according to the image data of the cloud application window and the merged image.

[0021] The application further provides a cloud application screenshot system, comprising a local device and a virtual machine, wherein the local device is in communication connection with the virtual machine; the local device comprises an information acquisition module and an image analysis module, and the virtual machine comprises a screenshot synthesis module.

[0022] The information acquisition module is configured to perform full-screen screenshot when a cloud application is started to obtain a first screenshot.

[0023] The image analysis module is configured to segment the first screenshot to obtain image data of a first local window and a first local desktop, and transmit the image data of the first local desktop to a screenshot synthesis module of a virtual machine where the cloud application is located.

[0024] The information acquisition module is further configured to perform full-screen screenshot when a screenshot notification is received to obtain a second screenshot.

[0025] The image analysis module is further configured to segment the second screenshot to obtain image data of a second local window and a second local desktop.

[0026] The image analysis module is further configured to perform difference analysis on the image data of the first local desktop and the second local desktop to obtain desktop difference data, and send the desktop difference data and the image data of the second local window to the virtual machine.

[0027] The screenshot synthesis module is configured to restore image data of the second local desktop according to the image data of the first local desktop and the desktop difference data, and merge the image data of the second local window and the second local desktop to obtain a merged image.

[0028] The screenshot synthesis module is further configured to obtain image data of a cloud application window, and synthesize a second screenshot according to the image data of the cloud application window and the merged image.

[0029] The application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method.

[0030] The application has the advantages that by dividing the screenshot into different image elements, the image data of the local window is directly transmitted, the image data of the local desktop is only transmitted with the difference, and the image data of the cloud application window is not transmitted, so that the amount of data to be transmitted is reduced, the operation delay during the screenshot of the cloud application is reduced, the bandwidth and traffic usage during the screenshot of the cloud application are reduced, and the use experience of the screenshot of the cloud application is improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of a cloud application screenshot method of the first embodiment of the application;

[0032] Figure 2 A screenshot splitting schematic diagram of the first embodiment of the application;

[0033] Figure 3 A structural schematic diagram of a cloud application screenshot system of the first embodiment of the application. DETAILED DESCRIPTION

[0034] To make the technical content, the achieved purposes and effects of the application clear, the following will be described in detail in combination with the embodiments and the accompanying drawings.

[0035] Please refer to Figure 1 A cloud application screenshot method, comprising:

[0036] When the cloud application is started, the local device performs full-screen screenshot to obtain a first screenshot;

[0037] The local device divides the first screenshot to obtain image data of a first local window and a first local desktop, and transmits the image data of the first local desktop to a virtual machine where the cloud application is located, wherein the image data of the first local desktop comprises image data of a first local desktop background and a first local desktop taskbar;

[0038] When the local device receives the screenshot notification, the local device performs full-screen screenshot to obtain a second screenshot;

[0039] The local device divides the second screenshot to obtain image data of a second local window and a second local desktop, wherein the image data of the second local desktop comprises image data of a second local desktop background and a second local desktop taskbar;

[0040] The local device performs difference analysis on the image data of the first local desktop and the second local desktop to obtain desktop difference data, and sends the desktop difference data and the image data of the second local window to the virtual machine.

[0041] The virtual machine restores image data of a second local desktop according to the image data of the first local desktop and the desktop difference data, and merges the second local window and the image data of the second local desktop to obtain a merged image;

[0042] The virtual machine obtains image data of the cloud application window, and synthesizes a second screenshot according to the image data of the cloud application window and the merged image.

[0043] As can be seen from the above description, the application has the advantages of reducing the amount of data to be transmitted, reducing the operation delay during cloud application screenshot, and reducing the bandwidth and traffic usage during cloud application screenshot.

[0044] Further, the local device segments the first screenshot to obtain image data of the first local window and the first local desktop, specifically as follows:

[0045] The local device obtains position information of each window through a window manager, and cuts the first local window image data from the first screenshot according to the position information of the windows other than the desktop background, the desktop taskbar and the cloud application window;

[0046] The position information of the desktop taskbar is obtained through the window manager, and the first local desktop taskbar image data is cut from the first screenshot according to the position information of the desktop taskbar;

[0047] The image data of the cloud application window is determined in the first screenshot according to the position information of the cloud application window, and the remaining image data in the first screenshot, except for the image data of the first local window, the first desktop taskbar and the cloud application window, is taken as the image data of the first local desktop background;

[0048] The image data includes RGB values of each pixel point and corresponding position information.

[0049] As can be seen from the above description, the image data of the local window, the local desktop taskbar and the local desktop background can be accurately segmented.

[0050] Further, when the local device receives the screenshot notification, full-screen screenshot is performed to obtain a second screenshot, specifically as follows:

[0051] After receiving the screenshot instruction, the virtual machine sends a screenshot notification to the local device;

[0052] After receiving the screenshot notification, the local device performs full-screen screenshot to obtain a second screenshot.

[0053] Further, the local device compresses and transmits the desktop difference data and the image data of the second local window to the virtual machine.

[0054] The local device compresses and transmits the desktop difference data and the image data of the second local window to the virtual machine.

[0055] As can be seen from the above description, by compressing and then transmitting, the amount of data transmitted can be further reduced.

[0056] The application further provides a cloud application screenshot system, comprising a local device and a virtual machine, wherein the local device is in communication connection with the virtual machine; the local device comprises an information acquisition module and an image analysis module, and the virtual machine comprises a screenshot synthesis module;

[0057] The information acquisition module is configured to perform full-screen screenshot when the cloud application is started, and obtain a first screenshot.

[0058] The image analysis module is configured to segment the first screenshot, obtain image data of a first local window and a first local desktop, and transmit the image data of the first local desktop to the screenshot synthesis module of the virtual machine where the cloud application is located.

[0059] The information acquisition module is further configured to perform full-screen screenshot when a screenshot notification is received, and obtain a second screenshot.

[0060] The image analysis module is further configured to segment the second screenshot, and obtain image data of a second local window and a second local desktop.

[0061] The image analysis module is further configured to perform difference analysis on the image data of the first local desktop and the second local desktop, obtain desktop difference data, and transmit the desktop difference data and the image data of the second local window to the virtual machine.

[0062] The screenshot synthesis module is configured to restore the image data of the second local desktop according to the image data of the first local desktop and the desktop difference data, and combine the image data of the second local window and the second local desktop to obtain a combined image.

[0063] The screenshot synthesis module is further configured to acquire image data of a cloud application window, and synthesize the second screenshot according to the image data of the cloud application window and the combined image.

[0064] Further, the virtual machine further comprises a screenshot interception module.

[0065] The screenshot interception module is configured to send a screenshot notification to the information acquisition module of the local device after capturing a screenshot instruction.

[0066] Further, the image data of the first local desktop includes image data of a first local desktop background and a first local desktop taskbar; and the image data of the second local desktop includes image data of a second local desktop background and a second local desktop taskbar.

[0067] The image analysis module is specifically configured to acquire position information of each window through a window manager, and to obtain image data of the first local window by cropping the first screenshot according to position information of windows other than the desktop background, the desktop taskbar and the cloud application window.

[0068] The image analysis module is specifically configured to acquire position information of the desktop taskbar through the window manager, and to obtain image data of the first local desktop taskbar by cropping the first screenshot according to the position information of the desktop taskbar.

[0069] The image analysis module is specifically configured to determine image data of the cloud application window in the first screenshot according to position information of the cloud application window, and to take remaining image data in the first screenshot other than the image data of the first local window, the first desktop taskbar and the cloud application window as image data of the first local desktop background.

[0070] The image data includes RGB values of each pixel point and corresponding position information.

[0071] Further, the image analysis module is specifically configured to compress and send the desktop difference data and the image data of the second local window to the virtual machine.

[0072] The application further provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the method.

[0073] Embodiment one

[0074] Please refer to Figures 1-3 Embodiment one of the application is a cloud application screenshot method, which is based on image element segmentation and can be applied to a cloud desktop.

[0075] As shown in Figure 2 The screenshot content of a cloud application only includes the following parts:

[0076] Part 1: content of a cloud application window;

[0077] Part 2: content of a local normal window;

[0078] Part 3: content of a local desktop background;

[0079] Part 4: content of a local desktop taskbar.

[0080] Of these four parts, part 1 does not need to be transmitted at all, as the virtual machine already contains this part's image information, while the other parts do need to be transmitted. Meanwhile, since parts 3 and 4 rarely change, it is possible to choose to transmit only the differences.

[0081] Therefore, in this embodiment, the image elements of the above four parts are segmented, and the image elements of parts 2, 3 and 4 are transmitted separately during the screenshot. Finally, the four parts are combined together to complete the screenshot.

[0082] Specifically, this embodiment is based on a cloud application screenshot system, such as... Figure 3 As shown, the cloud application screenshot system includes a local device and a virtual machine connected by communication. The local device includes an information acquisition module and an image analysis module, while the virtual machine includes a screenshot interception module and a screenshot synthesis module.

[0083] like Figure 1 As shown, the method in this embodiment includes the following steps:

[0084] S1: When the cloud application starts, the local device takes a full-screen screenshot to obtain the first screenshot.

[0085] Specifically, when the cloud application initiates a connection, the information acquisition module takes a screenshot and sends the result to the image analysis module. At this point, it can be assumed that the cloud application window is not yet displayed on the screen.

[0086] S2: The local device segments the first screenshot to obtain image data of the first local window and the first local desktop, and transmits the image data of the first local desktop to the virtual machine where the cloud application is located. The image data of the first local desktop includes image data of the first local desktop background and image data of the first local desktop taskbar.

[0087] That is, the image analysis module analyzes and follows... Figure 2 The screenshot is divided into three parts: part 2, part 3, and part 4. Specifically, the segmentation method used in this embodiment is as follows:

[0088] The local device enumerates all windows through the window manager, obtains the position information of all other windows except the desktop, taskbar, and cloud applications, and crops the image data of this portion from the first screenshot to obtain the image data of the first local window. Simultaneously, the window manager can also obtain the position information of the desktop taskbar, and then crop the image data of this portion from the first screenshot to obtain the image data of the first local desktop taskbar.

[0089] Since the cloud application window is a special local application window and can be identified, the image data of the cloud application window can be determined in the first screenshot according to the position information of the cloud application window.

[0090] Finally, the image data of the first local window, the first local desktop taskbar and the cloud application window are removed from the first screenshot, and the image data of the first local desktop background is obtained.

[0091] In this embodiment, the image data includes the RGB value of each pixel point in the image and the corresponding position information.

[0092] The local device saves the image data of the first local desktop taskbar and the image data of the first local desktop background locally and transmits them to the virtual machine through network communication, so that the image data of the first local desktop is stored in the image analysis module of the local device and the screenshot synthesis module of the virtual machine, for subsequent difference comparison and image merging.

[0093] S3: When the local device receives the screenshot notification, full-screen screenshot is performed to obtain a second screenshot.

[0094] That is, when the user performs screenshot in the cloud application, the screenshot interception module of the virtual machine captures the screenshot instruction and sends the screenshot notification to the local device through network communication; after receiving the screenshot notification, the information acquisition module of the local device performs full-screen screenshot to obtain a second screenshot.

[0095] S4: The local device segments the second screenshot to obtain the image data of the second local window and the second local desktop, wherein the image data of the second local desktop includes the image data of the second local desktop background and the second local desktop taskbar.

[0096] That is, the image analysis module analyzes and segments the image data of the first local desktop and the second local desktop according to the position information of the first local desktop taskbar and the second local desktop taskbar. Figure 2 The second screenshot is segmented into three parts: part 2, part 3 and part 4. The segmentation method is referred to step S2.

[0097] S5: The local device performs difference analysis on the image data of the first local desktop and the second local desktop to obtain desktop difference data, and sends the desktop difference data and the image data of the second local window to the virtual machine.

[0098] Specifically, the local device performs difference analysis on the image data of the first local desktop background and the second local desktop background and the image data of the first local desktop taskbar and the second local desktop taskbar respectively to obtain desktop difference data, and sends the compressed desktop difference data and the image data of the second local window to the virtual machine.

[0099] That is, the image analysis module compares the image data of part 3 and part 4 obtained in step S2 with the image data of part 3 and part 4 obtained in step S4, and then sends the image data of the desktop difference and part 2 obtained in step S4 to the screenshot synthesis module on the virtual machine through the network communication after compression.

[0100] S6: The virtual machine restores the image data of the second local desktop according to the image data of the first local desktop and the desktop difference data, and merges the image data of the second local desktop and the second local window to obtain a merged image.

[0101] Specifically, the virtual machine first decompresses the received data to obtain the image data of the second local window and the desktop difference data, and then restores the image data of the second local desktop according to the image data of the first local desktop received in step S2 and the desktop difference data; and then merges the image data of the second local window and the second local desktop to obtain a merged image.

[0102] That is, the screenshot synthesis module decompresses the image data of the desktop difference and part 2 to obtain the image data of part 3 and part 4, and then restores the image data of part 3 and part 4 according to the difference part. Then, the image data of part 2, part 3 and part 4 are merged, and after merging, there is still a part of the image that is not filled, which is part 1 (the image data of the cloud application window).

[0103] S7: The virtual machine obtains the image data of the cloud application window, and synthesizes a second screenshot according to the image data of the cloud application window and the merged image.

[0104] That is, the screenshot synthesis module fills the image data of the cloud application window into the unfilled part of the merged image to synthesize a second screenshot. The synthesized screenshot is returned to the screenshot interception module and filled into the screenshot software to complete the screenshot.

[0105] The embodiment can greatly reduce the content required to be transmitted, especially when the cloud application window itself occupies most of the screen content. For example, when a cloud application window is maximized and then screenshot is taken, the cloud application window (part 1) almost fills the entire screen. Using this screenshot method, the image data of part 1 does not need to be transmitted at all, and the data transmission amount is almost zero. Even in the worst case, the data to be transmitted will not be more than the existing cloud application screenshot method.

[0106] The embodiment not only reduces the operation delay of the cloud application screenshot, but also reduces the bandwidth and traffic usage of the cloud application screenshot, achieving better cloud application screenshot effect.

[0107] Embodiment two

[0108] The embodiment is a computer readable storage medium corresponding to the above embodiment, and a computer program is stored on the computer readable storage medium. The program is executed by a processor to implement each step of the cloud application screenshot method in the above embodiment and achieve the same technical effects. Here, the same technical effects are not repeated.

[0109] In summary, the cloud application screenshot method, system and computer readable storage medium provided by the application can reduce the amount of data to be transmitted, reduce the operation delay during cloud application screenshot, reduce the bandwidth and traffic usage during cloud application screenshot, and improve the use experience of cloud application screenshot by dividing the screenshot into different image elements, directly compressing and transmitting the image data of the local window, only transmitting the differential part of the image data of the local desktop, and not transmitting the image data of the cloud application window.

[0110] The above description is only an embodiment of the application, and does not limit the patent scope of the application. Any equivalent transformation or direct or indirect application in the related technical field based on the content of the specification and drawings is also included in the patent protection scope of the application.

Claims

1. A cloud application screenshot method, characterized by, The method comprises the following steps: When the cloud application is started, the local device takes a full-screen screenshot to obtain a first screenshot; The local device divides the first screenshot to obtain image data of a first local window and a first local desktop, and transmits the image data of the first local desktop to a virtual machine where the cloud application is located; After the virtual machine captures the screenshot instruction, the virtual machine sends a screenshot notification to the local device; When the local device receives the screenshot notification, the local device takes a full-screen screenshot to obtain a second screenshot; The local device divides the second screenshot to obtain image data of a second local window and a second local desktop; The local device performs differential analysis on the image data of the first local desktop and the second local desktop to obtain desktop differential data, and transmits the desktop differential data and the image data of the second local window to the virtual machine; The virtual machine restores the image data of the second local desktop according to the image data of the first local desktop and the desktop differential data, and combines the image data of the second local window and the second local desktop to obtain a combined image; The virtual machine obtains image data of a cloud application window, and synthesizes the second screenshot according to the image data of the cloud application window and the combined image. The image data of the first local desktop comprises image data of a first local desktop background and a first local desktop taskbar, and the image data of the second local desktop comprises image data of a second local desktop background and a second local desktop taskbar.

2. The cloud application screenshot method of claim 1, wherein, The local device divides the first screenshot to obtain image data of a first local window and a first local desktop, specifically as follows: The local device obtains position information of each window through a window manager, and cuts the first screenshot to obtain image data of the first local window according to position information of other windows except a desktop background, a desktop taskbar and a cloud application window; The local device obtains position information of the desktop taskbar through the window manager, and cuts the first screenshot to obtain image data of the first local desktop taskbar according to the position information of the desktop taskbar; According to position information of the cloud application window, the local device determines image data of the cloud application window in the first screenshot, and takes remaining image data in the first screenshot except the image data of the first local window, the first desktop taskbar and the cloud application window as image data of the first local desktop background. The image data comprises RGB values of each pixel point and corresponding position information.

3. The cloud application screenshot method of claim 1, wherein, The local device transmits the desktop differential data and the image data of the second local window to the virtual machine after compression. The method comprises the following steps:

4. A cloud application screenshot system, characterized by, The local device and the virtual machine are in communication connection; the local device comprises an information acquisition module and an image analysis module, and the virtual machine comprises a screenshot interception module and a screenshot synthesis module; The information acquisition module is configured to take a full-screen screenshot to obtain a first screenshot when a cloud application is started; The image analysis module is configured to divide the first screenshot to obtain image data of a first local window and a first local desktop, and transmit the image data of the first local desktop to a virtual machine where the cloud application is located. The image analysis module is configured to segment the first screenshot to obtain image data of a first local window and a first local desktop, and transmit the image data of the first local desktop to a screenshot synthesis module of a virtual machine where the cloud application is located. The screenshot interception module is configured to send a screenshot notification to an information acquisition module of the local device after capturing the screenshot instruction. The information acquisition module is further configured to perform full-screen screenshot to obtain a second screenshot when the screenshot notification is received. The image analysis module is further configured to segment the second screenshot to obtain image data of a second local window and a second local desktop. The image analysis module is further configured to perform difference analysis on the image data of the first local desktop and the second local desktop to obtain desktop difference data, and transmit the desktop difference data and the image data of the second local window to the virtual machine. The screenshot synthesis module is configured to restore the image data of the second local desktop according to the image data of the first local desktop and the desktop difference data, and combine the image data of the second local window and the second local desktop to obtain a combined image. The screenshot synthesis module is further configured to acquire image data of a cloud application window, and synthesize the second screenshot according to the image data of the cloud application window and the combined image. The image data of the first local desktop includes image data of a first local desktop background and a first local desktop taskbar, and the image data of the second local desktop includes image data of a second local desktop background and a second local desktop taskbar.

5. The cloud application screenshot system of claim 4, wherein, The image analysis module is specifically configured to acquire position information of each window through a window manager, and cut the image data of the first local window from the first screenshot according to position information of other windows except the desktop background, the desktop taskbar and the cloud application window. The window manager is used to acquire position information of the desktop taskbar, and cut the image data of the first local desktop taskbar from the first screenshot according to the position information of the desktop taskbar. The image data of the cloud application window is determined in the first screenshot according to position information of the cloud application window, and the remaining image data in the first screenshot except the image data of the first local window, the first desktop taskbar and the cloud application window is taken as the image data of the first local desktop background. The image data includes RGB values of each pixel point and corresponding position information.

6. The cloud application screenshot system of claim 4, wherein, The image analysis module is specifically configured to compress and transmit the desktop difference data and the image data of the second local window to the virtual machine.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-3.

Images