Virtual desktop display method and related equipment

Abstract

The invention discloses a virtual desktop display method and related equipment, and relates to the technical field of cloud computing. The terminal equipment receives initial image data and area indication information from the virtual desktop of the computing equipment, wherein the area indication information comprises coordinate information or size information of a to-be-operated object in the virtual desktop in the virtual desktop; the terminal device detects a target operation used for adjusting a display area of the to-be-operated object in the virtual desktop according to the area indication information; the terminal device renders to obtain a target image and displays the target image, a display area of the to-be-operated object in the target image is determined based on the target operation, and the target image is obtained based on rendering of the initial image data. Thus, the terminal device renders based on the received initial image data to obtain the target image to be displayed without temporarily waiting for the initial image data to be transmitted from the computing device to the terminal device, so that the time delay of the terminal device in response to the target operation can be effectively reduced, and the experience of a user in using a virtual desktop is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a virtual desktop display method and related equipment. Background Art

[0002] A cloud desktop is a desktop that is virtualized. Specifically, the operating system, applications, user data, etc. are all stored on a computing device, such as a server (or server cluster) on the cloud side, and the desktop resources on the computing device are distributed to the terminal device on the user side through the network, so that the user experience of using the virtual desktop is basically the same as that of using the desktop on the local terminal device.

[0003] At present, when a user uses a virtual desktop, the terminal device on the user side usually transmits the user's interactive operation on the virtual desktop to the computing device, and the interactive operation may be, for example, a drag operation on a window in the virtual desktop. Then, the computing device renders a virtual desktop image according to the interactive operation, and sends the virtual desktop image to the terminal device on the user side for display.

[0004] However, this method of displaying a virtual desktop often results in a large response delay, which seriously affects the user experience of using the virtual desktop. Summary of the invention

[0005] The present application provides a virtual desktop display method, aiming to improve the user experience of using the virtual desktop. In addition, the present application also provides a corresponding terminal device, a computing device, a computer-readable storage medium and a computer program product.

[0006] In the first aspect, the present application provides a virtual desktop display method, which can be executed by a terminal device. Specifically, the terminal device receives initial image data and first area indication information of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated, which can be, for example, an icon or a window in the virtual desktop, and the first area indication information includes coordinate information (such as coordinate values) or size information (such as length value and width value, etc.) of the object to be operated in the virtual desktop. Then, the terminal device detects a target operation for the object to be operated according to the first area indication information, and the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, such as an operation of dragging the object to be operated in the virtual desktop, so that the display area of ​​the object to be operated in the virtual desktop can change based on the drag operation performed by the user. Then, the terminal device renders a target image and displays the target image, wherein the display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data.

[0007] After detecting the target operation, the terminal device renders the target operation and the initial image data currently received to obtain the target image of the virtual desktop, and displays the target image on the terminal device. That is, the terminal device renders the target image to be displayed based on the initial image data that has been received, without temporarily waiting for the initial image data to be transmitted from the computing device to the terminal device, and without transmitting the rendered target image between the computing device and the terminal device through the network, which can effectively reduce the time delay of responding to the target operation performed by the user on the terminal device and presenting the rendered target image. Thus, in the process of the user operating the object to be operated in the virtual desktop, the display area of ​​the object to be operated in the virtual desktop displayed by the terminal device can change in time with the change of the user operation, and the change of the display area of ​​the object to be operated in the virtual desktop will be more fluid, which can effectively improve the user's experience of using the virtual desktop.

[0008] In a possible implementation, after detecting the target operation, the terminal device may also send a first notification message to the computing device, the first notification message being used to indicate that the target operation has begun to be performed on the object to be operated. In this way, the computing device may determine based on the first notification message that there is currently a target operation for the object to be operated, so as to perform subsequent processing based on the first notification message. For example, when the computing device learns that the target operation has begun to be performed on the object to be operated based on the first notification message, it may not continue to send image data to the terminal device, so as to reduce network resource consumption between the computing device and the terminal device.

[0009] In a possible implementation, after detecting that the target operation is completed, the terminal device can also send a second notification message to the computing device, the second notification message is used to indicate that the target operation is completed, and the second notification message also includes a second area indication information, the second area indication information includes the coordinate information (such as coordinate value), size information (such as length value and width value) or rotation angle information (such as angle value) of the object to be operated in the virtual desktop when the target operation is completed. In this way, the computing device can determine the completion of the target operation and the display area of ​​the object to be operated in the virtual desktop when the target operation is completed based on the second notification message. In this way, when the computing device provides the image data of the virtual desktop rendered on the computing device to the terminal device, it can send the correct area indication information, so that the display area of ​​the object to be operated viewed by the user on the terminal device is consistent with the display area of ​​the object to be operated when the user ends the target operation, so that before and after the user performs the target operation on the object to be operated, the position change of the object to be operated in the virtual desktop can maintain a high level of fluency, thereby ensuring the user's experience of using the virtual desktop.

[0010] In a possible implementation, when the terminal device renders the target image, it can specifically determine the movement trajectory of the pointer in the virtual desktop according to the target operation, and the pointer is used to indicate the operation position in the virtual desktop, so that the terminal device renders the target image according to the movement trajectory of the pointer and the received initial image data. In this way, the terminal device can determine the movement position of the pointer according to the target operation, that is, determine the position of the object to be operated during the process of executing the target operation, so that the display area of ​​the object to be operated in the target image rendered by the terminal device based on the movement trajectory can be a display area that meets the user's expectations.

[0011] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is the image data obtained by rendering on the computing device; then, when the terminal device renders the target image according to the movement trajectory of the pointer and the received initial image data, it can specifically generate border image data according to the target operation, and the area within the border indicated by the border image data is the display area of ​​the object to be operated in the target image, so that the terminal device can render the target image of the virtual desktop according to the border image data, the movement trajectory of the pointer and the initial image data. In this way, in the screen content of the target image rendered and presented by the terminal device, the border can be used to indicate the display area of ​​the object to be operated in the virtual desktop after the target operation is performed.

[0012] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data. In this way, the terminal device can render the target image locally according to the background image data and window image data provided by the computing device and the detected target operation, so that the window on which the target operation is executed in the screen of the virtual desktop presented by the terminal device can change the display area in time with the change of the target operation performed by the user, thereby improving the user's experience of using the virtual desktop.

[0013] In a possible implementation, the initial image data includes target background image data and historical window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated. In this way, the terminal device can present different background image contents of the virtual desktop in the two frames of images before and after, thereby improving the user's experience of using the virtual desktop. Moreover, in the process of responding to the target operation performed by the user, the image data sent by the computing device to the terminal device can be reduced from the two types of data, background image data and window image data, to one type of data, background image data, which can effectively reduce the amount of data transmitted between the computing device and the terminal device, thereby reducing the data transmission delay, thereby reducing the delay of the terminal device responding to the target operation and presenting the target image corresponding to the target operation, and further improving the user's experience of using the virtual desktop.

[0014] Alternatively, the initial image data includes historical background image data and target window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated. In this way, the terminal device can present different background image contents in the window in the previous and next two frames of images (such as playing a video in the window, etc.), and can reduce the delay of the terminal device responding to the target operation and presenting the target image corresponding to the target operation, thereby improving the user experience of using the virtual desktop.

[0015] Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated. In this way, the terminal device can present different background image contents of the virtual desktop and different background image contents in the window in the two frames before and after, and can reduce the delay of the terminal device responding to the target operation and presenting the target image corresponding to the target operation, thereby improving the user experience of using the virtual desktop.

[0016] Alternatively, the initial image data includes historical background image data and historical window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is started on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated. In this way, the terminal device can reduce the delay in responding to the target operation and presenting the target image corresponding to the target operation, thereby improving the user experience of using the virtual desktop.

[0017] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0018] In a second aspect, the present application provides a virtual desktop display method, which is applied to a computing device. Specifically, the computing device generates initial image data and first area indication information of a virtual desktop, the virtual desktop includes an object to be operated (such as an icon or a window, etc.), and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; and before a target operation is performed on the object to be operated, the computing device sends the initial image data and the first area indication information to a terminal device, wherein the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, the target image displayed on the terminal device is generated based on the initial image data and the first area indication information, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0019] In a possible implementation, the computing device may also receive a first notification message from the terminal device, where the first notification message is used to indicate that a target operation begins to be performed on the object to be operated.

[0020] In a possible implementation, after the target operation is completed, the computing device may also receive a second notification message from the terminal device, where the second notification message is used to indicate that the target operation is completed. The second notification message also includes second area indication information, and the second area indication information includes coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

[0021] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on a computing device.

[0022] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0023] In a possible implementation, the initial image data includes target background image data and historical window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated; or, the initial image data includes historical background image data and target window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is started on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated. image data; or, initial image data, including target background image data and target window image data, the target background image data is the background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated; or, initial image data, including historical background image data and historical window image data, the historical background image data is the background image data received by the terminal device before the target operation is started to be performed on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated.

[0024] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0025] The virtual desktop display method provided in the second aspect corresponds to the virtual desktop display method provided in the first aspect. Therefore, the technical effects of the second aspect and any implementation method of the second aspect can be referred to the relevant description of the technical effects of the first aspect and the corresponding implementation method of the first aspect, and will not be repeated here.

[0026] In a third aspect, the present application provides a virtual desktop display method, which is applied to a terminal device. Specifically, the terminal device receives initial image data of a virtual desktop from a computing device, and the virtual desktop includes an object to be operated; and the terminal device also detects a target operation for the object to be operated, and the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop; thus, the terminal device renders a target image and displays the target image, wherein the display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data. After detecting the target operation, the terminal device renders the target image of the virtual desktop based on the target operation and the initial image data currently received, and displays the target image on the terminal device. That is, the terminal device renders the target image to be displayed based on the initial image data already received, without temporarily waiting for the initial image data to be transmitted from the computing device to the terminal device, and without transmitting the rendered target image between the computing device and the terminal device through a network, which can effectively reduce the delay in responding to the target operation performed by the user on the terminal device and presenting the rendered target image. Therefore, when the user operates the object to be operated in the virtual desktop, the display area of ​​the object to be operated in the virtual desktop displayed by the terminal device can change in time with the change of the user operation, and the change of the display area of ​​the object to be operated in the virtual desktop will be smoother, thereby effectively improving the user experience of using the virtual desktop.

[0027] In a fourth aspect, the present application provides a virtual desktop display method, which is applied to a computing device. Specifically, the computing device generates initial image data of a virtual desktop, which includes an object to be operated; and the computing device also sends the initial image data to a terminal device before a target operation is performed on the object to be operated, the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop, and a target image displayed on the terminal device is generated based on the initial image data, and a display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0028] The virtual desktop display method provided in the fourth aspect corresponds to the virtual desktop display method provided in the third aspect. Therefore, the technical effects of the fourth aspect can be referred to the relevant description of the technical effects of the third aspect mentioned above, and will not be repeated here.

[0029] In a fifth aspect, the present application provides a terminal device, including a communication module, used to receive initial image data and first area indication information of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; a detection module, used to detect a target operation for the object to be operated according to the first area indication information, and the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; a rendering module, used to render a target image, wherein the display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; and a display module, used to display the target image.

[0030] In a possible implementation, the communication module is further configured to send a first notification message to the computing device after detecting the target operation, where the first notification message is used to indicate that the target operation begins to be executed on the object to be operated.

[0031] In one possible implementation, the communication module is further used to send a second notification message to the computing device after detecting that the target operation is completed, the second notification message is used to indicate that the target operation is completed, and the second notification message also includes second area indication information, the second area indication information includes the coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

[0032] In a possible implementation, the rendering module is specifically used to: determine a movement trajectory of a pointer in the virtual desktop according to a target operation, where the pointer is used to indicate an operation position in the virtual desktop; and render a target image according to the movement trajectory of the pointer and the received initial image data.

[0033] In one possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on a computing device; the rendering module is specifically used to: generate border image data according to the target operation, and the area within the border indicated by the border image data is the display area of ​​the object to be operated in the target image; render the target image of the virtual desktop according to the border image data, the movement trajectory of the pointer and the initial image data.

[0034] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0035] In a possible implementation, the initial image data includes target background image data and historical window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated; or, the initial image data includes historical background image data and target window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is started on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated. image data; or, initial image data, including target background image data and target window image data, the target background image data is the background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated; or, initial image data, including historical background image data and historical window image data, the historical background image data is the background image data received by the terminal device before the target operation is started to be performed on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated.

[0036] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0037] The terminal device provided in the fifth aspect corresponds to the virtual desktop display method provided in the first aspect. Therefore, the technical effects of the fifth aspect and any implementation method in the fifth aspect can be referred to the relevant description of the technical effects of the above-mentioned first aspect and the corresponding implementation method in the first aspect, and will not be repeated here.

[0038] In a sixth aspect, the present application provides a computing device, comprising: a generation module, used to generate initial image data and first area indication information of a virtual desktop, the virtual desktop including an object to be operated, and the first area indication information including coordinate information or size information of the object to be operated in the virtual desktop; a communication module, used to send the initial image data and the first area indication information to a terminal device before a target operation is performed on the object to be operated, the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, the target image displayed on the terminal device is generated based on the initial image data and the first area indication information, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0039] In a possible implementation, the communication module is also used to receive a second notification message from the terminal device after the target operation is completed, the second notification message is used to indicate that the target operation is completed, and the second notification message also includes second area indication information, the second area indication information includes the coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

[0040] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on a computing device.

[0041] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0042] In a possible implementation manner, the initial image data includes target background image data and historical window image data, wherein the target background image data is background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the historical window image data is window image data received by the terminal device before the target operation is performed on the object to be operated;

[0043] Alternatively, the initial image data includes historical background image data and target window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated;

[0044] Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated;

[0045] Alternatively, the initial image data includes historical background image data and historical window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated.

[0046] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0047] The computing device provided in the sixth aspect corresponds to the virtual desktop display method provided in the second aspect. Therefore, the technical effects of the sixth aspect and any implementation method in the sixth aspect can be found in the relevant description of the technical effects of the above-mentioned second aspect and the corresponding implementation method in the second aspect, and will not be repeated here.

[0048] In a seventh aspect, the present application provides a terminal device, including a communication module for receiving initial image data of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated; a detection module for detecting a target operation for the object to be operated, wherein the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; a rendering module for rendering to obtain a target image, wherein the display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; and a display module for displaying the target image.

[0049] In an eighth aspect, the present application provides a computing device, comprising: a generation module, used to generate initial image data of a virtual desktop, wherein the virtual desktop includes an object to be operated; a communication module, used to send the initial image data to a terminal device before a target operation is performed on the object to be operated, wherein the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop, and a target image displayed on the terminal device is generated based on the initial image data, and a display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0050] In a ninth aspect, the present application provides a computing device comprising a processor and a memory; wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to execute the operating steps of the virtual desktop display method described in any one of the implementation methods of the first to fourth aspects above.

[0051] In a tenth aspect, the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computing device, enables the computing device to execute the operating steps of the virtual desktop display method described in any one of the implementations of the first to fourth aspects above.

[0052] In an eleventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the operating steps of the virtual desktop display method described in any one of the implementations of the first to fourth aspects above.

[0053] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1A schematic diagram of the structure of an exemplary data processing system provided by the present application;

[0055] Figure 2 A schematic diagram of the structure of another exemplary data processing system provided by the present application;

[0056] Figure 3 A flowchart of a virtual desktop display method provided by this application;

[0057] Figure 4a A schematic diagram of dragging the display area of ​​a window from area 1 to area 2;

[0058] Figure 4b A schematic diagram of scaling the display area of ​​the window from area 3 to area 4;

[0059] Figure 4c A schematic diagram of scaling the display area of ​​the window from area 5 to area 6;

[0060] Figure 5a A schematic diagram of obtaining a target image including a border by rendering based on initial image data in a window dragging scenario;

[0061] Figure 5b A schematic diagram of rendering a target image including a border based on initial image data in a window zoom scenario;

[0062] Figure 5c A schematic diagram of rendering a target image including a border based on initial image data in a window rotation scenario;

[0063] Figure 6 A flowchart of another virtual desktop display method provided by the present application;

[0064] Figure 7 A schematic diagram showing that after executing the target operation, the background screen content and the window display area change, but the window screen content does not change;

[0065] Figure 8 A flowchart of another virtual desktop display method provided by the present application;

[0066] Fig. 9 It is a schematic diagram that after executing the target operation, the window screen content and the window display area are changed, but the background screen content is not changed;

[0067] Fig.10 A flowchart of another virtual desktop display method provided by the present application;

[0068] Fig.11 A schematic diagram showing changes in the background screen content, the window screen content and the window display area after the target operation is performed;

[0069] Fig.12 A flowchart of another virtual desktop display method provided by the present application;

[0070] Fig.13 It is a schematic diagram that after executing the target operation, the background screen content and the window screen content remain unchanged, but the window display area changes;

[0071] Fig.14 A schematic diagram of the structure of a terminal device provided for this application;

[0072] Fig.15 A schematic diagram of the structure of a computing device provided for this application;

[0073] Fig.16 A schematic diagram of the hardware structure of a computing device provided for this application. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, various non-limiting implementation methods in the embodiments of the present application are exemplarily described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained based on the above content belong to the scope of protection of the present application.

[0075] See also Figure 1 , which shows a schematic diagram of the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes at least one server and at least one terminal device. For ease of understanding, Figure 1 The example of computing node servers 101 to 102 and terminal devices 201 to 203 is used for explanation. The servers and terminal devices may be interconnected through a network, such as a communication network including at least one switch.

[0076] In actual application scenarios, the servers 101 and 102 may be deployed on the cloud side to support the provision of virtual desktop services. The terminal devices 201 to 203 may be deployed on the user side to support users to use virtual desktops using the terminal devices.

[0077] Among them, the virtual desktop can be a cloud desktop, wherein the cloud desktop, for example, can be a desktop provided by cloud products such as a cloud computer, a cloud laptop, a cloud set-top box, or can be other virtual desktops implemented by computing devices deployed on the cloud side.

[0078] Terminal device 201 (other terminal devices are similar) can be any device that can use a virtual desktop, such as a thin terminal (a terminal device with relatively strong computing and storage capabilities), a fat terminal (a terminal device with relatively weak computing and storage capabilities), cloud extended reality (XR) glasses, a cloud terminal (a terminal device designed specifically for virtual desktop technology), a mobile phone, an iPad, a laptop computer, etc., or other types of terminal devices, without limitation.

[0079] In the process of using the virtual desktop provided by the server 101 using the terminal device 201, the user often performs the operation of adjusting the display area of ​​objects such as windows or icons in the virtual desktop. For example, the user can drag the window in the virtual desktop, or the user can scale the window in the virtual desktop. In the process of adjusting the display area of ​​the object in the virtual desktop by the user, if the terminal device 201 sends the operation performed by the user to the server 101, and the server 101 renders a new image according to the operation, and then sends the image to the terminal device 201 for display, it is easy to cause the image of the virtual desktop displayed by the terminal device 201 to appear in the operation jam or operation delay. The operation jam refers to the process of operating the object in the virtual desktop, the display area of ​​the object in the virtual desktop changes rapidly and significantly, such as the display area of ​​the object in the virtual desktop suddenly moves from display area A to display area B. Operation delay refers to the process of the user operating the object in the virtual desktop, the display area of ​​the object does not change in time with the user's operation. In this way, it will seriously affect the user's experience of using the virtual desktop.

[0080] Based on this, in the data processing system 10 provided in the present application, the terminal device 201 receives the initial image data and area indication information of the virtual desktop sent by the server 101. Among them, the initial image data, for example, can be the background image data of the virtual desktop and the image data corresponding to the object to be operated in the virtual desktop, or it can be the image data of the virtual desktop rendered by the server 101, etc. The area indication information includes the coordinate information or size information of the object to be operated in the virtual desktop, or the area indication information includes the coordinate information and size information at the same time. In the process of the user performing the target operation on the object to be operated in the virtual desktop based on the terminal device 201, the terminal device 201 will detect the target operation performed by the user on the object to be operated according to the area indication information, and the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop (such as the above-mentioned drag operation, etc.). Then, the terminal device 201 renders the target image of the virtual desktop, which is rendered based on the initial image data. For example, the terminal device 201 can determine the moving trajectory of the pointer in the virtual desktop based on the target operation, and render the target image according to the moving trajectory and the initial image data. Finally, the terminal device 201 displays the target image.

[0081] After the target operation is detected, the terminal device 201 will render an image, specifically, the target image of the virtual desktop is rendered according to the target operation and the initial image data currently received, and the target image is displayed on the terminal device 201. That is, the terminal device 201 renders the target image to be displayed based on the initial image data that has been received, without temporarily waiting for the initial image data to be transmitted from the server 101 to the terminal device 201, and without transmitting the rendered target image between the server 101 and the terminal device 201 through the network, which can effectively reduce the time delay of responding to the target operation and presenting the rendered target image on the terminal device 201. Therefore, in the process of the user operating the object to be operated in the virtual desktop, the display area of ​​the object to be operated in the virtual desktop displayed by the terminal device 201 can change in time with the change of the user operation, and the change of the display area of ​​the object to be operated in the virtual desktop will be more smooth, so as to effectively improve the user's experience of using the virtual desktop.

[0082] It is worth noting that the above Figure 1 The data processing system 10 shown is only used as an exemplary description and is not intended to be limiting. For example, in other possible data processing systems, the number of terminal devices and servers may be other numbers, or other types of devices may be included, such as devices for managing server 101 and server 102.

[0083] Alternatively, in other possible data processing systems, in addition to the server, other types of computing devices may be used to provide virtual desktop services to the user's terminal device. Figure 2 In the data processing system 20 shown, a computing device 301 may provide a virtual desktop for a terminal device 302 on the user side.

[0084] Among them, the computing device 301, for example, can be a physical device implementation including a processor, such as a host-side device of a split XR glasses. Among them, the processor can be a CPU, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a DPU, etc., or it can be any combination of the above-mentioned various processors. In addition, the number of processors included in the computing device 301 can be one or more, and the type of processors included can be one or more. The number and type of processors can be set according to the business requirements of the actual application, and this embodiment does not limit this. The terminal device 302, for example, can be a glasses-side device of a split XR glasses, or other types of devices. Furthermore, the terminal device 302 and the computing device 301 can communicate via a network, so that the user can use the virtual desktop on the terminal device 302. At this time, the computing device 301 may not need to be deployed on the cloud side, but may still be deployed on the user side.

[0085] For ease of understanding, an embodiment of displaying a virtual desktop on a terminal device provided by the present application is described below in conjunction with the accompanying drawings.

[0086] See also Figure 3 , Figure 3 A flowchart of an exemplary virtual desktop display method provided in an embodiment of the present application. Figure 3 The virtual desktop display method shown can be applied to Figure 1The data processing system 10 shown in the figure may be applied to other possible data processing systems. Figure 1 The data processing system 10 shown and the server 101 providing a virtual desktop for the terminal device 201 are used as an example for explanation.

[0087] like Figure 3 As shown, the virtual desktop display method may specifically include the following steps.

[0088] S301: The server 101 sends initial image data of the virtual desktop and area indication information 1 to the terminal device 201, wherein the virtual desktop includes an object to be operated, the initial image data is image data rendered on the server 101, and the area indication information 1 includes coordinate information or size information of the object to be operated in the virtual desktop.

[0089] Exemplarily, the object to be operated in the virtual desktop refers to an object to be operated that can be controlled and its display area in the virtual desktop can be adjusted, for example, it can be a window, an icon, etc., or it can be other types of objects to be operated.

[0090] In this embodiment, before the user adjusts the display area of ​​the object to be operated in the virtual desktop, the server 101 can be responsible for executing the image rendering process for the virtual desktop and sending the rendered image to the terminal device 201, that is, the initial image data sent by the server 101 to the terminal device 201 is specifically the data of the image rendered on the server 101.

[0091] In a possible implementation, the terminal device 201 presents a virtual desktop to the user, and the user can perform corresponding interactive operations on the virtual desktop, and the interactive operations are specifically operations that do not change the display area of ​​the object to be operated in the virtual desktop, such as moving the mouse, single-clicking / double-clicking the object to be operated in the virtual desktop, and the like. Then, the terminal device 201 can send the interactive operations performed by the user to the server 101. The server 101 can render and generate initial image data of the virtual desktop based on the received interactive operations, and send the initial image data to the terminal device 201 through the network. In actual application, in addition to performing operations on the object to be operated through the mouse, operations on the object to be operated can also be performed through a remote control, a handle, a touch screen, a gesture detection device, and the like.

[0092] In addition, the server 101 will also send area indication information 1, so as to use the area indication information 1 to indicate the display area of ​​the object to be operated in the virtual desktop in the initial image data. Exemplarily, the area indication information 1 can be, for example, the coordinate information and size information of the object to be operated in the virtual desktop. Among them, the coordinate information can be, for example, a coordinate value, such as the coordinate value of the center point of the object to be operated in the virtual desktop. The size information can be, for example, information such as length and width, which can indicate the size of the object to be operated in the virtual desktop. Alternatively, the area indication information 1 can only include the coordinate information of the object to be operated in the virtual desktop (the size of the object to be operated in the virtual desktop is fixed); or, the area indication information 1 can only be the size information of the object to be operated in the virtual desktop (the coordinates of the object to be operated in the virtual desktop are fixed), etc., and this is not limited.

[0093] In actual application, the server 101 may also send the identifier of the object to be operated to the terminal device 201, so as to use the identifier to notify the terminal device 201 that the area indication information 1 is used to describe the coordinate information or size information of which object to be operated in the virtual desktop.

[0094] S302: The terminal device 201 displays an image corresponding to the initial image data of the virtual desktop.

[0095] When the user uses the virtual desktop using the terminal device 201 , the terminal device 201 displays a corresponding image according to the initial image data sent by the server 101 , so that the user can continue to use the virtual desktop based on the image of the virtual desktop currently presented by the terminal device 201 .

[0096] In a specific implementation, the terminal device 201 can decode the received initial image data, and output the decoded image data to a display device (such as a display screen, etc.) on the terminal device 201, so that the image corresponding to the initial image data can be displayed through the display device.

[0097] In actual application scenarios, the terminal device 201 can present multiple frames of images of the virtual desktop per second, such as 30 frames of images of the virtual desktop per second, so as to ensure the smoothness of the screen content of the virtual desktop seen by the user. Before the user adjusts the display area of ​​the object to be operated in the virtual desktop, the screen content in the virtual desktop presented by the terminal device 201 usually changes at a low speed (such as the screen content remains basically unchanged), so the images of the virtual desktop presented by the terminal device 201 can all be images rendered by the server 101.

[0098] S303: The terminal device 201 detects a target operation on the object to be operated in the virtual desktop according to the received area indication information 1, where the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop.

[0099] The display area of ​​the object to be operated in the virtual desktop is specifically an imaging area of ​​the object to be operated in the image of the virtual desktop.

[0100] In actual application scenarios, when using a virtual desktop, users often need to adjust the display area of ​​the object to be operated in the virtual desktop. For example, when multiple windows in the virtual desktop overlap each other, the user can adjust the display area of ​​the partial window in the virtual desktop by dragging part of the window so that the contents of multiple windows do not block each other. For another example, when the display area of ​​the image in the virtual desktop does not meet the user's expectations, the user can drag the icon to another location in the virtual desktop. For another example, when the window size in the virtual desktop does not meet the user's expectations, the user can scale the window to enlarge or reduce the size of the window.

[0101] When the user needs to adjust the display area of ​​the object to be operated in the virtual desktop, the user can use the mouse or other devices to perform a target operation on the object to be operated, and the target operation can be, for example, a drag operation, a zoom operation, or a rotation operation, or other possible operations. The terminal device 201 can detect in real time whether there is an operation for the object to be operated based on the received area indication information 1. In actual application scenarios, the object operated by the user (such as the icon / window selected by the user clicking or moving the mouse), or the object determined based on the operation performed by the user (such as the window popped up by the user double-clicking the icon), can be called a focus object (such as a focus window, a focus image, etc.), wherein the object to be operated is the focus object. In addition, the operation performed by the user can be other operations in addition to the above-mentioned target operation, such as a single click or double-click operation performed by the user on the object.

[0102] In the first implementation example, the image of the virtual desktop presented by the terminal device 201 may include a pointer, which is used to indicate the operation position in the virtual desktop, and the imaging position of the pointer in the virtual desktop may change with the target operation during the user's execution of the target operation. Then, when the object to be operated is specifically an icon (or other types of objects to be operated), the terminal device 201 may detect whether the operation position indicated by the pointer is located within the display area of ​​the icon. If so, the terminal device 201 may determine that a target operation for the icon is detected. If not, the terminal device 201 may determine that there is currently no target operation for the image.

[0103] In the second implementation example, when the object to be operated is a window (or other types of objects to be operated), the terminal device 201 can detect whether the operation position indicated by the pointer is located in the controllable area of ​​the window. The controllable area refers to a part of the display area of ​​the window, and the part is used to detect whether there is an operation to control the window. For example, Figure 4a In the window shown, the gray area is the controllable area in the window; when the user moves the operation position indicated by the pointer to the controllable area by operating a device such as a mouse, the window can be controlled, such as dragging the entire display area of ​​the window from Figure 4a The area 1 shown in the figure moves to the area 2, and so on. At this time, when the user performs operations such as dragging or zooming, the terminal device 201 can determine that the operation is valid and control the window accordingly. When it is detected that the operation position indicated by the pointer is within the controllable area of ​​the window, the terminal device 201 can determine that the target operation for the window is detected. However, when the operation position indicated by the pointer is not within the controllable area of ​​the window, the terminal device 201 can determine that the target operation for the window is detected. Figure 4a If the gray area in the window, that is, the operation position indicated by the pointer is located in the non-controllable area in the window or in the area outside the window, the terminal device 201 can determine that there is no operation on the window at present.

[0104] Among them, before the target operation is performed on the window, the terminal device 201 can obtain the indication information of the controllable area in the window, so as to determine the controllable area in the window by using the indication information. As a first example, in the process of sending the initial image data to the terminal device 201, the server 101 can send the indication information of the controllable area corresponding to the window on which the target operation is performed to the terminal device 201, and the indication information can be, for example, the coordinate information (such as coordinate value) and size information (such as length and width) of the controllable area, or only the coordinate information of the controllable area (the size of the controllable area is fixed), or only the size information of the controllable area (the coordinates of the controllable area are fixed relative to the window coordinates). In the process of the window being performed on the target operation, the server 101 may not send the indication information of the controllable area. As a second example, when the controllable area is relatively fixed in the area within the window, the terminal device 201 can also determine the controllable area in the target display area after determining the target display area corresponding to the window, such as the controllable area is fixed to the first N rows of pixels in the target display area of ​​the window (N is a positive integer). In this way, the terminal device 201 can determine whether there is a target operation for the window based on the acquired indication information of the controllable area.

[0105] In actual application scenarios, after detecting that there is a target operation on the object to be operated, the terminal device 201 can also send a notification message 1 to the server 101, so as to use the notification message 1 to notify the server 101 that the target operation starts to be performed on the object to be operated, such as Figure 3 Alternatively, after detecting that there is a target operation on the object to be operated, the terminal device 201 may not send the notification message 1 to the server 101 .

[0106] S304: The terminal device 201 renders and obtains a target image of the virtual desktop, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image.

[0107] In this embodiment, after the terminal device 201 detects the target operation for adjusting the display area of ​​the object to be operated, it does not send the target operation to the server 101, but instead renders the target image to be displayed on the terminal device 201 based on the target operation and the initial image data currently received.

[0108] In a possible implementation, the terminal device 201 may first generate border image data according to the target operation, and the area within the border indicated by the border image data is the target display area, which is the display area of ​​the object to be operated in the target image to be generated.

[0109] In a specific implementation, after detecting the target operation, the terminal device 201 may record the movement track of the pointer in the virtual desktop, and calculate the target display area according to the movement track and the received area indication information 1.

[0110] For example, when the target operation performed by the user is specifically a drag operation, and the object to be operated is specifically a window, assuming that the display area of ​​the window in the initial image data is as follows: Figure 4a The area 1 shown in the figure can be indicated by the coordinates 1 and size of the window. Then, in the process of adjusting the display area of ​​the window, the terminal device 201 can calculate the coordinates 2 of the window in the virtual desktop when the user performs the target operation to reach the current stage according to the coordinates 1, size and movement track of the pointer of the area 1, and determine the target display area of ​​the window in the virtual desktop according to the coordinates 2 and the size of the window (the size of the window remains unchanged in the dragging scenario). Figure 4a Thus, after the window is dragged, the display area of ​​the window in the virtual desktop is changed from area 1 to area 2.

[0111] For another example, when the target operation performed by the user is specifically a zoom operation (specifically an operation of reducing a window), and the object to be operated is specifically a window, assuming that the display area of ​​the object to be operated in the initial image data is as follows: Figure 4b Then, in the process of adjusting the display area of ​​the window, the terminal device 201 can calculate the size 2 of the window after scaling according to the movement track of the pointer and the window size 1, and determine the target display area of ​​the window in the virtual desktop according to the size 2 and the coordinates of the window (assuming that the window coordinates remain unchanged in the scaling scenario). Figure 4b Thus, after the window is scaled, the display area of ​​the window in the virtual desktop is changed from area 3 to area 4.

[0112] For another example, when the target operation performed by the user is specifically a rotation operation, and the object to be operated is specifically a window, assuming that the display area of ​​the object to be operated in the initial image data is as follows: Figure 4c Then, in the process of adjusting the display area of ​​the window, the terminal device 201 can calculate the rotation angle of the window after scaling according to the movement trajectory of the pointer, and determine the target display area of ​​the object to be operated in the virtual desktop according to the rotation angle and the coordinates and size of the window (assuming that the coordinates and size of the window remain unchanged in the rotation scenario). Figure 4c Thus, after the window is scaled, the display area of ​​the window in the virtual desktop is changed from area 5 to area 6.

[0113] Then, the terminal device 201 can generate the frame image data corresponding to the target display area, and the area within the frame indicated by the frame image data is the target display area. Among them, the frame image data can be used to draw a frame in the image, and the attribute information such as the width, color, style, and line (dashed / solid line) of the drawn frame can be pre-configured in the terminal device 201, for example, the attribute information can be pre-configured in the terminal device 201 by a technician, or the attribute information can be configured in the terminal device 201 by the server 101, etc.

[0114] Finally, the terminal device 201 can render the target image of the virtual desktop according to the border image data, the movement trajectory of the pointer, and the initial image data of the virtual desktop that has been received. Specifically, the position of the border in the image can be determined according to the movement trajectory of the pointer, and the border image data and the initial image data are superimposed and drawn based on the position to achieve the rendering of the target image. The display area of ​​the object to be operated in the rendered target image is different from the display area of ​​the object to be operated in the image corresponding to the initial image data, so that the display area of ​​the object to be operated in the virtual desktop can be adjusted.

[0115] For example, assuming that the target operation is a drag operation, the object to be operated is a window, and the image corresponding to the initial image data can be, for example, Figure 5a Then, after detecting the drag operation, the terminal device 201 generates frame image data according to the drag operation, and renders the frame image data and the initial image data to obtain the following image: Figure 5a The target image is shown below. Figure 5a The black frame shown in is the frame drawn based on the frame image data corresponding to the drag operation, and the area within the black frame is the area where the window should be located in the virtual desktop after the drag operation is performed. That is, during the process of dragging the window, the window frame will appear in the image screen presented by the terminal device 201, and the position of the frame will continue to change with the user's drag operation, while the size of the frame remains unchanged. It should be noted that Figure 5a The dotted line in FIG. 2 is only used to reflect the trajectory of the window being dragged (ie, the movement trajectory of the pointer), and the screen content of the target image rendered by the terminal device 201 does not include the dotted line.

[0116] For another example, assuming that the target operation is a zoom operation, the object to be operated is a window, and the image corresponding to the initial image data can be, for example, Figure 5b Then, after detecting the zoom operation, the terminal device 201 generates frame image data according to the zoom operation, and renders the frame image data and the initial image data to obtain the following image: Figure 5b The target image shown below. In the process of zooming the window, a window border will appear in the image screen presented by the terminal device 201, and the size of the border will continue to change with the user's zooming operation. It should be noted that Figure 5b The dotted line in FIG. 2 is only used to reflect the trajectory of the window being scaled (ie, the movement trajectory of the pointer), and the screen content of the target image rendered by the terminal device 201 does not include the dotted line.

[0117] For another example, assuming that the target operation is a rotation operation, the object to be operated is a window, and the image corresponding to the initial image data can be, for example, Figure 5c Then, after detecting the rotation operation, the terminal device 201 generates frame image data according to the rotation operation, and renders the frame image data and the initial image data to obtain the following image: Figure 5c The target image shown below. In the process of rotating the window, a window frame will appear in the image displayed by the terminal device 201, and the rotation angle of the frame will continue to change with the user's rotation operation. It should be noted that Figure 5cThe dotted line in FIG. 2 is only used to reflect the rotation angle of the window (determined based on the movement trajectory of the pointer), and the screen content of the target image rendered by the terminal device 201 does not include the dotted line.

[0118] It is worth noting that the above implementation method of rendering the target image according to the target operation and the initial image data is only an exemplary description. In other embodiments, the terminal device 201 can also use other methods to render the target image, which is not limited to this.

[0119] S305: The terminal device 201 displays the rendered target image.

[0120] In this embodiment, after the terminal device 201 obtains the target image through local rendering, the target image may be displayed so that the user can see the changes in the display area of ​​the object to be operated under the control of the target operation on the terminal device 201 .

[0121] Since, in the process of responding to the target operation performed by the user, the terminal device 201 renders and displays the target image based on the initial image data that has been received, there is no need to temporarily wait for the server 101 to send new image data to render the target image, that is, there is no need to perform the process of transmitting the image data between the server 101 and the terminal device 201, which can effectively reduce the time delay of the terminal device 201 responding to the target operation and presenting the target image corresponding to the target operation. Thus, in the process of the user operating the object to be operated in the virtual desktop, the display area of ​​the object to be operated displayed by the terminal device 201 in the virtual desktop can change in real time with the operation performed by the user, and the change of the display area of ​​the object to be operated in the virtual desktop will be more fluid, which can effectively improve the user's experience of using the virtual desktop.

[0122] Understandably, Figure 3 An example of an example of a target image of a virtual desktop rendered by the terminal device 201 is used for illustrative explanation. In actual application, the terminal device 201 can generate and display multiple consecutive images of the virtual desktop based on the above implementation process and the target operation continuously performed by the user. As the user continues to perform the target operation, the user can see the continuous dynamic changes in the display area of ​​the object to be operated on the terminal 201, such as the border position and size of the display area for indicating the object to be operated in the image screen continuously presented by the terminal device 201 changes with the user's operation.

[0123] In a further possible implementation, when the terminal device 201 detects that the target operation on the object to be operated in the virtual desktop is completed, such as the terminal device 201 detects a lift event on the object to be operated (such as a mouse button being lifted), the terminal device 201 can send a notification message 2 to the server 101, such as Figure 3 As shown, the notification message 2 can indicate that the target operation is completed. In addition, the notification message 2 can also include area indication information 2, which can include coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed. When the target operation is a drag operation, the area indication information 2 can be coordinate information; when the target operation is a zoom operation, the area indication information can be size information; when the target operation is a rotation operation, the area indication information 2 can be rotation angle information, which is used to indicate the angle at which the object to be operated is rotated. In this way, after receiving the notification message 2, the server 101 can render a new image based on the area indication information 2 in the notification message 2, and send the new image to the terminal device 201 for display.

[0124] In this way, after the target operation is completed, the server 101 can continue to render the image of the virtual desktop and display the image on the terminal device 201, so that the display area of ​​the object to be operated viewed by the user on the terminal device 201 (that is, the display area of ​​the object to be operated in the image rendered by the server 101) is consistent with the display area of ​​the object to be operated when the user ends the target operation (that is, the display area of ​​the object to be operated in the image rendered by the terminal device 201). This ensures that the position change of the object to be operated in the virtual desktop can maintain a high level of smoothness before and after the user performs the target operation on the object to be operated (avoiding the display area of ​​the object to be operated in the virtual desktop suddenly changing to the display area where it was before the target operation was performed after the target operation is performed on the object, thereby ensuring the user's experience of using the virtual desktop.

[0125] In addition, the screen presented by the terminal device 201 may also include a pointer, and the notification message 2 sent by the terminal device 201 may also include the display position of the pointer in the screen, that is, the position in the virtual desktop. In this way, in the newly rendered image sent by the server 101 to the terminal device 201, the imaging position of the pointer in the image is consistent with the position of the pointer in the virtual desktop after the user completes the target operation.

[0126] In actual application, during the process of executing the target operation on the object to be operated, the server 101 can continuously send the data of the image rendered on the server 101 to the terminal device 201. Thus, each time the terminal device 201 needs to render and generate a new image (the terminal device 201 renders and generates an image at regular intervals and displays it to ensure the smoothness of the change of the display area of ​​the object to be operated displayed on the terminal device 201), the new image can be rendered and displayed according to the data of the latest image received and saved locally in advance (sent by the server 101 to the terminal device 201). Among them, after detecting the target operation, the terminal device 201 can send a notification message 1 to the server 101 to notify the server 101 that the object to be operated begins to be executed. Alternatively, the terminal device 201 may not send the notification message 1.

[0127] Alternatively, after detecting the target operation, the terminal device 201 may send a notification message 1 to the server 101 to notify the server 101 that the target operation has begun to be performed on the object to be operated. At this time, upon receiving the notification message 1 (i.e., during the process of performing the target operation on the object to be operated), the server 101 may stop sending the data of the image rendered on the server 101 to the terminal device 201. Among them, before the terminal device 201 sends the notification message 1, the server 101 will continue to send image data to the terminal device 201. Thus, each time the terminal device 201 needs to render and generate a new image, it can render and display a new image based on the initial image data that has been saved locally. In this way, only the position of the border indicating the display area of ​​the object to be operated changes in the continuous multiple images of the virtual desktop presented by the terminal device 201.

[0128] It is worth noting that the above Figure 3 The embodiment shown is only used as an exemplary description. In actual application, Figure 3 Some steps may be added, deleted or replaced on the basis of the illustrated embodiment, but this is not limited to the above.

[0129] For example, in other embodiments, before the user performs the target operation, the server 101 only sends the initial image data of the rendered virtual desktop, and does not send the area indication information 1. After the terminal device 201 detects the target operation, the terminal device 201 can send a notification message 1 to the server 101 to notify the server 101 that the target operation has begun to be performed on the object to be operated. At this time, upon receiving the notification message 1, the server 101 will send the area indication information 1 to the terminal device 201, so that the terminal device 201 determines the display area of ​​the object to be operated in the initial image data of the virtual desktop according to the received area indication information 1, and further renders the target image according to the area indication information 1.

[0130] For another example, in this embodiment, the terminal device 201 determines the display area (or controllable area) of the object to be operated in the image corresponding to the initial image data according to the area indication information 1 (or the indication information of the controllable area) sent by the server 101, and determines that there is currently a target operation for the object to be operated by detecting that the operation position indicated by the pointer is within the display area of ​​the object to be operated (or the controllable area included in the display area), thereby detecting the target operation. In other embodiments, the server 101 may not send the area indication information 1 (and the indication information of the controllable area) to the terminal device 201. At this time, the terminal device 201 can identify the display area of ​​the object to be operated in the image corresponding to the initial image data through artificial intelligence (AI) models or computer vision (CV) and other technologies.

[0131] In a specific implementation, the terminal device 201 can input the initial image data into a pre-trained AI model, and the AI ​​model can perform reasoning based on the initial image data and output a reasoning result, which is used to indicate the focus object in the image corresponding to the initial image data and the display area of ​​the focus object in the initial image data. Alternatively, the terminal device 201 can use CV technology to identify the focus object in the initial image data and the display area of ​​the focus object in the image corresponding to the initial image data, such as identifying the focus object and its display area in the image through an edge detection algorithm or a shape matching algorithm. Among them, the identified focus object is the above-mentioned object to be operated. After determining the object to be operated and the display area of ​​the object to be operated in the image corresponding to the initial image data, the terminal device 201 can determine that there is currently an operation on the object to be operated by detecting that the operation position indicated by the pointer is within the controllable area included in the display area of ​​the object to be operated, and further determine whether the operation is a target operation. If the terminal device 201 does not detect a lift event for the object to be operated within a preset time period after detecting a press event for the object to be operated, the operation can be determined to be a target operation, thereby achieving detection of the target operation for the object to be operated.

[0132] In this way, the server 101 only needs to send the initial image data to the terminal device 201 (without sending the area indication information). Accordingly, the terminal device 201 can detect the target operation based on the above method, and render the target image according to the target operation and the received initial image data, and display the target image, thereby reducing the delay of the terminal device 201 responding to the target operation and presenting the target image corresponding to the target operation, and improving the user experience of using the virtual desktop.

[0133] Above Figure 3 In the illustrated embodiment, an example is taken in which the server 101 renders an image and sends the image data (ie, initial image data) to the terminal device 201 .

[0134] In other embodiments, the initial image data sent by the server 101 to the terminal device 201 may specifically be background image data and window image data that have not been rendered. Figure 6 , Figure 8 , Fig.10 as well as Fig.12 This is explained in detail. Figure 6 , Figure 8 , Fig.10 as well as Fig.12 In the illustrated embodiment, the object to be operated may specifically be a window.

[0135] See also Figure 6 , showing another flow chart of virtual desktop display. Figure 6 The illustrated embodiment may specifically include the following steps.

[0136] S601 : The server 101 sends historical background image data, historical window image data, and area indication information 1 of the virtual desktop to the terminal device 201 .

[0137] The background image data refers to the image data used as the background in the virtual desktop. The background image described by the background image data may not include any type of object; or the background image described by the background image data may include objects other than the window on which the target operation is performed. When the background image described by the background image data includes multiple objects other than the window on which the target operation is performed, the display areas of the multiple objects may overlap or may not overlap. The historical background image data refers to the background image data received by the terminal device 201 before the window is subjected to the target operation.

[0138] The window image data refers to the image data used to describe the window in the virtual desktop. The historical window image data refers to the window image data received by the terminal device 201 before the target operation is performed on the object to be operated (ie, the window).

[0139] The region indication information 1 includes coordinate information or size information of the window in the virtual desktop, or includes both coordinate information and size information. In actual application, the region indication information 1 can be used to indicate the display region of the window in the virtual desktop.

[0140] In this embodiment, the image of the virtual desktop can be rendered on the terminal device 201. Therefore, before the user performs a target operation on a window in the virtual desktop, the server 101 can send the terminal device 201 relevant data required for participating in generating the image of the virtual desktop, which can specifically include historical background image data, historical window image data, and area indication information 1 of the virtual desktop.

[0141] S602: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data and the area indication information 1, and displays the initial image.

[0142] When the user uses the virtual desktop using the terminal device 201, the terminal device 201 can determine the display area of ​​the window in the virtual desktop according to the area indication information, and overlay the historical window image data with the historical background image data according to the display area to render an image, which is referred to as the initial image in this embodiment for the sake of distinction. Then, the terminal device 201 will display the initial image so that the user can continue to use the virtual desktop based on the initial image of the virtual desktop currently presented by the terminal device 201.

[0143] In actual application, the terminal device 201 can locally save the received historical window image data and the area indication information 1.

[0144] S603: The terminal device 201 detects a target operation on a window in the virtual desktop according to the area indication information 1.

[0145] In this embodiment, the terminal device 201 detects the target operation in a manner similar to the above Figure 3 The implementation method of the terminal device 201 detecting the target operation in the illustrated embodiment is similar, and the details may refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0146] S604: The terminal device 201 sends a notification message 1 to the server 101, where the notification message 1 is used to indicate that the window starts to be executed with the target operation.

[0147] S605 : The server 101 sends the target background image data of the virtual desktop to the terminal device 201 .

[0148] In this embodiment, after detecting the target operation, the terminal device 201 may notify the server 101. At this time, the image data sent by the server 101 to the terminal device 201 may not include the window image data and the area indication information, but may only include the target background image data of the virtual desktop. The target background image data refers to the background image data received by the terminal device 201 after the target operation is performed on the window. In addition, the background image of the virtual desktop described by the target background image data may be different from the background image of the virtual desktop described by the historical background image data.

[0149] S606: The terminal device 201 renders a target image of the virtual desktop, wherein a display area of ​​the window in the target image is determined based on a target operation, and the target image is obtained based on target background image data and historical window image data.

[0150] S607: The terminal device 201 displays the rendered target image.

[0151] In this embodiment, the terminal device 201 can render a target image (i.e., a new image of the virtual desktop) based on the pre-saved historical window image data and the received target background image data, and display the target image. At this time, the screen content of the window image in the target image displayed by the terminal device 201 remains unchanged, but the display area of ​​the window in the target image (i.e., the target display area) is different from the display area of ​​the window in the initial image, and the background screen content in the target image is different from the background screen content in the initial image. In this way, the user can not only see the changes in the display area of ​​the window under the control of the target operation on the terminal device 201, but also see the changes in the background screen content of the virtual desktop during the operation of the window (for example, the background screen of the virtual desktop can be a continuously changing dynamic wallpaper, etc.). For example, Figure 7 As shown, before the user performs the target operation, the initial image displayed by the terminal device 201 is Figure 7 The image shown above, but after the user performs the target operation, the target image displayed by the terminal device 201 is Figure 7 The image shown below shows that the background content and the window's display area in the virtual desktop change, but the content in the window remains unchanged.

[0152] Furthermore, in the process of responding to the target operation performed by the user, the image data sent by the server 101 to the terminal device 201 can be reduced from the two types of data, namely, background image data and window image data, to one type of data, namely, background image data. This can effectively reduce the amount of data transmitted between the server 101 and the terminal device 201, that is, it can reduce the transmission delay of data between the server 101 and the terminal device 201, thereby reducing the delay of the terminal device 201 responding to the target operation and presenting the target image corresponding to the target operation. In this way, in the process of the user operating the window in the virtual desktop, the display area of ​​the window in the virtual desktop displayed by the terminal device 201 can change in real time with the operation performed by the user, and the fluency of the change of the display area of ​​the window in the virtual desktop will also be higher, thereby effectively improving the user's experience of using the virtual desktop.

[0153] In actual application, during the process of the user performing the target operation, the server 101 can detect whether the target background image data currently required to be sent to the terminal device 201 is consistent with the historical background image data sent previously. And, when the two background image data are inconsistent, the server 101 will send new background image data (i.e., the above-mentioned target background image data) to the terminal device 201, so that the terminal device 201 can render a new image based on the new background image data. And when the target background image data is consistent with the historical background image data, the server 101 may not send the target background image data. At this time, after the terminal device 201 does not receive the target background image data within a preset time period (the duration is less than the duration of the transmission background image data), it can render the target image based on the historical background image data (previously saved), the historical window image data, and the area indication information 1.

[0154] In a further possible implementation, when the terminal device 201 detects that the target operation on the window in the virtual desktop is completed, the terminal device 201 may send a notification message 2 to the server 101, such as Figure 6As shown, the notification message 2 can indicate that the target operation is completed. In addition, the notification message 2 can also include area indication information 2, which can be used to indicate the display area of ​​the window in the virtual desktop when the target operation is completed. Specifically, the area indication information 2 can include the coordinate information, size information or rotation angle information of the window in the virtual desktop when the target operation is completed. Further, the notification message 2 sent by the terminal device 201 can also carry the position of the pointer in the virtual desktop (which can be determined according to the movement trajectory of the pointer). In this way, after receiving the notification message 2, the server 101 can resume sending window image data, background image data, and area indication information 2 to the terminal device 201, so that the terminal device 201 can render a new image based on the received image data and display the image. In addition, before and after the user performs the target operation on the window, the position change of the window in the virtual desktop can have a high level of fluency, which can ensure the user's experience of using the virtual desktop.

[0155] Above Figure 6 In the embodiment shown, the background of the virtual desktop displayed by the terminal device 201 changes, but the window screen content remains unchanged. Figure 8 In the embodiment shown, the background of the virtual desktop displayed by the terminal device 201 remains unchanged, while the window screen content changes during the user's operation of the window. Figure 8 This is described in detail. The object to be operated on which the target operation is performed may specifically be a window.

[0156] See also Figure 8 , showing another flow chart of virtual desktop display. Figure 8 The illustrated embodiment may specifically include the following steps.

[0157] S801: The server 101 sends historical background image data, historical window image data, and area indication information 1 of the virtual desktop to the terminal device 201.

[0158] In this embodiment, the image data sent by the server 101 to the terminal device 201 is specifically an unrendered image.

[0159] S802: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data and the area indication information 1, and displays the initial image.

[0160] In actual application, the terminal device 201 can locally save the received historical background image data and the area indication information 1.

[0161] S803: The terminal device 201 detects a target operation on a window in the virtual desktop according to the area indication information 1.

[0162] S804: The terminal device 201 sends a notification message 1 to the server 101, where the notification message 1 is used to indicate that the window starts to be executed with the target operation.

[0163] In this embodiment, the implementation method of steps S801 to S804 is the same as that of the above Figure 6 The implementation methods of steps S601 to S604 in the illustrated embodiment are similar, and the details may refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0164] S805 : The server 101 sends the target window image data of the virtual desktop to the terminal device 201 .

[0165] In this embodiment, after detecting the target operation, the terminal device 201 may notify the server 101. At this time, the data sent by the server 101 to the terminal device 201 may not include the background image data and the area indication information, but may only include the target window image data. The target window image data refers to the window image data received by the terminal device 201 after the target operation is performed on the window. The target window image data, and the window screen content described may be different from the window screen content described by the historical background image data.

[0166] S806: The terminal device 201 renders a target image of the virtual desktop, wherein a display area of ​​the window in the target image is determined based on a target operation, and the target image is obtained based on historical background image data and target window image data.

[0167] S807: The terminal device 201 displays the rendered target image.

[0168] In this embodiment, the terminal device 201 can generate a target image (that is, a new image of the virtual desktop) based on the pre-saved historical window image data and the newly received target background image data, and display the target image. At this time, the background screen content in the target image displayed by the terminal device 201 remains unchanged, but the display area of ​​the window in the target image (that is, the target display area) is different from the display area of ​​the window in the initial image, and the window screen content in the target image is different from the window screen content in the initial image. In this way, the user can not only see the changes in the display area of ​​the window under the control of the target operation on the terminal device 201, but also see the changes in the window screen content in the virtual desktop during the operation of the window (for example, a continuously changing video frame can be played in the window of the virtual desktop, etc.). For example, Fig. 9As shown, before the user performs the target operation, the initial image displayed by the terminal device 201 is Fig. 9 The image shown above, but after the user performs the target operation, the target image displayed by the terminal device 201 is Fig. 9 The image shown below shows that the window content and the window display area in the virtual desktop change, but the background content of the virtual desktop remains unchanged.

[0169] Furthermore, in the process of responding to the target operation performed by the user, the image data sent by the server 101 to the terminal device 201 can be reduced from two types of data, namely, background image data and window image data, to one type of data, namely, window image data. This can effectively reduce the amount of data transmitted between the server 101 and the terminal device 201, that is, it can reduce the transmission delay of data between the server 101 and the terminal device 201, thereby reducing the delay of the terminal device 201 responding to the target operation and presenting the target image corresponding to the target operation. In this way, in the process of the user operating the window in the virtual desktop, the display area of ​​the window in the virtual desktop displayed by the terminal device 201 can change in real time with the operation performed by the user, and the fluency of the change of the display area of ​​the window in the virtual desktop will also be higher, thereby effectively improving the user's experience of using the virtual desktop.

[0170] In actual application, when the user performs the target operation, the server 101 can detect whether the target window image data currently required to be sent to the terminal device 201 is consistent with the historical window image data sent previously. And, when the two window image data are inconsistent, the server 101 will send new window image data (i.e., the above-mentioned target window image data) to the terminal device 201, so that the terminal device 201 can render a new image based on the new window image data. When the target window image data is consistent with the historical window image data, the server 101 may not send the target window image data. At this time, after the terminal device 201 does not receive the target window image data within a preset time period (the duration is less than the duration of transmitting the background image data), it can render the target image based on the historical window image data (previously saved), the historical background image data, and the area indication information 1.

[0171] In a further possible implementation, when the terminal device 201 detects that the target operation on the window in the virtual desktop is completed, the terminal device 201 may send a notification message 2 to the server 101, such as Figure 8As shown, the notification message 2 can indicate that the target operation is completed. In addition, the notification message 2 can also include area indication information 2, which can include coordinate information, size information or rotation angle information of the window in the virtual desktop when the target operation is completed. Further, the notification message 2 sent by the terminal device 201 can also carry the position of the pointer in the virtual desktop (which can be determined according to the movement trajectory of the pointer). In this way, after receiving the notification message 2, the server 101 can resume sending the window image data, background image data, and the area indication information 2 carried by the notification message 2 to the terminal device 201, so that the terminal device 201 renders a new image based on the received data and displays the image. In addition, before and after the user performs the target operation on the window, the position change of the window in the virtual desktop can have a high level of fluency, which can ensure the user's experience of using the virtual desktop.

[0172] See also Fig.10 , shows another flow chart of virtual desktop display. Fig.10 In the illustrated embodiment, the background of the virtual desktop and the window screen content displayed by the terminal device 201 can change during the process of the user operating the window. Figure 8 The illustrated embodiment may specifically include the following steps.

[0173] S1001 : The server 101 sends historical background image data, historical window image data, and area indication information 1 of the virtual desktop to the terminal device 201 .

[0174] In this embodiment, the image data sent by the server 101 to the terminal device 201 is specifically an unrendered image.

[0175] S1002: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data and the area indication information 1, and displays the initial image.

[0176] S1003: The terminal device 201 detects a target operation on a window in the virtual desktop according to the area indication information 1.

[0177] S1004: The terminal device 201 sends a notification message 1 to the server 101, where the notification message 1 is used to start executing the target operation on the window.

[0178] In this embodiment, the implementation method of steps S1001 to S1004 is the same as that of the above Figure 6 The implementation methods of steps S601 to S604 in the illustrated embodiment are similar, and the details may refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0179] S1005 : The server 101 sends the target background image data, target window image data and area indication information 1 of the virtual desktop to the terminal device 201 .

[0180] In this embodiment, the server 101 continuously sends the background image data, the window image data, and the region indication information to the terminal device 201, that is, the process of the server 101 sending the background image data, the window image data, and the region indication information 1 to the terminal device 201 is independent of the process of the terminal device 201 detecting the target operation. Therefore, during the process of the terminal device 201 detecting the target operation, the server 101 can receive the target background image data, the target window image data, and the region indication information 1 newly sent by the server 101, and save them locally in the terminal device 201.

[0181] S1006: The terminal device 201 renders a target image of the virtual desktop, wherein a display area of ​​the window in the target image is determined based on a target operation, and the target image is obtained based on target background image data and target window image data.

[0182] S1007: The terminal device 201 displays the rendered target image.

[0183] In this embodiment, the terminal device 201 can generate a target image based on the saved target window image data and target background image data, and display the target image. At this time, the background image content, window image content, and the display area of ​​the window in the virtual desktop in the target image displayed by the terminal device 201 all change. In this way, the user can not only see the changes in the display area of ​​the window under the control of the target operation on the terminal device 201, but also see the changes in the background image content and window image content in the virtual desktop during the window operation. For example, Fig.11 As shown, before the user performs the target operation, the initial image displayed by the terminal device 201 is Fig.11 The image shown above, but after the user performs the target operation, the target image displayed by the terminal device 201 is Fig.11 The image shown below shows that the background content, window content, and the window's display area in the virtual desktop have all changed.

[0184] In actual application, during the process of the user performing the target operation, the server 101 can detect whether the target window image data currently needed to be sent to the terminal device 201 is consistent with the historical window image data sent previously, and detect whether the target background image data is consistent with the historical background image data sent previously. And, when the two window image data are inconsistent, the server 101 will send new window image data (i.e. the above-mentioned target window image data) to the terminal device 201, and when the two background image data are inconsistent, the server 101 will send new background image data (i.e. the above-mentioned target background image data) to the terminal device 201, so that the terminal device 201 renders a new image based on the new background image data and the window image data. And when the target window image data is consistent with the historical window image data, the server 101 may not send the target window image data, and at the same time, when the target background image data is consistent with the historical background image data, the server 101 may not send the target background image data. At this time, after receiving the target background image data and the target window image data within the preset time period, the terminal device 201 can render the target image based on the newly received target window image data and target background image data (and the area indication information 1).

[0185] In a further possible implementation, when the terminal device 201 detects that the target operation on the window in the virtual desktop is completed, the terminal device 201 may send a notification message 2 to the server 101, such as Fig.10 As shown, the notification message 2 can indicate that the target operation is completed. In addition, the notification message 2 can also include area indication information 2, which can include the coordinate information, size information or rotation angle information of the window in the virtual desktop when the target operation is completed. Furthermore, the notification message 2 sent by the terminal device 201 can also carry the position of the pointer in the virtual desktop (which can be determined based on the movement trajectory of the pointer). In addition, before and after the user performs the target operation on the window, the position change of the window in the virtual desktop can have a high level of smoothness, which can ensure the user's experience of using the virtual desktop.

[0186] See also Fig.12 , shows another flow chart of virtual desktop display. Fig.12 In the illustrated embodiment, the background of the virtual desktop and the window screen content displayed by the terminal device 201 can remain unchanged during the process of the user operating the window. Fig.12 The illustrated embodiment may specifically include the following steps.

[0187] S1201 : The server 101 sends historical background image data, historical window image data, and area indication information 1 of the virtual desktop to the terminal device 201 .

[0188] In this embodiment, the image data sent by the server 101 to the terminal device 201 is specifically an unrendered image.

[0189] S1202: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data and the area indication information 1, and displays the initial image.

[0190] S1203: The terminal device 201 detects a target operation on a window in the virtual desktop according to the area indication information 1.

[0191] S1204: The terminal device 201 sends a notification message 1 to the server 101, where the notification message 1 is used to indicate that the window starts to be executed with the target operation.

[0192] In this embodiment, the implementation method of steps S1201 to S1204 is the same as that of the above Figure 6 The implementation methods of steps S601 to S604 in the illustrated embodiment are similar, and the details may refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0193] S1205: The terminal device 201 renders a target image of the virtual desktop, wherein a display area of ​​the window in the target image is determined based on a target operation, and the target image is obtained based on historical background image data and historical window image data.

[0194] S1206: The terminal device 201 displays the rendered target image.

[0195] In this embodiment, after determining that the window has started to perform the target operation, the server 101 may not send the background image data, the window image data, and the area indication information to the terminal device 201. Therefore, the terminal device 201 can generate a target image based on the saved historical window image data and historical background image data, and display the target image. At this time, the background image content and the window image content in the target image displayed by the terminal device 201 remain unchanged, but the display area of ​​the window in the virtual desktop changes. For example, Fig.13 As shown, before the user performs the target operation, the initial image displayed by the terminal device 201 is Fig.13 The image shown above, but after the user performs the target operation, the target image displayed by the terminal device 201 is Fig.13 The image shown below shows that the display area of ​​the background image content and the window image content in the virtual desktop remains unchanged, while the display area of ​​the window in the virtual desktop changes.

[0196] In other embodiments, after determining that the window has started to perform the target operation, the server 101 can still send the background image data, the window image data, and the area indication information to the terminal device 201. However, when the terminal device 201 needs to generate and display the target image, if the terminal device 201 does not receive new background image data and window image data, the terminal device 201 can generate and display the target image using the historical background image data and historical window image data (and the area indication information 1) stored locally, so as to ensure that the position change of the window in the virtual desktop can have a high level of smoothness, which can ensure the user's experience of using the virtual desktop.

[0197] In a further possible implementation, when the terminal device 201 detects that the target operation on the window in the virtual desktop is completed, the terminal device 201 may send a notification message 2 to the server 101, such as Fig.12 As shown, the notification message 2 may indicate that the target operation is completed. In addition, the notification message 2 may also include area indication information 2, which may include coordinate information, size information, or rotation angle information of the window in the virtual desktop when the target operation is completed. Furthermore, the notification message 2 sent by the terminal device 201 may also carry the position of the pointer in the virtual desktop (which may be determined based on the movement trajectory of the pointer).

[0198] It should be noted that the above Figure 3 , Figure 6 as well as Figure 8 , Fig.10 as well as Fig.12 The embodiments shown are all based on the application Figure 1 The data processing system 10 is used as an example for illustration. Figure 2 In the data processing system 20 shown, when the user performs an operation to adjust the display area of ​​a window in the virtual desktop displayed on the terminal device 302, the terminal device 302 and the computing device 301 can also display the image of the virtual desktop in real time based on a similar method as described above, so as to improve the user experience of using the virtual desktop on the terminal device 302. The specific implementation process can be found in the description of the relevant parts above and will not be repeated here.

[0199] It is worth noting that other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0200] Combination of the above Figures 1 to 13The virtual desktop display method provided in the embodiment of the present application is introduced, and then the structure of the terminal device and the computing device provided in the embodiment of the present application are introduced in conjunction with the accompanying drawings.

[0201] See also Fig.14 , shows a schematic diagram of the structure of a terminal device. Fig.14 As shown, the terminal device 1400 includes:

[0202] The communication module 1401 is used to receive initial image data and first area indication information of a virtual desktop from a computing device, where the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop;

[0203] A detection module 1402 is configured to detect a target operation on the object to be operated according to the first area indication information, where the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop;

[0204] A rendering module 1403 is used for rendering to obtain a target image, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data;

[0205] The display module 1404 is used to display the target image.

[0206] Alternatively, the communication module 1401 is used to receive initial image data of a virtual desktop from a computing device, where the virtual desktop includes an object to be operated;

[0207] A detection module 1402 is used to detect a target operation on the object to be operated, where the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop;

[0208] A rendering module 1403 is used to obtain a target image according to the rendering, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data;

[0209] The display module 1404 is used to display the target image.

[0210] In a possible implementation, the communication module 1401 is further configured to send a first notification message to the computing device after detecting the target operation, where the first notification message is used to indicate that the target operation begins to be executed on the object to be operated.

[0211] In one possible implementation, the communication module 1401 is also used to send a second notification message to the computing device after detecting that the target operation is completed, the second notification message is used to indicate that the target operation is completed, and the second notification message also includes second area indication information, the second area indication information includes the coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

[0212] In a possible implementation, the rendering module 1403 is specifically configured to:

[0213] Determine a moving track of a pointer in the virtual desktop according to the target operation, where the pointer is used to indicate an operation position in the virtual desktop;

[0214] The target image is rendered according to the movement trajectory of the pointer and the received initial image data.

[0215] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on a computing device;

[0216] The rendering module 1403 is specifically used for:

[0217] Generate frame image data according to the target operation, the area within the frame indicated by the frame image data is the display area of ​​the object to be operated in the target image;

[0218] The target image of the virtual desktop is rendered according to the frame image data, the movement track of the pointer and the initial image data.

[0219] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0220] In a possible implementation manner, the initial image data includes target background image data and historical window image data, wherein the target background image data is background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the historical window image data is window image data received by the terminal device before the target operation is performed on the object to be operated;

[0221] Alternatively, the initial image data includes historical background image data and target window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated;

[0222] Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated;

[0223] Alternatively, the initial image data includes historical background image data and historical window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated.

[0224] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0225] because Fig.14 The terminal device 1400 shown corresponds to the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 The terminal device 201 in the embodiment shown, therefore Fig.14 For the specific implementation of the terminal device 1400 and its technical effects, see the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0226] See also Fig.15 , which shows a schematic diagram of the structure of a computing device. Fig.15 As shown, the computing device 1500 includes:

[0227] A generating module 1501 is used to generate initial image data of a virtual desktop and first area indication information, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop;

[0228] Communication module 1502 is used to send initial image data and first area indication information to the terminal device before the target operation is performed on the object to be operated. The target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop. The target image displayed on the terminal device is generated based on the initial image data and the first area indication information. The display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0229] Alternatively, the generating module 1501 is used to generate initial image data of a virtual desktop, wherein the virtual desktop includes an object to be operated;

[0230] The communication module 1502 is used to send initial image data to the terminal device before the target operation is performed on the object to be operated, the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, the target image displayed on the terminal device is generated based on the initial image data, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device. In a possible implementation, the communication module 1502 is also used to receive a first notification message from the terminal device, and the first notification message is used to indicate that the target operation has begun to be performed on the object to be operated.

[0231] In a possible implementation, the communication module 1502 is also used to receive a second notification message from the terminal device after the target operation is completed, the second notification message is used to indicate that the target operation is completed, and the second notification message also includes second area indication information, the second area indication information includes the coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

[0232] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on a computing device.

[0233] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0234] In a possible implementation manner, the initial image data includes target background image data and historical window image data, wherein the target background image data is background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the historical window image data is window image data received by the terminal device before the target operation is performed on the object to be operated;

[0235] Alternatively, the initial image data includes historical background image data and target window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated;

[0236] Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated;

[0237] Alternatively, the initial image data includes historical background image data and historical window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated.

[0238] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0239] because Fig.15 The computing device 1500 shown corresponds to the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 The server 101 in the embodiment shown, therefore Fig.15 For the specific implementation of the computing device 1500 and its technical effects, see the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0240] Fig.16 The hardware structure diagram of a computing device 1600 provided in this application is as follows. The computing device 1600 can implement the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 The terminal device 201 in the embodiment shown may implement the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 The server 101 etc. in the illustrated embodiment.

[0241] like Fig.16As shown, the computing device 1600 includes a processor 1601, a memory 1602, and a communication interface 1603. The processor 1601, the memory 1602, and the communication interface 1603 communicate via a bus 1604, and may also communicate via other means such as wireless transmission. The memory 1602 is used to store instructions, and the processor 1601 is used to execute the instructions stored in the memory 1602. Further, the computing device 1600 may also include a memory unit 1605, and the memory unit 1605 may be connected to the processor 1601, the storage medium 1602, and the communication interface 1603 via the bus 1604. The memory 1602 stores program code.

[0242] When the computing device 1600 is specifically used to implement the terminal device 201, the processor 1601 may call the program code stored in the memory 1602 to perform the following operations:

[0243] Receive initial image data and first area indication information of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; detect a target operation for the object to be operated based on the first area indication information, and the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; render to obtain a target image, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; and display the target image.

[0244] Alternatively, the processor 1601 may call the program code stored in the memory 1602 to perform the following operations:

[0245] Receive initial image data of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated; detect a target operation for the object to be operated, wherein the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; render a target image, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; and display the target image.

[0246] When the computing device 1600 is specifically used to implement the above server 101, the processor 1601 may call the program code stored in the memory 1602 to perform the following operations:

[0247] Generate initial image data and first area indication information of a virtual desktop, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; before a target operation is performed on the object to be operated, send the initial image data and the first area indication information to a terminal device, wherein the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop, and a target image displayed on the terminal device is generated based on the initial image data and the first area indication information, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0248] Alternatively, the processor 1601 may call the program code stored in the memory 1602 to perform the following operations:

[0249] Generate initial image data of a virtual desktop, which includes an object to be operated; before a target operation is performed on the object to be operated, send the initial image data to a terminal device, the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop, a target image displayed on the terminal device is generated based on the initial image data, and a display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0250] It should be understood that in this embodiment, the processor 1601 may be a CPU, and the processor 1601 may also be other general-purpose processors, deep learning processors (DPUs), GPUs, NPUs, TPUs, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc., or may be a combination of the above-mentioned various processors. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0251] The memory 1602 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1601. The memory 1602 may also include a nonvolatile random access memory.

[0252] The memory 1602 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0253] The communication interface 1603 is used to communicate with other devices connected to the computing device 1600. The bus 1604 may include a power bus, a control bus, a status signal bus, etc. in addition to a data bus. However, for the sake of clarity, various buses are labeled as bus 1604 in the figure.

[0254] It should be understood that the computing device 1600 according to the embodiment of the present application may correspond to the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 The server 101 in the illustrated embodiment can be used to execute the method executed by the server 101. The above and other operations and / or functions implemented by the computing device 1600 are respectively to implement the above Figure 3 , Figure 6 , Figure 8 , Fig.10 as well as Fig.12 For the sake of brevity, the process of the corresponding method executed by the server 101 in the illustrated embodiment will not be repeated here.

[0255] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned virtual desktop display method.

[0256] The embodiment of the present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.

[0257] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0258] The computer program product may be a software installation package. When any of the aforementioned virtual desktop display methods is required, the computer program product may be downloaded and executed on a computing device.

[0259] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0260] The terms used in the above embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the attached claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; the character " / " generally indicates that the objects associated with each other are in an "or" relationship. In the embodiments of the present application. "Simultaneously" refers to the same time period, including the situation at the same time. The terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application.

[0261] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0262] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A virtual desktop display method, characterized in that: The method is applied to a terminal device, and the method comprises: Receiving initial image data and first area indication information of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; According to the first area indication information, a target operation for the object to be operated is detected, and the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; Rendering to obtain a target image, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; The target image is displayed.

2. The method according to claim 1, characterized in that The method further comprises: After the target operation is detected, a first notification message is sent to the computing device, where the first notification message is used to indicate that the target operation begins to be executed on the object to be operated.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: After detecting that the target operation is completed, a second notification message is sent to the computing device, the second notification message is used to indicate that the target operation is completed, and the second notification message also includes second area indication information, the second area indication information includes the coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

4. The method according to any one of claims 1 to 3, characterized in that: The rendering to obtain a target image includes: Determine a moving track of a pointer in the virtual desktop according to the target operation, wherein the pointer is used to indicate an operation position in the virtual desktop; The target image is rendered according to the movement trajectory of the pointer and the received initial image data.

5. The method according to any one of claims 1 to 4, characterized in that: The object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on a computing device; The rendering to obtain the target image according to the movement trajectory of the pointer and the received initial image data includes: Generate frame image data according to the target operation, wherein the area within the frame indicated by the frame image data is the display area of ​​the object to be operated in the target image; The target image of the virtual desktop is rendered according to the frame image data, the movement track of the pointer and the initial image data.

6. The method according to any one of claims 1 to 4, characterized in that: The object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

7. The method according to claim 6, characterized in that The initial image data includes target background image data and historical window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated; Alternatively, the initial image data includes historical background image data and target window image data, wherein the historical background image data is background image data received by the terminal device before the target operation is performed on the object to be operated, and the target window image data is window image data received by the terminal device after the target operation is performed on the object to be operated; Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated; Alternatively, the initial image data includes historical background image data and historical window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated.

8. The method according to any one of claims 1 to 7, characterized in that: The target operation is a drag operation, a zoom operation or a rotation operation.

9. A virtual desktop display method, characterized in that: The method is applied to a computing device, and the method comprises: Generate initial image data and first area indication information of the virtual desktop, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; Before the target operation is performed on the object to be operated, the initial image data and the first area indication information are sent to the terminal device, the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, the target image displayed on the terminal device is generated based on the initial image data and the first area indication information, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

10. The method according to claim 9, characterized in that The method further comprises: A first notification message is received from the terminal device, where the first notification message is used to indicate that the target operation begins to be performed on the object to be operated.

11. The method according to claim 9 or 10, characterized in that: The method further comprises: After the target operation is completed, a second notification message is received from the terminal device, the second notification message is used to indicate that the target operation is completed, and the second notification message also includes second area indication information, and the second area indication information includes the coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

12. The method according to any one of claims 9 to 11, characterized in that: The object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on a computing device.

13. The method according to any one of claims 9 to 11, characterized in that: The object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

14. The method according to claim 13, characterized in that The initial image data includes target background image data and historical window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated; Alternatively, the initial image data includes historical background image data and target window image data, wherein the historical background image data is background image data received by the terminal device before the target operation is performed on the object to be operated, and the target window image data is window image data received by the terminal device after the target operation is performed on the object to be operated; Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started to be performed on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated; Alternatively, the initial image data includes historical background image data and historical window image data, wherein the historical background image data is the background image data received by the terminal device before the target operation is performed on the object to be operated, and the historical window image data is the window image data received by the terminal device before the target operation is performed on the object to be operated.

15. The method according to any one of claims 9 to 14, characterized in that: The target operation is a drag operation, a zoom operation or a rotation operation.

16. A virtual desktop display method, characterized in that: The method is applied to a terminal device, and the method comprises: Receiving initial image data of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated; detecting a target operation on the object to be operated, wherein the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; Rendering to obtain a target image, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; The target image is displayed.

17. A virtual desktop display method, characterized in that: The method is applied to a computing device, and the method comprises: Generating initial image data of the virtual desktop, wherein the virtual desktop includes an object to be operated; Before the target operation is performed on the object to be operated, the initial image data is sent to the terminal device, the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, the target image displayed on the terminal device is generated based on the initial image data, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

18. A terminal device, characterized in that: The terminal device comprises: A communication module, configured to receive initial image data and first area indication information of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; a detection module, configured to detect a target operation for the object to be operated according to the first area indication information, wherein the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; A rendering module, used for rendering to obtain a target image, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; A display module is used to display the target image.

19. A computing device, characterized in that The computing device comprises: A generating module, configured to generate initial image data of the virtual desktop and first area indication information, wherein the virtual desktop includes an object to be operated, and the first area indication information includes coordinate information or size information of the object to be operated in the virtual desktop; A communication module is used to send the initial image data and the first area indication information to the terminal device before the target operation is performed on the object to be operated, the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, the target image displayed on the terminal device is generated based on the initial image data and the first area indication information, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

20. A terminal device, characterized in that: The terminal device comprises: A communication module, configured to receive initial image data of a virtual desktop from a computing device, wherein the virtual desktop includes an object to be operated; A detection module, configured to detect a target operation on the object to be operated, wherein the target operation is used to adjust a display area of ​​the object to be operated in the virtual desktop; A rendering module, used for rendering to obtain a target image, wherein a display area of ​​the object to be operated in the target image is determined based on the target operation, and the target image is obtained based on the initial image data; A display module is used to display the target image.

21. A computing device, characterized in that The computing device comprises: A generating module, used to generate initial image data of the virtual desktop, wherein the virtual desktop includes an object to be operated; A communication module is used to send the initial image data to the terminal device before the target operation is performed on the object to be operated, the target operation is used to adjust the display area of ​​the object to be operated in the virtual desktop, the target image displayed on the terminal device is generated based on the initial image data, and the display area of ​​the object to be operated in the target image is determined based on the target operation detected on the terminal device.

22. A computing device, characterized in that The computing device includes a processor and a memory; The memory is used to store instructions, and the processor executes the instructions stored in the memory to enable the computing device to perform the method according to any one of claims 1 to 17.

23. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computing device, cause the computing device to perform the method according to any one of claims 1 to 17.

24. A computer program product comprising instructions, characterized in that When the method is executed on a computing device, the computing device is caused to execute the method according to any one of claims 1 to 17.

Images