Virtual desktop display method and related device

Abstract

A virtual desktop display method and related device relate to the field of cloud computing technology. A terminal device receives initial image data of a virtual desktop and region indication information from a computing device, the region indication information including coordinate information or size information of a to-be-operated object in the virtual desktop; the terminal device detects a target operation for adjusting a display region of the to-be-operated object in the virtual desktop according to the region indication information; and the terminal device renders a target image and displays the target image, a display region of the to-be-operated object in the target image being determined based on the target operation, the target image being rendered based on the initial image data. In this way, the terminal device renders the target image to be displayed based on the initial image data that has been received, without temporarily waiting for the initial image data to be transmitted from the computing device to the terminal device, which can effectively reduce the time delay of the terminal device in responding to the target operation, thereby effectively improving the user experience of using the virtual desktop.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and particularly relates to a virtual desktop display method and related equipment. BACKGROUND

[0002] A virtual desktop refers to virtualization of a desktop, and specifically can refer to storing an operating system, an application program, user data and the like on a computing device, such as a server (or a server cluster) at a cloud side, and distributing desktop resources on the computing device to a terminal device at a user side through a network, so that the user has a virtual desktop experience basically the same as that of using a desktop on a local terminal device.

[0003] At present, in the process of using a virtual desktop by a user, the terminal device at the user side usually transmits an interactive operation of the user on the virtual desktop to the computing device, which can be, for example, a drag operation on a window in the virtual desktop. Then, the computing device renders a virtual desktop image according to the interactive operation and sends the virtual desktop image to the terminal device at the user side for display.

[0004] However, this way of displaying a virtual desktop often causes a large response delay, which seriously affects the user experience of using the virtual desktop. SUMMARY

[0005] The present application provides a virtual desktop display method, which aims to improve the user experience of using a virtual desktop. In addition, the present application also provides a corresponding terminal device, a computing device, a computer readable storage medium and a computer program product.

[0006] In a first aspect, the present application provides a virtual desktop display method, which can be executed by a terminal device. Specifically, the terminal device receives initial image data of a virtual desktop and first region indication information from a computing device, wherein the virtual desktop includes a to-be-operated object, which can be, for example, an icon or a window in the virtual desktop, and the first region indication information includes coordinate information (such as coordinate values) or size information (such as length values and width values) of the to-be-operated object in the virtual desktop. Then, the terminal device detects a target operation on the to-be-operated object according to the first region indication information, which is used to adjust the display region of the to-be-operated object in the virtual desktop, for example, an operation of dragging the to-be-operated object in the virtual desktop, so that the display region of the to-be-operated object in the virtual desktop can change based on the drag operation performed by the user. Next, the terminal device renders a target image and displays the target image, wherein the display region of the to-be-operated object in the target image is determined based on the target operation, and the target image is obtained based on the initial image data.

[0007] After detecting the target operation, the terminal device renders a target image of the virtual desktop based on the target operation and the initial image data that has been received, and displays the target image on the terminal device. That is, the terminal device renders the target image to be displayed based on the initial image data that has been received, without temporarily waiting for the initial image data to be transmitted from the computing device to the terminal device, and without transmitting the rendered target image between the computing device and the terminal device through the network, which can effectively reduce the time delay of responding to the target operation performed by the user on the terminal device and presenting the rendered target image. Thus, in the process of the user operating the to-be-operated object in the virtual desktop, the display area of the to-be-operated object in the virtual desktop displayed by the terminal device can change in time with the change of the user operation, and the smoothness of the change of the display area of the to-be-operated object in the virtual desktop is also higher, so as to effectively improve the user experience of using the virtual desktop.

[0008] In a possible implementation, after detecting the target operation, the terminal device can further send a first notification message to the computing device, where the first notification message is used to indicate that the to-be-operated object starts to perform the target operation. In this way, the computing device can determine that there is currently a target operation for the to-be-operated object based on the first notification message, so as to subsequently process based on the first notification message. For example, the computing device can not continue to send image data to the terminal device based on the first notification message, so as to reduce the network resource consumption between the computing device and the terminal device.

[0009] In a possible implementation, after detecting that the target operation is completed, the terminal device can further send a second notification message to the computing device, where the second notification message is used to indicate that the target operation is completed, and the second notification message further includes second area indication information, where the second area indication information includes coordinate information (such as a coordinate value), size information (such as a length value and a width value), or rotation angle information (such as an angle value) of the to-be-operated object in the virtual desktop when the target operation is completed. In this way, the computing device can determine that the target operation is completed and the display area of the to-be-operated object in the virtual desktop when the target operation is completed based on the second notification message. In this way, when the computing device provides the image data of the virtual desktop rendered on the computing device to the terminal device, the computing device can issue correct area indication information, so that the display area of the to-be-operated object viewed by the user on the terminal device is consistent with the display area of the to-be-operated object when the user ends the target operation, thereby maintaining a high level of smoothness of the change of the position of the to-be-operated object in the virtual desktop before and after the user performs the target operation on the to-be-operated object, and further guaranteeing the user experience of using the virtual desktop.

[0010] In a possible implementation, when the terminal device renders the target image, specifically, the terminal device can determine a moving track of a pointer in the virtual desktop according to the target operation, the pointer being used to indicate an operation position in the virtual desktop, so that the terminal device renders the target image according to the moving track of the pointer and the received initial image data. In this way, the terminal device can determine the moving position of the pointer according to the target operation, that is, determine the position of the to-be-operated object in the process of performing the target operation, so that the display area of the to-be-operated object in the target image rendered by the terminal device based on the moving track can be the display area expected by the user.

[0011] In a possible implementation, the to-be-operated object is a window, and the initial image data of the virtual desktop is image data rendered on the computing device. When the terminal device renders the target image according to the moving track of the pointer and the received initial image data, specifically, the terminal device can generate border image data according to the target operation, the area in the border indicated by the border image data being the display area of the to-be-operated object in the target image, so that the terminal device can render the target image of the virtual desktop according to the border image data, the moving track of the pointer and the initial image data. In this way, the picture content of the target image rendered and presented by the terminal device can use the border to indicate the display area of the to-be-operated object in the virtual desktop after the to-be-operated object is performed with the target operation.

[0012] In a possible implementation, the to-be-operated object is a window, and the initial image data of the virtual desktop includes background image data and window image data. In this way, the terminal device can render the target image locally according to the background image data and the window image data provided by the computing device and the detected target operation, so that the window performed with the target operation in the picture of the virtual desktop presented by the terminal device can change the display area in a timely manner along with the change of the target operation performed by the user, improving the experience of the user using the virtual desktop.

[0013] In a possible implementation, the initial image data includes target background image data and historical window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to perform the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated performs the target operation. In this way, the terminal device can present different background image contents of the virtual desktop in two frames of images before and after, improving the experience of the user using the virtual desktop. Moreover, in the process of responding to the target operation performed by the user, the image data sent by the computing device to the terminal device can be reduced from the two types of data, i.e., the background image data and the window image data, to one type of data, i.e., the background image data, which can effectively reduce the amount of data transmitted between the computing device and the terminal device, thereby reducing the data transmission delay, reducing the time delay of the terminal device in responding to the target operation and presenting the target image corresponding to the target operation, and further improving the experience of the user using the virtual desktop.

[0014] Alternatively, the initial image data includes historical background image data and target window image data, the historical background image data being background image data received by the terminal device before the object to be operated starts to perform the target operation, and the target window image data being window image data received by the terminal device after the object to be operated performs the target operation. In this way, the terminal device can present different background image contents in the window (such as playing a video in the window) in two frames of images before and after, and can reduce the time delay of the terminal device in responding to the target operation and presenting the target image corresponding to the target operation, improving the experience of the user using the virtual desktop.

[0015] Alternatively, the initial image data includes target background image data and target window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to perform the target operation, and the target window image data being window image data received by the terminal device after the object to be operated performs the target operation. In this way, the terminal device can present different background image contents of the virtual desktop and different background image contents in the window in two frames of images before and after, and can reduce the time delay of the terminal device in responding to the target operation and presenting the target image corresponding to the target operation, improving the experience of the user using the virtual desktop.

[0016] Alternatively, the initial image data includes historical background image data and historical window image data, the historical background image data being background image data received by the terminal device before the object to be operated starts to perform the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated performs the target operation. In this way, the terminal device can reduce the time delay of responding to the target operation and presenting the target image corresponding to the target operation, improving the experience of the user using the virtual desktop.

[0017] In a possible implementation, the target operation is a dragging operation, a zooming operation, or a rotating operation.

[0018] In a second aspect, the present application provides a virtual desktop display method, which is applied to a computing device. Specifically, the computing device generates initial image data of a virtual desktop and first region indication information, the virtual desktop including an object to be operated (such as an icon or a window, etc.), and the first region indication information including coordinate information or size information of the object to be operated in the virtual desktop; and before the object to be operated is executed a target operation, the computing device sends the initial image data and the first region indication information to a terminal device, wherein the target operation is used to adjust a display region of the object to be operated in the virtual desktop, a target image displayed on the terminal device is generated based on the initial image data and the first region indication information, and a display region of the object to be operated in the target image is determined based on the target operation detected on the terminal device.

[0019] In a possible implementation, the computing device can further receive a first notification message from the terminal device, the first notification message being used to indicate that the object to be operated starts to be executed the target operation.

[0020] In a possible implementation, after the target operation is executed, the computing device can further receive a second notification message from the terminal device, the second notification message being used to indicate that the target operation is executed, and the second notification message further including second region indication information, the second region indication information including coordinate information, size information, or rotating angle information of the object to be operated in the virtual desktop when the target operation is executed.

[0021] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on the computing device.

[0022] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0023] In a possible implementation, the initial image data comprises target background image data and historical window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to perform the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated performs the target operation; or, the initial image data comprises historical background image data and target window image data, the historical background image data being background image data received by the terminal device before the object to be operated starts to perform the target operation, and the target window image data being window image data received by the terminal device after the object to be operated performs the target operation; or, the initial image data comprises target background image data and target window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to perform the target operation, and the target window image data being window image data received by the terminal device after the object to be operated performs the target operation; or, the initial image data comprises historical background image data and historical window image data, the historical background image data being background image data received by the terminal device before the object to be operated starts to perform the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated performs the target operation.

[0024] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0025] The virtual desktop display method provided in the second aspect corresponds to the virtual desktop display method provided in the first aspect, and therefore the technical effects of the second aspect and any implementation manner of the second aspect can be referred to the description of the corresponding implementation manners of the first aspect, which is not repeated here.

[0026] In a third aspect, the present application provides a virtual desktop display method, which is applied to a terminal device. Specifically, the terminal device receives initial image data of a virtual desktop from a computing device, and the virtual desktop includes a to-be-operated object; and the terminal device further detects a target operation on the to-be-operated object, and the target operation is used to adjust a display area of the to-be-operated object in the virtual desktop; and then the terminal device renders a target image based on the target operation and the initial image data, and displays the target image, wherein the display area of the to-be-operated object in the target image is determined based on the target operation, and the target image is obtained based on the initial image data. After detecting the target operation, the terminal device renders the target image of the virtual desktop based on the target operation and the initial image data that has been received, and displays the target image on the terminal device. That is, the terminal device renders the target image to be displayed based on the initial image data that has been received, without temporarily waiting for the initial image data to be transmitted from the computing device to the terminal device, and without transmitting the rendered target image between the computing device and the terminal device through the network, which can effectively reduce the time delay of responding to the target operation performed by the user and presenting the rendered target image on the terminal device. Therefore, during the operation of the to-be-operated object in the virtual desktop by the user, the display area of the to-be-operated object in the virtual desktop displayed by the terminal device can be changed in time along with the change of the user operation, and the smoothness of the change of the display area of the to-be-operated object in the virtual desktop is also higher, so as to effectively improve the user experience of using the virtual desktop.

[0027] In a fourth aspect, the present application provides a virtual desktop display method, which is applied to a computing device. Specifically, the computing device generates initial image data of a virtual desktop, and the virtual desktop includes a to-be-operated object; and the computing device further sends the initial image data to a terminal device before the to-be-operated object is operated by a target operation, and the target operation is used to adjust a display area of the to-be-operated object in the virtual desktop, and a target image displayed on the terminal device is generated based on the initial image data, and the display area of the to-be-operated object in the target image is determined based on the target operation detected by the terminal device.

[0028] The virtual desktop display method provided in the fourth aspect corresponds to the virtual desktop display method provided in the third aspect, and the technical effects of the fourth aspect can be referred to the description of the related technical effects of the third aspect, which will not be repeated here.

[0029] In a fifth aspect, the present application provides a terminal device, comprising a communication module, configured to receive initial image data and first area indication information of a virtual desktop from a computing device, the virtual desktop comprising an object to be operated, and the first area indication information comprising coordinate information or size information of the object to be operated in the virtual desktop; a detection module, configured to detect a target operation on the object to be operated according to the first area indication information, the target operation being used to adjust a display area of the object to be operated in the virtual desktop; a rendering module, configured to render a target image, a display area of the object to be operated in the target image being determined based on the target operation, and the target image being obtained based on the initial image data; and a display module, configured to display the target image.

[0030] In a possible implementation, the communication module is further configured to send a first notification message to the computing device after detecting the target operation, the first notification message being used to indicate that the object to be operated starts to be operated by the target operation.

[0031] In a possible implementation, the communication module is further configured to send a second notification message to the computing device after detecting that the target operation is completed, the second notification message being used to indicate that the target operation is completed, and the second notification message further comprising second area indication information, the second area indication information comprising coordinate information, size information or rotation angle information of the object to be operated in the virtual desktop when the target operation is completed.

[0032] In a possible implementation, the rendering module is specifically configured to: determine a moving track of a pointer in the virtual desktop according to the target operation, the pointer being used to indicate an operation position in the virtual desktop; and render the target image according to the moving track of the pointer and the received initial image data.

[0033] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on the computing device; the rendering module is specifically configured to: generate border image data according to the target operation, an area within a border indicated by the border image data being the display area of the object to be operated in the target image; and render the target image of the virtual desktop according to the border image data, the moving track of the pointer and the initial image data.

[0034] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop comprises background image data and window image data.

[0035] In a possible implementation, the initial image data includes target background image data and historical window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to perform the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated performs the target operation; or, the initial image data includes historical background image data and target window image data, the historical background image data being background image data received by the terminal device before the object to be operated starts to perform the target operation, and the target window image data being window image data received by the terminal device after the object to be operated performs the target operation; or, the initial image data includes target background image data and target window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to perform the target operation, and the target window image data being window image data received by the terminal device after the object to be operated performs the target operation; or, the initial image data includes historical background image data and historical window image data, the historical background image data being background image data received by the terminal device before the object to be operated starts to perform the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated performs the target operation.

[0036] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0037] The terminal device provided in the fifth aspect corresponds to the virtual desktop display method provided in the first aspect, and therefore has the technical effects of the first aspect and the corresponding implementation manners of the first aspect, which are not described herein again.

[0038] In the sixth aspect, the application provides a computing device, including: a generation module configured to generate initial image data of a virtual desktop and first area indication information, the virtual desktop including an object to be operated, and the first area indication information including coordinate information or size information of the object to be operated in the virtual desktop; and a communication module configured to send, to a terminal device, the initial image data and the first area indication information before the object to be operated performs a target operation, the target operation being used to adjust a display area of the object to be operated in the virtual desktop, a target image displayed on the terminal device being generated based on the initial image data and the first area indication information, and a display area of the object to be operated in the target image being determined based on a target operation detected on the terminal device.

[0039] In a possible implementation, the communication module is further configured to receive a second notification message from the terminal device after the target operation is completed, the second notification message being used to indicate that the target operation is completed, and the second notification message further comprising second region indication information, the second region indication information comprising coordinate information, size information, or rotation angle information of the to-be-operated object in the virtual desktop when the target operation is completed.

[0040] In a possible implementation, the to-be-operated object is a window, and the initial image data of the virtual desktop is image data rendered on the computing device.

[0041] In a possible implementation, the to-be-operated object is a window, and the initial image data of the virtual desktop comprises background image data and window image data.

[0042] In a possible implementation, the initial image data comprises target background image data and historical window image data, the target background image data being background image data received by the terminal device after the to-be-operated object starts to be operated, and the historical window image data being window image data received by the terminal device before the to-be-operated object is operated;

[0043] In a possible implementation, the initial image data comprises historical background image data and target window image data, the historical background image data being background image data received by the terminal device before the to-be-operated object starts to be operated, and the target window image data being window image data received by the terminal device after the to-be-operated object is operated;

[0044] In a possible implementation, the initial image data comprises target background image data and target window image data, the target background image data being background image data received by the terminal device after the to-be-operated object starts to be operated, and the target window image data being window image data received by the terminal device after the to-be-operated object is operated;

[0045] In a possible implementation, the initial image data comprises historical background image data and historical window image data, the historical background image data being background image data received by the terminal device before the to-be-operated object starts to be operated, and the historical window image data being window image data received by the terminal device before the to-be-operated object is operated.

[0046] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0047] The computing device of the sixth aspect corresponds to the virtual desktop display method of the second aspect, and thus the technical effects of the sixth aspect and any implementation manner of the sixth aspect are the same as those of the second aspect and the corresponding implementation manner of the second aspect, which are not described herein.

[0048] In the seventh aspect, the present application provides a terminal device, which comprises a communication module, configured to receive initial image data of a virtual desktop from a computing device, the virtual desktop comprising a to-be-operated object; a detection module, configured to detect a target operation on the to-be-operated object, the target operation being configured to adjust a display area of the to-be-operated object in the virtual desktop; a rendering module, configured to render a target image, a display area of the to-be-operated object in the target image being determined based on the target operation, the target image being obtained based on the initial image data; and a display module, configured to display the target image.

[0049] In the eighth aspect, the present application provides a computing device, which comprises a generation module, configured to generate initial image data of a virtual desktop, the virtual desktop comprising a to-be-operated object; and a communication module, configured to send the initial image data to a terminal device before a target operation on the to-be-operated object is performed, the target operation being configured to adjust a display area of the to-be-operated object in the virtual desktop, a target image displayed on the terminal device being generated based on the initial image data, and a display area of the to-be-operated object in the target image being determined based on the target operation detected on the terminal device.

[0050] In the ninth aspect, the present application provides a computing device, which comprises a processor and a memory; the memory is configured to store instructions, and the processor is configured to execute the instructions stored in the memory to perform the operation steps of the virtual desktop display method of any implementation manner of the first aspect to the fourth aspect.

[0051] In the tenth aspect, the present application provides a computer readable storage medium, which stores instructions, and when the instructions are executed on a computing device, the computing device performs the operation steps of the virtual desktop display method of any implementation manner of the first aspect to the fourth aspect.

[0052] In the eleventh aspect, the present application provides a computer program product comprising instructions, and when the instructions are executed on a computing device, the computing device performs the operation steps of the virtual desktop display method of any implementation manner of the first aspect to the fourth aspect.

[0053] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1A structural schematic diagram of an example data processing system provided in the present application;

[0055] Figure 2 A structural schematic diagram of another example data processing system provided in the present application;

[0056] Figure 3 A flowchart of a virtual desktop display method provided in the present application;

[0057] Figure 4a A schematic diagram of dragging the display area of a window from region 1 to region 2;

[0058] Figure 4b A schematic diagram of scaling the display area of a window from region 3 to region 4;

[0059] Figure 4c A schematic diagram of scaling the display area of a window from region 5 to region 6;

[0060] Figure 5a A schematic diagram of rendering a target image including a frame based on initial image data in a window dragging scenario;

[0061] Figure 5b A schematic diagram of rendering a target image including a frame based on initial image data in a window scaling scenario;

[0062] Figure 5c A schematic diagram of rendering a target image including a frame based on initial image data in a window rotating scenario;

[0063] Figure 6 A flowchart of another virtual desktop display method provided in the present application;

[0064] Figure 7 A schematic diagram of a scenario in which the background picture content and the window display area change after performing a target operation, but the window picture content does not change;

[0065] Figure 8 A flowchart of yet another virtual desktop display method provided in the present application;

[0066] Figure 9 A schematic diagram of a scenario in which the window picture content and the window display area change after performing a target operation, but the background picture content does not change;

[0067] Figure 10 A flowchart of still another virtual desktop display method provided in the present application;

[0068] Figure 11 A schematic diagram of a scenario in which the background picture content, the window picture content and the window display area all change after performing a target operation;

[0069] Figure 12 A flowchart illustrating yet another virtual desktop display method provided in this application;

[0070] Figure 13 This is a diagram illustrating how the background and window content remain unchanged after the target operation is performed, but the window display area changes.

[0071] Figure 14 A schematic diagram of the structure of a terminal device provided in this application;

[0072] Figure 15 A schematic diagram of the structure of a computing device provided in this application;

[0073] Figure 16 This is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0074] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, various non-limiting embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained based on the embodiments in this application and based on the above content are within the scope of protection of this application.

[0075] See Figure 1 The diagram illustrates the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes at least one server and at least one terminal device. For ease of understanding, Figure 1 The following description uses computing node servers 101 to 102 and terminal devices 201 to 203 as an example. The servers and terminal devices can be interconnected via a network, such as a communication network including at least one switch.

[0076] In practical applications, servers 101 and 102 can be deployed on the cloud side to support the provision of virtual desktop services. Terminal devices 201 to 203 can be deployed on the user side to support users in using virtual desktops.

[0077] The virtual desktop can be a cloud desktop, such as the desktop of a cloud computer, a cloud laptop, a cloud set-top box, or other cloud products, or it can be a virtual desktop implemented through computing devices deployed on the cloud side.

[0078] The terminal device 201 (the rest of the terminal devices are similar) can be any device capable of using a virtual desktop, for example, can be a thin terminal (a terminal device with relatively strong computing and storage capabilities), a fat terminal (a terminal device with relatively weak computing and storage capabilities), a cloud extended reality (XR) glasses, a cloud terminal (a terminal device specially designed for virtual desktop technology), a mobile phone, an iPad, a notebook computer, and the like, or can be other types of terminal devices, and no limitation is made thereto.

[0079] In the process in which the user uses the terminal device 201 to use the virtual desktop provided by the server 101, an operation of adjusting a display area of an object such as a window or an icon in the virtual desktop is often performed. For example, the user can drag a window in the virtual desktop, or the user can scale the window in the virtual desktop, and the like. In the process in which the user adjusts the display area of the object in the virtual desktop, if the terminal device 201 sends the operation performed by the user to the server 101, and the server 101 renders a new image according to the operation, and then sends the image to the terminal device 201 for display, it is easy to cause problems such as operation lag or operation delay in the image of the virtual desktop displayed by the terminal device 201. The operation lag refers to that in the process of operating the object in the virtual desktop, the display area of the object in the virtual desktop changes rapidly and significantly, for example, the display area of the object in the virtual desktop suddenly moves from the display area A to the display area B. The operation delay refers to that in the process of operating the object in the virtual desktop, the display area of the object does not change in time with the operation of the user. In this way, the use experience of the user for the virtual desktop is seriously affected.

[0080] Based on this, in the data processing system 10 provided in this application, the terminal device 201 receives the initial image data of the virtual desktop and the region indication information sent by the server 101. The initial image data may be, for example, the background image data of the virtual desktop and the image data corresponding to the to-be-operated object in the virtual desktop, or may be the image data of the virtual desktop rendered by the server 101, etc. The region indication information includes the coordinate information or size information of the to-be-operated object in the virtual desktop, or the region indication information includes both the coordinate information and the size information, etc. In the process of the user performing a target operation on the to-be-operated object in the virtual desktop based on the terminal device 201, the terminal device 201 detects the target operation performed by the user on the to-be-operated object according to the region indication information, and the target operation is an operation (such as the above-mentioned drag operation, etc.) for adjusting the display region of the to-be-operated object in the virtual desktop. Then, the terminal device 201 renders a target image of the virtual desktop, and the target image is rendered based on the initial image data. For example, the terminal device 201 can determine the moving track of the pointer in the virtual desktop based on the target operation, and render the target image according to the moving track and the initial image data, etc. Finally, the terminal device 201 displays the target image.

[0081] Since after detecting the target operation, the terminal device 201 will render the image, specifically, render the target image of the virtual desktop according to the target operation and the initial image data that has been received, and display the target image on the terminal device 201. That is, the terminal device 201 renders the target image to be displayed based on the initial image data that has been received, without temporarily waiting for the initial image data to be transmitted from the server 101 to the terminal device 201, and without transmitting the rendered target image between the server 101 and the terminal device 201 through the network, which can effectively reduce the time delay of responding to the target operation and presenting the rendered target image on the terminal device 201. Therefore, in the process of the user operating the to-be-operated object in the virtual desktop, the display region of the to-be-operated object in the virtual desktop displayed by the terminal device 201 can change in time with the change of the user operation, and the fluency of the change of the display region of the to-be-operated object in the virtual desktop will also be higher, so as to effectively improve the user's experience of using the virtual desktop.

[0082] It is worth noting that the above-mentioned Figure 1 The data processing system 10 shown in the figure is only an exemplary illustration and is not used for limitation. For example, in other possible data processing systems, the number of terminal devices and servers can also be other numbers, or can include other types of devices, such as devices for managing the server 101 and the server 102, etc.

[0083] Alternatively, in other possible data processing systems, in addition to the server, a virtual desktop service can also be provided for a terminal device on the user side by other types of computing devices. For example, in the data processing system 20 shown in Figure 2 The computing device 301 can provide a virtual desktop for the terminal device 302 on the user side.

[0084] The computing device 301, for example, can be a physical device implementation including a processor, such as a host-side device of a split XR glasses. The processor can be a CPU, an ASIC, a PLD, a CPLD, an FPGA, a GAL, a SoC, an SDI chip, an AI chip, or a DPU, or any combination of the above. The number of processors included in the computing device 301 can be one or more, and the types of processors included can be one or more. The number and types of processors can be set according to the actual business requirements, and the present embodiment does not limit the number and types of processors. The terminal device 302, for example, can be a glasses-side device of the split XR glasses, or other types of devices. The terminal device 302 and the computing device 301 can communicate through a network, so that the user can use the virtual desktop on the terminal device 302. At this time, the computing device 301 can not be deployed on the cloud side, but can still be deployed on the user side, etc.

[0085] For ease of understanding, the embodiments provided by the present application for displaying a virtual desktop on a terminal device are described below with reference to the accompanying drawings.

[0086] Referring to Figure 3 , Figure 3 A flowchart of an exemplary virtual desktop display method provided by the embodiments of the present application is shown. Figure 3 The virtual desktop display method shown in Figure 1The data processing system 10 shown, or other possible data processing systems, can be applied. For the purpose of understanding and description, the following is taken as an example of the data processing system 10 shown, and the server 101 provides a virtual desktop for the terminal device 201. Figure 1 The data processing system 10 shown, and the server 101 provides a virtual desktop for the terminal device 201 as an example for description.

[0087] As shown in the figure, the virtual desktop display method can specifically include the following steps. Figure 3

[0088] S301: The server 101 sends initial image data of a virtual desktop and region indication information 1 to the terminal device 201, wherein the virtual desktop includes an object to be operated, the initial image data is image data rendered on the server 101, and the region indication information 1 includes coordinate information or size information of the object to be operated in the virtual desktop.

[0089] Exemplarily, the object to be operated in the virtual desktop refers to an object to be operated that can be controlled and adjusted in the display region in the virtual desktop, which can be a window, an icon, or other types of objects to be operated.

[0090] In this embodiment, before the user adjusts the display region of the object to be operated in the virtual desktop, the server 101 can be responsible for performing the image rendering process of the virtual desktop and sending the rendered image to the terminal device 201, that is, the initial image data sent by the server 101 to the terminal device 201 is specifically the data of the image rendered on the server 101.

[0091] In a possible implementation, the terminal device 201 presents the virtual desktop to the user, and the user can perform a corresponding interactive operation on the virtual desktop, which is specifically an operation of not changing the display region of the object to be operated in the virtual desktop, such as a mouse operation, a single / double click operation on the object to be operated in the virtual desktop, etc. Then, the terminal device 201 can send the interactive operation performed by the user to the server 101. The server 101 can render the initial image data of the virtual desktop according to the received interactive operation and send the initial image data to the terminal device 201 through the network. In actual application, in addition to performing operations on the object to be operated through the mouse, operations on the object to be operated can also be realized through a remote controller, a handle, a touch screen, a gesture detection device, etc.

[0092] ​Further, the server 101 also sends region indication information 1, so as to indicate the display region of the to-be-operated object in the initial image data in the virtual desktop by using the region indication information 1. For example, the region indication information 1 can be coordinate information and size information of the to-be-operated object in the virtual desktop. For example, the coordinate information can be a coordinate value, for example, a coordinate value of a center point of the to-be-operated object in the virtual desktop. The size information can be length and width information, and can indicate the size of the to-be-operated object in the virtual desktop. Alternatively, the region indication information 1 can only include the coordinate information of the to-be-operated object in the virtual desktop (the size of the to-be-operated object in the virtual desktop is fixed); or the region indication information 1 can only be size information of the to-be-operated object in the virtual desktop (the coordinate of the to-be-operated object in the virtual desktop is fixed), and the like, which is not limited herein.

[0093] In actual application, the server 101 can also send an identifier of the to-be-operated object to the terminal device 201, so as to notify the terminal device 201 by using the identifier that the region indication information 1 is used to describe the coordinate information or the size information of which to-be-operated object in the virtual desktop.

[0094] S302: The terminal device 201 displays the image corresponding to the initial image data of the virtual desktop.

[0095] In the process that the user uses the virtual desktop by using the terminal device 201, the terminal device 201 displays the corresponding image according to the initial image data sent by the server 101, so that the user can continue to use the virtual desktop based on the image of the virtual desktop currently presented by the terminal device 201.

[0096] In specific implementation, the terminal device 201 can decode the received initial image data, and output the decoded image data to a display device (such as a display screen, etc.) on the terminal device 201, so as to display the image corresponding to the initial image data by using the display device.

[0097] In actual application scenarios, the terminal device 201 can present multiple frames of images of the virtual desktop per second, for example, the terminal device 201 can present 30 frames of images of the virtual desktop per second, and the like, so as to ensure the fluency of the picture content of the virtual desktop seen by the user. Before the user adjusts the display region of the to-be-operated object in the virtual desktop, the change speed of the picture content in the virtual desktop presented by the terminal device 201 is usually small (for example, the picture content basically remains unchanged, etc.), and therefore, the images of the virtual desktop presented by the terminal device 201 can all be the images rendered by the server 101.

[0098] S303: The terminal device 201 detects, according to the received area indication information 1, a target operation on the to-be-operated object in the virtual desktop, the target operation being used to adjust a display area of the to-be-operated object in the virtual desktop.

[0099] The display area of the to-be-operated object in the virtual desktop is specifically an imaging area of the to-be-operated object in an image of the virtual desktop.

[0100] In actual application scenarios, a user often needs to adjust a display area of a to-be-operated object in a virtual desktop in the process of using the virtual desktop. For example, when multiple windows in the virtual desktop overlap with each other, the user can adjust a display area of a part of the windows in the virtual desktop by dragging the part of the windows, so that the contents of the multiple windows do not block each other. For another example, when a display area of an image in the virtual desktop does not meet the user's expectation, the user can drag the icon to another position in the virtual desktop. For yet another example, when a size of a window in the virtual desktop does not meet the user's expectation, the user can scale the window to enlarge or reduce the size of the window.

[0101] When the user needs to adjust the display area of the to-be-operated object in the virtual desktop, the user can perform a target operation on the to-be-operated object by using a mouse or another device, the target operation being, for example, a dragging operation, a scaling operation, or a rotating operation, or another possible operation. The terminal device 201 can detect, in real time, whether there is currently an operation on the to-be-operated object according to the received area indication information 1. In actual application scenarios, an object on which the user performs an operation (for example, an icon / window selected by the user by clicking or moving a mouse) or an object determined based on the operation performed by the user (for example, a window popped up by the user by double-clicking an icon) can be referred to as a focus object (for example, a focus window, a focus image, or the like), and the to-be-operated object is the focus object. In addition to the target operation, the operation performed by the user can be another operation, such as a single-click operation or a double-click operation performed by the user on the object.

[0102] In a first implementation example, the image of the virtual desktop presented by the terminal device 201 can include a pointer used to indicate an operation position in the virtual desktop, and during the process in which the user performs the target operation, an imaging position of the pointer in the virtual desktop can change with the target operation. When the to-be-operated object is specifically an icon (or another type of to-be-operated object), the terminal device 201 can detect whether the operation position indicated by the pointer is located in the display area of the icon. If yes, the terminal device 201 can determine that the target operation on the icon is detected. If no, the terminal device 201 can determine that there is currently no target operation on the image.

[0103] In the second implementation example, when the object to be operated on is specifically a window (or other type of object to be operated on), the terminal device 201 can detect whether the operation position indicated by the pointer is within the controllable area of ​​the window. The controllable area refers to a portion of the window's display area, and this portion is used to detect whether there is an operation controlling the window. For example, ... Figure 4a In the window shown, the gray area represents the controllable area. When a user moves the pointer to this controllable area using a mouse or other device, they can control the window, such as dragging the entire display area from... Figure 4a The window is moved from area 1 to area 2, etc. At this time, when the user performs dragging or zooming operations, the terminal device 201 can determine that the operation is valid and control the window accordingly. When the operation position indicated by the pointer is detected to be within the controllable area of ​​the window, the terminal device 201 can determine that a target operation has been detected for that window. However, when the operation position indicated by the pointer is not within the controllable area... Figure 4a If the gray area in the diagram, i.e., the operation position indicated by the pointer, is located in an uncontrollable area within the window or in an area outside the window, then the terminal device 201 can determine that there is currently no operation targeting that window.

[0104] Before the target operation is performed on the window, the terminal device 201 can obtain indication information of the controllable area within the window to determine the controllable area within the window. As a first example, when the server 101 sends initial image data to the terminal device 201, it can also send the indication information of the controllable area corresponding to the window to be targeted. This indication information may be, for example, the coordinate information (such as coordinate values) and size information (such as length and width) of the controllable area, or only the coordinate information of the controllable area (the size of the controllable area is fixed), or only the size information of the controllable area (the coordinates of the controllable area are fixed relative to the window coordinates). During the target operation is performed on the window, the server 101 may not send the indication information of the controllable area. As a second example, when the area of ​​the controllable area within the window is relatively fixed, the terminal device 201 can also determine the controllable area within the target display area after determining the target display area corresponding to the window, such as the controllable area being fixed as the first N rows of pixels in the target display area of ​​the window (N is a positive integer). Thus, the terminal device 201 can determine whether there is a target operation on the window based on the acquired indication information of the controllable area.

[0105] In an actual application scenario, after detecting that there is a target operation on the to-be-operated object, the terminal device 201 can further send a notification message 1 to the server 101, so as to notify the server 101 that the to-be-operated object starts to be executed by the target operation, as shown in Figure 3 Alternatively, after detecting that there is a target operation on the to-be-operated object, the terminal device 201 can also not send a notification message 1 to the server 101.

[0106] S304: The terminal device 201 renders a target image of the virtual desktop, wherein a display region of the to-be-operated object in the target image is determined based on the target operation, and the target image is obtained based on the initial image.

[0107] In this embodiment, after detecting the target operation for adjusting the display region of the to-be-operated object, the terminal device 201 does not send the target operation to the server 101, but renders a target image to be displayed on the terminal device 201 according to the target operation and the initial image data that has been received.

[0108] In a possible implementation, the terminal device 201 can first generate border image data according to the target operation, wherein a region in a border indicated by the border image data is a target display region, and the target display region is a display region of the to-be-operated object in the target image to be generated.

[0109] In specific implementation, after detecting the target operation, the terminal device 201 can record a moving track of a pointer in the virtual desktop, and calculate the target display region according to the moving track and the received region indication information 1.

[0110] For example, when the target operation performed by the user is a drag operation, and the to-be-operated object being operated is a window, it is assumed that a display region of the window in the initial image data is a region 1 as shown in Figure 4a which can be indicated by a coordinate 1 and a size of the window. Then, in the process of adjusting the display region of the window, the terminal device 201 can calculate a coordinate 2 of the window in the virtual desktop when the user performs the target operation to the current stage according to the coordinate 1, the size of the region 1, and the moving track of the pointer, and determine a target display region of the window in the virtual desktop as a region 2 as shown in Figure 4a according to the coordinate 2 and the size of the window (the size of the window does not change in the drag scenario). In this way, after the drag operation on the window, the display region of the window in the virtual desktop changes from the region 1 to the region 2.

[0111] For example, when the target operation performed by the user is a zoom operation (specifically, an operation of zooming out a window), and the to-be-operated object being operated is a window, assume that the display region of the to-be-operated object in the initial image data is region 3 as shown in FIG. 6. Figure 4b In the process of adjusting the display region of the window, the terminal device 201 can calculate the size 2 of the window after zooming according to the movement track of the pointer and the size 1 of the window, and determine the target display region of the window in the virtual desktop as region 4 according to the size 2 and the coordinates of the window (assuming that the coordinates of the window do not change in the zooming scenario). Figure 4b Thus, after the zoom operation on the window, the display region of the window in the virtual desktop changes from region 3 to region 4.

[0112] For example, when the target operation performed by the user is a rotation operation, and the to-be-operated object being operated is a window, assume that the display region of the to-be-operated object in the initial image data is region 5 as shown in FIG. 7. Figure 4c In the process of adjusting the display region of the window, the terminal device 201 can calculate the rotation angle of the window after zooming according to the movement track of the pointer, and determine the target display region of the to-be-operated object in the virtual desktop as region 6 according to the rotation angle and the coordinates and size of the window (assuming that the coordinates and size of the window do not change in the rotation scenario). Figure 4c Thus, after the zoom operation on the window, the display region of the window in the virtual desktop changes from region 5 to region 6.

[0113] Then, the terminal device 201 can generate the border image data corresponding to the target display region, and the region within the border indicated by the border image data is the target display region. The border image data can be used to draw a border in an image, and the attribute information of the drawn border, such as the width, color, style, and line (dashed / real) of the border, can be pre-configured in the terminal device 201, such as pre-configured by a technician in the terminal device 201, or configured by the server 101 in the terminal device 201, and the like.

[0114] Finally, the terminal device 201 can render the target image of the virtual desktop according to the border image data, the movement track of the pointer, and the initial image data of the virtual desktop that has been received. Specifically, the position of the border in the image can be determined according to the movement track of the pointer, and the border image data and the initial image data are superimposed and drawn based on the position to achieve the rendering of the target image. The display region of the to-be-operated object in the rendered target image is different from the display region of the to-be-operated object in the image corresponding to the initial image data, so as to achieve the adjustment of the display region of the to-be-operated object in the virtual desktop.

[0115] For example, assuming that the target operation is specifically a drag operation, the object to be operated is specifically a window, and the image corresponding to the initial image data can be, for example, the image shown in the upper part of FIG. 7. Figure 5a After detecting the drag operation, the terminal device 201 generates the border image data according to the drag operation, and renders the target image shown in the lower part of FIG. 7 according to the border image data and the initial image data. In this case, the black frame shown in the middle part of FIG. 7 is the border drawn based on the border image data corresponding to the drag operation, and the area within the black frame is the area in which the window after the drag operation should be located in the virtual desktop. That is, in the process of dragging the window, the border of the window appears in the image presented by the terminal device 201, and the position of the border continuously changes with the drag operation of the user, while the size of the border remains unchanged. It should be noted that the dashed line in the middle part of FIG. 7 is only used to reflect the trajectory of the window being dragged (i.e., the moving trajectory of the pointer), and the content of the image rendered by the terminal device 201 does not include the dashed line. Figure 5a Figure 5a Figure 5a

[0116] For example, assuming that the target operation is specifically a zoom operation, the object to be operated is specifically a window, and the image corresponding to the initial image data can be, for example, the image shown in the upper part of FIG. 8. Figure 5b After detecting the zoom operation, the terminal device 201 generates the border image data according to the zoom operation, and renders the target image shown in the lower part of FIG. 8 according to the border image data and the initial image data. In this case, in the process of zooming the window, the border of the window appears in the image presented by the terminal device 201, and the size of the border continuously changes with the zoom operation of the user. It should be noted that the dashed line in the middle part of FIG. 8 is only used to reflect the trajectory of the window being zoomed (i.e., the moving trajectory of the pointer), and the content of the image rendered by the terminal device 201 does not include the dashed line. Figure 5b Figure 5b

[0117] For example, assuming that the target operation is specifically a rotation operation, the object to be operated is specifically a window, and the image corresponding to the initial image data can be, for example, the image shown in the upper part of FIG. 9. Figure 5c After detecting the rotation operation, the terminal device 201 generates the border image data according to the rotation operation, and renders the target image shown in the lower part of FIG. 9 according to the border image data and the initial image data. In this case, in the process of rotating the window, the border of the window appears in the image presented by the terminal device 201, and the rotation angle of the border continuously changes with the rotation operation of the user. It should be noted that the dashed line in the middle part of FIG. 9 is only used to reflect the trajectory of the window being rotated (i.e., the moving trajectory of the pointer), and the content of the image rendered by the terminal device 201 does not include the dashed line. Figure 5c Figure 5c ​​​​​​The dashed line in the figure is only used to reflect that the window is rotated by a rotation angle (determined based on the moving track of the pointer), and the picture content of the target image rendered by the terminal device 201 does not include the dashed line.

[0118] It is worth noting that the above implementation manner of rendering the target image according to the target operation and the initial image data is only an example, and in other embodiments, the terminal device 201 can also render the target image in other manners, which is not limited.

[0119] S305: The terminal device 201 displays the rendered target image.

[0120] In this embodiment, after rendering the target image locally, the terminal device 201 can display the target image, so that the user can see the change of the display area of the to-be-operated object under the control of the target operation on the terminal device 201.

[0121] Since the terminal device 201 renders the target image according to the initial image data that has been received and displays it in response to the target operation performed by the user, it can not need to temporarily wait for the server 101 to send new image data to render the target image, that is, it does not need to perform the process of transmitting image data between the server 101 and the terminal device 201, which can effectively reduce the time delay of the terminal device 201 in responding to the target operation and presenting the target image corresponding to the target operation. Thus, in the process of the user operating the to-be-operated object in the virtual desktop, the display area of the to-be-operated object in the virtual desktop displayed by the terminal device 201 can change in real time along with the operation performed by the user, and the smoothness of the change of the display area of the to-be-operated object in the virtual desktop will be higher, so as to effectively improve the user's experience of using the virtual desktop.

[0122] It can be understood that, Figure 3 The terminal device 201 renders a target image of the virtual desktop as an example, and in actual application, the terminal device 201 can generate and display a plurality of continuous images of the virtual desktop based on the above implementation process and the target operation continuously performed by the user, so that the user can see the continuous dynamic change of the display area of the to-be-operated object on the terminal 201 along with the continuous target operation performed by the user, such as the change of the position and size of the frame indicating the display area of the to-be-operated object in the image picture continuously presented by the terminal device 201.

[0123] In a further possible implementation, when terminal device 201 detects that a target operation on an object to be operated on in the virtual desktop has been completed, such as when terminal device 201 detects a release event on the object to be operated on (e.g., a mouse button being released), terminal device 201 can send notification message 2 to server 101, such as... Figure 3 As shown, notification message 2 can indicate that the target operation has been completed. Furthermore, notification message 2 may also include area indication information 2, which can include the coordinates, size, or rotation angle information of the object to be operated on in the virtual desktop when the target operation is completed. Specifically, when the target operation is a drag operation, the area indication information 2 can be coordinate information; when the target operation is a zoom operation, the area indication information can be size information; and when the target operation is a rotation operation, the area indication information 2 can be rotation angle information, which indicates the angle by which the object to be operated is rotated. Thus, after receiving notification message 2, server 101 can render a new image based on the area indication information 2 in notification message 2 and send the new image to terminal device 201 for display.

[0124] In this way, after the target operation is completed, the server 101 can continue to render the image of the virtual desktop and display the image on the terminal device 201. Thus, the display area of ​​the object to be operated on that the user sees on the terminal device 201 (that is, the display area of ​​the object to be operated in the image rendered by the server 101) is consistent with the display area of ​​the object to be operated when the user ends the target operation (that is, the display area of ​​the object to be operated in the image rendered by the terminal device 201). This ensures that the position change of the object to be operated in the virtual desktop can maintain a high level of smoothness before and after the user performs the target operation (avoiding the display area of ​​the object to be operated in the virtual desktop from abruptly changing to the display area before the target operation was performed), thereby ensuring the user's experience of using the virtual desktop.

[0125] Furthermore, the screen presented by the terminal device 201 may also include a pointer. Therefore, the notification message 2 sent by the terminal device 201 may also include the display position of the pointer in the screen, that is, its position in the virtual desktop. In this way, the image position of the pointer in the newly rendered image sent by the server 101 to the terminal device 201 is consistent with the position of the pointer in the virtual desktop after the user completes the target operation.

[0126] In practical applications, during the execution of the target operation on the object to be operated on, server 101 can continuously send data of the image rendered on server 101 to terminal device 201. Thus, each time terminal device 201 needs to render and generate a new image (terminal device 201 renders and displays an image at regular intervals to ensure smooth changes in the display area of ​​the object to be operated on terminal device 201), it can render and display a new image based on the latest image data (sent to terminal device 201 by server 101) pre-received and stored locally. Upon detecting a target operation, terminal device 201 can send notification message 1 to server 101 to notify server 101 that the object to be operated on has begun to undergo the target operation; alternatively, terminal device 201 may choose not to send notification message 1.

[0127] Alternatively, after detecting a target operation, terminal device 201 can send notification message 1 to server 101 to notify server 101 that the target operation has begun to be performed on the object to be operated on. Upon receiving notification message 1 (i.e., during the execution of the target operation on the object to be operated on), server 101 can stop sending data of the image rendered on server 101 to terminal device 201. Before terminal device 201 sends notification message 1, server 101 will continuously send image data to terminal device 201. Therefore, each time terminal device 201 needs to render a new image, it can render and display a new image based on the locally saved initial image data. In this way, in the multiple consecutive images of the virtual desktop presented by terminal device 201, only the position of the border indicating the display area of ​​the object to be operated on changes.

[0128] It is worth noting that the above Figure 3 The embodiments shown are for illustrative purposes only. In actual applications, the specific implementation can be different. Figure 3 The steps may be added, deleted, or replaced based on the illustrated embodiments, and there are no limitations on this.

[0129] For example, in other embodiments, before the user performs the target operation, the server 101 only sends the initial image data of the rendered virtual desktop and does not send the region indication information 1. After the terminal device 201 detects the target operation, the terminal device 201 can send a notification message 1 to the server 101 to notify the server 101 that the target operation has begun. At this time, upon receiving the notification message 1, the server 101 will send the region indication information 1 to the terminal device 201 so that the terminal device 201 can determine the display area of ​​the target object in the initial image data of the virtual desktop according to the received region indication information 1, and further render the target image according to the region indication information 1.

[0130] For example, in this embodiment, the terminal device 201 determines the display region (or the controllable region) of the to-be-operated object in the image corresponding to the initial image data according to the region indication information 1 (or the indication information of the controllable region) sent by the server 101, and determines that the target operation exists for the to-be-operated object by detecting that the operation position indicated by the pointer is located in the display region (or the controllable region included in the display region) of the to-be-operated object, so as to realize the detection of the target operation. In other embodiments, the server 101 can also not send the region indication information 1 (and the indication information of the controllable region) to the terminal device 201. At this time, the terminal device 201 can recognize the display region of the to-be-operated object in the image corresponding to the initial image data by using an artificial intelligence (AI) model or a computer vision (CV) technology.

[0131] In a specific implementation, the terminal device 201 can input the initial image data into the AI model that has been trained in advance, and the AI model performs inference according to the initial image data and outputs an inference result, where the inference result is used to indicate the focal object in the image corresponding to the initial image data and the display region of the focal object in the initial image data. Alternatively, the terminal device 201 can recognize the focal object in the initial image data and the display region of the focal object in the image corresponding to the initial image data by using a CV technology, such as recognizing the focal object and the display region of the focal object in the image by using an edge detection algorithm or a shape matching algorithm. The recognized focal object is the to-be-operated object. After the to-be-operated object and the display region of the to-be-operated object in the image corresponding to the initial image data are determined, the terminal device 201 can determine that the operation exists for the to-be-operated object by detecting that the operation position indicated by the pointer is located in the controllable region included in the display region of the to-be-operated object, and further determine whether the operation is the target operation. For example, after detecting the press event for the to-be-operated object, if the terminal device 201 does not detect the lift event for the to-be-operated object within a preset time period, it is determined that the operation is the target operation, so as to realize the detection of the target operation for the to-be-operated object.

[0132] In this way, the server 101 only needs to send the initial image data to the terminal device 201 (without sending the region indication information). Accordingly, the terminal device 201 can detect the target operation based on the above manner, render the target image according to the target operation and the received initial image data, and display the target image, so as to reduce the time delay of the terminal device 201 in responding to the target operation and presenting the target image corresponding to the target operation, and improve the user experience of using the virtual desktop.

[0133] The above Figure 3 In the embodiment shown, the server 101 renders an image and sends data of the image (i.e., initial image data) to the terminal device 201.

[0134] In other embodiments, the initial image data sent by the server 101 to the terminal device 201 can be background image data and window image data that have not been rendered. Details are described below in connection with Figure 6 、 Figure 8 、 Figure 10 and Figure 12 In the embodiment shown, the object to be operated can be a window. Figure 6 、 Figure 8 、 Figure 10 and Figure 12 In the embodiment shown, the object to be operated can be a window.

[0135] Referring to Figure 6 , another flowchart for displaying a virtual desktop is shown. Figure 6 The embodiment shown can include the following steps.

[0136] S601: The server 101 sends historical background image data, historical window image data, and region indication information 1 of a virtual desktop to the terminal device 201.

[0137] The background image data refers to image data that is the background in the virtual desktop. The background image described by the background image data can not include any type of object. Alternatively, the background image described by the background image data can include objects other than the window on which the target operation is performed. When the background image described by the background image data includes multiple objects other than the window on which the target operation is performed, the display regions of the multiple objects can overlap or not overlap. The historical background image data refers to background image data received by the terminal device 201 before the window on which the target operation is performed.

[0138] The window image data refers to image data used to describe a window in the virtual desktop. The historical window image data refers to window image data received by the terminal device 201 before the object to be operated (i.e., the window) on which the target operation is performed.

[0139] The region indication information 1 includes coordinate information or size information of the window in the virtual desktop, or both coordinate information and size information. In actual applications, the region indication information 1 can be used to indicate the display region of the window in the virtual desktop.

[0140] In this embodiment, the image of the virtual desktop can be rendered on the terminal device 201. Therefore, before the user performs the target operation on the window in the virtual desktop, the server 101 can send the terminal device 201 the relevant data required for participating in generating the image of the virtual desktop, which can specifically include the historical background image data of the virtual desktop, the historical window image data, and the region indication information 1.

[0141] S602: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data, and the region indication information 1, and displays the initial image.

[0142] In the process of using the virtual desktop by the user using the terminal device 201, the terminal device 201 can determine the display region of the window in the virtual desktop according to the region indication information, and superimpose and draw the historical window image data and the historical background image data according to the display region to render an image, which is referred to as an initial image in this embodiment for ease of distinction. Then, the terminal device 201 displays the initial image, so that the user can continue to use the virtual desktop based on the initial image of the virtual desktop presented by the terminal device 201.

[0143] In actual application, the terminal device 201 can locally save the received historical window image data and the region indication information 1.

[0144] S603: The terminal device 201 detects the target operation on the window in the virtual desktop according to the region indication information 1.

[0145] The implementation manner of detecting the target operation by the terminal device 201 in this embodiment is similar to the implementation manner of detecting the target operation by the terminal device 201 in the above Figure 3 embodiment, and details are referred to the description of the related part in the above embodiment, which is not described herein.

[0146] S604: The terminal device 201 sends a notification message 1 to the server 101, where the notification message 1 is used to indicate that the window starts to be operated.

[0147] S605: The server 101 sends the target background image data of the virtual desktop to the terminal device 201.

[0148] In this embodiment, after detecting the target operation, the terminal device 201 can notify the server 101 of the target operation. At this time, the image data sent by the server 101 to the terminal device 201 can not include the window image data and the region indication information, but can only include the target background image data of the virtual desktop. The target background image data refers to the background image data received by the terminal device 201 after the window is executed by the target operation. In addition, the background image of the virtual desktop described by the target background image data can be different from the background image of the virtual desktop described by the historical background image data.

[0149] S606: The terminal device 201 renders the target image of the virtual desktop, wherein the display region of the window in the target image is determined based on the target operation, and the target image is obtained based on the target background image data and the historical window image data.

[0150] S607: The terminal device 201 displays the target image rendered.

[0151] In this embodiment, the terminal device 201 can render the target image (i.e., the new image of the virtual desktop) based on the pre-stored historical window image data and the target background image data received, and display the target image. At this time, the picture content of the window image in the target image displayed by the terminal device 201 remains unchanged, but the display region of the window in the target image (i.e., the target display region) is different from the display region of the window in the initial image, and the background picture content in the target image is different from the background picture content in the initial image. In this way, the user can not only see the change of the display region of the window under the control of the target operation on the terminal device 201, but also see the change of the background picture content of the virtual desktop during the operation of the window (for example, the background picture of the virtual desktop can be a dynamic wallpaper that changes continuously). For example, as shown in FIG. 6A, before the user performs the target operation, the initial image displayed by the terminal device 201 is the image shown in the upper part of FIG. 6A, but after the user performs the target operation, the target image displayed by the terminal device 201 is the image shown in the lower part of FIG. 6A. Among them, the background picture content and the display region of the window in the virtual desktop change, but the picture content in the window remains unchanged. Figure 7 Figure 7 Figure 7

[0152] ​​​In response to the target operation performed by the user, the image data sent by the server 101 to the terminal device 201 can be reduced from the background image data and the window image data to the background image data, which can effectively reduce the amount of data transmitted between the server 101 and the terminal device 201, that is, reduce the transmission delay between the server 101 and the terminal device 201, thereby reducing the time delay of the terminal device 201 in response to the target operation and presenting the target image corresponding to the target operation. In this way, during the user's operation of the window in the virtual desktop, the display area of the window in the virtual desktop displayed by the terminal device 201 can change in real time with the operation performed by the user, and the smoothness of the change of the display area of the window in the virtual desktop will be higher, thereby effectively improving the user's experience of using the virtual desktop.

[0153] In actual application, during the user's operation of the target operation, the server 101 can detect whether the target background image data currently needed to be sent to the terminal device 201 is consistent with the historical background image data sent before. When the two background image data are inconsistent, the server 101 sends new background image data (i.e., the target background image data) to the terminal device 201, so that the terminal device 201 renders a new image based on the new background image data. When the target background image data is consistent with the historical background image data, the server 101 can not send the target background image data, and at this time, the terminal device 201 can render a target image based on the historical background image data (previously saved), the historical window image data, and the region indication information 1 within a preset time period (the time length is less than the time length of transmitting the background image data) without receiving the target background image data.

[0154] In a further possible implementation, when the terminal device 201 detects that the target operation on the window in the virtual desktop is completed, the terminal device 201 can send a notification message 2 to the server 101, such as Figure 6As shown, the notification message 2 can indicate that the target operation execution is completed. In addition, the notification message 2 can further include region indication information 2, which can be used to indicate the display region of the window in the virtual desktop when the target operation is executed to completion. Specifically, the region indication information 2 can include coordinate information, size information or rotation angle information of the window in the virtual desktop when the target operation is executed to completion. Further, the notification message 2 sent by the terminal device 201 can further carry the position of the pointer in the virtual desktop (which can be determined according to the moving track of the pointer). In this way, after receiving the notification message 2, the server 101 can resume sending the window image data, the background image data and the region indication information 2 to the terminal device 201, so that the terminal device 201 renders a new image based on the received image data and displays the image. In addition, the position of the window in the virtual desktop before and after the user performs the target operation on the window can have a high level of smoothness, which can ensure the user experience of using the virtual desktop.

[0155] The above Figure 6 In the embodiment shown, the background of the virtual desktop displayed by the terminal device 201 changes, but the window picture content remains unchanged. In the following Figure 8 In the embodiment shown, the background of the virtual desktop displayed by the terminal device 201 remains unchanged, but the window picture content changes during the user operation on the window. The following Figure 8 will be described in detail. The to-be-operated object on which the target operation is performed can be specifically the window.

[0156] Referring to Figure 8 , another flowchart of virtual desktop display is shown. Figure 8 The embodiment shown can specifically include the following steps.

[0157] S801: The server 101 sends the historical background image data, the historical window image data and the region indication information 1 of the virtual desktop to the terminal device 201.

[0158] In this embodiment, the image data sent by the server 101 to the terminal device 201 is specifically an unrendered image.

[0159] S802: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data and the region indication information 1, and displays the initial image.

[0160] In actual application, the terminal device 201 can locally save the received historical background image data and the region indication information 1.

[0161] S803: The terminal device 201 detects the target operation on the window in the virtual desktop according to the region indication information 1.

[0162] S804: The terminal device 201 sends a notification message 1 to the server 101, where the notification message 1 is used to indicate that the window starts to be executed the target operation.

[0163] In this embodiment, the implementation manners of steps S801 to S804 are similar to those of steps S601 to S604 in the above-described Figure 6 The implementation manners of steps S601 to S604 in the above-described

[0164] S805: The server 101 sends target window image data of the virtual desktop to the terminal device 201.

[0165] In this embodiment, after detecting the target operation, the terminal device 201 can notify the server 101. At this time, the data sent by the server 101 to the terminal device 201 can not include the background image data and the region indication information, but can only include the target window image data. The target window image data refers to the window image data received by the terminal device 201 after the window is executed the target operation. The target window image data, and the described window picture content can be different from the window picture content described by the historical background image data.

[0166] S806: The terminal device 201 renders a target image of the virtual desktop, where the display region of the window in the target image is determined based on the target operation, and the target image is obtained based on the historical background image data and the target window image data.

[0167] S807: The terminal device 201 displays the rendered target image.

[0168] In this embodiment, the terminal device 201 can generate a target image (i.e., a new image of the virtual desktop) according to the pre-stored historical window image data and the newly received target background image data, and display the target image. At this time, the background picture content in the target image displayed by the terminal device 201 remains unchanged, but the display region of the window in the target image (i.e., the target display region) is different from the display region of the window in the initial image, and the window picture content in the target image is different from the window picture content in the initial image. In this way, the user can not only see the change of the display region of the window under the control of the target operation on the terminal device 201, but also see the change of the window picture content in the virtual desktop during the operation of the window (for example, a continuously changing video frame picture can be played in the window of the virtual desktop). For example, as shown in FIG. 8, the display region of the window in the target image is different from the display region of the window in the initial image, and the window picture content in the target image is different from the window picture content in the initial image. Figure 9As shown, the initial image displayed by the terminal device 201 before the user performs the target operation is Figure 9 As shown above, but after the user performs the target operation, the target image displayed by the terminal device 201 is Figure 9 As shown below. Among them, the window picture content and the display area of the window in the virtual desktop change, but the background picture content of the virtual desktop remains unchanged.

[0169] And in the process of responding to the target operation performed by the user, the image data sent by the server 101 to the terminal device 201 can be reduced from two kinds of data, background image data and window image data, to one kind of data, window image data, which can effectively reduce the amount of data transmitted between the server 101 and the terminal device 201, that is, reduce the transmission delay between the server 101 and the terminal device 201, thereby reducing the delay of the terminal device 201 in responding to the target operation and presenting the target image corresponding to the target operation. In this way, in the process of the user operating the window in the virtual desktop, the display area of the window in the virtual desktop displayed by the terminal device 201 can change in real time with the operation performed by the user, and the smoothness of the change of the display area of the window in the virtual desktop will be higher, thereby effectively improving the user's experience of using the virtual desktop.

[0170] In actual application, in the process of the user performing the target operation, the server 101 can detect whether the target window image data currently needed to be sent to the terminal device 201 is consistent with the historical window image data sent before. And when the two window image data are inconsistent, the server 101 will send new window image data (i.e. the above-mentioned target window image data) to the terminal device 201, so that the terminal device 201 renders a new image based on the new window image data. When the target window image data is consistent with the historical window image data, the server 101 can not send the target window image data, at this time, the terminal device 201 can render a target image based on the historical window image data (previously saved), historical background image data and region indication information 1 within a preset time period (the time length is less than that of transmitting background image data) without receiving the target window image data.

[0171] In a further possible implementation, when the terminal device 201 detects that the target operation on the window in the virtual desktop is completed, the terminal device 201 can send a notification message 2 to the server 101, such as Figure 8As shown, the notification message 2 can indicate that the target operation execution is completed. Further, the notification message 2 can also include region indication information 2, which can include coordinate information, size information or rotation angle information of the window in the virtual desktop at the time when the target operation execution is completed. Further, the notification message 2 sent by the terminal device 201 can also carry the position of the pointer in the virtual desktop (which can be determined according to the moving track of the pointer). In this way, after receiving the notification message 2, the server 101 can send the window image data, the background image data and the region indication information 2 carried in the notification message 2 to the terminal device 201, so that the terminal device 201 renders a new image based on the received data and displays the image. Further, the position of the window in the virtual desktop before and after the user performs the target operation on the window can have a high level of smoothness, which can ensure the user experience of using the virtual desktop.

[0172] Referring to Figure 10 , another flow diagram of displaying a virtual desktop is shown. In Figure 10 the embodiment shown, the background of the virtual desktop and the window picture content displayed by the terminal device 201 can all change during the user operation on the window. Figure 8 The embodiment shown can specifically include the following steps.

[0173] S1001: The server 101 sends historical background image data, historical window image data and region indication information 1 of the virtual desktop to the terminal device 201.

[0174] In this embodiment, the image data sent by the server 101 to the terminal device 201 is specifically an unrendered image.

[0175] S1002: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data and the region indication information 1, and displays the initial image.

[0176] S1003: The terminal device 201 detects a target operation on the window in the virtual desktop according to the region indication information 1.

[0177] S1004: The terminal device 201 sends a notification message 1 to the server 101, which is used to indicate that the window starts to be operated by the target operation.

[0178] In this embodiment, the implementation manners of steps S1001 to S1004 are similar to the implementation manners of steps S601 to S604 in the above Figure 6 embodiment, and details are described in the above embodiment, which will not be repeated here.

[0179] S1005: The server 101 sends target background image data, target window image data and region indication information 1 of the virtual desktop to the terminal device 201.

[0180] In this embodiment, the server 101 continuously sends background image data, window image data and region indication information to the terminal device 201, i.e., the process of the server 101 sending background image data, window image data and region indication information 1 to the terminal device 201 is independent of the process of the terminal device 201 detecting the target operation. Therefore, in the process of the terminal device 201 detecting the target operation, the server 101 can receive and save the target background image data, target window image data and region indication information 1 newly sent by the server 101 in the terminal device 201.

[0181] S1006: The terminal device 201 renders a target image of the virtual desktop, wherein the display region of the window in the target image is determined based on the target operation, and the target image is obtained based on the target background image data and the target window image data.

[0182] S1007: The terminal device 201 displays the rendered target image.

[0183] In this embodiment, the terminal device 201 can generate a target image according to the target window image data and the target background image data that have been saved, and display the target image. At this time, the background picture content, the window picture content and the display region of the window in the virtual desktop in the target image displayed by the terminal device 201 all change. In this way, the user can not only see the change of the display region of the window under the control of the target operation on the terminal device 201, but also see the change of the background picture content and the window picture content in the virtual desktop during the operation of the window. For example, as shown in FIG. 8, before the user performs the target operation, the initial image displayed by the terminal device 201 is the image shown in the upper part of FIG. 8, but after the user performs the target operation, the target image displayed by the terminal device 201 is the image shown in the lower part of FIG. 8. Among them, the background picture content, the window picture content and the display region of the window in the virtual desktop all change. Figure 11 Figure 11 Figure 11

[0184] ​​​In actual application, in the process that the user performs the target operation, the server 101 can detect whether the target window image data currently required to be sent to the terminal device 201 is consistent with the historical window image data previously sent, and whether the target background image data is consistent with the historical background image data previously sent. When the two window image data are inconsistent, the server 101 sends the terminal device 201 the new window image data (i.e., the target window image data described above), and when the two background image data are inconsistent, the server 101 sends the terminal device 201 the new background image data (i.e., the target background image data described above), so that the terminal device 201 renders a new image based on the new background image data and the window image data. When the target window image data is consistent with the historical window image data, the server 101 can not send the target window image data, and when the target background image data is consistent with the historical background image data, the server 101 can not send the target background image data. At this time, after the terminal device 201 receives the target background image data and the target window image data within a preset time period, the terminal device 201 can render a target image based on the newly received target window image data and target background image data (and the region indication information 1).

[0185] In a further possible implementation, when the terminal device 201 detects that the target operation on the window in the virtual desktop is performed, the terminal device 201 can send a notification message 2 to the server 101, as shown in Figure 10 The notification message 2 can indicate that the target operation is performed. In addition, the notification message 2 can further include region indication information 2, which can include coordinate information, size information or rotation angle information of the window in the virtual desktop when the target operation is performed. Further, the notification message 2 sent by the terminal device 201 can further carry the position of the pointer in the virtual desktop (which can be determined according to the moving track of the pointer). In addition, the position of the window in the virtual desktop before and after the user performs the target operation on the window can have a high level of smoothness, which can ensure the user experience of using the virtual desktop.

[0186] Referring to Figure 12 , another flow diagram of virtual desktop display is shown. In the embodiment shown in Figure 12 , the background and window picture content of the virtual desktop displayed by the terminal device 201 can remain unchanged in the process that the user operates the window. Figure 12 The embodiment shown in

[0187] S1201: The server 101 sends the terminal device 201 the historical background image data, the historical window image data and the region indication information 1 of the virtual desktop.

[0188] In this embodiment, the image data sent by the server 101 to the terminal device 201 is specifically an unrendered image.

[0189] S1202: The terminal device 201 renders an initial image of the virtual desktop according to the historical background image data, the historical window image data, and the region indication information 1, and displays the initial image.

[0190] S1203: The terminal device 201 detects a target operation on the window in the virtual desktop according to the region indication information 1.

[0191] S1204: The terminal device 201 sends a notification message 1 to the server 101, where the notification message 1 is used to indicate that the window starts to be executed by the target operation.

[0192] In this embodiment, the implementation manners of steps S1201 to S1204 are similar to the implementation manners of steps S601 to S604 in the above-described Figure 6 embodiment, and details are described in the above-described embodiment, which will not be described here.

[0193] S1205: The terminal device 201 renders a target image of the virtual desktop, where a display region of the window in the target image is determined based on the target operation, and the target image is obtained based on the historical background image data and the historical window image data.

[0194] S1206: The terminal device 201 displays the target image rendered.

[0195] In this embodiment, after determining that the window is started to be executed by the target operation, the server 101 can not send the background image data, the window image data, and the region indication information to the terminal device 201. Therefore, the terminal device 201 can generate the target image according to the historical window image data and the historical background image data that have been saved, and display the target image. At this time, the background picture content and the window picture content in the target image displayed by the terminal device 201 remain unchanged, but the display region of the window in the virtual desktop changes. For example, as shown in FIG. 8, Figure 13 FIG. 8 shows an image displayed by the terminal device 201 before the user performs the target operation, and FIG. 9 shows an image displayed by the terminal device 201 after the user performs the target operation. The background picture content and the window picture content in the virtual desktop remain unchanged, but the display region of the window in the virtual desktop changes. Figure 13 Figure 13

[0196] ​​In other embodiments, the server 101 can still send the background image data, the window image data and the region indication information to the terminal device 201 after determining that the window starts to perform the target operation. However, when the terminal device 201 needs to generate and display the target image, if the terminal device 201 receives no new background image data and window image data, the terminal device 201 can generate and display the target image by using the historical background image data and the historical window image data (and the region indication information 1) saved locally, so as to ensure that the position change of the window in the virtual desktop can have a high level of smoothness, which can ensure the user experience of using the virtual desktop.

[0197] In further possible embodiments, when the terminal device 201 detects that the target operation performed on the window in the virtual desktop is completed, the terminal device 201 can send a notification message 2 to the server 101, as shown in Figure 12 The notification message 2 can indicate that the target operation is completed. In addition, the notification message 2 can further include region indication information 2, which can include the coordinate information, the size information or the rotation angle information of the window in the virtual desktop when the target operation is completed. Further, the notification message 2 sent by the terminal device 201 can further carry the position of the pointer in the virtual desktop (which can be determined according to the moving track of the pointer).

[0198] It should be noted that the embodiments shown in Figure 3 , Figure 6 and Figure 8 , Figure 10 and Figure 12 are exemplary described by taking the data processing system 10 shown in Figure 1 as an example. In the data processing system 20 shown in Figure 2 , when the user performs the operation of adjusting the display region on the window in the virtual desktop displayed by the terminal device 302, the terminal device 302 and the computing device 301 can also display the image of the virtual desktop in real time based on the similar manner described above, so as to improve the user experience of using the virtual desktop on the terminal device 302. The specific implementation process can be referred to the description of the related part above, and will not be described here.

[0199] It should be noted that other reasonable combinations of steps that can be thought of by those skilled in the art based on the above description are also within the protection scope of the present application. Secondly, those skilled in the art should be familiar that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the present application.

[0200] The above is described in combination with Figure 1 to Figure 13The virtual desktop display method provided by the embodiments of the present application is introduced, and then the structure of the terminal device and the computing device provided by the embodiments of the present application is introduced in combination with the drawings.

[0201] Referring to Figure 14 , a structural schematic diagram of a terminal device is shown. As Figure 14 indicated, the terminal device 1400 includes:

[0202] The communication module 1401 is configured to receive initial image data of a virtual desktop and first region indication information from a computing device, the virtual desktop including a to-be-operated object, and the first region indication information including coordinate information or size information of the to-be-operated object in the virtual desktop.

[0203] The detection module 1402 is configured to detect a target operation on the to-be-operated object according to the first region indication information, the target operation being used to adjust a display region of the to-be-operated object in the virtual desktop.

[0204] The rendering module 1403 is configured to render a target image, a display region of the to-be-operated object in the target image being determined based on the target operation, and the target image being obtained based on the initial image data.

[0205] The display module 1404 is configured to display the target image.

[0206] Alternatively, the communication module 1401 is configured to receive initial image data of a virtual desktop from a computing device, the virtual desktop including a to-be-operated object.

[0207] The detection module 1402 is configured to detect a target operation on the to-be-operated object, the target operation being used to adjust a display region of the to-be-operated object in the virtual desktop.

[0208] The rendering module 1403 is configured to render a target image according to the rendering, a display region of the to-be-operated object in the target image being determined based on the target operation, and the target image being obtained based on the initial image data.

[0209] The display module 1404 is configured to display the target image.

[0210] In a possible implementation, the communication module 1401 is further configured to send a first notification message to the computing device after detecting the target operation, the first notification message being used to indicate that the to-be-operated object starts to be operated according to the target operation.

[0211] In a possible implementation, the communication module 1401 is further configured to send a second notification message to the computing device after detecting that the target operation is completed, the second notification message being used to indicate that the target operation is completed, and the second notification message further including second region indication information, the second region indication information including coordinate information, size information, or rotation angle information of the to-be-operated object in the virtual desktop when the target operation is completed.

[0212] In a possible implementation, the rendering module 1403 is specifically configured to:

[0213] determine a moving track of a pointer in the virtual desktop according to the target operation, the pointer being used to indicate an operation position in the virtual desktop;

[0214] render the target image according to the moving track of the pointer and the received initial image data.

[0215] In a possible implementation, the to-be-operated object is a window, and the initial image data of the virtual desktop is image data rendered on the computing device.

[0216] The rendering module 1403 is specifically configured to:

[0217] generate border image data according to the target operation, a region in a border indicated by the border image data being a display region of the to-be-operated object in the target image;

[0218] render the target image of the virtual desktop according to the border image data, the moving track of the pointer, and the initial image data.

[0219] In a possible implementation, the to-be-operated object is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0220] In a possible implementation, the initial image data includes target background image data and historical window image data, the target background image data being background image data received by the terminal device after the to-be-operated object is started to be operated, and the historical window image data being window image data received by the terminal device before the to-be-operated object is operated.

[0221] Alternatively, the initial image data includes historical background image data and target window image data, the historical background image data being background image data received by the terminal device before the to-be-operated object is started to be operated, and the target window image data being window image data received by the terminal device after the to-be-operated object is operated.

[0222] Or, the initial image data comprises target background image data and target window image data, the target background image data is background image data received by the terminal device after the object to be operated starts to perform the target operation, and the target window image data is window image data received by the terminal device after the object to be operated performs the target operation.

[0223] Or, the initial image data comprises history background image data and history window image data, the history background image data is background image data received by the terminal device before the object to be operated starts to perform the target operation, and the history window image data is window image data received by the terminal device before the object to be operated performs the target operation.

[0224] In a possible implementation, the target operation is a drag operation, a zoom operation, or a rotation operation.

[0225] Because Figure 14 As shown in the terminal device 1400, corresponding to the above Figure 3 , Figure 6 , Figure 8 , Figure 10 and Figure 12 As shown in the terminal device 201 in the embodiment, therefore Figure 14 The specific implementation of the terminal device 1400 and the technical effects thereof are described above with reference to the above Figure 3 , Figure 6 , Figure 8 , Figure 10 and Figure 12 , and will not be described here.

[0226] Referring to Figure 15 , a structural schematic diagram of a computing device is shown. As Figure 15 shown, the computing device 1500 comprises:

[0227] The generating module 1501 is configured to generate initial image data of a virtual desktop and first region indication information, the virtual desktop comprising an object to be operated, and the first region indication information comprising coordinate information or size information of the object to be operated in the virtual desktop.

[0228] The communication module 1502 is configured to send, to a terminal device, the initial image data and the first region indication information before the object to be operated performs a target operation, the target operation being used to adjust a display region of the object to be operated in the virtual desktop, a target image displayed on the terminal device being generated based on the initial image data and the first region indication information, and a display region of the object to be operated in the target image being determined based on a target operation detected on the terminal device.

[0229] Alternatively, the generating module 1501 is configured to generate initial image data of a virtual desktop, the virtual desktop including an object to be operated;

[0230] The communication module 1502 is configured to send the initial image data to a terminal device before the object to be operated is executed by a target operation, the target operation being used to adjust a display area of the object to be operated in the virtual desktop, a target image displayed on the terminal device being generated based on the initial image data, and a display area of the object to be operated in the target image being determined based on the target operation detected on the terminal device. In a possible implementation, the communication module 1502 is further configured to receive a first notification message from the terminal device, the first notification message being used to indicate that the object to be operated starts to be executed by the target operation.

[0231] In a possible implementation, the communication module 1502 is further configured to receive a second notification message from the terminal device after the target operation is executed, the second notification message being used to indicate that the target operation is executed, and the second notification message further including second area indication information, the second area indication information including coordinate information, size information, or rotation angle information of the object to be operated in the virtual desktop when the target operation is executed.

[0232] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop is image data rendered on the computing device.

[0233] In a possible implementation, the object to be operated is a window, and the initial image data of the virtual desktop includes background image data and window image data.

[0234] In a possible implementation, the initial image data includes target background image data and historical window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to be executed by the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated is executed by the target operation.

[0235] In a possible implementation, the initial image data includes historical background image data and target window image data, the historical background image data being background image data received by the terminal device before the object to be operated starts to be executed by the target operation, and the target window image data being window image data received by the terminal device after the object to be operated is executed by the target operation.

[0236] In a possible implementation, the initial image data includes target background image data and target window image data, the target background image data being background image data received by the terminal device after the object to be operated starts to be executed by the target operation, and the target window image data being window image data received by the terminal device after the object to be operated is executed by the target operation.

[0237] Alternatively, the initial image data includes historical background image data and historical window image data, the historical background image data being background image data received by the terminal device before the object to be operated is started to perform the target operation, and the historical window image data being window image data received by the terminal device before the object to be operated is performed the target operation.

[0238] In a possible implementation, the target operation is a drag operation, a zoom operation or a rotation operation.

[0239] Since Figure 15 the computing device 1500 shown in the above Figure 3 , Figure 6 , Figure 8 , Figure 10 and Figure 12 embodiments, the server 101 has the technical effects as described above Figure 15 , Figure 3 , Figure 6 , Figure 8 , Figure 10 and Figure 12 embodiments, and details thereof are not described herein again.

[0240] Figure 16 Fig. 16 is a schematic diagram of a hardware structure of a computing device 1600 provided in the present application, which may, for example, implement the terminal device 201 in the above Figure 3 , Figure 6 , Figure 8 , Figure 10 and Figure 12 embodiments, or may implement the server 101 in the above Figure 3 , Figure 6 , Figure 8 , Figure 10 and Figure 12 embodiments, etc.

[0241] As Figure 16As shown, the computing device 1600 includes a processor 1601, a memory 1602, a communication interface 1603. Among them, the processor 1601, the memory 1602, the communication interface 1603 communicate through the bus 1604, and can also be realized by other means such as wireless transmission. The memory 1602 is used to store instructions, and the processor 1601 is used to execute the instructions stored in the memory 1602. Further, the computing device 1600 can also include a memory unit 1605, and the memory unit 1605 can also be connected with the processor 1601, the storage medium 1602 and the communication interface 1603 through the bus 1604. Among them, the memory 1602 stores program code.

[0242] When the computing device 1600 is specifically used to implement the terminal device 201 described above, the processor 1601 can call the program code stored in the memory 1602 to perform the following operations:

[0243] Receive initial image data and first area indication information of a virtual desktop from a computing device, the virtual desktop including a to-be-operated object, the first area indication information including coordinate information or size information of the to-be-operated object in the virtual desktop; detect a target operation for the to-be-operated object according to the first area indication information, the target operation being used to adjust a display area of the to-be-operated object in the virtual desktop; render a target image, a display area of the to-be-operated object in the target image being determined based on the target operation, the target image being obtained based on the initial image data; and display the target image.

[0244] Alternatively, the processor 1601 can call the program code stored in the memory 1602 to perform the following operations:

[0245] Receive initial image data of a virtual desktop from a computing device, the virtual desktop including a to-be-operated object; detect a target operation for the to-be-operated object, the target operation being used to adjust a display area of the to-be-operated object in the virtual desktop; render a target image, a display area of the to-be-operated object in the target image being determined based on the target operation, the target image being obtained based on the initial image data; and display the target image.

[0246] When the computing device 1600 is specifically used to implement the server 101 described above, the processor 1601 can call the program code stored in the memory 1602 to perform the following operations:

[0247] The initial image data of the virtual desktop and the first region indication information are generated, the virtual desktop includes a to-be-operated object, and the first region indication information includes coordinate information or size information of the to-be-operated object in the virtual desktop; before the to-be-operated object is executed with a target operation, the initial image data and the first region indication information are sent to a terminal device, the target operation is used to adjust a display region of the to-be-operated object in the virtual desktop, a target image displayed on the terminal device is generated based on the initial image data and the first region indication information, and a display region of the to-be-operated object in the target image is determined based on the target operation detected on the terminal device.

[0248] Alternatively, the processor 1601 can invoke program code stored in the memory 1602 to perform the following operations:

[0249] The initial image data of the virtual desktop is generated, the virtual desktop includes a to-be-operated object; before the to-be-operated object is executed with a target operation, the initial image data is sent to a terminal device, the target operation is used to adjust a display region of the to-be-operated object in the virtual desktop, a target image displayed on the terminal device is generated based on the initial image data, and a display region of the to-be-operated object in the target image is determined based on the target operation detected on the terminal device.

[0250] It should be understood that, in the present embodiment, the processor 1601 can be a CPU, and can also be other general-purpose processors, deep learning processors (DPUs), GPUs, NPUs, TPUs, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, or the like, or a combination of the above various processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0251] The memory 1602 can include read-only memory and random access memory, and provide instructions and data to the processor 1601. The memory 1602 can also include non-volatile random access memory.

[0252] The memory 1602 can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory, among others. The volatile memory can be random access memory (RAM), which acts as the external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double-data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM), among others.

[0253] The communication interface 1603 is configured to communicate with other devices connected to the computing device 1600. The bus 1604 can include, in addition to the data bus, a power bus, a control bus, and a state signal bus, among others. However, for the sake of clarity, the various buses are labeled as the bus 1604 in the figure.

[0254] It should be understood that the computing device 1600 according to the embodiments of the present application can correspond to the server 101 in the embodiments shown in Figure 3 、 Figure 6 、 Figure 8 、 Figure 10 and Figure 12 above, and can be configured to perform the methods performed by the server 101, and the computing device 1600 can be configured to perform the above and other operations and / or functions to implement the corresponding methods performed by the server 101 in the embodiments shown in Figure 3 、 Figure 6 、 Figure 8 、 Figure 10 and Figure 12 above, and for brevity, will not be described again here.

[0255] The embodiment of the present application further provides a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be used to store data which can be accessed by a computing device. The computer readable storage medium can be a magnetic-based medium (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical-based medium (e.g., a compact disc (CD), a digital versatile disc (DVD)), a semiconductor-based medium (e.g., a solid state drive (SSD) or a flash memory device), and the like. The computer readable storage medium includes instructions that are executable by a computing device to implement the virtual desktop display method described above.

[0256] The embodiment of the present application further provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the processes or functions described in the embodiment of the present application are generated in whole or in part.

[0257] The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0258] The computer program product can be a software installation package, which can be downloaded and executed on a computing device when any of the methods described above is needed.

[0259] The above-described embodiments can be implemented in part or in whole through software, hardware, firmware or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions produce the above-described processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, from a website site, a computer, a server, or a data center to another website site, a computer, a server, or a data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing a set of one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0260] The terms used in the above-described embodiments are only for the purpose of describing particular embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the application, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "one or more" as used in the application embodiments refers to one, two, or more than two. The character " / " generally indicates that the associated objects before and after the " / " are in an "or" relationship. In the embodiments of the present application, "simultaneously" means in the same time period, including the case of being at the same time. The terms "first", "second", and the like as used in the specification and claims of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms used in this way can be interchanged as appropriate, and this is merely a way of distinguishing the objects of the same attribute used in the description of the embodiments of the present application.

[0261] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but can refer to one or more but not all embodiments. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner on one or more embodiments. The terms "including," "containing," and "having" are intended to be open-ended and mean that there can be additional elements other than the listed elements. The terms "a," "an," and "the" are intended to be inclusive when they

[0262] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for displaying a virtual desktop, characterized in that, The method is applied to a terminal device, and the method includes: Receive initial image data and first area indication information from a virtual desktop of a computing device, wherein the virtual desktop includes an object to be operated, and the first area indication information includes the coordinate information or size information of the object to be operated in the virtual desktop; Based on the first area indication information, a target operation is detected for the object to be operated on, and the target operation is used to adjust the display area of ​​the object to be operated on in the virtual desktop. A target image is obtained by rendering, and the display area of ​​the object to be operated on in the target image is determined based on the target operation. The target image is obtained based on the initial image data. Display the target image.

2. The method according to claim 1, characterized in that, The method further includes: Upon detecting the target operation, a first notification message is sent to the computing device, the first notification message being used to instruct the object to be operated on to begin executing the target operation.

3. The method according to claim 1, characterized in that, The method further includes: Upon detection that the target operation has been completed, a second notification message is sent to the computing device. The second notification message is used to indicate that the target operation has been completed. The second notification message also includes second area indication information, which includes the coordinate information, size information, or rotation angle information of the object to be operated on in the virtual desktop when the target operation is completed.

4. The method according to claim 1, characterized in that, The rendering process yields the target image, including: The movement trajectory of the pointer in the virtual desktop is determined based on the target operation, and the pointer is used to indicate the operation position in the virtual desktop; The target image is rendered based on the movement trajectory of the pointer and the received initial image data.

5. The method according to claim 4, characterized in that, The object to be operated on is a window, and the initial image data of the virtual desktop is image data rendered on a computing device; The step of rendering the target image based on the movement trajectory of the pointer and the received initial image data includes: According to the target operation, border image data is generated, and the area within the border indicated by the border image data is the display area of ​​the object to be operated on in the target image; The target image of the virtual desktop is rendered based on the border image data, the movement trajectory of the pointer, and the initial image data.

6. The method according to claim 1, characterized in that, The object to be operated on is a window, and the initial image data of the virtual desktop includes background image data and window image data.

7. The method according to claim 6, characterized in that, The initial image data includes target background image data and historical window image data. The target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated. The historical window image data is the window image data received by the terminal device before the target operation is started on the object to be operated. Alternatively, the initial image data may include historical background image data and target window image data, wherein the historical background image data is background image data received by the terminal device before the target operation is started on the object to be operated, and the target window image data is window image data received by the terminal device after the target operation is performed on the object to be operated. Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated. Alternatively, the initial image data may include historical background image data and historical window image data, wherein the historical background image data is background image data received by the terminal device before the target operation is started on the object to be operated, and the historical window image data is window image data received by the terminal device before the target operation is started on the object to be operated.

8. The method according to any one of claims 1 to 7, characterized in that, The target operation is a drag operation, a zoom operation, or a rotation operation.

9. A method for displaying a virtual desktop, characterized in that, The method is applied to a computing device, and the method includes: Generate initial image data and first area indication information for the virtual desktop. The virtual desktop includes an object to be operated on, and the first area indication information includes the coordinate information or size information of the object to be operated on the virtual desktop. Before the target operation is performed on the object to be operated, the initial image data and the first area indication information are sent to the terminal device. The target operation is used to adjust the display area of ​​the object to be operated on the virtual desktop. The target image displayed on the terminal device is generated based on the initial image data and the first area indication information. The display area of ​​the object to be operated on the target image is determined based on the target operation detected on the terminal device.

10. The method according to claim 9, characterized in that, The method further includes: A first notification message is received from the terminal device, the first notification message being used to instruct the object to be operated to begin performing the target operation.

11. The method according to claim 9, characterized in that, The method further includes: After the target operation is completed, a second notification message is received from the terminal device. The second notification message is used to indicate that the target operation has been completed. The second notification message also includes second area indication information, which includes the coordinate information, size information, or rotation angle information of the object to be operated on in the virtual desktop when the target operation is completed.

12. The method according to claim 9, characterized in that, The object to be operated on is a window, and the initial image data of the virtual desktop is image data rendered on a computing device.

13. The method according to claim 9, characterized in that, The object to be operated on is a window, and the initial image data of the virtual desktop includes background image data and window image data.

14. The method according to claim 13, characterized in that, The initial image data includes target background image data and historical window image data. The target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated. The historical window image data is the window image data received by the terminal device before the target operation is started on the object to be operated. Alternatively, the initial image data may include historical background image data and target window image data, wherein the historical background image data is background image data received by the terminal device before the target operation is started on the object to be operated, and the target window image data is window image data received by the terminal device after the target operation is performed on the object to be operated. Alternatively, the initial image data includes target background image data and target window image data, wherein the target background image data is the background image data received by the terminal device after the target operation is started on the object to be operated, and the target window image data is the window image data received by the terminal device after the target operation is performed on the object to be operated. Alternatively, the initial image data may include historical background image data and historical window image data, wherein the historical background image data is background image data received by the terminal device before the target operation is started on the object to be operated, and the historical window image data is window image data received by the terminal device before the target operation is started on the object to be operated.

15. The method according to any one of claims 9 to 14, characterized in that, The target operation is a drag operation, a zoom operation, or a rotation operation.

16. A method for displaying a virtual desktop, characterized in that, The method is applied to a terminal device, and the method includes: Receive initial image data from a virtual desktop of a computing device, the virtual desktop including an object to be operated; A target operation is detected targeting the object to be operated on, the target operation being used to adjust the display area of ​​the object to be operated on in the virtual desktop; A target image is obtained by rendering, and the display area of ​​the object to be operated on in the target image is determined based on the target operation. The target image is obtained based on the initial image data. Display the target image.

17. A method for displaying a virtual desktop, characterized in that, The method is applied to a computing device, and the method includes: Generate initial image data for the virtual desktop, which includes an object to be operated on; Before the target operation is performed on the object to be operated on, the initial image data is sent to the terminal device. The target operation is used to adjust the display area of ​​the object to be operated on in the virtual desktop. The target image displayed on the terminal device is generated based on the initial image data, and the display area of ​​the object to be operated on in the target image is determined based on the target operation detected on the terminal device.

18. A terminal device, characterized in that, The terminal device includes: A communication module is used to receive initial image data and first area indication information from a virtual desktop of a computing device. The virtual desktop includes an object to be operated, and the first area indication information includes the coordinate information or size information of the object to be operated in the virtual desktop. The detection module is used to detect a target operation on the object to be operated based on the first area indication information, wherein the target operation is used to adjust the display area of ​​the object to be operated on in the virtual desktop; A rendering module is used to render a target image. The display area of ​​the object to be operated on in the target image is determined based on the target operation. The target image is obtained based on the initial image data. The display module is used to display the target image.

19. A computing device, characterized in that, The computing device includes: The generation module is used to generate initial image data of the virtual desktop and first area indication information. The virtual desktop includes an object to be operated, and the first area indication information includes the coordinate information or size information of the object to be operated in the virtual desktop. A communication module is used to send the initial image data and the first area indication information to a terminal device before the target operation is performed on the object to be operated. The target operation is used to adjust the display area of ​​the object to be operated on the virtual desktop. The target image displayed on the terminal device is generated based on the initial image data and the first area indication information. The display area of ​​the object to be operated on the target image is determined based on the target operation detected on the terminal device.

20. A terminal device, characterized in that, The terminal device includes: A communication module is used to receive initial image data from a virtual desktop of a computing device, wherein the virtual desktop includes an object to be operated; The detection module is used to detect a target operation on the object to be operated on, the target operation being used to adjust the display area of ​​the object to be operated on in the virtual desktop; A rendering module is used to render a target image. The display area of ​​the object to be operated on in the target image is determined based on the target operation. The target image is obtained based on the initial image data. The display module is used to display the target image.

21. A computing device, characterized in that, The computing device includes: A generation module is used to generate initial image data for a virtual desktop, which includes objects to be operated on. A communication module is used to send the initial image data to a terminal device before the target operation is performed on the object to be operated on. The target operation is used to adjust the display area of ​​the object to be operated on in the virtual desktop. The target image displayed on the terminal device is generated based on the initial image data, and the display area of ​​the object to be operated on in the target image is determined based on the target operation detected on the terminal device.

22. A computing device, characterized in that, The computing device includes a processor and memory; The memory is used to store instructions, and the processor executes the instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 17.

23. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 17.

24. A computer program product containing instructions, characterized in that, When it is run on a computing device, it causes the computing device to perform the method as described in any one of claims 1 to 17.

Images