Cloud desktop display method, apparatus, device and storage medium

Abstract

The embodiments of the present application provide a method, apparatus, device, and storage medium for displaying a cloud desktop. In the cloud desktop display method, a real-world image in a real environment where a wearable device is located and a cloud desktop image provided by a cloud server can be obtained, and the real-world image can be displayed in synchronization with the cloud desktop image in a virtual scene of the wearable device. In this manner, the real-world image can be displayed in synchronization with the cloud desktop image in the virtual scene, so that the user can recognize the real-world scene even after wearing the wearable device, and the user does not need to remove the wearable device when using a tool in the real world. This improves the user's immersion in the virtual world.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application references Chinese patent application No. 202210329680.6, entitled "Cloud Desktop Display Method, Apparatus, Device and Storage Medium," filed on March 31, 2022, which is incorporated herein by reference in its entirety.

[0002] The embodiments of the present application relate to the field of smart wearable technology, and in particular to a method, apparatus, device and storage medium for displaying a cloud desktop. [Background technology]

[0003] With the rapid development of related technologies such as virtual reality, augmented reality, and mixed reality, new head-mounted smart devices, such as smart glasses including head-mounted virtual reality glasses and head-mounted mixed reality glasses, are being developed one after another, and the user experience is gradually improving.

[0004] In the prior art, smart glasses can be used to display cloud desktop videos sent by a cloud server, and a steering wheel or other controller paired with the smart glasses can be used to interact with the cloud desktop, allowing users to telework or engage in leisure activities in a virtual world through the cloud desktop.

[0005] However, in this method, the user cannot sense the actual scene in the real world after wearing the smart glasses, so when the user wants to use a tool in the real world, the user needs to remove the smart glasses, which reduces the user's immersion in the virtual world. Therefore, a solution needs to be proposed. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a cloud desktop display method, apparatus, device and storage medium, which allows the user to still perceive the real world scene after wearing the wearable device, and further enhances the user's sense of immersion in the virtual world.

[0007] An embodiment of the present application provides a method for displaying a cloud desktop, which includes acquiring a real-world image of a real environment in which a wearable device is located and a cloud desktop image provided by a cloud server; and displaying the real-world image in synchronization with the cloud desktop image in a virtual scene of the wearable device.

[0008] Further optionally, acquiring a real-world image of the real environment in which the wearable device is located and a cloud desktop image provided by a cloud server includes acquiring a timestamp of the cloud desktop image for any frame of the cloud desktop image in a cloud desktop video stream; and selecting a frame image from a real-world video stream capturing the real environment in which the wearable device is located that has the same timestamp as the cloud desktop image as the real-world image.

[0009] Further optionally, displaying the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device includes fusing the cloud desktop image and the real scene image to obtain a fused image; and displaying the fused image in the virtual scene of the wearable device.

[0010] Further optionally, fusing the cloud desktop image and the real world image to obtain a fused image includes overlaying the real world image on the cloud desktop image to obtain the fused image.

[0011] Further optionally, fusing the cloud desktop image and the real scene image to obtain a fused image includes splicing the real scene image and the cloud desktop image to obtain the fused image.

[0012] Further optionally, the real scene image includes a left-view real scene image and a right-view real scene image captured by a binocular camera, and fusing the cloud desktop image and the real scene image to obtain a fused image includes performing binocular rendering on the cloud desktop image to obtain a left-view virtual image and a right-view virtual image, fusing the left-view real scene image and the left-view virtual image to obtain a left-view fused image, and fusing the right-view real scene image and the right-view virtual image to obtain a right-view fused image.

[0013] Further optionally, the wearable device further includes a gaze detection module, and the method further includes detecting the user's gaze using the gaze detection module and obtaining a gaze direction; determining the user's gaze area in the virtual scene based on the gaze direction; and highlighting the real scene image if the gaze area is in the area where the real scene image is located.

[0014] An embodiment of the present application further provides a cloud desktop display device, which includes an acquisition module that acquires a real-world image in a real environment where a wearable device is located and a cloud desktop image provided by a cloud server, and a display module that displays the real-world image in synchronization with the cloud desktop image in a virtual scene of the wearable device.

[0015] An embodiment of the present application further provides a terminal device including a memory and a processor, wherein the memory is used to store one or more computer commands, and the processor is used to execute the one or more computer commands to perform steps in a cloud desktop display method.

[0016] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform steps in a method for displaying a cloud desktop.

[0017] The cloud desktop display method, apparatus, device, and storage medium provided in the embodiments of the present application can obtain a real-world image of the real environment where the wearable device is located and a cloud desktop image provided by a cloud server, and display the real-world image in synchronization with the cloud desktop image in the virtual scene of the wearable device. In this way, the real-world image is displayed in synchronization with the cloud desktop image in the virtual scene, so that the user can still sense the real world scene after wearing the wearable device, and does not need to remove the wearable device when using tools in the real world, thereby improving the user's sense of immersion in the virtual world. [Brief explanation of the drawings]

[0018] In order to more clearly describe the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description only show some embodiments of the present invention, and those skilled in the art can further obtain other drawings based on these drawings without any creative work.

[0019] [Figure 1] 1 is a flowchart of a cloud desktop display method provided by a representative embodiment of the present application; [Figure 2] 1 is a schematic diagram of an overlay provided by one exemplary embodiment of the present application; [Figure 3] 1 is a schematic diagram of splicing provided by a representative embodiment of the present application; [Figure 4] 1 is a schematic diagram of binocular rendering provided by an exemplary embodiment of the present application; [Figure 5]1 is a schematic diagram of binocular rendering modifications provided by an exemplary embodiment of the present application; [Figure 6] 1 is an architecture diagram of a mobile terminal provided by a representative embodiment of the present application; [Figure 7] 1 is a schematic diagram of a display terminal provided by a representative embodiment of the present application; [Figure 8] 1 is a schematic diagram of a cloud desktop display device provided by a representative embodiment of the present application; [Figure 9] 1 is a schematic diagram of a terminal device provided by a representative embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0020] In order to clarify the objectives, technical solutions and advantages of the embodiments of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without any creative work are all within the protection scope of the present invention.

[0021] In the prior art, smart glasses can be used to display cloud desktop videos sent by a cloud server, and a steering wheel or other controller paired with the smart glasses can be used to interact with the cloud desktop, allowing users to telework or engage in leisure activities in a virtual world through the cloud desktop. However, in this method, after wearing the smart glasses, users still cannot sense the actual scene in the real world, so they need to remove the smart glasses when they want to use tools in the real world, which reduces the user's sense of immersion in the virtual world.

[0022] To address the above technical problems, some embodiments of the present application provide solutions, and the technical solutions provided by each embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0023] FIG. 1 is a flowchart of a cloud desktop display method provided in a representative embodiment of the present application. As shown in FIG. 1, the method includes: Step 11: acquiring a real-world image of a real environment where the wearable device is located and a cloud desktop image provided by a cloud server; and step 12 of displaying the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device.

[0024] This embodiment may be executed by a terminal device, a wearable device, or a cloud server. The terminal device may include a computer, a tablet computer, a mobile phone, etc. The wearable device may include virtual reality (VR) glasses, mixed reality (MR) glasses, a VR head-mounted display (HMD), etc.

[0025] The real environment refers to an environment in the real world, and the real-scene image is used to reflect the real-world environment in which the wearable device is located. In this embodiment, a camera attached to the real-world environment captures the real-world environment in which the wearable device is located, thereby obtaining a real-scene video stream. The real-scene image is any frame image in the real-scene video stream. The camera for capturing the real environment can be attached to the wearable device, and its attachment position should ensure that the camera's field of view is within the field of view of the human eye, so that when the wearable device blocks the human eye's line of sight, the camera can replace the human eye in observing the real scene.

[0026] Cloud desktops, also known as desktop virtualization or cloud computing, use virtualization technology to perform virtual processing on various physical devices, thereby effectively improving resource utilization, cost savings, and application quality. After deploying a virtual desktop on a cloud server's cloud platform, users can access the virtual desktop and applications anywhere. The cloud server can transmit cloud desktop video streams to wearable devices for display. A cloud desktop image refers to any frame image in the cloud desktop video stream.

[0027] After the real scene image and the cloud desktop image are acquired, the real scene image can be displayed in synchronization with the cloud desktop image in the virtual scene of the wearable device.

[0028] When this embodiment is executed by a terminal device, a tablet computer is taken as an example. The tablet computer can acquire real-world images sent from a wearable device and cloud desktop images sent from a cloud server, and send the real-world images and cloud desktop images to the wearable device for synchronous display.

[0029] When this embodiment is implemented by a wearable device, VR glasses are taken as an example. The VR glasses can collect real-world images, obtain cloud desktop images sent from a cloud server, and synchronously display the real-world images and the cloud desktop images; or the terminal device can receive cloud desktop images sent from a cloud server and transfer them to the VR glasses; the VR glasses can obtain cloud desktop images sent from the terminal device, collect real-world images, and synchronously display the real-world images and the cloud desktop images.

[0030] When this embodiment is executed by a cloud server, the cloud server can obtain the real-world image sent from the wearable device, and send the real-world image and the cloud desktop image to the wearable device for synchronous display.

[0031] Among them, displaying the real scene image synchronously with the cloud desktop image means displaying the real scene image simultaneously with the cloud desktop image in the virtual scene of the wearable device, that is, the user can see the virtual cloud desktop image in the virtual scene and also see the actual real scene image.

[0032] Such an embodiment allows a real-world image to be displayed in the virtual scene in synchronization with a cloud desktop image, allowing the user to still sense the real-world scene after putting on the wearable device, and eliminating the need for the user to remove the wearable device when they want to use a real-world tool, thereby improving the user's sense of immersion in the virtual world.

[0033] At the same time, when a camera for collecting real-world images is attached to a wearable device, interaction with the cloud desktop can be achieved without the need for an external handle or other controller paired with the wearable device, which reduces hardware costs while also favoring a more integrated and lightweight wearable device.

[0034] In the following embodiment, a wearable device will be described as an example of an execution subject.

[0035] In some alternative embodiments, the wearable device can acquire a cloud desktop video stream and a real-world video stream displaying a scene in real time, transmitted from a cloud server, and perform frame-by-frame synchronized display of the cloud desktop image in the cloud desktop video stream and the real-world image in the real-world video stream.

[0036] For any frame of a cloud desktop image in the acquired cloud desktop video stream, the wearable device can acquire a timestamp for the cloud desktop image. For example, the timestamp for cloud desktop image P1 is 20:59:01. Furthermore, the wearable device can select a frame image with the same timestamp as the cloud desktop image from a video stream of a real scene captured in the real environment where the wearable device is located, and use it as a real scene image. For example, the wearable device can select image P1' with a timestamp of 20:59:01 from the video stream of the real scene and use it as a real scene image. This allows the wearable device to synchronously display different images with the same timestamp. In some possible practical application scenarios, several specific applications can be run on the cloud desktop. Since the acquired cloud desktop image and the real scene image have the same timestamp, a user can interact with the application in real time by interacting with specific items in the real scene image. For example, a user can touch a charger in the real scene image to synchronously perform a corresponding operation in the application (such as turning off the application, turning on the application, or restarting the application). If the application being executed is a game, the user can click on a cup, toothbrush, etc. in the real-world image to cause the virtual character in the game to synchronously perform the corresponding operation (drinking water, brushing teeth, etc.).

[0037] Such an embodiment allows the timestamps of the acquired cloud desktop image and the real scene image to be the same, further improving the synchronization effect between the cloud desktop image and the real scene image and reducing the sense of isolation when the cloud desktop image is displayed synchronously with the real scene image.

[0038] Optionally, the operation of step 12 of the above embodiment, “displaying a real scene image in synchronization with a cloud desktop image in a virtual scene on a wearable device”, can be: Step 121: fusing the cloud desktop image and the real scene image to obtain a fusion image; This can be achieved according to step 122 of displaying the fused image in the virtual scene on the wearable device.

[0039] This method allows users to simultaneously observe relevant information on the cloud desktop and relevant information in the real world from the fused image, eliminates the need for users to remove the wearable device when they want to use tools in the real world, and further allows users to more easily interact with objects in the real world when using the wearable device.

[0040] Optionally, the wearable device can: Embodiment 1: superimposing a real scene image on a cloud desktop image to obtain a fusion image; The cloud desktop image and the real scene image can be fused according to the second embodiment, in which the real scene image and the cloud desktop image are stitched to obtain a fusion image.

[0041] In the first embodiment, as shown in part A of Figure 2, the entire real scene image can be superimposed on the entire cloud desktop image. As shown in part B of Figure 2, a portion of the real scene image can be cropped from the real scene image and superimposed on a specified area of ​​the cloud desktop image. Alternatively, the real scene image can be reduced without being cropped and superimposed on a partial area of ​​the cloud desktop image. Optionally, the user can use entity keys on the wearable device or virtual keys in the virtual scene to perform operations such as zooming in, zooming out, or cropping on the real scene image, and adjust the area where the real scene image is superimposed on the cloud desktop image.

[0042] In embodiment 2, the real scene image and the cloud desktop image can be stitched together, as shown in Figure 3. Optionally, the user can use the entity keys on the wearable device or the virtual keys in the virtual scene to perform operations such as zooming in, zooming out, or cropping on the real scene image, and adjust the positions of the real scene image and the cloud desktop image.

[0043] It should be mentioned that the user can switch between the above two embodiments of overlay and splicing by using an entity key on the wearable device or a virtual key in the virtual scene.

[0044] According to the above embodiment, the cloud desktop image and the real scene image are merged by overlaying or splicing, and the user can freely adjust the fusion method of the cloud desktop image and the real scene image, and can more completely observe the cloud desktop image and the real scene image simultaneously.

[0045] Optionally, in a real scene, a wearable device is usually equipped with two cameras, one on the left and one on the right, which are referred to as a binocular camera for short. The wearable device can collect real scene images based on the binocular camera. The real scene images include a left-view real scene image and a right-view real scene image captured by the binocular camera. The left-view real scene image is the real scene image collected by the left camera of the binocular camera, and corresponds to the user's left eye; the right-view real scene image is the real scene image collected by the right camera of the binocular camera, and corresponds to the user's right eye.

[0046] Based on this, the wearable device can fuse the cloud desktop image and the real scene image to obtain the fused image based on the following steps: Step S1 performs binocular rendering on the cloud desktop image to obtain a left-eye virtual image and a right-eye virtual image, where the left-eye virtual image is a virtual image corresponding to the user's left eye, and the right-eye virtual image is a virtual image corresponding to the user's right eye. Step S2 fuses the left-eye real scene image and the left-eye virtual image to obtain a left-eye fused image; and fuses the right-eye real scene image and the right-eye virtual image to obtain a right-eye fused image. Based on this step, when the user's left eye and right eye see the left-eye fused image and the right-eye fused image respectively, the user's brain can automatically synthesize the images seen by the left and right eyes into a 3D image.

[0047] The binocular rendering in step S1 will be described in detail below with reference to FIG.

[0048] As shown in FIG. 4, R is the display distance required by the glasses, FOV (Field of View), and w is the distance between the user's eyes.

[0049] D is the pixel width output to the left and right screens, i.e., the maximum width of the screen that the user can see with one eye, and can be calculated based on the following formula 1: Number 1 JPEG0007808375000001.jpg29166

[0050] The distance from a pixel point on the cloud desktop image to the central axis is S, and the coordinates of this point are (S, R).

[0051] Based on the above process, the ratio of the x-coordinate value of this pixel point in the left-eye virtual image (represented by B1) and the ratio of the x-coordinate value of this pixel point in the right-eye virtual image (represented by B2) can be calculated using the following equations 2 and 3, respectively. Number 2 JPEG0007808375000002.jpg21132 count 3 JPEG0007808375000003.jpg22131

[0052] After calculating the ratio of the x-coordinate value of this pixel point in the left-eye virtual image and the right-eye virtual image, the final x-coordinate value can be obtained by multiplying it by the actual pixel width a of the screen, and the y-coordinate value does not change.

[0053] This binocular rendering method can provide left-eye virtual images and right-eye virtual images corresponding to the user's left eye and right eye, respectively, and further allows the user to use the wearable device to view the cloud desktop more realistically, improving the sense of immersion in the user's interaction with the cloud desktop.

[0054] After binocular rendering of the cloud desktop image, further image fusion can be performed. However, as shown in Figure 5, in the above-mentioned binocular rendering, the distance between the binocular cameras is usually different from the distance between the user's eyes, so after binocular rendering, further correction can be performed using the following formula:

[0055] As shown in Figure 5, d is the deviation distance between the camera and the eye, which can be calculated based on the following formula: Number 4 JPEG0007808375000004.jpg18169Here, e is the distance between the eyes and w' is the distance between the cameras.

[0056] Based on this, the offset amount rx1 of the left side x coordinate in the right screen of the camera and the offset amount rx2 of the right side x coordinate in the right screen of the camera can be calculated using the following equations 5 and 6: Number 5 JPEG0007808375000005.jpg20169 Number 6 JPEG0007808375000006.jpg16165 where FOV d is the camera's field of view, FOV e is the visual angle of the eye and R is the viewing distance required by the glasses.

[0057] By calculating the above offset amount, the left-eye virtual image and the right-eye virtual image obtained after binocular rendering can be further corrected, and the image quality of the fusion image obtained by subsequent fusion can be improved based on relatively accurate virtual images.

[0058] In some embodiments, the wearable device may be equipped with a gaze detection module, which may perform gaze detection on the user to obtain the user's gaze direction. Furthermore, the wearable device may identify the user's gaze area in the virtual scene based on the gaze direction.

[0059] If the gaze area is located in the area where the real scene image is located, the real scene image is highlighted; if the gaze area is located in the area where the cloud desktop image is located, the cloud desktop image is highlighted.

[0060] It should be noted that when a user gazes at the cloud desktop image for a long time, the real-world image can be hidden and the cloud desktop image can be displayed in full screen; when a user gazes at the real-world image for a long time, the cloud desktop image can be hidden and the real-world image can be displayed in full screen. Optionally, a virtual key / area can be preset in the virtual scene of the wearable device, and when a user gazes at the key / area, a corresponding function can be performed, such as displaying the cloud desktop image in full screen, or displaying the real-world image in full screen, or displaying the cloud desktop image and the real-world image according to a preset layout style.

[0061] The following will further explain how to display a cloud desktop with reference to Figures 6 and 7 and actual application scenarios.

[0062] FIG. 6 is an architecture diagram of a video stream processing method for a mobile terminal (ie, the above-mentioned terminal device), and FIG. 7 is an architecture diagram of a video stream processing method for a display terminal (ie, the above-mentioned wearable device).

[0063] In practice, a specific application program (hereinafter referred to as Mapp) can be installed on a mobile terminal, and the application program can obtain authorized account information. After a user enters an account password, the application program can call a cloud desktop API (Application Programming Interface) to perform authorization authentication. After the account password entered by the user is authenticated, Mapp can obtain the IP (Internet Protocol) address and port number of the cloud desktop virtual machine corresponding to the user's account.

[0064] Mapp can attempt to establish a connection with a cloud desktop virtual machine via wireless transmission based on a remote desktop connection protocol. For illustrative purposes, the remote desktop connection protocol includes, but is not limited to, the TCP / IP (Transmission Control Protocol / Internet Protocol) protocol, the NetBEUI (NetBios Enhanced User Interface) protocol, the IPX / SPX (Internetwork Packet Exchange / Sequences Packet Exchange) protocol, and the RDP (Remote Display Protocol) protocol, and the present embodiment is not limited thereto. Among these, the RDP protocol is a remote desktop protocol established on the TCP / IP protocol.

[0065] After Mapp successfully connects to the cloud desktop virtual machine, data communication between the two can begin based on the ISO-layer remote desktop connection protocol. If a display device is not connected to Mapp, Mapp operates as a normal cloud desktop client. When it detects that a display device is connected to Mapp, Mapp does not directly display the cloud desktop interface, but instead performs image processing on the cloud desktop video stream returned from the cloud desktop virtual machine and sends it to the display device for display.

[0066] After the above-mentioned remote desktop connection protocol is successfully established, the Mapp program can identify the image channel from the established virtual channels, which has a special identifier. Mapp can then obtain the image data sent from the image channel and obtain the original bitmap stream data from the cloud desktop.

[0067] Furthermore, Mapp can obtain the bitmap resolution, and perform compression optimization on the obtained original bitmap stream data based on parameters such as the resolution and frame rate parameters of the display terminal, to improve the efficiency of subsequent image processing.

[0068] Mapp can determine input parameters for binocular rendering based on other parameters of the display terminal (such as interpupillary distance (offset), field of view (FOV), rendering screen width / height, depth far / depth near of the viewing frustum, lens focal length (convergence), or anti-distortion coefficient (anti-distortion), etc.) and can generate left-eye virtual images and right-eye virtual images for each frame of the cloud desktop image through image calculation based on the input parameters.

[0069] Mapp can acquire left-view real scene images and right-view real scene images captured by a binocular camera on a display terminal, and fuses the left-view real scene image with the left-view virtual image to obtain a left-view fused image, and fuses the right-view real scene image with the right-view virtual image to obtain a right-view fused image.

[0070] For example, Mapp can establish a transmission channel with a specific application program (hereinafter referred to as Vapp) running on a display terminal via a wired connection (Type-C or Lightning). The transmission channel can then be packaged and sent to the Vapp of the display terminal via the corresponding channels. The Vapp of the display terminal identifies the fused images and then outputs and displays them on the two left and right screens of the display terminal.

[0071] Such an embodiment allows a real-world image to be displayed in the virtual scene in synchronization with a cloud desktop image, allowing the user to still sense the real-world scene after putting on the wearable device, and eliminating the need for the user to remove the wearable device when they want to use a real-world tool, thereby improving the user's sense of immersion in the virtual world.

[0072] FIG. 8 is a schematic diagram of a cloud desktop display device provided by a representative embodiment of the present application. As shown in FIG. 8, the display device includes: an acquisition module 801 for acquiring a real-world image in the real environment where the wearable device is located and a cloud desktop image provided by a cloud server; and a display module 802 for displaying the real-world image in synchronization with the cloud desktop image in the virtual scene of the wearable device.

[0073] Further optionally, when the acquisition module 801 acquires a real-world image of the real environment in which the wearable device is located and a cloud desktop image provided by the cloud server, it is specifically used to acquire a timestamp of the cloud desktop image for any frame of the cloud desktop image in the cloud desktop video stream; and to select a frame image from the real-world video stream capturing the real environment in which the wearable device is located that has the same timestamp as the cloud desktop image as the real-world image.

[0074] Further optionally, when the display module 802 displays the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device, specifically, it fuses the cloud desktop image and the real scene image to obtain a fused image; it is used to display the fused image in the virtual scene of the wearable device.

[0075] Further optionally, when the display module 802 fuses the cloud desktop image and the real scene image to obtain a fusion image, the display module 802 specifically superimposes the real scene image on the cloud desktop image to obtain the fusion image.

[0076] Further optionally, when the display module 802 fuses the cloud desktop image and the real scene image to obtain a fused image, it is specifically used to splice (combine) the real scene image and the cloud desktop image to obtain the fused image.

[0077] Optionally, the real scene image includes a left-view real scene image and a right-view real scene image captured by a binocular camera. When fusing the cloud desktop image and the real scene image to obtain a fusion image, the display module 802 is specifically used to perform binocular rendering on the cloud desktop image to obtain a left-view virtual image and a right-view virtual image; fuse the left-view real scene image and the left-view virtual image to obtain a left-view fusion image; and fuse the right-view real scene image and the right-view virtual image to obtain a right-view fusion image.

[0078] Optionally, the wearable device further includes a gaze detection means. The display module 802 further includes: detecting a user's gaze by the gaze detection module and obtaining a gaze direction; determining a gaze area of ​​the user in the virtual scene based on the gaze direction; and highlighting the real scene image when the gaze area is located in the real scene image.

[0079] In this embodiment, a real-world image of the real environment where the wearable device is located and a cloud desktop image provided by a cloud server can be acquired, and the real-world image is displayed in synchronization with the cloud desktop image in the virtual scene of the wearable device. In this manner, the real-world image is displayed in synchronization with the cloud desktop image in the virtual scene, allowing the user to recognize the real-world scene even after wearing the wearable device. Furthermore, the user does not need to remove the wearable device when using tools in the real world, thereby improving the user's sense of immersion in the virtual world.

[0080] 9 is a structural schematic diagram of a terminal device provided by a representative embodiment of this application. As shown in FIG. 9, the terminal device includes: a memory 901 and a processor 902.

[0081] The memory 901 is used to store computer programs and can also be configured to store various other data to support the operation of the terminal device, such as instructions for applications and methods running on the terminal device, contact data, phone book data, messages, photos, videos, etc.

[0082] Among them, memory 901 can be implemented by any type of volatile or non-volatile storage device, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk, or a combination thereof.

[0083] The processor 902 is connected to the memory 901 and is used to execute computer programs stored in the memory 901; to acquire real-world images of the real environment in which the wearable device is located and cloud desktop images provided by a cloud server; and to display the real-world images in synchronization with the cloud desktop images in the virtual scene of the wearable device.

[0084] Further optionally, when processor 902 acquires a real-world image of the real environment in which the wearable device is located and a cloud desktop image provided by a cloud server, it specifically acquires a timestamp of the cloud desktop image for any frame of the cloud desktop image in the cloud desktop video stream; this is used to select a frame image from the real-world video stream capturing the real environment in which the wearable device is located that has the same timestamp as the cloud desktop image as the real-world image.

[0085] Further optionally, the processor 902 is used to display the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device, specifically, to fuse the cloud desktop image with the real scene image to obtain a fused image; and to display the fused image in the virtual scene of the wearable device.

[0086] Further optionally, the processor 902 is used to fuse the cloud desktop image and the real scene image to obtain a fused image, specifically by overlaying the real scene image on the cloud desktop image to obtain the fused image.

[0087] Further optionally, the processor 902 is used to fuse the cloud desktop image and the real scene image to obtain a fused image, specifically by splicing (combining) the real scene image and the cloud desktop image to obtain the fused image.

[0088] Optionally, the real scene images include left-view real scene images and right-view real scene images captured by a binocular camera. When the processor 902 fuses the cloud desktop image and the real scene images to obtain a fused image, the processor 902 is specifically used for performing binocular rendering on the cloud desktop image to obtain a left-view virtual image and a right-view virtual image; fusing the left-view real scene image and the left-view virtual image to obtain a left-view fused image; and fusing the right-view real scene image and the right-view virtual image to obtain a right-view fused image.

[0089] Optionally, the wearable device further includes a gaze detection module, and the processor 902 is further configured to: detect the user's gaze using the gaze detection module and obtain a gaze direction; determine a gaze area of ​​the user in the virtual scene based on the gaze direction; and highlight the real-world image if the gaze area is located in the real-world image.

[0090] The memory shown in Figure 9 above can be implemented with any type of volatile or non-volatile storage device, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk, or a combination thereof.

[0091] The display 903 shown in FIG. 9 above includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). When the screen includes a touch panel, it is implemented as a touch screen and receives input signals from a user. The touch panel includes one or more touch sensors that detect touches, slides, and gestures on the touch panel. The touch sensors can not only detect the boundaries of a touch or slide motion, but also detect the duration and pressure associated with the touch or slide motion.

[0092] The audio component 904 shown in Figure 9 is configured to output and / or input audio signals. For example, the audio component may include a microphone (MIC) configured to receive external audio signals when the device in which the audio component is installed is in an operational mode such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component may also include a speaker for outputting audio signals.

[0093] 9, the terminal device further includes other components such as a communication component 905 and a power supply component 906. Note that FIG. 9 only schematically illustrates some of the components, and the terminal device does not necessarily include only the components illustrated in FIG.

[0094] The communication component 905 shown in FIG. 9 above is configured to enable wired or wireless communication between the device and other devices. The device on which the communication component is located may have access to a wireless network based on a communication standard such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0095] A power supply component 906 provides power to each component within the device and includes a power management system, one or more power supplies, and other power-related components within the device that generate, manage, and distribute electrical power.

[0096] In this embodiment, a real-world image of the real environment where the wearable device is located and a cloud desktop image provided by a cloud server can be acquired, and the real-world image can be displayed in synchronization with the cloud desktop image in the virtual scene of the wearable device. In this manner, the real-world image is displayed in synchronization with the cloud desktop image in the virtual scene, so that the user can still sense the real-world scene after wearing the wearable device, and the user does not need to remove the wearable device when using tools in the real world, thereby improving the user's sense of immersion in the virtual world.

[0097] Accordingly, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can realize each step executable by the terminal device in the above method embodiment.

[0098] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Also, the present invention can take the form of a computer program product embodied in one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0099] The present invention will be described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to generate an apparatus, and the instructions executed by the processor of the computer or other programmable data processing apparatus generate an apparatus for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0100] These computer program instructions can be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, and the instructions stored in the computer-readable memory can produce an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0101] These computer program instructions can be loaded into a computer or other programmable data processing apparatus and cause the computer or other programmable apparatus to perform a series of operational steps to generate a computer-implemented process, whereby the instructions executed by the computer or other programmable apparatus provide steps for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0102] In one typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0103] The memory may include non-persistent memory forms of a computer-readable medium such as read-only memory (ROM) or flash memory (flashRAM), random access memory (RAM), and / or non-volatile memory. The memory is one example of a computer-readable medium.

[0104] Computer-readable media include permanent and non-permanent, removable and non-removable media, and may implement any method or technology for information storage. Information may be computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disks (CD-ROMs), digital versatile disks (DVDs) or other optical storage devices, magnetic cassette tapes, etc. Magnetic disk storage devices or other magnetic storage devices or other non-transmission media may be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory storage computer-readable media such as modulated data signals and carrier waves.

[0105] It should be clarified that the terms "comprise" and "include" or any variation thereof are intended to be non-exclusive inclusive, whereby a process, method, article, or apparatus comprising a list of elements may include other elements not expressly listed, or may include elements inherent in such process, method, article, or apparatus. An element qualified by the phrase "comprises a..." does not exclude the inclusion of further identical elements in the process, method, article, or apparatus.

[0106] The above is only an embodiment of the present application and does not limit the present application. Those skilled in the art can make various changes and modifications to the present application. All modifications, equivalent replacements, improvements, etc. made based on the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. 1. A method for displaying a cloud desktop, comprising: For a cloud desktop image of any frame in the cloud desktop video stream, obtaining a timestamp of the cloud desktop image; selecting a frame image having the same timestamp as the cloud desktop image from a video stream of a real scene captured of a real environment where the wearable device is located as a real scene image; Using the real-world image and the cloud desktop image having the same timestamp, the user interacts with a specific item in the real-world image to perform real-time interaction with the application; A method for displaying a cloud desktop, characterized in that the real scene image is displayed in synchronization with the cloud desktop image in the virtual scene of the wearable device.

2. Displaying the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device includes: Fusing the cloud desktop image and the real scene image to obtain a fusion image; and displaying the fused image in a virtual scene on the wearable device.

3. Fusing the cloud desktop image and the real scene image to obtain a fusion image includes: The method of claim 2 , further comprising overlaying the real scene image onto the cloud desktop image to obtain the fused image.

4. Fusing the cloud desktop image and the real scene image to obtain a fusion image includes: The method of claim 2 , wherein the real scene image and the cloud desktop image are spliced ​​to obtain the fused image.

5. The real scene image includes obtaining a left-view real scene image and a right-view real scene image captured by a binocular camera, Fusing the cloud desktop image and the real scene image to obtain a fusion image includes: performing binocular rendering on the cloud desktop image to obtain a left-eye virtual image and a right-eye virtual image; The method according to any one of claims 2 to 4, characterized in that it includes fusing the left-view real scene image and the left-view virtual image to obtain a left-view fusion image, and fusing the right-view real scene image and the right-view virtual image to obtain a right-view fusion image.

6. The wearable device further comprises a gaze detection module, and the method further comprises: Detecting a user's gaze by the gaze detection module and acquiring a gaze direction; determining a gaze area of ​​the user in the virtual scene based on the gaze direction; The method according to any one of claims 1 to 4, further comprising highlighting the real scene image when the fixation area is located in an area where the real scene image is located.

7. A cloud desktop display device, comprising: an acquisition module for acquiring a timestamp of a cloud desktop image of any frame in the cloud desktop video stream, selecting a frame image having the same timestamp as the cloud desktop image from a video stream of a real scene captured of the real environment in which the wearable device is located as a real scene image, and using the real scene image and the cloud desktop image having the same timestamp to allow a user to interact with a specific item in the real scene image, thereby performing real-time interaction with an application; a display module for displaying the real scene image in synchronization with the cloud desktop image in a virtual scene of the wearable device; A cloud desktop display device comprising:

8. A terminal device including a memory and a processor, the memory is used to store one or more computer commands; A terminal device, characterized in that the processor executes the one or more computer commands and performs the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium storing a computer program, A computer-readable storage medium having stored thereon a computer program, the computer program causing the processor to perform the steps of the method according to any one of claims 1 to 4 when the computer program is executed by the processor.

Images