Cloud desktop display method, apparatus, device and storage medium

Abstract

The embodiments of the present application provide a method, apparatus, device, and storage medium for displaying a cloud desktop. In the cloud desktop display method, a real-world image in a real environment where a wearable device is located and a cloud desktop image provided by a cloud server can be obtained, and the real-world image can be displayed in synchronization with the cloud desktop image in a virtual scene of the wearable device. In this manner, the real-world image can be displayed in synchronization with the cloud desktop image in the virtual scene, so that the user can recognize the real-world scene even after wearing the wearable device, and the user does not need to remove the wearable device when using a tool in the real world. This improves the user's immersion in the virtual world.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application references Chinese patent application No. 202210329680.6, filed on March 31, 2022, entitled "Cloud Desktop Display Method, Apparatus, Device and Storage Medium," the entire contents of which are incorporated herein by reference.

[0002] The embodiments of the present application relate to the field of smart wearable technology, and in particular to a method, apparatus, device and storage medium for displaying a cloud desktop. [Background technology]

[0003] With the rapid development of related technologies such as virtual reality, augmented reality, and mixed reality, new head-mounted smart devices, such as smart glasses such as head-mounted virtual reality glasses and head-mounted mixed reality glasses, are being developed one after another, and the user experience is gradually improving.

[0004] In the prior art, smart glasses can be used to display cloud desktop videos sent by a cloud server, and a steering wheel or other controller paired with the smart glasses can be used to interact with the cloud desktop, allowing users to perform telework or leisure activities in a virtual world through the cloud desktop.

[0005] However, in this method, the user cannot sense the actual scene in the real world after wearing the smart glasses, so when the user wants to use a tool in the real world, the smart glasses need to be removed, which reduces the user's immersive feeling in the virtual world. Therefore, a solution needs to be proposed. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a cloud desktop display method, apparatus, device and storage medium, which allows a user to still sense the actual scene of the real world after wearing a wearable device, and further enhances the user's sense of immersion in the virtual world.

[0007] An embodiment of the present application provides a method for displaying a cloud desktop, which includes: acquiring a real scene image in a real environment where a wearable device is located and a cloud desktop image provided by a cloud server; and displaying the real scene image in synchronization with the cloud desktop image in a virtual scene of the wearable device.

[0008] Further optionally, acquiring a real-world image of the real environment in which the wearable device is located and a cloud desktop image provided by a cloud server includes acquiring a timestamp of the cloud desktop image for any frame of the cloud desktop image in a cloud desktop video stream; and selecting a frame image from the real-world video stream capturing the real-world environment in which the wearable device is located that has the same timestamp as the cloud desktop image as the real-world image.

[0009] Further optionally, displaying the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device includes fusing the cloud desktop image and the real scene image to obtain a fused image; and displaying the fused image in the virtual scene of the wearable device.

[0010] Further optionally, fusing the cloud desktop image and the real world image to obtain a fused image includes overlaying the real world image onto the cloud desktop image to obtain the fused image.

[0011] Further optionally, fusing the cloud desktop image and the real scene image to obtain a fused image includes splicing the real scene image and the cloud desktop image to obtain the fused image.

[0012] Further optionally, the real scene image includes a left-view real scene image and a right-view real scene image captured by a binocular camera, and fusing the cloud desktop image and the real scene image to obtain a fused image includes performing binocular rendering on the cloud desktop image to obtain a left-view virtual image and a right-view virtual image, fusing the left-view real scene image and the left-view virtual image to obtain a left-view fused image, and fusing the right-view real scene image and the right-view virtual image to obtain a right-view fused image.

[0013] Further optionally, the wearable device further includes a gaze detection module, and the method further includes detecting the user's gaze by the gaze detection module and obtaining a gaze direction; determining a gaze area of ​​the user in the virtual scene based on the gaze direction; and highlighting the real scene image if the gaze area is in an area where the real scene image is located.

[0014] An embodiment of the present application further provides a display device of a cloud desktop, which includes an acquisition module for acquiring a real scene image in a real environment in which a wearable device is located and a cloud desktop image provided by a cloud server, and a display module for displaying the real scene image in synchronization with the cloud desktop image in a virtual scene of the wearable device.

[0015] An embodiment of the present application further provides a terminal device including a memory and a processor, wherein the memory is used for storing one or more computer commands, and the processor is used for executing the one or more computer commands to perform steps in a cloud desktop display method.

[0016] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform steps in a method for displaying a cloud desktop.

[0017] The cloud desktop display method, apparatus, device and storage medium provided by the embodiments of the present application can obtain a real scene image in the real environment where the wearable device is located and a cloud desktop image provided by a cloud server, and display the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device. In this manner, the real scene image is displayed in synchronization with the cloud desktop image in the virtual scene, so that the user can still sense the real scene of the real world after wearing the wearable device, and does not need to take off the wearable device when the user wants to use a tool in the real world, thereby improving the user's immersion in the virtual world. [Brief description of the drawings]

[0018] In order to more clearly describe the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description only show some embodiments of the present invention, and those skilled in the art can further obtain other drawings based on these drawings without creative labor.

[0019] [Figure 1] 1 is a flowchart of a cloud desktop display method provided by an exemplary embodiment of the present application; [Diagram 2] FIG. 1 is a schematic diagram of an overlay provided by an exemplary embodiment of the present application; [Diagram 3] 1 is a schematic diagram of splicing provided by a representative embodiment of the present application; [Figure 4] FIG. 1 is a schematic diagram of binocular rendering provided by an exemplary embodiment of the present application; [Diagram 5]FIG. 1 is a schematic diagram of binocular rendering corrections provided by an exemplary embodiment of the present application; [Figure 6] FIG. 1 is an architecture diagram of a mobile terminal provided by a representative embodiment of the present application; [Figure 7] FIG. 1 is a schematic diagram of a display terminal provided by an exemplary embodiment of the present application; [Figure 8] FIG. 1 is a schematic diagram of a cloud desktop display device provided by an exemplary embodiment of the present application; [Figure 9] 1 is a schematic diagram of a terminal device provided by a representative embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention, obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative labor are all within the protection scope of the present invention.

[0021] In the prior art, smart glasses can be used to display cloud desktop videos sent by a cloud server, and a steering wheel or other controller paired with the smart glasses can be used to interact with the cloud desktop, allowing users to telework or perform leisure activities in the virtual world through the cloud desktop. However, in this manner, users still cannot sense the actual scene in the real world after wearing the smart glasses, so when users want to use tools in the real world, they need to remove the smart glasses, which reduces the user's immersive feeling in the virtual world.

[0022] In order to solve the above technical problems, some embodiments of the present application provide solutions. The following describes in detail the technical solutions provided by each embodiment of the present application with reference to the drawings.

[0023] FIG. 1 is a flowchart of a cloud desktop display method provided in a representative embodiment of the present application. As shown in FIG. 1, the method includes: Step 11 of acquiring a real scene image in a real environment where the wearable device is located and a cloud desktop image provided by a cloud server; and step 12 of displaying the real scene image in the virtual scene of the wearable device in synchronization with the cloud desktop image.

[0024] The present embodiment may be executed by a terminal device, a wearable device, or a cloud server. The terminal device may include a computer, a tablet computer, a mobile phone, etc. The wearable device may include VR (Virtual Reality) glasses, MR (Mixed Reality) glasses, a VR head-mounted display (Head-Mounted Display, HMD), etc.

[0025] The real environment is an environment in the real world, and the real-scene image is used to reflect the real environment in which the wearable device is located. In this embodiment, a camera attached to the real environment can capture the real environment in which the wearable device is located to obtain a real-scene video stream. The real-scene image is any frame image in the real-scene video stream. The camera for capturing the real environment can be attached to the wearable device, and the mounting position should ensure that the viewing range of the camera is located within the viewing range of the human eye, so that when the line of sight of the human eye is blocked by the wearable device, the camera can replace the human eye to observe the real scene.

[0026] Among them, cloud desktop, also called desktop virtualization or cloud computer, can use virtual technology to perform virtualization processing on various physical devices, thereby effectively improving resource utilization, saving costs and improving application quality. After deploying the virtual desktop on the cloud platform in the cloud server, users can access the virtual desktop and applications anywhere. The cloud server can send the cloud desktop video stream to the wearable device for display. Among them, the cloud desktop image refers to any frame image in the cloud desktop video stream.

[0027] After acquiring the real scene image and the cloud desktop image, the real scene image can be displayed in synchronization with the cloud desktop image in the virtual scene of the wearable device.

[0028] When this embodiment is executed by a terminal device, a tablet computer is taken as an example, and the tablet computer can obtain the real scene image sent from the wearable device and the cloud desktop image sent from the cloud server, and send the real scene image and the cloud desktop image to the wearable device for synchronous display.

[0029] When this embodiment is implemented by a wearable device, the VR glasses are taken as an example, and the VR glasses can collect real scene images, obtain cloud desktop images sent from the cloud server, and synchronously display the real scene images and the cloud desktop images; or the terminal device can receive the cloud desktop images sent from the cloud server and transmit them to the VR glasses; the VR glasses can obtain the cloud desktop images sent from the terminal device, collect real scene images, and synchronously display the real scene images and the cloud desktop images.

[0030] When this embodiment is executed by a cloud server, the cloud server can obtain the real scene image sent from the wearable device, and send the real scene image and the cloud desktop image to the wearable device for synchronous display.

[0031] Among them, displaying the real scene image synchronously with the cloud desktop image means displaying the real scene image at the same time as displaying the cloud desktop image in the virtual scene of the wearable device, that is, the user can view the virtual cloud desktop image in the virtual scene and also view the actual real scene image.

[0032] Such an embodiment allows real-world images to be displayed in the virtual scene in synchronization with the cloud desktop image, allowing the user to still sense the real-world scene after putting on the wearable device, and eliminating the need for the user to remove the wearable device when attempting to use a real-world tool, thereby enhancing the user's sense of immersion in the virtual world.

[0033] At the same time, when a camera for collecting real-world images is attached to a wearable device, interaction with the cloud desktop can be achieved without the need to install a handle or other controller that is paired with an external wearable device, which reduces hardware costs while also being advantageous for better integration and weight reduction of the wearable device.

[0034] In the following embodiment, a wearable device will be described as an example of an execution subject.

[0035] In some alternative embodiments, the wearable device can obtain the cloud desktop video stream and the real-world video stream displaying the scene in real time sent from the cloud server, and perform frame-by-frame synchronous display of the cloud desktop image in the cloud desktop video stream and the real-world image in the real-world video stream. An example is given below.

[0036] For any frame of the cloud desktop image in the acquired cloud desktop video stream, the wearable device can acquire the timestamp of the cloud desktop image. For example, the timestamp of the cloud desktop image P1 is 20:59:01. Furthermore, the wearable device can select a frame image with the same timestamp as the cloud desktop image from a video stream of a real scene taken in the real environment where the wearable device is located, and use it as a real scene image. For example, the wearable device can select an image P1' with a timestamp of 20:59:01 from the video stream of the real scene, and use it as a real scene image. Thereby, the wearable device can synchronously display different images with the same timestamp. In some possible actual application scenarios, some specific applications can be executed on the cloud desktop, and since the timestamps of the acquired cloud desktop image and the real scene image are the same, the user can perform real-time interaction with the application by interacting with a specific item in the real scene image. For example, the user can synchronously perform a corresponding operation (such as turning off the application, turning on the application, or restarting the application) in the application by touching a charger in the real scene image. If the application being executed is a game, the user can click on a cup, toothbrush, etc. in the real-world image to synchronously cause the virtual character in the game to perform the corresponding operation (drinking water, brushing teeth, etc.).

[0037] Such an embodiment allows the timestamps of the acquired cloud desktop image and the real scene image to be the same, further improving the synchronization effect between the cloud desktop image and the real scene image, and reducing the sense of isolation when the cloud desktop image is displayed synchronously with the real scene image.

[0038] Optionally, the operation of step 12 of the above embodiment “displaying a real scene image in synchronization with a cloud desktop image in a virtual scene on a wearable device” can be: Step 121: fusing the cloud desktop image and the real scene image to obtain a fusion image; This can be achieved based on step 122 of displaying the fused image in the virtual scene on the wearable device.

[0039] This method allows users to simultaneously observe relevant information on the cloud desktop and relevant information in the real world from the fused image, does not require users to remove the wearable device when they want to use real-world tools, and can more easily interact with real-world objects when using the wearable device.

[0040] Optionally, the wearable device can: Embodiment 1 in which a real scene image is superimposed on a cloud desktop image to obtain a fusion image; The cloud desktop image and the real scene image can be fused according to embodiment 2, in which the real scene image and the cloud desktop image are stitched to obtain a fusion image.

[0041] In the first embodiment, as shown in part A of Fig. 2, the entire real scene image and the entire cloud desktop image can be superimposed. As shown in part B of Fig. 2, a part of the real scene image is cut out from the real scene image and superimposed in a specified area of ​​the cloud desktop image. Or, the real scene image is not cut out, but is reduced and superimposed in a part of the cloud desktop image. Optionally, the user can perform operations such as enlarging, reducing or cropping on the real scene image by using an entity key on the wearable device or a virtual key in the virtual scene, and can adjust the area where the real scene image is superimposed on the cloud desktop image.

[0042] In embodiment 2, the real scene image and the cloud desktop image can be stitched together, as shown in Fig. 3. Optionally, the user can use the entity keys on the wearable device or the virtual keys in the virtual scene to perform operations such as zooming in, zooming out, or cropping on the real scene image, and adjust the positions of the real scene image and the cloud desktop image.

[0043] It is to be mentioned that the user can switch between the above two embodiments of overlay and splicing by an entity key on the wearable device or a virtual key in the virtual scene.

[0044] According to the above embodiment, the cloud desktop image and the real scene image are fused by overlaying or splicing, and the user can freely adjust the fusion method of the cloud desktop image and the real scene image, and can more completely observe the cloud desktop image and the real scene image simultaneously.

[0045] Optionally, in a real scene, the wearable device is usually equipped with two cameras, one on the left and one on the right, which is called binocular camera for short. The wearable device can collect real scene images based on the binocular camera. The real scene images include a left-view real scene image and a right-view real scene image captured by the binocular camera. The left-view real scene image is a real scene image collected by the left camera of the binocular camera, and the left-view real scene image corresponds to the left eye of the user; the right-view real scene image is a real scene image collected by the right camera of the binocular camera, and the right-view real scene image corresponds to the right eye of the user.

[0046] Based on this, the wearable device can fuse the cloud desktop image and the real scene image to obtain the fusion image according to the following steps: Step S1 performs binocular rendering on the cloud desktop image to obtain a left-eye virtual image and a right-eye virtual image, where the left-eye virtual image is a virtual image corresponding to a user's left eye, and the right-eye virtual image is a virtual image corresponding to a user's right eye. Step S2: fusing the left-eye real scene image and the left-eye virtual image to obtain a left-eye fusion image; and fusing the right-eye real scene image and the right-eye virtual image to obtain a right-eye fusion image. Based on this step, when the user's left eye and right eye respectively view the left-eye fusion image and the right-eye fusion image, the user's brain can automatically synthesize the images viewed by the left and right eyes into a 3D image.

[0047] The binocular rendering in step S1 will be described in detail below with reference to FIG.

[0048] As shown in FIG. 4, R is the display distance required by the glasses, FOV (Field of View), and w is the distance between the user's eyes.

[0049] D is the pixel width output to the left and right screens, i.e., the maximum width of the screen that the user can see with one eye, and can be calculated based on the following formula 1: Number 1 JPEG2025510949000002.jpg29166

[0050] The distance from a pixel point in the cloud desktop image to the central axis is S, and the coordinates of the point are (S, R).

[0051] Based on the above process, the ratio of the x-coordinate value of this pixel point in the left-eye virtual image (represented by B1) and the ratio of the x-coordinate value of this pixel point in the right-eye virtual image (represented by B2) can be calculated using the following Equations 2 and 3, respectively. Number 2 JPEG2025510949000003.jpg21132Number3 JPEG2025510949000004.jpg22131

[0052] After calculating the ratio of the x-coordinate value of this pixel point in the left-eye virtual image and the right-eye virtual image, the final x-coordinate value can be obtained by multiplying it by the actual pixel width a of the screen, and the y-coordinate value does not change.

[0053] This binocular rendering method can obtain left-eye virtual images and right-eye virtual images corresponding to the user's left eye and right eye respectively, and further allows the user to view the cloud desktop more realistically using the wearable device, thereby improving the sense of immersion in the user's interaction with the cloud desktop.

[0054] After binocular rendering is performed on the cloud desktop image, further image fusion can be performed. However, as shown in Figure 5, in the above-mentioned binocular rendering, the distance between the binocular cameras is usually different from the distance between the user's eyes, so after binocular rendering, further correction can be performed according to the following formula:

[0055] As shown in Figure 5, d is the deviation distance between the camera and the eye, which can be calculated based on the following formula: Number 4 JPEG2025510949000005.jpg18169Here, e is the distance between the eyes and w' is the distance between the cameras.

[0056] Based on this, the offset amount rx1 of the left side x coordinate in the right screen of the camera and the offset amount rx2 of the right side x coordinate in the right screen of the camera can be calculated by the following formulas 5 and 6: Number 5 JPEG2025510949000006.jpg20169Number6 JPEG2025510949000007.jpg16165, where FOV d is the camera's viewing angle, FOV e is the visual angle of the eye and R is the viewing distance required by the glasses.

[0057] By calculating the above offset amount, the left-eye virtual image and the right-eye virtual image obtained after binocular rendering can be further corrected, and the image quality of the fusion image obtained by subsequent fusion can be improved based on the relatively accurate virtual images.

[0058] In some embodiments, the wearable device can be equipped with a gaze detection module. The wearable device can perform gaze detection on the user through the gaze detection module to obtain the user's gaze direction. Furthermore, the wearable device can identify the user's gaze area in the virtual scene based on the gaze direction.

[0059] If the gaze area is located in an area where the real scene image is located, the real scene image is highlighted; if the gaze area is located in an area where the cloud desktop image is located, the cloud desktop image is highlighted.

[0060] It should be explained that when a user gazes at a cloud desktop image for a long time, the real scene image can be hidden and the cloud desktop image can be displayed in full screen; when a user gazes at a real scene image for a long time, the cloud desktop image can be hidden and the real scene image can be displayed in full screen. Optionally, a virtual key / area is preset in the virtual scene of the wearable device, and when a user gazes at the key / area, a corresponding function can be executed, such as displaying the cloud desktop image in full screen, or displaying the real scene image in full screen, or displaying the cloud desktop image and the real scene image according to a preset layout style, etc.

[0061] The following further explains the method for displaying a cloud desktop with reference to Figures 6 and 7 and actual application scenarios.

[0062] FIG. 6 is an architecture diagram of a video stream processing method of a mobile terminal (i.e., the above-mentioned terminal device), and FIG. 7 is an architecture diagram of a video stream processing method of a display terminal (i.e., the above-mentioned wearable device).

[0063] In a practical scenario, a specific application program (hereinafter referred to as Mapp) can be installed on a mobile terminal, and the application program can obtain authorized account information. After a user inputs an account password, the application program can call a cloud desktop API (Application Programming Interface) to perform authorization authentication. After the account password input by the user is authenticated, Mapp can obtain the IP (Internet Protocol) address and port number of the cloud desktop virtual machine corresponding to the user's account.

[0064] Mapp can attempt to establish a connection with the cloud desktop virtual machine through wireless transmission based on a remote desktop connection protocol. For illustrative purposes, the remote desktop connection protocol includes, but is not limited to, the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol, the NetBios Enhanced User Interface (NetBEUI) protocol, the Internetwork Packet Exchange / Sequences Packet Exchange (IPX / SPX) protocol, the Remote Display Protocol (RDP) protocol, etc., and the present embodiment is not limited thereto. Among them, the RDP protocol is a remote desktop protocol established on the TCP / IP protocol.

[0065] After Mapp successfully connects to the cloud desktop virtual machine, data communication between the two can begin based on the ISO layer remote desktop connection protocol. When a display terminal is not connected to Mapp, Mapp operates as a normal cloud desktop client; when it detects that a display terminal is connected to Mapp, Mapp does not directly display the cloud desktop interface, but performs image processing on the cloud desktop video stream returned from the cloud desktop virtual machine and sends it to the display terminal for display.

[0066] After the above-mentioned remote desktop connection protocol is successfully established, the Mapp program can identify the image channel from the established virtual channels, among which the image channel has a special identifier, and then Mapp can obtain the image data sent from the image channel to obtain the original bitmap stream data of the cloud desktop.

[0067] Furthermore, Mapp can obtain the resolution of the bitmap, and perform compression optimization on the obtained original bitmap stream data based on parameters such as the resolution and frame rate parameters of the display terminal, so as to improve the efficiency of subsequent image processing.

[0068] Mapp can determine the input parameters of binocular rendering based on other parameters of the display terminal (such as interpupillary distance (offset), field of view (FOV), rendering screen width / height, depth far / depth near of the viewing frustum, lens focal length (Convergence) or anti-distortion coefficient (Anti-distortion), etc.) and can generate a left-eye virtual image and a right-eye virtual image on the cloud desktop image of each frame through image calculation based on the input parameters.

[0069] Mapp can acquire left-eye real scene images and right-eye real scene images captured by a binocular camera on a display terminal, and fuses the left-eye real scene image with the left-eye virtual image to obtain a left-eye fused image, and fuses the right-eye real scene image with the right-eye virtual image to obtain a right-eye fused image.

[0070] It should be noted that Mapp can establish a transmission channel with a specific application program (hereinafter referred to as Vapp) running on a display terminal through a wired connection (type-c or lightning). The transmission channel can be further packaged and then send the left-eye fusion image and the right-eye fusion image to the Vapp of the display terminal through the corresponding channels. After identifying the fusion image, the Vapp of the display terminal outputs and displays the fusion image on the two left and right screens of the display terminal, respectively.

[0071] Such an embodiment allows a real-world image to be displayed in the virtual scene in synchronization with a cloud desktop image, allowing the user to still sense the real-world scene after putting on the wearable device, and eliminating the need for the user to remove the wearable device when they want to use a real-world tool, thereby enhancing the user's immersion in the virtual world.

[0072] FIG. 8 is a schematic diagram of a cloud desktop display device provided in a representative embodiment of the present application. As shown in FIG. 8, the display device includes: an acquisition module 801 for acquiring a real scene image in a real environment where a wearable device is located and a cloud desktop image provided by a cloud server; and a display module 802 for displaying the real scene image in synchronization with the cloud desktop image in a virtual scene of the wearable device.

[0073] Further optionally, when the acquisition module 801 acquires a real-world image of the real environment in which the wearable device is located and a cloud desktop image provided by a cloud server, it is specifically used to acquire a timestamp of the cloud desktop image for any frame of the cloud desktop image in the cloud desktop video stream; and to select a frame image from the real-world video stream capturing the real-world environment in which the wearable device is located that has the same timestamp as the cloud desktop image as the real-world image.

[0074] Further optionally, when the display module 802 displays the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device, specifically, the display module 802 fuses the cloud desktop image and the real scene image to obtain a fused image; the display module 802 is used to display the fused image in the virtual scene of the wearable device.

[0075] Further optionally, when the display module 802 fuses the cloud desktop image and the real scene image to obtain a fusion image, the display module 802 is specifically used for overlaying the real scene image on the cloud desktop image to obtain the fusion image.

[0076] Further optionally, when the display module 802 fuses the cloud desktop image and the real scene image to obtain a fusion image, the display module 802 is specifically used for splicing (combining) the real scene image and the cloud desktop image to obtain the fusion image.

[0077] Optionally, the real scene image includes a left-view real scene image and a right-view real scene image captured by a binocular camera. When the display module 802 fuses the cloud desktop image and the real scene image to obtain a fusion image, the display module 802 is specifically used for performing binocular rendering on the cloud desktop image to obtain a left-view virtual image and a right-view virtual image; fusing the left-view real scene image and the left-view virtual image to obtain a left-view fusion image; and fusing the right-view real scene image and the right-view virtual image to obtain a right-view fusion image.

[0078] Optionally, the wearable device further includes a gaze detection means. The display module 802 further includes: detecting a user's gaze by the gaze detection module and obtaining a gaze direction; determining a gaze area of ​​the user in the virtual scene according to the gaze direction; and highlighting the real scene image when the gaze area is in the area where the real scene image is located.

[0079] In this embodiment, a real-world image of the real environment where the wearable device is located and a cloud desktop image provided by a cloud server can be obtained, and the real-world image is displayed in synchronization with the cloud desktop image in the virtual scene of the wearable device. In this manner, the real-world image is displayed in synchronization with the cloud desktop image in the virtual scene, so that the user can recognize the real-world scene even after wearing the wearable device, and the user does not need to remove the wearable device when using tools in the real world, thereby improving the user's immersion in the virtual world.

[0080] 9 is a structural schematic diagram of a terminal device provided by a representative embodiment of this application. As shown in FIG. 9, the terminal device includes: a memory 901 and a processor 902.

[0081] The memory 901 is used to store computer programs and can also be configured to store various other data to support the operation of the terminal device, such as applications and method instructions running on the terminal device, contact data, phone book data, messages, photos, videos, etc.

[0082] Among them, the memory 901 can be realized by any type of volatile or non-volatile storage device, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk, or a combination thereof.

[0083] The processor 902 is connected to the memory 901 and is used for executing computer programs stored in the memory 901; and for acquiring real-world images of the real environment in which the wearable device is located and a cloud desktop image provided by a cloud server; and for displaying the real-world images in synchronization with the cloud desktop image in the virtual scene of the wearable device.

[0084] Further optionally, when processor 902 acquires a real-world image of the real environment in which the wearable device is located and a cloud desktop image provided by a cloud server, it specifically acquires a timestamp of the cloud desktop image for any frame of the cloud desktop image in the cloud desktop video stream; this is used to select a frame image from the real-world video stream capturing the real-world environment in which the wearable device is located that has the same timestamp as the cloud desktop image as the real-world image.

[0085] Further optionally, the processor 902 is used when displaying the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device, specifically, to fuse the cloud desktop image and the real scene image to obtain a fused image; and to display the fused image in the virtual scene of the wearable device.

[0086] Further optionally, the processor 902 is used to fuse the cloud desktop image and the real world image to obtain a fused image, specifically by overlaying the real world image on the cloud desktop image to obtain the fused image.

[0087] Further optionally, the processor 902 is used to fuse the cloud desktop image and the real scene image to obtain a fused image, specifically by splicing (combining) the real scene image and the cloud desktop image to obtain the fused image.

[0088] Optionally, the real scene image includes a left-view real scene image and a right-view real scene image captured by a binocular camera. When the processor 902 fuses the cloud desktop image and the real scene image to obtain a fusion image, the processor 902 is specifically used for performing binocular rendering on the cloud desktop image to obtain a left-view virtual image and a right-view virtual image; fusing the left-view real scene image and the left-view virtual image to obtain a left-view fusion image; and fusing the right-view real scene image and the right-view virtual image to obtain a right-view fusion image.

[0089] Optionally, the wearable device further includes a gaze detection means. The processor 902 is further used for: detecting the gaze of the user by the gaze detection module and obtaining a gaze direction; determining a gaze area of ​​the user in the virtual scene based on the gaze direction; and highlighting the real scene image when the gaze area is in the area where the real scene image is located.

[0090] The memory shown in FIG. 9 above may be implemented with any type of volatile or non-volatile storage device, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic or optical disk, or any combination thereof.

[0091] The display 903 shown in FIG. 9 above includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it is realized as a touch screen and receives input signals from a user. The touch panel includes one or more touch sensors to detect touches, slides, and gestures on the touch panel. The touch sensors can not only detect the boundaries of a touch or slide motion, but also detect the duration and pressure associated with the touch or slide motion.

[0092] The audio component 904 shown in Fig. 9 is configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device in which the audio component is installed is in an operation mode such as a call mode, a record mode, and a voice recognition mode. The received audio signals can be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

[0093] In addition, as shown in Fig. 9, the terminal device further includes other components, such as a communication component 905 and a power supply component 906. Note that Fig. 9 shows only a schematic representation of some of the components, and the terminal device does not include only the components shown in Fig. 9.

[0094] The communication component 905 shown in FIG. 9 above is configured to enable wired or wireless communication between the device and other devices. The device in which the communication component is located can access a wireless network based on a communication standard such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0095] A power supply component 906 provides power to each component in the device and includes a power management system, one or more power supplies, and other power-related components in the device that generate, manage, and distribute electrical power.

[0096] In this embodiment, the real scene image in the real environment where the wearable device is located and the cloud desktop image provided by the cloud server can be obtained, and the real scene image can be displayed in synchronization with the cloud desktop image in the virtual scene of the wearable device. In this manner, the real scene image is displayed in synchronization with the cloud desktop image in the virtual scene, so that the user can still sense the real scene of the real world after wearing the wearable device, and the user does not need to take off the wearable device when using tools in the real world, thereby improving the user's immersive feeling in the virtual world.

[0097] Accordingly, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when the computer program is executed, can realize each step executable by a terminal device in the above method embodiment.

[0098] It should be understood by those skilled in the art that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. The present invention can also take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing one or more computer usable program codes.

[0099] The present invention will be described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to generate an apparatus, and the instructions executed by the processor of the computer or other programmable data processing apparatus generate an apparatus for implementing a function specified in a flow or flows of the flowcharts and / or a block or blocks of the block diagrams.

[0100] These computer program instructions can be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, and the instructions stored in the computer-readable memory generate an article of manufacture that includes an instruction apparatus, which implements the functions specified in a flow or flows of the flowcharts and / or a block or blocks of the block diagrams.

[0101] These computer program instructions can be loaded into a computer or other programmable data processing apparatus and cause the computer or other programmable apparatus to perform a series of operational steps to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing a function specified in a flow or flows in the flowcharts and / or a block or blocks in the block diagrams.

[0102] In one typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0103] The memory may include non-persistent memory forms of computer readable media such as read only memory (ROM) or flash memory (flashRAM), random access memory (RAM) and / or non-volatile memory. The memory is one example of a computer readable medium.

[0104] Computer-readable media includes permanent and non-permanent, removable and non-removable media, and may be implemented by any method or technology for storing information. Information may be computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only optical disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette tape, and the like. Magnetic disk storage devices or other magnetic storage devices or other non-transmission media may be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory storage computer-readable media, such as modulated data signals and carrier waves.

[0105] It should be clarified that the terms "comprise" and "include" or any variation thereof are intended to be non-exclusive inclusive, whereby a process, method, article, or apparatus comprising a list of elements may include other elements not expressly recited, or may include elements inherent to such process, method, article, or apparatus. An element qualified by the phrase "comprises a" does not exclude the inclusion of further identical elements in the process, method, article, or apparatus.

[0106] The above is only an embodiment of the present application, and does not limit the present application. Those skilled in the art can make various changes and modifications to the present application. All modifications, equivalent replacements, improvements, etc. made based on the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. 1. A method for displaying a cloud desktop, comprising: Acquiring a real-world image of a real environment in which the wearable device is located and a cloud desktop image provided by a cloud server; A method for displaying a cloud desktop, comprising: displaying the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device.

2. Acquiring a real scene image in a real environment where the wearable device is located and a cloud desktop image provided by a cloud server For a cloud desktop image of any frame in the cloud desktop video stream, obtaining a timestamp of the cloud desktop image; The method of claim 1 , further comprising: selecting a frame image having the same timestamp as the cloud desktop image from a video stream of a real-world scene captured of the real-world environment in which the wearable device is located as the real-world scene image.

3. Displaying the real scene image in synchronization with the cloud desktop image in the virtual scene of the wearable device includes: Fusing the cloud desktop image and the real scene image to obtain a fusion image; and displaying the fused image in a virtual scene on the wearable device.

4. Fusing the cloud desktop image and the real scene image to obtain a fusion image, The method of claim 3 , further comprising: overlaying the real scene image onto the cloud desktop image to obtain the fused image.

5. Fusing the cloud desktop image and the real scene image to obtain a fusion image, The method of claim 3 , wherein the real scene image and the cloud desktop image are spliced ​​to obtain the fusion image.

6. The real scene image includes obtaining a left-view real scene image and a right-view real scene image captured by a binocular camera, Fusing the cloud desktop image and the real scene image to obtain a fusion image, performing binocular rendering on the cloud desktop image to obtain a left-eye virtual image and a right-eye virtual image; The method according to any one of claims 3 to 5, comprising fusing the left-view real scene image and the left-view virtual image to obtain a left-view fusion image, and fusing the right-view real scene image and the right-view virtual image to obtain a right-view fusion image.

7. The wearable device further comprises a gaze detection module, and the method further comprises: Detecting a user's gaze by the gaze detection module and acquiring a gaze direction; determining a gaze area of ​​the user in a virtual scene based on the gaze direction; The method according to any one of claims 1 to 5, further comprising highlighting the real scene image when the fixation area is located in an area in which the real scene image is located.

8. 1. A cloud desktop display device, comprising: An acquisition module for acquiring a real scene image in a real environment where the wearable device is located and a cloud desktop image provided by a cloud server; A display module for displaying the real scene image in synchronization with the cloud desktop image in a virtual scene of the wearable device; A cloud desktop display device comprising:

9. A terminal device including a memory and a processor, the memory is used to store one or more computer commands; A terminal device, characterized in that said processor executes said one or more computer commands and performs the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, A computer-readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 7.

Images