Display method of 3D application and 2D application and head-mounted display device

Abstract

The application relates to the technical field of three-dimensional display, and provides a display method of 3D application and 2D application and a head-mounted display device, which are used for realizing simultaneous display of 2D application and 3D application. Based on the support of the head-mounted display device to multiple types of screens, a virtual screen invisible to a user is created, and an image of the 2D application is rendered onto the virtual screen. In this way, the virtual screen serves as an intermediate medium for displaying the image of the 2D application, an Overlay service of the 2D application draws an image of a cover layer from the image of the 2D application on the virtual screen, the image of the cover layer is superimposed on a three-dimensional interface for binocular display, and thus the simultaneous display of the 2D application and the 3D application is realized. The user can access and operate the 2D application while enjoying the immersive experience, and the practicability and compatibility of the head-mounted display device are enhanced.

Description

Technical Field

[0001] This application relates to the field of three-dimensional display technology, and provides a display method for 3D applications and 2D applications, as well as a head-mounted display device. Background Technology

[0002] With the widespread adoption of head-mounted displays (HMDs), the demand for multitasking within these devices is increasing. One such demand is the simultaneous display of 2D applications within 3D applications. However, most current HMDs cannot display 3D and 2D applications simultaneously. To display a 2D application, the 3D experience must be exited and the 3D application placed in the background, preventing interaction between the two and thus failing to meet the demands of multitasking.

[0003] Therefore, achieving the joint display of 3D and 2D applications is of significant research value for head-mounted display devices. Summary of the Invention

[0004] This application provides a method for displaying 3D and 2D applications and a head-mounted display device for enabling the joint display of 3D and 2D applications.

[0005] In a first aspect, embodiments of this application provide a display method for 3D and 2D applications, applied to a head-mounted display device, comprising:

[0006] Control the display screen to show the three-dimensional interface of 3D applications;

[0007] Receive input commands to launch 2D applications;

[0008] In response to the operation command, the Overlay service is started and a virtual screen is created, and the image of the 2D application is drawn onto the virtual screen; the virtual screen is not visible to the user.

[0009] The Overlay service is controlled to acquire an image of the 2D application from the virtual screen to obtain an overlay image;

[0010] The overlay image is superimposed on the three-dimensional interface to obtain a target binocular image, which is then displayed on the display screen.

[0011] The beneficial effects of the above technical solution are as follows: Based on the head-mounted display device's support for multiple screen types, a virtual screen invisible to the user is created, and the images of 2D applications are rendered onto the virtual screen. In this way, the virtual screen can serve as an intermediate medium for displaying 2D application images. The Overlay service can obtain 2D application images from the virtual screen. Since the virtual screen is invisible and different from the display screen of the head-mounted display device, the intermediate display process of 2D applications on the virtual screen will not affect the user's immersive experience. On the other hand, based on the client-server architecture of the head-mounted display device, the Overlay service and the 3D application act as two clients of the head-mounted display device, while the image synthesizer acts as the server. Through ICP communication, it obtains the overlay image and the 3D interface from the Overlay service and the 3D application respectively, and overlays the overlay image onto the 3D interface for binocular display, thereby achieving simultaneous display of 2D and 3D applications. This allows users to enjoy an immersive experience while also accessing and operating 2D applications, enhancing the practicality and compatibility of the head-mounted display device.

[0012] Optionally, controlling the Overlay service to obtain an image of the 2D application from the virtual screen to obtain an overlay image includes:

[0013] Initialize the graphics rendering interface and create a second texture object for the graphics rendering interface;

[0014] The Overlay service is controlled to start a rendering loop, and the following operations are performed during each rendering:

[0015] Capture the current image frame of the 2D application on the virtual screen;

[0016] When rendering the current image frame, the pose information of the head-mounted display device relative to the reference space is obtained; the pose information is used for the overlay display of the overlay image;

[0017] The second texture object is rendered according to the current image frame, and the second texture object is rendered onto the exchange chain component according to the binding relationship between the second texture object and the exchange chain component.

[0018] Submit the exchange chain component and the pose information to obtain the overlay image of the current frame.

[0019] The beneficial effects of the above technical solution are as follows: by rendering the image of the 2D application on the virtual screen to the exchange chain component of the Overlay service, the image of the 2D application is located on the top layer of the display screen, realizing the fusion display with the three-dimensional interface. Furthermore, by acquiring and submitting the pose information of the head-mounted display device and the reference space, it can be ensured that the overlay image can be presented to the user in the appropriate position.

[0020] Optionally, starting the Overlay service includes:

[0021] Create and initialize an interface instance that provides operations for the Overlay service;

[0022] Based on the interface instance, obtain the configuration parameters of the operating system of the head-mounted display device;

[0023] Based on the configuration parameters, configure the Overlay service to adapt to the operating system of the head-mounted display device; create a session on the operating system of the head-mounted display device to manage the lifecycle of the Overlay service.

[0024] The beneficial effects of the above technical solution are as follows: By configuring the Overlay service through the configuration parameters of the operating system of the head-mounted display device, the Overlay service is adapted to the operating system of the head-mounted display device, thereby ensuring the normal display of 2D application images.

[0025] Optionally, starting the Overlay service further includes:

[0026] A reference space for the Overlay service is created, and the overlay properties of the reference space are set according to the display parameters of the 2D application and the 3D scene. The overlay properties are used to determine the display state of the Overlay service in the 3D scene.

[0027] The beneficial effects of the above technical solution are as follows: Since the overlay attribute is used to define the display state of the overlay service in the 3D scene, and the overlay attribute is set according to the display parameters of the 2D application and the 3D scene, it can ensure that the overlay image generated by the subsequent overlay service is properly placed in the 3D scene, reduce interference with the display of 3D applications, and ensure the user's immersive experience.

[0028] Optionally, the step of overlaying the overlay image onto the three-dimensional interface to obtain a target binocular image and displaying it on the display screen includes:

[0029] Obtain the spatial pose information of the head-mounted display device relative to the reference space;

[0030] Based on the pose information, the overlay image is superimposed on the three-dimensional interface to obtain a fused image;

[0031] A left-eye image and a right-eye image are generated based on the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

[0032] The beneficial effects of the above technical solution are: displaying the superimposed left-eye and right-eye images on the display screen to provide correct parallax and depth perception, thereby achieving a three-dimensional display effect for 2D applications.

[0033] Optionally, creating a virtual screen includes:

[0034] Create a media mapping object for capturing images of the 2D application on the virtual screen;

[0035] Create a first surface texture object, which is used to receive the image stream of the 2D application captured by the media mapping instance;

[0036] Based on the first surface texture object, a Surface object is created, which represents a drawn two-dimensional plane and serves as a bridge connecting the image stream and the rendering system.

[0037] Based on the media mapping object and the Surface object, the corresponding API is called to create a virtual screen.

[0038] The beneficial effects of the above technical solution are as follows: Since head-mounted display devices can support both display screens and virtual screens at the same time, and display screens serve as the display medium for 3D applications, virtual screens can be created as an intermediate medium for displaying 2D applications to achieve synchronous display of 2D and 3D applications.

[0039] Optionally, the binocular image includes a left-eye image and a right-eye image. When the rendering frame rate of the head-mounted display device is lower than the display frame rate, after rendering and displaying the binocular image on the display screen, the method further includes:

[0040] Based on the current head pose, determine the predicted head pose for the next frame;

[0041] Based on the predicted head pose, the left eye image and the right eye image are respectively subjected to frame interpolation to obtain the left eye intermediate frame and the right eye intermediate frame;

[0042] Before generating the left-eye and right-eye images for the next frame, the left-eye intermediate frame and the right-eye intermediate frame are generated via the display screen.

[0043] The beneficial effects of the above technical solution are as follows: by predicting head pose and interpolating frames, the display speed of head-mounted display devices can be improved to reduce stuttering. On the other hand, rendering based on head pose can improve the matching between the overlay image and the user's viewpoint and enhance the user's interactive experience.

[0044] Secondly, embodiments of this application provide a head-mounted display device, including a processor, a memory, a display screen, and a communication interface, wherein the communication interface, the display screen, the memory, and the processor are connected to the processor via a bus.

[0045] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0046] Control the display screen to show the three-dimensional interface of the 3D application;

[0047] The communication interface is used to receive input commands to launch 2D applications.

[0048] In response to the operation command, the Overlay service is started and a virtual screen is created, and the image of the 2D application is drawn onto the virtual screen; the virtual screen is not visible to the user.

[0049] The Overlay service is controlled to acquire an image of the 2D application from the virtual screen to obtain an overlay image;

[0050] The overlay image is superimposed on the three-dimensional interface to obtain a target binocular image, which is then displayed on the display screen.

[0051] Optionally, the processor controls the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image. Specifically, the operation is as follows:

[0052] Initialize the graphics rendering interface and create a second texture object for the graphics rendering interface;

[0053] The Overlay service is controlled to start a rendering loop, and the following operations are performed during each rendering:

[0054] Capture the current image frame of the 2D application on the virtual screen;

[0055] When rendering the current image frame, the pose information of the head-mounted display device relative to the reference space is obtained; the pose information is used for the overlay display of the overlay image;

[0056] The second texture object is rendered according to the current image frame, and the second texture object is rendered onto the exchange chain component according to the binding relationship between the second texture object and the exchange chain component.

[0057] Submit the exchange chain component and the pose information to obtain the overlay image of the current frame.

[0058] Optionally, the specific steps to start the Overlay service are as follows:

[0059] Create and initialize an interface instance that provides operations for the Overlay service;

[0060] Based on the interface instance, obtain the configuration parameters of the operating system of the head-mounted display device;

[0061] Based on the configuration parameters, configure the Overlay service to adapt to the operating system of the head-mounted display device; create a session on the operating system of the head-mounted display device to manage the lifecycle of the Overlay service.

[0062] Optionally, starting the Overlay service also includes:

[0063] A reference space for the Overlay service is created, and the overlay properties of the reference space are set according to the display parameters of the 2D application and the 3D scene. The overlay properties are used to determine the display state of the Overlay service in the 3D scene.

[0064] Optionally, the processor overlays the overlay image onto the 3D interface to obtain a target binocular image, which is then displayed on the display screen. Specifically, the operation is as follows:

[0065] Obtain the spatial pose information of the head-mounted display device relative to the reference space;

[0066] Based on the pose information, the overlay image is superimposed on the three-dimensional interface to obtain a fused image;

[0067] A left-eye image and a right-eye image are generated based on the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

[0068] Optionally, the specific steps for creating a virtual screen are as follows:

[0069] Create a media mapping object for capturing images of the 2D application on the virtual screen;

[0070] Create a first surface texture object, which is used to receive the image stream of the 2D application captured by the media mapping instance;

[0071] Based on the first surface texture object, a Surface object is created, which represents a drawn two-dimensional plane and serves as a bridge connecting the image stream and the rendering system.

[0072] Based on the media mapping object and the Surface object, the corresponding API is called to create a virtual screen.

[0073] Optionally, the binocular image includes a left-eye image and a right-eye image. When the rendering frame rate of the head-mounted display device is lower than the display frame rate, after rendering and displaying the binocular image on the display screen, the processor further executes:

[0074] Based on the current head pose, determine the predicted head pose for the next frame;

[0075] Based on the predicted head pose, the left eye image and the right eye image are respectively subjected to frame interpolation to obtain the left eye intermediate frame and the right eye intermediate frame;

[0076] Before generating the left-eye and right-eye images for the next frame, the left-eye intermediate frame and the right-eye intermediate frame are generated via the display screen.

[0077] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the 3D and 2D application display methods described in the first aspect.

[0078] The technical effects of any of the implementation methods in the second or third aspect can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 This is a schematic diagram of the safe zone of a VR device provided in an embodiment of this application;

[0081] Figure 2 A flowchart illustrating the display methods for 3D and 2D applications provided in embodiments of this application;

[0082] Figure 3This is a schematic diagram of the internal structure of the display screen provided in an embodiment of this application;

[0083] Figure 4 A flowchart illustrating the method for starting the Overlay service provided in this application embodiment;

[0084] Figure 5 A flowchart illustrating a method for creating a virtual screen as provided in an embodiment of this application;

[0085] Figure 6 A flowchart of the switching chain component rendering method provided in the embodiments of this application;

[0086] Figure 7 A flowchart illustrating a method for binocular fusion display of 2D images and a 3D interface provided in an embodiment of this application;

[0087] Figure 8 This is a flowchart of the frame interpolation processing method provided in the embodiments of this application;

[0088] Figure 9 A display system architecture diagram provided for embodiments of this application;

[0089] Figure 10 Timing diagrams of display methods for 3D and 2D applications provided in embodiments of this application;

[0090] Figure 11A A schematic diagram of an interactive interface for simultaneously displaying 2D and 3D applications, provided as an embodiment of this application;

[0091] Figure 11B A schematic diagram of an interactive interface for simultaneously displaying 2D and 3D applications, provided as an embodiment of this application; Figure 12 This is a structural diagram of a head-mounted display device provided in an embodiment of this application. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0093] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0094] Furthermore, the terms “including” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0095] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0096] The following terms are used in the embodiments of this application:

[0097] Overlay: An Android system-level UI element that can be overlaid on top of the display content to show specific information or interfaces without being perceived or affected by other applications, thus achieving a transparent overlay effect.

[0098] The design concept of the embodiments of this application will be summarized below in conjunction with application scenarios.

[0099] Synchronizing the display of 2D applications with 3D applications has a wide range of applications, such as:

[0100] 1. Users can run multiple applications simultaneously in a virtual environment, allowing them to browse the web, check emails, or use instant messaging software while experiencing 3D games;

[0101] 2. In some situations, a 2D interface may be more intuitive or better suited for completing a task. For example, typing text using a standard keyboard is faster than using a virtual keyboard in a virtual environment;

[0102] 3.2D applications can serve as navigation menus or settings interfaces, helping users operate within a virtual environment without needing to exit the 3D experience;

[0103] 4. Users can access external information provided by 2D applications (such as tutorials, maps, or game guides) within the 3D experience without interrupting the 3D experience;

[0104] 5. Displaying 2D applications in a virtual environment is one way to achieve a mixed reality experience, allowing elements from the real world to be overlaid on the virtual world;

[0105] 6. For developers, directly viewing the 2D application interface and debugging information in a virtual environment can greatly improve development efficiency;

[0106] 7. Allows users to access traditional 2D applications in a virtual environment, promoting interaction and integration between different platforms;

[0107] 8. For users who are not used to a fully immersive virtual environment, providing 2D applications can serve as a transition, allowing them to gradually adapt to the virtual environment.

[0108] Since displaying 2D applications simultaneously in 3D applications offers numerous advantages, the inability of head-mounted displays to simultaneously display both 3D and 2D applications will limit their widespread adoption.

[0109] In view of this, this application provides a method for displaying 3D and 2D applications based on an Overlay service for head-mounted display devices. During the display of a 3D application, upon receiving an operation command to launch a 2D application, the Overlay service is activated and an invisible virtual screen is created. The image of the 2D application is drawn onto the virtual screen. The virtual screen, acting as an intermediate display medium, can provide the Overlay service with the image of the 2D application, generating an overlay image. By superimposing the overlay image onto the 3D interface of the 3D application, synchronous display of the 3D and 2D applications is achieved. This allows users to obtain a richer and more flexible interactive experience while improving the convenience of interaction.

[0110] See Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this application. The head-mounted display device is equipped with 3D applications (such as 3D games, virtual try-on, and route guidance apps). After the 3D application is launched, the user experiences an immersive experience by wearing the head-mounted display device. The head-mounted display device can be a VR device, an AR device, or a MR device; these devices are collectively referred to as XR devices.

[0111] See Figure 2 This is a flowchart illustrating the display methods for 3D and 2D applications provided in this application embodiment. The process is executed by a head-mounted display device and mainly includes the following steps:

[0112] S20 1: Control the display screen to show the three-dimensional interface of the 3D application.

[0113] 3D applications, as clients of head-mounted display devices, can be displayed on a screen visible to the user.

[0114] The internal structure of the display screen is as follows: Figure 3As shown, it includes a left lens and a right lens. The left lens displays the left-eye image of the 3D application, and the right lens displays the right-eye image of the 3D application, thereby obtaining a three-dimensional interface of the 3D application and achieving a stereoscopic display effect.

[0115] S202: Receive the input operation command to start the 2D application.

[0116] When launching a 2D application while the 3D application is running, the user can send the operation command to launch the 2D application through the 3D interface or by voice.

[0117] S203: In response to the operation command, start the Overlay service and create a virtual screen to draw the image of the 2D application onto the virtual screen.

[0118] Among them, the Overlay service, as another client of the head-mounted display device, is responsible for implementing the overlay function in the virtual environment. Its startup process is as follows: Figure 4 As shown, it mainly includes the following steps:

[0119] S203_1: Create and initialize an interface instance that provides operations for the Overlay service.

[0120] The interface instance is an OpenXR instance, which forms the basis of applications on the head-mounted display device and provides an entry point for subsequent Overlay service operations. Specifically, an OpenXR instance can be created using the xrCreateInstance function, passing in the necessary information for creating the OpenXR instance.

[0121] S203_2: Based on the interface instance, obtain the configuration parameters of the operating system of the head-mounted display device, and configure the Overlay service to adapt to the operating system of the head-mounted display device according to the configuration parameters.

[0122] Before using the Overlay service to render 2D applications, check whether the operating system of the head-mounted display device supports the Overlay function at runtime by querying the extended string or using a specific extended function. If it does, no configuration of the Overlay service is required; otherwise, obtain the configuration parameters of the head-mounted display device's operating system based on the OpenXR instance, and configure the Overlay service accordingly to adapt to the head-mounted display device's operating system.

[0123] S203_3: Create sessions on the operating system of the head-mounted display device to manage the lifecycle of the Overlay service.

[0124] In OpenXR, a session is the context in which all graphics rendering and input processing are performed, and it manages the lifecycle of Overlay services. Specifically, the xrCreateSession function can be used to create a session on the operating system of the head-mounted display device.

[0125] After completing the OpenXR initialization process described above, OpenXR can support Overlay functionality at runtime. Furthermore, the following operations can be performed on the Overlay service:

[0126] S203_4: Create the reference space for the Overlay service.

[0127] The reference space is used to define the position, orientation, and size of the Overlay service in the 3D environment. Specifically, the xrCreateReferenceSpace function is used to create a reference space for the Overlay service.

[0128] S203_5: Set the overlay properties of the reference space based on the display parameters of the 2D application and the 3D scene.

[0129] After starting the Overlay service, you also need to configure the overlay properties. These properties determine the display status of the Overlay service in the 3D scene, including but not limited to the size, position, and transparency of the Overlay service. These properties can be specified in the Overlay service's creation information structure based on the display parameters of the 2D application and the 3D scene.

[0130] In the embodiments of this application, since the overlay attribute is used to define the display state of the overlay service in the three-dimensional scene, and the overlay attribute is set according to the display parameters of the 2D application and the three-dimensional scene, it can be ensured that the overlay image generated by the subsequent overlay service is properly placed in the three-dimensional scene.

[0131] S203_6: Create the exchange chain component for the Overlay service.

[0132] The swapchain component is a key component for displaying content in an Overlay service. It manages a series of image buffers used to store and render the content of the Overlay service. Specifically, a Swapchain component can be created using the xrCreateSwapchain function, specifying its format, size, and other rendering parameters.

[0133] Since 3D and 2D applications are two separate applications for a head-mounted display device, and it only has one display screen used to show the 3D interface of the 3D application, a virtual screen, not a physical display screen, can be created within the 3D environment to display the 2D application. This virtual screen is not visible to the user and serves as an intermediary for displaying the images of the 2D application. The specific process is as follows... Figure 5 As shown, it mainly includes the following steps:

[0134] S203_7: Create a media mapping object.

[0135] Most head-mounted display devices run on the Android operating system. MediaProjection is a service within Android used by Overlay services to capture images of 2D applications on a virtual screen. Specifically, it involves calling an API provided by the Android system to create a MediaProjection instance.

[0136] S203_8: Create the first surface texture object.

[0137] The first surface texture object is a special texture object used to receive image streams from 2D applications captured by the media mapping instance.

[0138] S203_9: Create a Surface object based on the first surface texture object.

[0139] The Surface object represents a drawn two-dimensional plane, serving as a bridge connecting the image stream and the rendering system.

[0140] S203_1 0: Create a virtual screen by calling the corresponding API based on the media mapping object and the Surface object.

[0141] Virtual screens are non-physical displays that are not visible to users.

[0142] S203_11: Draws images from a 2D application onto a virtual screen.

[0143] Virtual screens, serving as an intermediary for displaying images in 2D applications, can seamlessly embed images from 2D applications into 3D scenes.

[0144] In the embodiments of this application, since the head-mounted display device system can simultaneously support two types of screens, namely a display screen and a virtual screen, and the display screen serves as the display medium for 3D applications, a virtual screen can be created as an intermediate medium for displaying 2D applications to achieve synchronous display of 2D and 3D applications.

[0145] S204: Control the Overlay service to obtain an image of the 2D application from the virtual screen to obtain the overlay image.

[0146] After the 2D application's image is displayed on the virtual screen, a rendering loop is started in the Overlay service. This renders the 2D application's image from the virtual screen's Surface object onto the Swapchain component of the Overlay service, obtaining an overlay image on top of the display screen. The specific rendering process is as follows: Figure 6 As shown, it mainly includes the following steps:

[0147] S204_1: Initialize the graphics rendering interface and create the second texture object for the graphics rendering interface.

[0148] Among them, the graphics rendering interface serves as the foundation of graphics rendering. Appropriate APIs can be selected for initialization, which typically involves setting the graphics context and binding the Swapchain component as the rendering target.

[0149] Optionally, the graphics rendering interface includes, but is not limited to, OpenGL, Vulkan, etc.

[0150] The second texture object is used to render images on the virtual screen to the Swapchain component. When it is created, the unique identifier TextureID of the second texture object can be recorded to ensure that the images of 2D applications on the virtual screen can be correctly bound to the second texture object in the future.

[0151] In the Overlay service, a rendering loop is started. This loop is responsible for continuously rendering the 2D applied image on the Surface object of the virtual screen to the second texture object corresponding to the TextureID. Each rendering operation performs the following steps:

[0152] S204_2: Capture the current image frame of a 2D application on the virtual screen.

[0153] The Overlay service can capture the image stream of a 2D application from the virtual screen using the xrWaitFrame function. The xrWaitFrame function waits until the runtime is ready to accept the rendering of the current image frame, ensuring that the application produces and submits image frames synchronously with the runtime frame rate.

[0154] When the runtime is ready to accept the rendering of the current image frame, it starts rendering the current image frame through the xrBeginFrame function. The xrBeginFrame function is a key step in the OpenXR rendering process; it marks the start of frame rendering and ensures that the application stays in sync with the runtime's frame management.

[0155] S204_3: When rendering the current image frame, obtain the pose information of the head-mounted display device relative to the reference space.

[0156] The pose information is used for the overlay display of subsequent overlay images. Specifically, the xrLocateSpace function can be used to query the position and orientation of the head-mounted display device relative to the reference space of the Overlay service at a given time point.

[0157] S204_4: Render the second texture object according to the current image frame, and render the second texture object onto the swap chain component according to the binding relationship between the second texture object and the swap chain component.

[0158] The second texture object is displayed on top of the display using the Swapchain component.

[0159] S204_5: Submit the exchange chain components and pose information to obtain the overlay image of the current frame.

[0160] After the current frame is rendered, the swap chain components and pose information are submitted to the runtime through the xrEndFrame function to obtain the overlay image displayed on the top layer of the screen, so as to display the current frame of the 2D application to the user.

[0161] During the rendering cycle of the Overlay service, the Swapchain component can manage the acquisition, waiting, and release of images for the 2D application using functions such as xrSwapchainImageAcquireInfo, xrSwapchainImageWaitInfo, and xrSwapchainImageReleaseInfo.

[0162] In the embodiments of this application, by rendering the image of the 2D application on the virtual screen to the exchange chain component of the Overlay service, the image of the 2D application is placed on the top layer of the display screen, thereby achieving a fused display with the three-dimensional scene. Furthermore, by acquiring and submitting the pose information of the head-mounted display device and the reference space, it can be ensured that the overlay image can be presented to the user in the appropriate position.

[0163] S205: Overlay the overlay image onto the 3D interface to obtain a stereo image of the target and display it on the display screen.

[0164] In head-mounted display devices, the image compositor runs as a separate process within the operating system. It composites the overlay image with a 3D interface within a 3D scene, thus attaching the 2D application's overlay image to the 3D scene. This allows users to see an overlay image of a 2D application within a virtual environment, achieving seamless integration of 2D and 3D applications. Specifically, the display fusion process is as follows: Figure 7 As shown, it mainly includes the following steps:

[0165] S205_1: Obtain the pose information of the head-mounted display device relative to the reference space.

[0166] When the Overlay service starts, it sets the overlay properties of the reference space, taking into account the display parameters of both 2D and 3D applications. Therefore, during rendering, it can directly obtain the pose information of the head-mounted display device relative to the reference space.

[0167] S205_2: Based on the pose information, the overlay image is superimposed on the 3D interface to obtain a fused image.

[0168] Since the spatial location information of the head-mounted display device relative to the reference space of the Overlay service includes parameters such as the position and orientation of both, the image synthesizer can appropriately place the overlay image into the 3D scene and overlay it onto the 3D interface based on the pose information to obtain a fused image.

[0169] S205_3: Generate left-eye and right-eye images based on the fused image, and render and display the left-eye and right-eye images on the display screen.

[0170] The image synthesizer uses the pose information submitted to the runtime head-mounted display device space and reference space to perform coordinate transformation and view matrix calculations to display the superimposed left-eye and right-eye images on the display screen. This provides correct parallax and depth perception, enabling 3D display effects for 2D applications and ensuring that the overlay image is correctly displayed in the user's field of vision for proper interaction with other elements in the 3D scene.

[0171] In one example, when the rendering frame rate of the head-mounted display is lower than the display frame rate, to avoid stuttering, after rendering and displaying the left-eye and right-eye images on the screen, the following can also be performed: Figure 8 The following steps are shown:

[0172] S206: Determine the predicted head pose for the next frame based on the current head pose.

[0173] Optionally, head pose can be predicted using a Kalman filter. Since head pose prediction algorithms are relatively mature, this part is not the focus of this application and will not be described in detail.

[0174] S207: Based on the predicted head pose, perform frame interpolation on the left and right eye images respectively to obtain the left eye intermediate frame and the right eye intermediate frame.

[0175] The number of frames inserted can be set according to actual needs.

[0176] S207: Before generating the left-eye and right-eye images of the next frame, the left-eye intermediate frame and the right-eye intermediate frame are displayed on the screen.

[0177] In the embodiments of this application, head pose prediction and frame interpolation can improve the display speed of head-mounted display devices to reduce stuttering. On the other hand, rendering based on head pose can improve the matching between the overlay image and the user's viewpoint, thereby enhancing the user's interactive experience.

[0178] In one example, before displaying the superimposed left-eye and right-eye images, the left-eye and right-eye images can be optimized, such as by removing invisible objects, reducing draw calls, and using post-processing effects, thereby improving rendering efficiency and image quality.

[0179] In one example, when the 2D application shuts down or no longer needs the Overlay functionality, all resources associated with the Overlay service, including the Swapchain component, reference space, and other internal resources, are released by calling the corresponding function in OpenXR, thereby avoiding memory leaks and other resource management issues.

[0180] After 2D and 3D applications are displayed together, users can interact with 2D applications in the virtual environment without leaving the immersive experience. During the interaction, the response results of 2D applications will be updated and displayed in real time on the virtual screen, thereby updating the exchange chain components of the Overlay service and ensuring the real-time performance and accuracy of the 2D applications seen by users.

[0181] The system architecture diagram of the display methods for 3D and 2D applications provided in this application is as follows: Figure 9As shown, it includes a client and a server. The graphics processor acts as the server, while the 3D application, 2D application, and Overlay service act as clients. The graphics processor receives and merges the image data from the 2D and 3D applications through inter-process communication (ICP). Through the Overlay service, the 2D application is seamlessly overlaid on the 3D application, enabling interaction with the 2D application during the immersive experience and providing users with a more coherent and intuitive mixed reality environment.

[0182] based on Figure 9 The system architecture shown is illustrated in the timing diagram of the display methods for 3D and 2D applications provided in this application embodiment. Figure 10 As shown, it mainly includes the following steps:

[0183] S1: When the 3D interface of a 3D application is displayed on the screen, it receives an operation command to launch a 2D application.

[0184] Among them, 2D applications can provide users with interactive content for interacting with 3D applications.

[0185] Optionally, the 3D interface of a 3D application can be pre-rendered or rendered in real time.

[0186] S2: The Overlay service is started based on the operation commands of the 2D application.

[0187] The startup process of the Overlay service is described in S203_1-S203_6, and will not be elaborated here.

[0188] S3: The Overlay service creates virtual screens.

[0189] The process of creating a virtual screen is described in S203_7-S203_10 and will not be elaborated here.

[0190] S4: The Overlay service draws images from 2D applications onto a virtual screen.

[0191] S5: The Overlay service renders images of 2D applications on a virtual screen to the exchange chain component.

[0192] The process of rendering the exchange chain component is described in S204_1-S204_4 and will not be elaborated here.

[0193] S6: The exchange chain component of the Overlay service submits the image of the 2D application and obtains the overlay image.

[0194] S7: The image synthesizer obtains the overlay image of the Overlay service and the 3D interface of the 3D application through ICP communication.

[0195] S8: The image synthesizer acquires the head pose data collected by the head-mounted display device.

[0196] Head pose is used to determine the user's head position and orientation in order to render the scene correctly.

[0197] S9: The image synthesizer overlays the overlay image onto the 3D interface and generates left-eye and right-eye images.

[0198] S10: The image synthesizer integrates head pose and displays the superimposed left and right eye images on the screen for interaction.

[0199] The display process involves operations such as coordinate transformation and view matrix calculation.

[0200] S 11: The Overlay service injects interactive events into the XR architecture layer.

[0201] S12: The XR architecture layer updates the content of the 2D application on the virtual screen based on interactive events.

[0202] By updating the virtual screen, the Overlay service is updated synchronously, thereby enabling user interaction with the 2D application.

[0203] In the embodiments of this application, based on the head-mounted display device's support for multiple screen types, a virtual screen invisible to the user is created, and the images of 2D applications are rendered onto the virtual screen. Thus, the virtual screen can serve as an intermediate medium for displaying 2D application images. The Overlay service that renders the 2D application can obtain the 2D application images from the virtual screen. Since the virtual screen is invisible, unlike the display screen of the head-mounted display device, the intermediate display process of the 2D application on the virtual screen does not affect the user's immersive experience. On the other hand, based on the client-server architecture of the head-mounted display device, the Overlay service and the 3D application act as two clients of the head-mounted display device, while the image synthesizer acts as the server. Through ICP communication, it obtains the overlay image and the 3D interface from the Overlay service and the 3D application respectively, and overlays the overlay image onto the 3D interface for binocular display, thereby achieving simultaneous display of 2D and 3D applications. This allows users to enjoy an immersive experience while also accessing and operating 2D applications, enhancing the practicality and compatibility of the head-mounted display device.

[0204] like Figure 11AThe diagram shows an interactive interface where 2D and 3D applications are displayed simultaneously in a video playback scenario. During 3D video playback, a 2D video player is launched, and users can perform operations such as editing and recording 3D videos through interaction events with the video player.

[0205] like Figure 11B The diagram shows an interactive interface where 2D and 3D applications are displayed simultaneously in a VR streaming scenario. The head-mounted display device is physically connected to the computer and a 2D streaming assistant is activated to transmit the computer-generated 3D model to the head-mounted display device for display.

[0206] Based on the same technical concept, this application provides a head-mounted display device that can implement the steps of the above-mentioned display methods for 3D and 2D applications and achieve the same technical effect.

[0207] See Figure 12 The head-mounted display device includes a processor 1201, a memory 120, a display screen 1203, and a communication interface 1204. The communication interface 1204, the display screen 1203, the memory 1202, and the processor 1201 are connected via a bus 1205.

[0208] The memory 1202 stores a computer program, and the processor 1201 performs the following operations according to the computer program:

[0209] Control display screen 1203 to display the three-dimensional interface of 3D applications;

[0210] The system receives input commands to launch 2D applications via communication interface 1204.

[0211] In response to an operation command, the Overlay service is started and a virtual screen is created, on which the images of the 2D application are drawn; the virtual screen is not visible to the user.

[0212] Control the Overlay service to retrieve images of 2D applications from a virtual screen to obtain the overlay image;

[0213] The overlay image is superimposed on the 3D interface to obtain a stereo image of the target and display it on the screen.

[0214] Optionally, the processor 1201 controls the Overlay service to obtain images of 2D applications from the virtual screen to obtain overlay images. Specifically, the operation is as follows:

[0215] Initialize the graphics rendering interface and create a second texture object for the graphics rendering interface;

[0216] Control the Overlay service to start the rendering loop, and perform the following operations on each render:

[0217] Capture the current image frame of a 2D application on the virtual screen;

[0218] When rendering the current image frame, the pose information of the head-mounted display device relative to the reference space is obtained; the pose information is used for the overlay display of the overlay image.

[0219] The second texture object is rendered based on the current image frame, and the second texture object is rendered onto the swap chain component based on the binding relationship between the second texture object and the swap chain component.

[0220] Submit the exchange chain components and pose information to obtain the overlay image of the current frame.

[0221] Optionally, the specific steps for processor 1201 to start the Overlay service are as follows:

[0222] Create and initialize an interface instance that provides operations for the Overlay service;

[0223] Based on the interface instance, obtain the configuration parameters of the operating system of the head-mounted display device;

[0224] Configure the Overlay service according to the configuration parameters to adapt to the operating system of the head-mounted display device;

[0225] Create sessions on the operating system of the head-mounted display device to manage the lifecycle of the Overlay service.

[0226] Optionally, the process of processor 1201 starting the Overlay service also includes:

[0227] Create a reference space for the Overlay service and set the overlay properties of the reference space according to the display parameters of the 2D application and the 3D scene. The overlay properties are used to determine the display status of the Overlay service in the 3D scene.

[0228] Optionally, the processor 1201 overlays the overlay image onto the 3D interface to obtain a target binocular image, which is then displayed on the display screen 1203. The specific operation is as follows:

[0229] Acquire the pose information of the head-mounted display device relative to the reference space;

[0230] Based on the pose information, the overlay image is superimposed on the 3D interface to obtain a fused image;

[0231] The left-eye and right-eye images are generated from the fused images and rendered and displayed on the display screen 1203.

[0232] Optionally, the processor 1201 performs the following specific operations to create a virtual screen:

[0233] Create a media mapping object, which is used to capture images of 2D applications on a virtual screen;

[0234] Create a first surface texture object, which is used to receive the image stream of the 2D application captured by the media mapping instance;

[0235] Based on the first surface texture object, a Surface object is created. The Surface object represents a drawn two-dimensional plane and serves as a bridge connecting the image stream and the rendering system.

[0236] Create a virtual screen by calling the corresponding API based on the media mapping object and the Surface object.

[0237] Optionally, the binocular image includes a left-eye image and a right-eye image. When the rendering frame rate of the head-mounted display device is lower than the display frame rate, after rendering and displaying the binocular image on the display screen 1003, the processor 1201 further performs the following:

[0238] Based on the current head pose, determine the predicted head pose for the next frame;

[0239] Based on the predicted head pose, the left and right eye images are interpolated to obtain the left eye intermediate frame and the right eye intermediate frame.

[0240] Before generating the left-eye and right-eye images for the next frame, the left-eye intermediate frame and the right-eye intermediate frame are generated via the display screen 1203.

[0241] In this embodiment, the memory 1202 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and programs required for running instant messaging functions; the data storage area may store various instant messaging information and operation instruction sets. The memory 1202 may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any other medium capable of carrying or storing a desired computer program having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory 1202 may be a combination of the above-described memories.

[0242] The processor 1201 may include one or more central processing units (CPUs), GPUs, or digital processing units, etc. The processor 1201 is used to implement the display methods of any of the aforementioned 3D and 2D applications when invoking computer programs stored in memory 1202.

[0243] It should be noted that, Figure 12 This is merely an example illustrating the hardware necessary for a head-mounted display device to perform the display method steps for the 3D and 2D applications provided in the embodiments of this application. Not shown, the head-mounted display device may also include conventional hardware such as a microphone, camera, power supply, and handle.

[0244] This application embodiment does not limit the specific connection medium between the communication interface 1204, display screen 1203, memory 1202, and processor 1201. In this application embodiment, the communication interface 1204 and display screen 1203 are connected to the bus 1205 between memory 1202 and processor 1201. Figure 12 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1205 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 1 2 is described using only a thick line, but it does not describe that there is only one bus or one type of bus.

[0245] For ease of description, the head-mounted display device can be divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware.

[0246] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0247] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform the steps of any of the 3D and 2D application display methods described in the foregoing embodiments.

[0248] This application also provides a computer program product for storing a computer program for executing the steps of any of the display methods for 3D and 2D applications in the foregoing embodiments.

[0249] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0250] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0251] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0252] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0253] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A display method for 3D applications and 2D applications, characterized in that, Applications in head-mounted display devices, including: Control the display screen to show the three-dimensional interface of 3D applications; Receive input commands to launch 2D applications; In response to the operation command, the Overlay service is started and a virtual screen is created, and the image of the 2D application is drawn onto the virtual screen; the virtual screen is not visible to the user. The Overlay service is controlled to acquire an image of the 2D application from the virtual screen to obtain an overlay image; The overlay image is superimposed on the three-dimensional interface to obtain a target binocular image, which is then displayed on the display screen. The step of overlaying the overlay image onto the three-dimensional interface to obtain a target binocular image and displaying it on the display screen includes: Based on the pose information of the head-mounted display device relative to the reference space of the Overlay service, the overlay image is superimposed on the three-dimensional interface to obtain a fused image; the reference space is used to define at least one of the following information: position, orientation, and size of the Overlay service in the three-dimensional environment. A left-eye image and a right-eye image are generated based on the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

2. The method as described in claim 1, characterized in that, The process of controlling the Overlay service to obtain the image of the 2D application from the virtual screen and obtaining the overlay image includes: Initialize the graphics rendering interface and create a second texture object for the graphics rendering interface; The Overlay service is controlled to start a rendering loop, and the following operations are performed during each rendering: Capture the current image frame of the 2D application on the virtual screen; When rendering the current image frame, the pose information of the head-mounted display device relative to the reference space of the Overlay service is obtained; the pose information is used for the overlay display of the overlay image. The second texture object is rendered according to the current image frame, and the second texture object is rendered onto the swap chain component according to the binding relationship between the second texture object and the swap chain component; the swap chain component is a key component for the content displayed by the Overlay service. Submit the exchange chain component and the pose information to obtain the overlay image of the current image frame.

3. The method as described in claim 1, characterized in that, The activation of the Overlay service includes: Create and initialize an interface instance that provides operations for the Overlay service; Based on the interface instance, obtain the configuration parameters of the operating system of the head-mounted display device; Configure the Overlay service according to the configuration parameters to adapt to the operating system of the head-mounted display device; A session is created on the operating system of the head-mounted display device to manage the lifecycle of the Overlay service.

4. The method as described in claim 3, characterized in that, The activation of the Overlay service also includes: A reference space for the Overlay service is created, and the overlay properties of the reference space are set according to the display parameters of the 2D application and the 3D scene. The overlay properties are used to determine the display state of the Overlay service in the 3D scene.

5. The method according to any one of claims 1-4, characterized in that, The creation of the virtual screen includes: Create a media mapping object for capturing images of the 2D application on the virtual screen; Create a first surface texture object, which is used to receive the image stream of the 2D application captured by the media mapping instance; Based on the first surface texture object, a Surface object is created, which represents a drawn two-dimensional plane and serves as a bridge connecting the image stream and the rendering system. Based on the media mapping object and the Surface object, the corresponding API is called to create a virtual screen.

6. The method according to any one of claims 1-4, characterized in that, The binocular image includes a left-eye image and a right-eye image. When the rendering frame rate of the head-mounted display device is lower than the display frame rate, after rendering and displaying the binocular image on the display screen, the method further includes: Based on the current head pose, determine the predicted head pose for the next frame; Based on the predicted head pose, the left eye image and the right eye image are respectively subjected to frame interpolation to obtain the left eye intermediate frame and the right eye intermediate frame; Before generating the next frame of the left-eye and right-eye images, the left-eye intermediate frame and the right-eye intermediate frame are displayed on the display screen.

7. A head-mounted display device, characterized in that, It includes a processor, a memory, a display screen, and a communication interface, wherein the communication interface, the display screen, the memory, and the processor are connected to the processor via a bus. The memory stores a computer program, and the processor performs the following operations according to the computer program: Control the display screen to show the three-dimensional interface of the 3D application; The communication interface is used to receive input commands to launch 2D applications. In response to the operation command, the Overlay service is started and a virtual screen is created, and the image of the 2D application is drawn onto the virtual screen; the virtual screen is not visible to the user. The Overlay service is controlled to acquire an image of the 2D application from the virtual screen to obtain an overlay image; The overlay image is superimposed on the three-dimensional interface to obtain a target binocular image, which is then displayed on the display screen. The step of overlaying the overlay image onto the three-dimensional interface to obtain a target binocular image and displaying it on the display screen includes: Based on the pose information of the head-mounted display device relative to the reference space of the Overlay service, the overlay image is superimposed on the three-dimensional interface to obtain a fused image; the reference space is used to define at least one of the following information: position, orientation, and size of the Overlay service in the three-dimensional environment. A left-eye image and a right-eye image are generated based on the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

8. The head-mounted display device as described in claim 7, characterized in that, The processor controls the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image. The specific operation is as follows: Initialize the graphics rendering interface and create a second texture object for the graphics rendering interface; The Overlay service is controlled to start a rendering loop, and the following operations are performed during each rendering: Capture the current image frame of the 2D application on the virtual screen; When rendering the current image frame, the pose information of the head-mounted display device relative to the reference space of the Overlay service is obtained; The pose information is used for the overlay display of the overlay image; The second texture object is rendered according to the current image frame, and the second texture object is rendered onto the exchange chain component according to the binding relationship between the second texture object and the exchange chain component. The exchange chain component is a key component for the content displayed by the Overlay service; Submit the exchange chain component and the pose information to obtain the overlay image of the current image frame.

Images