Display method of 3D application and 2D application and head-mounted display device

Abstract

The invention relates to the technical field of three-dimensional display, and provides a display method of a 3D application and a 2D application and a head-mounted display device, which are used for realizing simultaneous display of the 2D application and the 3D application. On the basis of support of the head-mounted display device for multiple types of screens, a virtual screen invisible to a user is created, and an image of a 2D application is rendered to the virtual screen, so that the virtual screen serves as an intermediate medium for displaying the image of the 2D application, an Overlay service for drawing the 2D application obtains the image of the 2D application from the virtual screen to obtain a covering layer image, and the covering layer image is displayed on the head-mounted display device. According to the head-mounted display device, the covering layer image is overlaid on the three-dimensional interface for binocular display, so that the 2D application and the 3D application are displayed at the same time, a user can access and operate the 2D application while enjoying immersive experience, and the practicability and compatibility of the head-mounted display device are enhanced.

Description

Background Art

[0002] With the widespread promotion of head mounted display (HMD), the demand for multitasking in head mounted display devices is increasing. Among them, synchronously displaying 2D applications in 3D applications can meet the demand for multitasking. However, most head mounted display devices currently cannot display 3D applications and 2D applications at the same time. If you want to display a 2D application, you need to exit the 3D experience and put the 3D application in the background. The interaction between 3D applications and 2D applications cannot be achieved, and thus the demand for multitasking cannot be met.

[0003] Therefore, realizing the joint display of 3D applications and 2D applications has important research value for head-mounted display devices. Summary of the invention

[0004] The embodiments of the present application provide a display method of a 3D application and a 2D application and a head-mounted display device, which are used to realize the common display of a 3D application and a 2D application.

[0005] In a first aspect, an embodiment of the present application provides a display method for a 3D application and a 2D application, which is applied to a head-mounted display device, including:

[0006] Control the display screen to display the three-dimensional interface of the 3D application;

[0007] Receive an input operation instruction to start a 2D application;

[0008] In response to the operation instruction, start the Overlay service and create a virtual screen, and draw the image of the 2D application on the virtual screen; the virtual screen is not visible to the user;

[0009] Control the Overlay service to obtain the image of the 2D application from the virtual screen to obtain an overlay image;

[0010] The cover layer image is superimposed on the three-dimensional interface to obtain a target binocular image and display it through the display screen.

[0011] The beneficial effects of the above technical solution are as follows: based on the support of the head-mounted display device for multiple types of screens, a virtual screen invisible to the user is created and the image of the 2D application is rendered on the virtual screen. In this way, the virtual screen can be used as an intermediate medium for displaying the image of the 2D application, and the Overlay service can obtain the image of the 2D application from the virtual screen. Since the virtual screen is different from the display screen of the head-mounted display device and is invisible, the intermediate display process of the 2D application on the virtual screen will not affect the user's immersive experience. On the other hand, based on the Client-Server architecture of the head-mounted display device, the Overlay service and the 3D application are two clients of the head-mounted display device, and the image synthesizer is the server of the head-mounted display device. Through ICP communication, the overlay image and the three-dimensional interface are obtained from the Overlay service and the 3D application respectively, and the overlay image is superimposed on the three-dimensional interface for binocular display, thereby realizing the simultaneous display of 2D applications and 3D applications, which can support users to access and operate 2D applications while enjoying the immersive experience, thereby enhancing the practicality and compatibility of the head-mounted display device.

[0012] Optionally, the controlling the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image includes:

[0013] Initialize a graphics rendering interface and create a second texture object of the graphics rendering interface;

[0014] Control the Overlay service to start the rendering loop, and perform the following operations each time rendering:

[0015] Capturing a current image frame of the 2D application on the virtual screen;

[0016] When rendering the current image frame, obtaining the spatial position information of the head mounted display device relative to the reference space; the position information is used for the overlay display of the overlay layer image;

[0017] Rendering the second texture object according to the current image frame, and rendering the second texture object to the swap chain component according to a binding relationship between the second texture object and the swap chain component;

[0018] Submit the swap chain component and the pose information to obtain the overlay image of the current frame.

[0019] The beneficial effects of the above technical solution are: by rendering the image of the 2D application on the virtual screen to the exchange chain component of the Overlay service, the image of the 2D application is located at the top layer of the display screen, thereby achieving a fusion display with the three-dimensional interface, and by obtaining and submitting the posture information of the space and reference space of the head-mounted display device, it can be ensured that the overlay image can be presented to the user at the appropriate position.

[0020] Optionally, starting the Overlay service includes:

[0021] Create and initialize an interface instance that provides operations for the Overlay service;

[0022] According to the interface instance, obtaining configuration parameters of the operating system of the head mounted display device;

[0023] According to the configuration parameters, the Overlay service is configured to adapt to the operating system of the head mounted display device; and a session is created on the operating system of the head mounted display device to manage the life cycle of the Overlay service.

[0024] The beneficial effect of the above technical solution is: the Overlay service is configured through the configuration parameters of the operating system of the head-mounted display device, so that the Overlay service is adapted to the operating system of the head-mounted display device, thereby ensuring the normal display of the 2D application image.

[0025] Optionally, starting the Overlay service further includes:

[0026] A reference space of the Overlay service is created, and overlay layer properties of the reference space are set according to display parameters of the 2D application and the three-dimensional scene, wherein the overlay layer properties are used to determine the display state of the Overlay service in the three-dimensional scene.

[0027] The beneficial effect of the above technical solution is: since the overlay layer attributes are used to define the display status of the Overlay service in the three-dimensional scene, and the overlay layer attributes are set according to the display parameters of the 2D application and the three-dimensional scene, it can ensure that the overlay layer image generated by the subsequent Overlay service is properly placed in the three-dimensional scene, reducing interference with the 3D application display and ensuring the user's immersive experience.

[0028] Optionally, superimposing the cover layer image on the three-dimensional interface to obtain a target binocular image and displaying it on the display screen includes:

[0029] Acquire spatial position information of the head mounted display device relative to a reference space;

[0030] According to the posture information, superimposing the cover layer image on the three-dimensional interface to obtain a fused image;

[0031] A left-eye image and a right-eye image are generated according to the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

[0032] The beneficial effect of the above technical solution is: the superimposed left-eye image and right-eye image are displayed on the display screen to provide correct parallax and depth perception, thereby achieving a three-dimensional display effect of a 2D application.

[0033] Optionally, the creating a virtual screen includes:

[0034] Creating a media mapping object, wherein the media mapping object is used to capture an image of the 2D application on the virtual screen;

[0035] Creating a first surface texture object, where the first surface texture object is used to receive the image stream of the 2D application captured by the media mapping instance;

[0036] Creating a Surface object according to the first surface texture object, wherein the Surface object represents a drawn two-dimensional plane and serves as a bridge connecting an image stream and a rendering system;

[0037] According to the media mapping object and the Surface object, a corresponding API is called to create a virtual screen.

[0038] The beneficial effect of the above technical solution is: since the head-mounted display device can simultaneously support two types of screens, namely the display screen and the virtual screen, and the display screen serves as the display medium for 3D applications, it is possible to create a virtual screen as an intermediate medium for displaying 2D applications to achieve synchronous display of 2D applications and 3D applications.

[0039] Optionally, the binocular image includes a left-eye image and a right-eye image, and when a rendering frame rate of the head mounted display device is lower than a display frame rate, after rendering and displaying the binocular image on the display screen, the method further includes:

[0040] According to the current head pose, determine the predicted head pose for the next frame;

[0041] According to the predicted head pose, interpolating the left eye image and the right eye image respectively to obtain a left eye intermediate frame and a right eye intermediate frame;

[0042] Before generating the left eye image and the right eye image of the next frame, the left eye intermediate frame and the right eye intermediate frame are processed through the display screen.

[0043] The beneficial effects of the above technical solution are: through head posture prediction and interpolation processing, the display speed of the head-mounted display device can be improved to reduce the lag phenomenon. On the other hand, rendering and display based on head posture can improve the matching of the overlay image with the user's perspective and enhance the user interaction experience.

[0044] In a second aspect, an embodiment of the present application provides a head-mounted display device, including a processor, a memory, a display screen, and a communication interface, wherein the communication interface, the display screen, the memory, and the processor are connected to the processor via a bus:

[0045] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0046] Controlling the display screen to display a three-dimensional interface of a 3D application;

[0047] Receiving, via the communication interface, an operation instruction for starting a 2D application;

[0048] In response to the operation instruction, start the Overlay service and create a virtual screen, and draw the image of the 2D application on the virtual screen; the virtual screen is not visible to the user;

[0049] Control the Overlay service to obtain the image of the 2D application from the virtual screen to obtain an overlay image;

[0050] The cover layer image is superimposed on the three-dimensional interface to obtain a target binocular image and display it through the display screen.

[0051] Optionally, the processor controls the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image, and the specific operation is:

[0052] Initialize a graphics rendering interface and create a second texture object of the graphics rendering interface;

[0053] Control the Overlay service to start the rendering loop, and perform the following operations each time rendering:

[0054] Capturing a current image frame of the 2D application on the virtual screen;

[0055] When rendering the current image frame, obtaining the spatial position information of the head mounted display device relative to the reference space; the position information is used for the overlay display of the overlay layer image;

[0056] Rendering the second texture object according to the current image frame, and rendering the second texture object to the swap chain component according to a binding relationship between the second texture object and the swap chain component;

[0057] Submit the swap chain component and the pose information to obtain the overlay image of the current frame.

[0058] Optionally, the specific operations to start the Overlay service are:

[0059] Create and initialize an interface instance that provides operations for the Overlay service;

[0060] According to the interface instance, obtaining configuration parameters of the operating system of the head mounted display device;

[0061] According to the configuration parameters, the Overlay service is configured to adapt to the operating system of the head mounted display device; and a session is created on the operating system of the head mounted display device to manage the life cycle of the Overlay service.

[0062] Optionally, the operation of starting the Overlay service also includes:

[0063] A reference space of the Overlay service is created, and overlay layer properties of the reference space are set according to display parameters of the 2D application and the three-dimensional scene, wherein the overlay layer properties are used to determine the display state of the Overlay service in the three-dimensional scene.

[0064] Optionally, the processor superimposes the cover layer image on the three-dimensional interface to obtain a target binocular image and displays it on the display screen, and the specific operations are:

[0065] Acquire spatial position information of the head mounted display device relative to a reference space;

[0066] According to the posture information, superimposing the cover layer image on the three-dimensional interface to obtain a fused image;

[0067] A left-eye image and a right-eye image are generated according to the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

[0068] Optionally, the specific operations for creating a virtual screen are:

[0069] Creating a media mapping object, wherein the media mapping object is used to capture an image of the 2D application on the virtual screen;

[0070] Creating a first surface texture object, where the first surface texture object is used to receive the image stream of the 2D application captured by the media mapping instance;

[0071] Creating a Surface object according to the first surface texture object, wherein the Surface object represents a drawn two-dimensional plane and serves as a bridge connecting an image stream and a rendering system;

[0072] According to the media mapping object and the Surface object, a corresponding API is called to create a virtual screen.

[0073] Optionally, the binocular image includes a left-eye image and a right-eye image, and when a rendering frame rate of the head mounted display device is lower than a display frame rate, after rendering and displaying the binocular image on the display screen, the processor further executes:

[0074] According to the current head pose, determine the predicted head pose for the next frame;

[0075] According to the predicted head pose, interpolating the left eye image and the right eye image respectively to obtain a left eye intermediate frame and a right eye intermediate frame;

[0076] Before generating the left eye image and the right eye image of the next frame, the left eye intermediate frame and the right eye intermediate frame are processed through the display screen.

[0077] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the display method of any one of 3D applications and 2D applications in the first aspect are implemented.

[0078] The technical effects brought about by any one of the implementation methods in the second aspect to the third aspect can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0080] Figure 1 A schematic diagram of a safe area of ​​a VR device provided in an embodiment of the present application;

[0081] Figure 2 A flow chart of a display method for 3D applications and 2D applications provided in an embodiment of the present application;

[0082] Figure 3A schematic diagram of the internal structure of a display screen provided in an embodiment of the present application;

[0083] Figure 4 A flowchart of a method for starting an Overlay service provided in an embodiment of the present application;

[0084] Figure 5 A flowchart of a method for creating a virtual screen provided in an embodiment of the present application;

[0085] Figure 6 A flow chart of a swap chain component rendering method provided in an embodiment of the present application;

[0086] Figure 7 A flow chart of a method for binocular fusion display of 2D images and 3D interfaces provided in an embodiment of the present application;

[0087] Figure 8 A flow chart of the frame insertion processing method provided in an embodiment of the present application;

[0088] Fig. 9 A display system architecture diagram provided for an embodiment of the present application;

[0089] Fig.10 A timing diagram of a display method for a 3D application and a 2D application provided in an embodiment of the present application;

[0090] Fig.11A A schematic diagram of an interactive interface for simultaneously displaying a 2D application and a 3D application provided in an embodiment of the present application;

[0091] Fig. 11B A schematic diagram of another interactive interface for simultaneously displaying a 2D application and a 3D application provided in an embodiment of the present application; Fig.12 A structural diagram of a head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0093] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0094] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0095] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0096] The glossaries used in the embodiments of this application are as follows:

[0097] Overlay: An Android system-level UI element that can display specific information or interfaces on top of the display content without being perceived or affected by other applications, thus achieving a transparent overlay effect.

[0098] The following is an overview of the design concept of the embodiments of the present application in conjunction with application scenarios.

[0099] Synchronous display of 2D applications in 3D applications has a wide range of application scenarios, such as:

[0100] 1. Users can run multiple applications simultaneously in a virtual environment, so that users can browse the web, check email or use instant messaging software while playing 3D games;

[0101] 2. In some cases, a 2D interface may be more intuitive or better suited to completing a task. For example, using a standard keyboard to enter text is faster than operating a virtual keyboard in a virtual environment;

[0102] 3.2D applications can be used as navigation menus or settings interfaces to help users operate in the virtual environment without exiting the 3D experience;

[0103] 4. Users can view external information (such as tutorials, maps, or game guides) provided by 2D applications in the 3D experience without interrupting the 3D experience;

[0104] 5. Displaying 2D applications in a virtual environment is a way to achieve a mixed reality experience, where real-world elements can be superimposed on the virtual world;

[0105] 6. For developers, directly viewing the 2D application interface and debugging information in a virtual environment can greatly improve development efficiency;

[0106] 7. Allow users to access traditional 2D applications in a virtual environment, promoting interaction and integration between different platforms;

[0107] 8. For users who are not accustomed to fully immersive virtual environments, providing 2D applications can serve as a transition so that they can gradually adapt to the virtual environment.

[0108] Since synchronously displaying 2D applications in 3D applications has many conveniences, if the head-mounted display device cannot display 3D applications and 2D applications at the same time, it will limit its promotion and application.

[0109] In view of this, an embodiment of the present application provides a display method for 3D applications and 2D applications based on an Overlay service for a head-mounted display device. During the process of displaying a 3D application on a display screen, when an operation instruction to start a 2D application is received, the Overlay service is started and an invisible virtual screen is created. The image of the 2D application is drawn onto the virtual screen. The virtual screen serves as an intermediate display medium and can provide the Overlay service with the image of the 2D application and generate an overlay image. By superimposing the overlay image on the three-dimensional interface of the 3D application, the synchronous display of the 3D application and the 2D application is achieved. In this way, users can obtain a richer and more flexible interactive experience while improving the convenience of interaction.

[0110] See also Figure 1 , is a schematic diagram of an application scenario of an embodiment of the present application, in which a head-mounted display device is installed with a 3D application (such as 3D games, virtual fitting, route guidance and other APPs). After the 3D application is started, the user wears the head-mounted display device for an immersive experience. The head-mounted display device can be a VR device, an AR device, or an MR device, which are collectively referred to as XR devices.

[0111] See also Figure 2 , is a flow chart of a display method for 3D applications and 2D applications provided in an embodiment of the present application, the process is executed by a head mounted display device, and mainly includes the following steps:

[0112] S20 1: Control the display screen to display the three-dimensional interface of the 3D application.

[0113] As a client of the head-mounted display device, the 3D application can be displayed through the display screen visible to the user.

[0114] Among them, the internal structure of the display screen is as follows Figure 3As shown, it includes a left-eye lens and a right-eye lens. The left-eye image of the 3D application is displayed through the left-eye lens, and the right-eye image of the 3D application is displayed through the right-eye lens, so as to obtain a three-dimensional interface of the 3D application and achieve a stereoscopic display effect.

[0115] S202: Receive an input operation instruction to start a 2D application.

[0116] To start a 2D application while a 3D application is running, the user may send an operation instruction to start the 2D application through a 3D interface or by voice.

[0117] S203: In response to the operation instruction, start the Overlay service and create a virtual screen, and draw the image of the 2D application on the virtual screen.

[0118] Among them, the Overlay service is another client of the head-mounted display device, which is responsible for implementing the overlay function in the virtual environment. Its startup process is as follows: Figure 4 As shown, it mainly includes the following steps:

[0119] S203_1: Create and initialize an interface instance that provides operations for the Overlay service.

[0120] The interface instance is an OpenXR instance, which is the basis of applications on head-mounted display devices and provides an entry point for subsequent Overlay service operations. Specifically, you can use the xrCreateInstance function to create an OpenXR instance and pass in the necessary information to create an OpenXR instance.

[0121] S203_2: According to the interface instance, obtain configuration parameters of the operating system of the head mounted display device, and configure the Overlay service according to the configuration parameters to adapt to the operating system of the head mounted display device.

[0122] Before using the Overlay service to draw 2D applications, check whether the operating system of the head-mounted display device supports the Overlay function at runtime by querying the extension string or using a specific extension function. If it does, there is no need to configure the Overlay service. Otherwise, according to the OpenXR instance, obtain the configuration parameters of the operating system of the head-mounted display device, and configure the Overlay service to adapt to the operating system of the head-mounted display device based on the configuration parameters.

[0123] S203_3: Create a session on the operating system of the head mounted display device to manage the life cycle of the Overlay service.

[0124] The session is the context for performing all graphics rendering and input processing in OpenXR, and is used to manage the life cycle of the Overlay service. Specifically, the xrCreateSession function can be used to create a session on the operating system of the head-mounted display device.

[0125] After completing the above OpenXR initialization process, OpenXR can support the Overlay function at runtime. Further, the following operations can be performed on the Overlay service:

[0126] S203_4: Create a reference space for the Overlay service.

[0127] The reference space is used to define the position, direction, size, and other information of the Overlay service in a three-dimensional environment. Specifically, use the xrCreateReferenceSpace function to create a reference space for an Overlay service.

[0128] S203_5: Setting the overlay properties of the reference space according to the display parameters of the 2D application and the 3D scene.

[0129] After starting the Overlay service, you also need to set the overlay layer properties. The overlay layer properties are used to determine the display status of the Overlay service in the 3D scene, including but not limited to the size, position and transparency of the Overlay service. These properties can be specified in the Overlay service creation information structure based on the display parameters of the 2D application and the 3D scene.

[0130] In the embodiment of the present application, since the overlay layer attributes are used to define the display status of the Overlay service in the three-dimensional scene, and the overlay layer attributes are set according to the 2D application and the display parameters of the three-dimensional scene, it can be ensured that the overlay layer image generated by the subsequent Overlay service is properly placed in the three-dimensional scene.

[0131] S203_6: Create a swap chain component for the Overlay service.

[0132] The swapchain component is a key component for Overlay service display content. It is responsible for managing a series of image buffers for storing and rendering Overlay service content. Specifically, you can create a Swapchain component through the xrCreateSwapchain function and specify its format, size, and other rendering parameters.

[0133] Since the 3D application and the 2D application are two application programs of the head mounted display device, and it has only one display screen used to display the three-dimensional interface of the 3D application, in order to realize the display of the 2D application, a virtual screen that is not a physical display screen can be created in the three-dimensional environment. The virtual screen is invisible to the user and serves as an intermediate medium for the image display of the 2D application. The specific process is as follows Figure 5 As shown, it mainly includes the following steps:

[0134] S203_7: Create a media mapping object.

[0135] The operating system of most head-mounted display devices is Android. The media mapping object (MediaProjection) is a service in the Android system, which is used by the Overlay service to capture the image of the 2D application on the virtual screen. Specifically, call the API provided by the Android system to create a MediaProjection instance.

[0136] S203_8: Create a first surface texture object.

[0137] The first surface texture (SurfaceTexture) object is a special texture object used to receive the image stream of the 2D application captured by the media mapping instance.

[0138] S203_9: Create a Surface object according to the first surface texture object.

[0139] Among them, the Surface object represents a drawn two-dimensional plane, serving as a bridge connecting the image stream and the rendering system.

[0140] S203_1 0: According to the media mapping object and the Surface object, call the corresponding API to create a virtual screen.

[0141] Among them, the virtual screen is a non-physical display screen and is not visible to the user.

[0142] S203_11: Draw the image of the 2D application onto the virtual screen.

[0143] Among them, the virtual screen serves as an intermediate medium for displaying images of 2D applications, and can seamlessly embed the images of 2D applications into three-dimensional scenes.

[0144] In the embodiments of the present application, since the system of the head-mounted display device can simultaneously support two types of screens, namely, a display screen and a virtual screen, and the display screen serves as a display medium for 3D applications, it is possible to achieve synchronous display of 2D applications and 3D applications by creating a virtual screen as an intermediate medium for displaying 2D applications.

[0145] S204: Control the Overlay service to obtain the image of the 2D application from the virtual screen to obtain an overlay image.

[0146] After the image of the 2D application is displayed on the virtual screen, the rendering loop is started in the Overlay service to render the image of the 2D application on the Surface object of the virtual screen to the Swapchain component of the Overlay service to obtain the overlay image on the top layer of the display screen. The specific rendering process is as follows Figure 6 As shown, it mainly includes the following steps:

[0147] S204_1: Initialize the graphics rendering interface and create a second texture object of the graphics rendering interface.

[0148] Among them, the graphics rendering interface, as the basis of graphics rendering, can select a suitable API for initialization, which usually involves setting the graphics context, binding the Swapchain component as the rendering target, etc.

[0149] Optionally, the graphics rendering interface includes but is not limited to OpenGL, Vulkan, etc.

[0150] The second texture object is used to render the image on the virtual screen to the Swapchain component. The unique identifier TextureID of the second texture object can be recorded when it is created to ensure that the image of the 2D application on the virtual screen can be correctly bound to the second texture object later.

[0151] Start a rendering loop in the Overlay service. The loop is responsible for continuously rendering the image of the 2D application on the Surface object of the virtual screen to the second texture object corresponding to the TextureID. Each rendering operation performs the following operations:

[0152] S204_2: Capture the current image frame of the 2D application on the virtual screen.

[0153] The Overlay service can capture the image stream of the 2D application from the virtual screen through the xrWaitFrame function. The xrWaitFrame function can wait until the runtime is ready to accept the rendering of the current image frame, thereby ensuring that the application produces and submits image frames synchronously with the runtime's frame rhythm.

[0154] When the runtime is ready to accept the rendering of the current image frame, the xrBeginFrame function is used to start the rendering of the current image frame. The xrBeginFrame function is a key step in the OpenXR rendering process. It marks the beginning of frame rendering and ensures that the application is synchronized with the runtime frame management.

[0155] S204_3: When rendering the current image frame, obtain the spatial posture information of the head mounted display device relative to the reference space.

[0156] The pose information is used for the subsequent overlay image overlay display. Specifically, the xrLocateSpace function can be used to query the position and direction of the head-mounted display device relative to the reference space of the Overlay service at a given time point.

[0157] S204_4: Rendering a second texture object according to the current image frame, and rendering the second texture object onto the swap chain component according to a binding relationship between the second texture object and the swap chain component.

[0158] Through the Swapchain component, the second texture object is displayed on the top layer of the display.

[0159] S204_5: Submit the swap chain components and pose information to obtain the overlay image of the current frame.

[0160] After the current frame is rendered, the swap chain components and pose information are submitted to the runtime through the xrEndFrame function to obtain the overlay image displayed on the top layer of the display so that the current frame of the 2D application can be displayed to the user.

[0161] During the rendering loop of the Overlay service, the Swapchain component acquires, waits for, and releases the images of the 2D application, which can be managed using the xrSwapchainImageAcquireInfo function, xrSwapchainImageWaitInfo function, and xrSwapchainImageReleaseInfo function.

[0162] In an embodiment of the present application, by rendering the image of the 2D application on the virtual screen to the exchange chain component of the Overlay service, the image of the 2D application is located at the top layer of the display screen, thereby achieving a fusion display with the three-dimensional scene. In addition, by obtaining and submitting the spatial pose information of the head-mounted display device and the reference space, it can be ensured that the overlay image can be presented to the user at the appropriate position.

[0163] S205: Overlaying the cover layer image on the three-dimensional interface to obtain a target binocular image and displaying it on a display screen.

[0164] In a head-mounted display device, the image compositor runs as an independent process in the operating system. By synthesizing the overlay image with the 3D interface in the 3D scene, the 2D application is attached to the 3D scene in the overlay image. In this way, the user can see the overlay image of a 2D application in the virtual environment, realizing the seamless integration of 2D and 3D applications. Specifically, the process of fusion display is as follows: Figure 7 As shown, it mainly includes the following steps:

[0165] S205_1: Obtaining the spatial posture information of the head mounted display device relative to the reference space.

[0166] When the Overlay service is started, the overlay properties of the reference space are set, taking into account the display parameters of 2D and 3D applications. Therefore, during rendering, the position information of the head-mounted display device relative to the reference space can be directly obtained.

[0167] S205_2: According to the position information, the cover layer image is superimposed on the three-dimensional interface to obtain a fused image.

[0168] Since the position information of the head-mounted display device's space relative to the reference space of the Overlay service contains parameters such as the position and orientation of the two, the image synthesizer can properly place the overlay image in the three-dimensional scene based on the posture information, and superimpose it on the three-dimensional interface to obtain a fused image.

[0169] S205_3: Generate a left-eye image and a right-eye image according to the fused image, and render and display the left-eye image and the right-eye image on a display screen.

[0170] The image synthesizer displays the superimposed left-eye image and right-eye image on the display screen based on the pose information submitted to the runtime head-mounted display device space and the reference space, through coordinate transformation and view matrix calculation, to provide correct parallax and depth perception, achieve three-dimensional display effects for 2D applications, and ensure that the overlay image is displayed correctly in the user's field of view to facilitate correct interaction with other elements in the three-dimensional scene.

[0171] In one example, when the rendering frame rate of the head mounted display device is lower than the display frame rate, in order to avoid the display image from freezing, after the left eye image and the right eye image are rendered and displayed on the display screen, the following steps may be performed: Figure 8 The steps shown are:

[0172] S206: Determine the predicted head pose of the next frame according to the current head pose.

[0173] Optionally, the head posture may be predicted by a Kalman filter. Since the head posture prediction algorithm is relatively mature, this part is not the focus of this application and will not be described in detail.

[0174] S207: performing interpolation processing on the left eye image and the right eye image respectively according to the predicted head pose to obtain a left eye intermediate frame and a right eye intermediate frame.

[0175] The number of interpolated frames can be set according to actual needs.

[0176] S207: Before generating the left eye image and the right eye image of the next frame, a left eye intermediate frame and a right eye intermediate frame are generated through the display screen.

[0177] In the embodiments of the present application, through head pose prediction and interpolation processing, the display speed of the head-mounted display device can be improved to reduce the jamming phenomenon. On the other hand, rendering and display based on the head pose can improve the matching of the overlay image with the user's perspective and enhance the user interaction experience.

[0178] In one example, before displaying the superimposed left-eye image and right-eye image, the left-eye image and the right-eye image may be optimized, such as eliminating invisible objects, reducing drawing calls, using post-processing effects, etc., so as to improve rendering efficiency and image quality.

[0179] In one example, when a 2D application is closed or no longer needs the Overlay function, all resources related to the Overlay service, including Swapchain components, reference space, and other internal resources, are released by calling the corresponding function in OpenXR, thereby avoiding memory leaks and other resource management issues.

[0180] After 2D applications and 3D applications are displayed together, users can interact with 2D applications in the virtual environment without exiting the immersive experience. During the interaction, the response results of the 2D application will be updated and displayed in real time on the virtual screen, thereby updating the exchange chain components of the Overlay service to ensure the real-time and accuracy of the 2D application seen by the user.

[0181] The system architecture diagram of the display method of 3D application and 2D application provided in the embodiment of the present application is as follows: Fig. 9As shown, it includes a client and a server, wherein the image processor serves as the server, and the 3D application, 2D application and Overlay service serve as the client. The graphics processor receives and fuses the image data of the 2D application and the 3D application through inter-process communication (ICP). Through the Overlay service, the 2D application is seamlessly superimposed on the 3D application, realizing the interaction with the 2D application during the immersive experience, and providing users with a more coherent and intuitive mixed reality environment.

[0182] based on Fig. 9 The system architecture shown in FIG. 1 and the timing diagram of the display method of the 3D application and the 2D application provided in the embodiment of the present application are as follows: Fig.10 As shown, it mainly includes the following steps:

[0183] S 1: When the three-dimensional interface of the 3D application is presented on the display screen, an operation instruction to start the 2D application is received.

[0184] Among them, 2D applications can provide operation content for users to interact with 3D applications.

[0185] Optionally, the three-dimensional interface of the 3D application may be pre-rendered or rendered in real time.

[0186] S2: The Overlay service is started based on the operation instructions of the 2D application.

[0187] Among them, the startup process of the Overlay service refers to the relevant description of S203_1-S203_6, which will not be described in detail here.

[0188] S3: Overlay service creates a virtual screen.

[0189] The process of creating the virtual screen refers to the relevant description of S203_7-S203_10, which will not be described in detail here.

[0190] S4: Overlay service draws the image of the 2D application onto the virtual screen.

[0191] S5: The Overlay service renders the image of the 2D application on the virtual screen to the swap chain component.

[0192] The process of rendering the swap chain component refers to the relevant description of S204_1-S204_4, which will not be described in detail here.

[0193] S6: The swap chain component of the Overlay service submits the image of the 2D application to obtain the overlay image.

[0194] S7: The image synthesizer obtains the overlay image of the Overlay service and the three-dimensional interface of the 3D application through ICP communication.

[0195] S8: The image synthesizer obtains the head posture collected by the head mounted display device.

[0196] Among them, head pose is used to determine the user's head position and orientation in order to render the scene correctly.

[0197] S9: The image synthesizer superimposes the overlay image on the three-dimensional interface and generates a left-eye image and a right-eye image.

[0198] S 1 0: The image synthesizer integrates the head posture and displays the superimposed left-eye image and right-eye image on the display screen for interaction.

[0199] Among them, the display process involves operations such as coordinate transformation and view matrix calculation.

[0200] S 11: The Overlay service injects interaction events into the XR architecture layer.

[0201] S12: The XR architecture layer updates the content of the 2D application on the virtual screen based on the interaction events.

[0202] By updating the virtual screen, the Overlay service is updated synchronously, thereby enabling interaction between users and 2D applications.

[0203] In the embodiment of the present application, based on the support of the head-mounted display device for multiple types of screens, a virtual screen invisible to the user is created and the image of the 2D application is rendered on the virtual screen. In this way, the virtual screen can be used as an intermediate medium for displaying the 2D application image, and the Overlay service that draws the 2D application can obtain the image of the 2D application from the virtual screen. Since the virtual screen is different from the display screen that displays the head-mounted display device and is invisible, the intermediate display process of the 2D application on the virtual screen will not affect the user's immersive experience. On the other hand, based on the Client-Server architecture of the head-mounted display device, the Overlay service and the 3D application are two clients of the head-mounted display device, and the image synthesizer is the server of the head-mounted display device. Through ICP communication, the overlay image and the three-dimensional interface are obtained from the Overlay service and the 3D application respectively, and the overlay image is superimposed on the three-dimensional interface for binocular display, thereby realizing the simultaneous display of 2D applications and 3D applications, which can support users to access and operate 2D applications while enjoying the immersive experience, thereby enhancing the practicality and compatibility of the head-mounted display device.

[0204] like Fig.11AAs shown, it is a schematic diagram of the interactive interface in which 2D applications and 3D applications are displayed simultaneously in a video playback scenario. During the 3D video playback process, the 2D video player is started, and the user implements operations such as editing and recording of the 3D video through interactive events with the video player.

[0205] like Fig. 11B As shown, it is a schematic diagram of the interactive interface in which 2D applications and 3D applications are displayed simultaneously in a VR streaming scenario. The head-mounted display device is connected to the computer through a physical connection, and the 2D streaming assistant is started to transmit the three-dimensional model generated by the computer to the head-mounted display device for display.

[0206] Based on the same technical concept, an embodiment of the present application provides a head-mounted display device that can implement the steps of the display method of the above-mentioned 3D application and 2D application, and can achieve the same technical effect.

[0207] See also Fig.12 , the head mounted display device comprises a processor 1201, a memory 120, a display screen 1203 and a communication interface 1204, wherein the communication interface 1204, the display screen 1203, the memory 1202 and the processor 1201 are connected via a bus 1205;

[0208] The memory 1202 stores a computer program, and the processor 1201 performs the following operations according to the computer program:

[0209] Controlling the display screen 1203 to display a three-dimensional interface of the 3D application;

[0210] Receiving an input operation instruction for starting a 2D application through the communication interface 1204;

[0211] In response to the operation instruction, the Overlay service is started and a virtual screen is created, and the image of the 2D application is drawn on the virtual screen; the virtual screen is not visible to the user;

[0212] Control the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image;

[0213] The overlay image is superimposed on the three-dimensional interface to obtain a target binocular image and display it on a display screen.

[0214] Optionally, the processor 1201 controls the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image, and the specific operations are:

[0215] Initialize the graphics rendering interface and create a second texture object of the graphics rendering interface;

[0216] Controls the Overlay service to start the rendering loop, and performs the following operations each time rendering:

[0217] Capture the current image frame of the 2D application on the virtual screen;

[0218] When rendering the current image frame, obtaining the position information of the head mounted display device relative to the reference space; the position information is used for the overlay display of the overlay layer image;

[0219] Rendering the second texture object according to the current image frame, and rendering the second texture object to the swap chain component according to a binding relationship between the second texture object and the swap chain component;

[0220] Submit the swap chain components and pose information to obtain the overlay image of the current frame.

[0221] Optionally, the specific operation of the processor 1201 starting the Overlay service is:

[0222] Create and initialize an interface instance that provides operations for the Overlay service;

[0223] According to the interface instance, obtain configuration parameters of the operating system of the head mounted display device;

[0224] Configure the Overlay service to adapt to the operating system of the head-mounted display device according to the configuration parameters;

[0225] Create a session on the HMD's operating system to manage the lifecycle of the overlay service.

[0226] Optionally, the operation of the processor 1201 starting the Overlay service further includes:

[0227] Create a reference space for the Overlay service and set the overlay layer properties of the reference space according to the display parameters of the 2D application and the 3D scene. The overlay layer properties are used to determine the display status of the Overlay service in the 3D scene.

[0228] Optionally, the processor 1201 superimposes the cover layer image on the three-dimensional interface to obtain the target binocular image and displays it through the display screen 1203. The specific operations are:

[0229] Obtaining the spatial pose information of the head mounted display device relative to the reference space;

[0230] According to the posture information, the cover layer image is superimposed on the three-dimensional interface to obtain a fused image;

[0231] A left-eye image and a right-eye image are generated according to the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen 1203 .

[0232] Optionally, the specific operation of the processor 1201 to create a virtual screen is:

[0233] Create a media mapping object, which is used to capture the image of the 2D application on the virtual screen;

[0234] Creating a first surface texture object, where the first surface texture object is used to receive an image stream of a 2D application captured by a media mapping instance;

[0235] According to the first surface texture object, a Surface object is created, wherein the Surface object represents a drawn two-dimensional plane and serves as a bridge connecting the image stream and the rendering system;

[0236] According to the media mapping object and Surface object, call the corresponding API to create a virtual screen.

[0237] Optionally, the binocular image includes a left-eye image and a right-eye image. When the rendering frame rate of the head mounted display device is lower than the display frame rate, after rendering and displaying the binocular image on the display screen 1003, the processor 1201 further executes:

[0238] According to the current head pose, determine the predicted head pose for the next frame;

[0239] According to the predicted head posture, the left eye image and the right eye image are interpolated to obtain the left eye intermediate frame and the right eye intermediate frame;

[0240] Before generating the left eye image and the right eye image of the next frame, the left eye intermediate frame and the right eye intermediate frame are processed through the display screen 1203 .

[0241] In the embodiment of the present application, the memory 1202 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, and programs required for running an instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory 1202 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1202 may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (fllash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1202 may be any other medium that can be used to carry or store a desired computer program in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1202 may be a combination of the above memories.

[0242] The processor 1201 may include one or more central processing units (CPU), GPU or digital processing unit, etc. The processor 1201 is used to implement the steps of any of the above-mentioned display methods of 3D applications and 2D applications when calling the computer program stored in the memory 1202 .

[0243] It should be noted that Fig.12 It is only an example to provide the necessary hardware for the head mounted display device to execute the display method steps of the 3D application and 2D application provided in the embodiment of the present application. If not shown, the head mounted display device may also include conventional hardware such as a pickup, a microphone, a camera, a power supply, and a handle.

[0244] In the embodiment of the present application, the specific connection medium between the communication interface 1204, the display screen 1203, the memory 1202 and the processor 1201 is not limited. In the embodiment of the present application, the communication interface 1204 and the display screen 1203 are connected to the bus 1205 between the memory 1202 and the processor 1201. Fig.12 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1205 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 1 2 is described with only one thick line, but it does not describe that there is only one bus or one type of bus.

[0245] For the convenience of description, the head mounted display device can be divided into modules (or units) according to their functions and described separately. Of course, when implementing the present application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0246] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0247] The embodiment of the present application further provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the steps of any one of the display methods of 3D applications and 2D applications in the aforementioned embodiments.

[0248] The embodiment of the present application further provides a computer program product for storing a computer program, wherein the computer program is used to execute the steps of any one of the display methods for 3D applications and 2D applications in the aforementioned embodiments.

[0249] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0250] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0251] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0252] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0253] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for displaying 3D applications and 2D applications, characterized in that: Applied to head-mounted display devices, including: Control the display screen to display the three-dimensional interface of the 3D application; Receive an input operation instruction to start a 2D application; In response to the operation instruction, start the Overlay service and create a virtual screen, and draw the image of the 2D application on the virtual screen; the virtual screen is not visible to the user; The Overlay service is controlled to obtain the image of the 2D application from the virtual screen to obtain an overlay image; the overlay image is superimposed on the three-dimensional interface to obtain a target binocular image and display it through the display screen.

2. The method according to claim 1, characterized in that The controlling the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image includes: Initialize a graphics rendering interface and create a second texture object of the graphics rendering interface; Control the Overlay service to start the rendering loop, and perform the following operations each time rendering: Capturing a current image frame of the 2D application on the virtual screen; When rendering the current image frame, obtaining the spatial position information of the head mounted display device relative to the reference space; the position information is used for the overlay display of the overlay layer image; Rendering the second texture object according to the current image frame, and rendering the second texture object to the swap chain component according to a binding relationship between the second texture object and the swap chain component; Submit the swap chain component and the pose information to obtain the overlay image of the current frame.

3. The method according to claim 1, characterized in that The starting of the Overlay service includes: Create and initialize an interface instance that provides operations for the Overlay service; According to the interface instance, obtaining configuration parameters of the operating system of the head mounted display device; According to the configuration parameters, configure the Overlay service to adapt to the operating system of the head mounted display device; A session is created on the operating system of the head mounted display device to manage the life cycle of the Overlay service.

4. The method according to claim 3, characterized in that The starting of the Overlay service also includes: A reference space of the Overlay service is created, and overlay layer properties of the reference space are set according to display parameters of the 2D application and the three-dimensional scene, wherein the overlay layer properties are used to determine the display state of the Overlay service in the three-dimensional scene.

5. The method according to claim 1, characterized in that The step of superimposing the cover layer image on the three-dimensional interface to obtain a target binocular image and displaying the target image on the display screen includes: Acquire spatial position information of the head mounted display device relative to a reference space; According to the posture information, superimposing the cover layer image on the three-dimensional interface to obtain a fused image; A left-eye image and a right-eye image are generated according to the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

6. The method according to any one of claims 1 to 5, characterized in that The creating of the virtual screen comprises: Creating a media mapping object, wherein the media mapping object is used to capture an image of the 2D application on the virtual screen; Creating a first surface texture object, where the first surface texture object is used to receive the image stream of the 2D application captured by the media mapping instance; Creating a Surface object according to the first surface texture object, wherein the Surface object represents a drawn two-dimensional plane and serves as a bridge connecting an image stream and a rendering system; According to the media mapping object and the Surface object, a corresponding API is called to create a virtual screen.

7. The method according to any one of claims 1 to 5, characterized in that The binocular image includes a left-eye image and a right-eye image. When the rendering frame rate of the head-mounted display device is lower than the display frame rate, after rendering and displaying the binocular image on the display screen, the method further includes: According to the current head pose, determine the predicted head pose for the next frame; According to the predicted head pose, interpolating the left eye image and the right eye image respectively to obtain a left eye intermediate frame and a right eye intermediate frame; Before generating the left eye image and the right eye image of the next frame, the left eye intermediate frame and the right eye intermediate frame are processed through the display screen.

8. A head mounted display device, characterized in that: It includes a processor, a memory, a display screen and a communication interface, wherein the communication interface, the display screen, the memory and the processor are connected to the processor via a bus: The memory stores a computer program, and the processor performs the following operations according to the computer program: Controlling the display screen to display a three-dimensional interface of a 3D application; Receiving, via the communication interface, an operation instruction for starting a 2D application; In response to the operation instruction, start the Overlay service and create a virtual screen, and draw the image of the 2D application on the virtual screen; the virtual screen is not visible to the user; The Overlay service is controlled to obtain the image of the 2D application from the virtual screen to obtain an overlay image; the overlay image is superimposed on the three-dimensional interface to obtain a target binocular image and display it through the display screen.

9. The head mounted display device according to claim 8, wherein: The processor controls the Overlay service to obtain the image of the 2D application from the virtual screen to obtain the overlay image, and the specific operation is: Initialize a graphics rendering interface and create a second texture object of the graphics rendering interface; Control the Overlay service to start the rendering loop, and perform the following operations each time rendering: Capturing a current image frame of the 2D application on the virtual screen; When rendering the current image frame, obtaining the spatial position information of the head mounted display device relative to the reference space; The position information is used for superimposed display of the cover layer image; Rendering the second texture object according to the current image frame, and rendering the second texture object to the swap chain component according to a binding relationship between the second texture object and the swap chain component; Submit the swap chain component and the pose information to obtain the overlay image of the current frame.

10. The head mounted display device according to claim 8, wherein: The processor superimposes the cover layer image on the three-dimensional interface, obtains the target binocular image and displays it through the display screen, and the specific operation is: Acquire spatial position information of the head mounted display device relative to a reference space; According to the posture information, superimposing the cover layer image on the three-dimensional interface to obtain a fused image; A left-eye image and a right-eye image are generated according to the fused image, and the left-eye image and the right-eye image are rendered and displayed on the display screen.

Images