A binocular fusion display method based on virtual display and independent layer synthesis

Abstract

This invention provides a binocular fusion display method based on virtual display and independent layer synthesis, comprising the following steps: creating a virtual display screen at the Android application framework layer to carry the target application's display content, and redirecting the target application to the virtual display screen to form an application display layer; constructing an offline rendering system for augmented reality user interfaces, rendering system-level user interface components as augmented reality user interface layers; uniformly managing the application display layer and the augmented reality user interface layer through a layer synthesis service, and synthesizing the output image according to a preset hierarchical order; generating left and right eye binocular display images through a rendering engine and outputting them to a mask surface covering the main physical display screen; simultaneously performing coordinate mapping and input focus determination for input events. This invention enables unified binocular display and interaction of 2D applications and system-level user interface components without modifying the target application and the system's underlying graphics synthesizer.

Description

Technical Field

[0001] This invention relates to the field of AR / VR display technology, and in particular to a binocular fusion display method based on virtual display and independent layer synthesis. Background Technology

[0002] With the development of augmented reality and virtual reality display devices, Android-based binocular AR / VR glasses are increasingly being used in scenarios such as information display, assisted interaction, mobile office, and immersive entertainment. These devices typically include two optical display modules, one for the left eye and one for the right eye, and output display images to the left and right eye display modules respectively via dual display interfaces. To ensure clear and stable visual perception for the user, the left and right eye display areas usually need to receive complete, mutually matching images, or corresponding images conforming to binocular display rules.

[0003] In existing dual-view display devices based on the Android system, the display output from the system's underlying graphics synthesizer often uses a horizontally spliced ​​logical resolution. For example, the left-eye display area and the right-eye display area logically occupy a portion of the display buffer. When a typical Android 2D application directly renders in full-screen mode according to this logical resolution, the output frame buffer image may be mechanically split into left and right parts by the display processing unit and sent to the left-eye display path and the right-eye display path respectively. At this time, the left and right eyes do not receive two identical or matching complete images, but rather different fragments of the same application image. This makes it difficult for the human eye to fuse the left and right eye images into a clear, single visual perception, easily leading to problems such as display fragmentation, visual fatigue, and dizziness.

[0004] To address the adaptation issues of standard Android 2D applications on binocular display devices, existing technologies typically employ a virtual display redirection scheme. This scheme creates an independent virtual display output channel for the target application through the virtual display interface provided by the Android system, and outputs the rendered content of the target application to the corresponding graphics surface of the virtual display. Subsequently, the graphics rendering interface captures, performs texture conversion, and performs left-right copying or mapping processing on the image in the virtual display, thereby generating left-eye and right-eye images suitable for output on binocular display devices.

[0005] However, existing virtual display redirection solutions are primarily suitable for displaying content in regular application windows. For system-level user interface components such as status bars, dialog boxes, notifications, and tooltips, they are typically not entirely dependent on the target application's Activity window, but are managed independently by the Android native window management system or system user interface services. Therefore, when creating a virtual display only for the target application, these system-level user interface components are difficult to capture completely and display in a dual-view split-screen manner along with the target application's screen.

[0006] To further support system-level user interface components, existing technologies also employ a method of creating separate virtual displays for different types of user interface components. For example, separate virtual display output channels are established for status bars, dialog boxes, notifications, or prompts. The graphics compositing module then captures the content of each virtual display and overlays it onto the final binocular display. While this method can expand the coverage of binocular display content to some extent, because each virtual display corresponds to an independent display compositing link, when there are many system-level user interface components, multiple virtual display instances need to be maintained simultaneously. This significantly increases the compositing load of the underlying graphics compositor, the bandwidth usage of the graphics processor, and memory transfer overhead, leading to decreased frame rates, screen stuttering, and increased device power consumption.

[0007] Furthermore, multi-virtual display architecture increases the complexity of window management, hierarchy management, and input event routing. Different user interface components reside in different display contexts, and their occlusion relationships, hierarchy order, and display priorities require maintenance through additional cross-display session coordination logic. When multiple application windows or system-level user interface components exist simultaneously, issues such as layer order disorder, abnormal display occlusion, loss of input focus, unresponsive clicks, or incorrect input event distribution can easily occur, affecting the interaction stability and user experience of binocular AR / VR devices. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a binocular fusion display method based on virtual display and independent layer synthesis. This method addresses the problem of how to enable ordinary Android 2D applications, as well as system-level user interface components such as status bars, dialog boxes, notifications, and prompts, to be uniformly and stably displayed and interacted with in binocular AR / VR devices without significantly increasing the number of virtual displays.

[0009] To achieve the above objectives, in a first aspect, the present invention provides: a binocular fusion display method based on virtual display and independent layer synthesis, comprising the following steps: A virtual display screen is created at the Android application framework layer to carry the content displayed by the target application, and the Activity instance of the target application is redirected to the virtual display screen, so that the graphics rendering output of the target application forms an application display layer that is separate from the main physical display screen; Construct an offline rendering system for augmented reality user interfaces that is independent of Android's native WindowManager. Perform offline measurement, layout, and drawing of system-level user interface components, and output the drawn pixel data to a graphic surface bound to the augmented reality user interface layer. The application display layer and the augmented reality user interface layer are managed uniformly by a layer compositing service that works in the Android application framework layer, and the visible layers are sorted and composited according to a preset layer order to obtain the composited image to be output. The rendering engine performs binocular split-screen rendering on the composite image to be output, generating binocular display images corresponding to the left eye display area and the right eye display area respectively, and outputs the binocular display images to the mask surface covering the main physical display screen; The system acquires input events and determines the target application or augmented reality user interface component corresponding to the input event based on the coordinates of the input event and the current input focus. The processed input event is then distributed to the corresponding target application or augmented reality user interface component.

[0010] According to one aspect of the above technical solution, the steps of creating a virtual display screen at the Android application framework layer to carry the content displayed by the target application, and redirecting the Activity instance of the target application to the virtual display screen specifically include: A virtual display screen dedicated to binocular display is dynamically created during system initialization; Register a launch interceptor with the Android system's activity management service; When an Activity launch request from the target application is detected, the launch interceptor directs the Activity instance to the virtual display screen for launch, so that the rendered content of the target application is output to the display buffer corresponding to the virtual display screen.

[0011] According to one aspect of the above technical solution, the step of outputting the binocular display image to the mask surface covering the main physical display screen specifically includes: A mask surface is created residing at the top of the display layer through a surface control interface, and the mask surface covers the main physical display screen; When the application display layer or the augmented reality user interface layer undergoes content updates, the layer composition service captures the updated layer content and converts the layer content into a graphics processor texture. The rendering engine generates the binocular display image based on the graphics processor texture, and renders the binocular display image onto the mask surface in real time.

[0012] According to one aspect of the above technical solution, the specific steps for constructing an offline rendering system for augmented reality user interfaces independent of the Android native WindowManager include: Set the root view of the augmented reality user interface component as the augmented reality view container; The root view and its subviews are measured, laid out, and drawn using the augmented reality view root implementation module; The pixel data generated by the drawing is output to a graphics surface bound to the augmented reality user interface layer, where each augmented reality user interface layer corresponds to a logical display layer.

[0013] According to one aspect of the above technical solution, the system-level user interface component includes at least one of a status bar, a dialog box, a notification, and a prompt box; The system-level user interface component accesses the layer compositing service through the corresponding augmented reality user interface layer, so that the system-level user interface component is overlaid with the application display layer in the binocular field of view.

[0014] According to one aspect of the above technical solution, the augmented reality user interface offline rendering system includes an augmented reality user interface manager; The augmented reality user interface manager is used to create corresponding system-level user interface components according to the calling instructions of the business layer, and generate corresponding augmented reality user interface layers for the system-level user interface components, so that the augmented reality user interface layers are connected to the layer compositing service for unified management.

[0015] According to one aspect of the above technical solution, the layer compositing service includes a layer compositor and a vertical synchronization scheduler; The layer compositor is used to manage the lifecycle of the application display layer that carries the content of the virtual display screen and the augmented reality user interface layer that carries the augmented reality user interface components; The vertical synchronization scheduler is used to monitor the vertical synchronization signal and trigger the layer compositor to composite the visible layers in each display refresh cycle.

[0016] According to one aspect of the above technical solution, the steps of sorting and compositing visible layers according to a preset layer order specifically include: Obtain the layer order information and visibility status information of each visible layer; The occlusion relationship between the application display layer and each augmented reality user interface layer is determined based on the hierarchical order information. The application display layer and each augmented reality user interface layer are combined according to the occlusion relationship to form the composite image to be output.

[0017] According to one aspect of the above technical solution, the step of performing binocular split-screen rendering on the composite image to be output using a rendering engine specifically includes: The rendering engine merges the sorted visible layers through a graphics rendering interface, which includes an OpenGL ES interface or a Vulkan interface. The merged image is copied or mapped according to the left and right eye display areas of the binocular display device to obtain the left eye image and the right eye image; The left-eye image and the right-eye image are output to the mask surface according to the physical display path correspondence of the binocular display device.

[0018] According to one aspect of the above technical solution, the steps of acquiring an input event, determining the target application or augmented reality user interface component corresponding to the input event based on the coordinate position of the input event and the current input focus, and distributing the processed input event to the corresponding target application or augmented reality user interface component specifically include: The system receives raw touch events and key events through the input event management module; The touch coordinates occurring in the main physical display coordinate system are mapped to the logical coordinate system of the virtual display screen, or to the logical coordinate system of the layer where the corresponding augmented reality user interface component is located, according to the proportional relationship between the virtual display screen and the main physical display screen. The window task management module listens for window state change events and physical button trigger events, and maintains the input focus between the virtual display screen, the target application, and the augmented reality user interface components.

[0019] Compared with existing technologies, the advantages of this invention are as follows: By creating a virtual display screen at the Android application framework layer and redirecting the target application to this virtual display screen, ordinary Android 2D applications can form an application display layer suitable for binocular display without modification, improving the compatibility of existing applications in AR / VR glasses; at the same time, by constructing an offline rendering system for augmented reality user interfaces independent of the Android native window manager, system-level user interface components such as status bars, dialog boxes, notifications, and prompts are converted into augmented reality user interface layers, and these layers are uniformly composited together with the application display layer by a layer compositing service according to a preset hierarchical order, avoiding the compositing chain caused by creating multiple virtual displays for different system-level user interface components in existing technologies. The system addresses the issues of complex routing, high system overhead, and high power consumption. Furthermore, by using a rendering engine to perform binocular split-screen rendering of the composited image and outputting it to a mask surface covering the main physical display, the left and right eye display areas obtain mutually matching display images. This improves upon the difficulties in binocular fusion, visual fragmentation, and dizziness caused by the mechanical splitting of traditional images. In addition, by performing coordinate mapping and input focus determination on input events, the processed input events can be accurately distributed to the corresponding target applications or augmented reality user interface components, improving the accuracy and stability of interaction in a binocular display environment. Moreover, the above processing is mainly implemented at the Android application framework layer, without requiring intrusive modifications to the underlying system graphics compositor, which helps reduce the difficulty of system modification and improves subsequent maintenance and expansion capabilities. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the binocular fusion display method based on virtual display and independent layer synthesis in the first embodiment of the present invention. The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0021] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention is implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0023] Example 1 The first embodiment of this solution provides a binocular fusion display method based on virtual display and independent layer synthesis, applied to a binocular AR / VR display device based on the Android system. This binocular AR / VR display device typically includes display modules corresponding to the left and right eye display areas respectively. Logically, the main physical display screen can form a horizontally spliced ​​display area, so that the left and right eye display areas each occupy a portion of the logical screen of the main physical display screen.

[0024] In this embodiment, the binocular split-screen display software framework includes a virtual display redirection module, an augmented reality user interface offline rendering module, a layer compositing service, a rendering engine, an input event management module, and a window task management module. Specifically, the virtual display redirection module directs the display output of the target application to a virtual display screen; the augmented reality user interface offline rendering module performs offline rendering of system-level user interface components such as status bars, dialog boxes, notifications, and prompts; the layer compositing service uniformly manages application display layers and augmented reality user interface layers; the rendering engine generates left-eye and right-eye images; and the input event management module and window task management module handle input event distribution and input focus maintenance.

[0025] Please see Figure 1 The figure shows a flowchart of a binocular fusion display method based on virtual display and independent layer synthesis in the first embodiment of the present invention. As shown in the figure, the method includes the following steps: Step S100: Create a virtual display screen in the Android application framework layer to carry the content displayed by the target application, and redirect the Activity instance of the target application to the virtual display screen, so that the graphics rendering output of the target application forms an application display layer that is separate from the main physical display screen.

[0026] Preferably, when the system starts or the binocular display mode is activated, the binocular split-screen display software framework dynamically creates a virtual display screen dedicated to binocular display. The virtual display screen can be a VirtualDisplay provided by the Android system, which has preset logical resolution, refresh rate, and pixel format.

[0027] Furthermore, the dual-camera split-screen display software framework registers a launch interceptor with the Android system's activity management service. When a launch request for the target application's Activity is detected, the launch interceptor intercepts the Activity launch request and redirects the Activity instance to the virtual display screen for launch. In this way, the target application still renders as a normal Android 2D application, but its rendered content is not directly output to the main physical display screen; instead, it is output to the display buffer or graphics surface corresponding to the virtual display screen. Thus, the target application's graphics rendering output forms an application display layer separate from the main physical display screen.

[0028] For example, when a user launches a map application, video application, or office application, the activity management service receives an Activity launch request. After the launch interceptor identifies that the application is a target application requiring binocular display, it switches the launch target from the main physical display screen to the virtual display screen. The target application perceives a normal display environment, therefore, no modification to the target application's source code is required to redirect the application content output.

[0029] Step S200: Construct an offline rendering system for augmented reality user interface that is independent of the native Android WindowManager, perform offline measurement, layout and drawing of system-level user interface components, and output the drawn pixel data to the graphic surface bound to the augmented reality user interface layer.

[0030] Preferably, the augmented reality user interface offline rendering system does not rely on the Android native WindowManager to directly create windows. Instead, it achieves offline rendering of system-level user interface components through an augmented reality view container, an augmented reality view root implementation module, and an augmented reality user interface layer. The augmented reality view container can be specifically implemented as ARViewGroup, the augmented reality view root implementation module can be specifically implemented as ARViewRootImpl, and the augmented reality user interface layer can be specifically implemented as ARLayer.

[0031] Specifically, when a system-level user interface component needs to be displayed, the corresponding augmented reality user interface component is first created, and its root view is set as the augmented reality view container. The augmented reality view root implementation module takes over the measurement, layout, and drawing processes of the root view and its subviews. The measurement process is used to determine the width, height, and display boundaries of the augmented reality user interface component; the layout process is used to determine the relative positions of each sub-control within the augmented reality user interface component; and the drawing process is used to convert the augmented reality user interface component into pixel data.

[0032] Subsequently, the pixel data generated is output to the graphics surface bound to the augmented reality user interface layer. Each augmented reality user interface layer corresponds to a logical display layer and has layer identifier, size information, display position, visibility status, and hierarchy order information. As a result, system-level user interface components such as status bars, dialog boxes, notifications, and prompts no longer rely on multiple independent virtual displays for separate display, but are instead integrated into a unified layer composition system in the form of augmented reality user interface layers.

[0033] Specifically, the system-level user interface components include at least one of a status bar, dialog boxes, notifications, and alert boxes. Dialog boxes can be created using a dialog box manager, notifications can be created using a notification manager, and alert boxes can be created using an alert box manager. The dialog box manager can be specifically implemented as ARDIallogManager, the notification manager can be specifically implemented as ARNotificationManager, and the alert box manager can be specifically implemented as ARToastManager. The status bar can access a layer compositing service through the corresponding augmented reality user interface layer.

[0034] When the business layer needs to display a dialog box, it can call the interface provided by the dialog box manager, passing in parameters such as the dialog box title, body content, button content, display position, and interaction strategy. The dialog box manager creates the corresponding augmented reality user interface component based on the business layer's call instructions and generates a corresponding augmented reality user interface layer for that component. This augmented reality user interface layer is then registered with the layer compositing service and overlaid on the application display layer in the binocular field of view. The creation process for notifications and alerts is similar to that of dialog boxes and will not be described further here.

[0035] Step S300: The application display layer and the augmented reality user interface layer are uniformly managed by the layer compositing service working in the Android application framework layer, and the visible layers are sorted and composited according to the preset layer order to obtain the composite screen to be output.

[0036] Preferably, the layer compositing service includes a layer compositor and a vertical synchronization scheduler. The layer compositor can be specifically implemented as ARLayerFlinger, and the vertical synchronization scheduler can be specifically implemented as ARVsyncScheduler. The layer compositor is used to uniformly manage the lifecycle of all layers to be displayed within the system. These layers include application display layers that carry virtual display screen content, and augmented reality user interface layers that carry augmented reality user interface components.

[0037] The layer compositor can maintain a layer list. This list records each layer's layer identifier, layer type, graphic surface, layer order information, visibility status information, display area information, and update status information. Layer types can include application display layers, status bar layers, dialog box layers, notification layers, and tooltip layers, etc. Layer order information represents the hierarchical relationship between different layers and may also include Z-order information. Visibility status information represents whether a layer needs to participate in the compositor's composition for the current display refresh cycle.

[0038] The vertical synchronization scheduler monitors the system's vertical synchronization signal. Upon entering a display refresh cycle, the vertical synchronization scheduler triggers the layer compositor to execute the compositing process. The layer compositor first iterates through the layer list, selecting layers that are visible; then it obtains the layer order and visibility information of each visible layer; next, it determines the occlusion relationship between the application display layer and each augmented reality user interface layer based on the layer order information; finally, it composites the application display layer and each augmented reality user interface layer according to the occlusion relationship to obtain the final composite image to be output.

[0039] For example, the application display layer can be located at the bottom, the status bar layer can be above the application display layer, the dialog box layer can be above the status bar layer, and the tooltip layer can be above or below the dialog box layer depending on the display priority. When a dialog box is displayed, the dialog box layer covers part of the application display layer; when a tooltip appears, the tooltip layer can cover the application display layer or dialog box layer for a preset time. In this way, the hierarchical relationship between each system-level user interface component and the target application can be determined within the same display context, avoiding the confusion of occlusion relationships caused by cross-display session coordination between multiple virtual displays.

[0040] Step S400: The rendering engine performs binocular split-screen rendering on the composite image to be output, generating binocular display images corresponding to the left eye display area and the right eye display area respectively, and outputs the binocular display images to the mask surface covering the main physical display screen.

[0041] Preferably, the binocular split-screen display software framework creates a mask surface that resides at the top of the display layer through a surface control interface, and makes the mask surface cover the main physical display screen. The surface control interface can specifically be SurfaceControl in the Android system, and the mask surface can specifically be MaskSurface. The mask surface is used to receive the binocular display image output by the rendering engine and cover the original output area of ​​the main physical display screen.

[0042] When the content of an application's display layer or augmented reality user interface layer is updated, the layer compositing service captures the updated layer content and converts it into a graphics processor texture. The graphics processor texture can be a texture object used by the OpenGL ES interface or the Vulkan interface. The rendering engine blends the sorted visible layers through a graphics rendering interface, including the OpenGL ES interface or the Vulkan interface.

[0043] Specifically, the rendering engine first receives the texture data corresponding to the composite image to be output, and then copies or maps it according to the left and right eye display areas of the binocular display device to obtain the left and right eye images. For example, when the target application needs to display the same image in binocular fusion, the rendering engine copies the composite image to be output into two copies and maps them to the left and right eye display areas respectively.

[0044] After generating the left-eye and right-eye images, the rendering engine outputs them to the mask surface according to the physical display path mapping of the binocular display device. For devices using horizontally spliced ​​display areas, the left-eye image can be written to the left display area of ​​the mask surface, and the right-eye image can be written to the right display area of ​​the mask surface. For other display layouts, the left-eye and right-eye images can also be written to the corresponding areas according to the physical display path mapping of the device.

[0045] Through the above processing, the complete screen of the target application and the system-level user interface components can be uniformly synthesized into a screen suitable for binocular display, instead of being mechanically split into discontinuous left and right segments by the main physical display screen, thereby improving the binocular fusion display effect.

[0046] Step S500: Obtain the input event, and determine the target application or augmented reality user interface component corresponding to the input event based on the coordinate position of the input event and the current input focus, and distribute the processed input event to the corresponding target application or augmented reality user interface component.

[0047] Preferably, the input event management module receives raw system touch events and key events. The raw system touch events may include press events, move events, and release events, and the key events may include physical button events, gamepad button events, or key events generated by other external input devices. The input event management module can be specifically implemented as ARInputManager.

[0048] For touch events, the input event management module obtains the touch coordinates of the touch event in the coordinate system of the main physical display screen, and performs coordinate mapping according to the ratio between the virtual display screen and the main physical display screen.

[0049] When a touch event falls within the display area of ​​the layer containing the augmented reality user interface component, the input event management module maps the touch coordinates to the logical coordinate system of the corresponding layer and distributes the mapped input event to that component. For example, when a dialog box layer is visible and a touch event falls within the dialog box button area, the input event management module distributes the touch event to the dialog box component; when a touch event does not fall within any interactive augmented reality user interface layer, the input event management module maps the touch event to the virtual display's logical coordinate system and distributes it to the current target application.

[0050] Preferably, the window task management module listens for window state change events and physical button trigger events, and maintains the input focus between the virtual display screen, the target application, and the augmented reality user interface components. The window task management module can be specifically implemented as ARWindowTaskManager. Window state change events can include events such as target application startup, target application switching, target application exit, augmented reality user interface component display, and augmented reality user interface component closing.

[0051] In this embodiment, since the target application's display content forms an application display layer through a virtual display screen, and the system-level user interface components form an augmented reality user interface layer through an augmented reality user interface offline rendering system, and the application display layer and the augmented reality user interface layer are uniformly sorted and composited through the same layer compositing service, it is unnecessary to create multiple virtual display screens separately for system-level user interface components such as status bars, dialog boxes, notifications, and prompts. This reduces the number of independent display compositing links, lowers the compositing load of the underlying graphics compositor and the graphics processor resource consumption, and improves the stability of input event routing and input focus management.

[0052] In this embodiment, the surface control interface can be SurfaceControl, the mask surface can be MaskSurface, the augmented reality view container can be ARViewGroup, the augmented reality view root implementation module can be ARViewRootImpl, the augmented reality user interface layer can be ARLayer, the layer compositor can be ARLayerFlinger, the vertical synchronization scheduler can be ARVsyncScheduler, the rendering engine can be ARRenderEngine, the input event management module can be ARInputManager, and the window task management module can be ARWindowTaskManager. The above English names are only used to describe a specific software implementation method and do not constitute a limitation on the corresponding functional modules.

[0053] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A binocular fusion display method based on virtual display and independent layer synthesis, characterized in that, Includes the following steps: A virtual display screen is created at the Android application framework layer to carry the content displayed by the target application, and the Activity instance of the target application is redirected to the virtual display screen, so that the graphics rendering output of the target application forms an application display layer that is separate from the main physical display screen; Construct an offline rendering system for augmented reality user interfaces that is independent of Android's native WindowManager. Perform offline measurement, layout and drawing of system-level user interface components, and output the drawn pixel data to the graphic surface bound to the augmented reality user interface layer. The application display layer and the augmented reality user interface layer are managed uniformly by a layer compositing service that works in the Android application framework layer, and the visible layers are sorted and composited according to a preset layer order to obtain the composited image to be output. The rendering engine performs binocular split-screen rendering on the composite image to be output, generating binocular display images corresponding to the left eye display area and the right eye display area respectively, and outputs the binocular display images to the mask surface covering the main physical display screen; The system acquires input events and determines the target application or augmented reality user interface component corresponding to the input event based on the coordinates of the input event and the current input focus. The processed input event is then distributed to the corresponding target application or augmented reality user interface component.

2. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 1, characterized in that, The steps of creating a virtual display screen at the Android application framework layer to host the content displayed by the target application, and redirecting the Activity instance of the target application to the virtual display screen specifically include: A virtual display screen dedicated to binocular display is dynamically created during system initialization; Register a launch interceptor with the Android system's activity management service; When an Activity launch request from the target application is detected, the launch interceptor directs the Activity instance to the virtual display screen for launch, so that the rendered content of the target application is output to the display buffer corresponding to the virtual display screen.

3. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 1, characterized in that, The specific steps of outputting the binocular display image to the mask surface covering the main physical display screen include: A mask surface is created residing at the top of the display layer through a surface control interface, and the mask surface covers the main physical display screen; When the application display layer or the augmented reality user interface layer undergoes content updates, the layer compositing service captures the updated layer content and converts the layer content into a graphics processor texture. The rendering engine generates the binocular display image based on the graphics processor texture, and renders the binocular display image onto the mask surface in real time.

4. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 1, characterized in that, The specific steps for building an offline rendering system for augmented reality user interfaces that is independent of Android's native WindowManager include: Set the root view of the augmented reality user interface component as the augmented reality view container; The root view and its subviews are measured, laid out, and drawn using the augmented reality view root implementation module; The pixel data generated by the drawing is output to a graphics surface bound to the augmented reality user interface layer, where each augmented reality user interface layer corresponds to a logical display layer.

5. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 4, characterized in that, The system-level user interface components include at least one of a status bar, dialog box, notification, and prompt box; The system-level user interface component accesses the layer compositing service through the corresponding augmented reality user interface layer, so that the system-level user interface component is overlaid with the application display layer in the binocular field of view.

6. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 5, characterized in that, The augmented reality user interface offline rendering system includes an augmented reality user interface manager; The augmented reality user interface manager is used to create corresponding system-level user interface components according to the calling instructions of the business layer, and generate corresponding augmented reality user interface layers for the system-level user interface components, so that the augmented reality user interface layers are connected to the layer compositing service for unified management.

7. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 1, characterized in that, The layer compositing service includes a layer compositer and a vertical synchronization scheduler; The layer compositor is used to manage the lifecycle of the application display layer that carries the content of the virtual display screen and the augmented reality user interface layer that carries the augmented reality user interface components; The vertical synchronization scheduler is used to monitor the vertical synchronization signal and trigger the layer compositor to composite the visible layers in each display refresh cycle.

8. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 1, characterized in that, The specific steps for sorting and compositing visible layers according to a preset layer order include: Obtain the layer order information and visibility status information of each visible layer; The occlusion relationship between the application display layer and each augmented reality user interface layer is determined based on the hierarchical order information. The application display layer and each augmented reality user interface layer are combined according to the occlusion relationship to form the composite image to be output.

9. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 1, characterized in that, The specific steps of performing binocular split-screen rendering of the composite image to be output using a rendering engine include: The rendering engine merges the sorted visible layers through a graphics rendering interface, which includes an OpenGL ES interface or a Vulkan interface. The merged image is copied or mapped according to the left and right eye display areas of the binocular display device to obtain the left eye image and the right eye image; The left-eye image and the right-eye image are output to the mask surface according to the physical display path correspondence of the binocular display device.

10. The binocular fusion display method based on virtual display and independent layer synthesis according to claim 1, characterized in that, The steps of acquiring an input event, determining the target application or augmented reality user interface component corresponding to the input event based on the coordinate position of the input event and the current input focus, and distributing the processed input event to the corresponding target application or augmented reality user interface component specifically include: The system receives raw touch events and key events through the input event management module; The touch coordinates occurring in the main physical display coordinate system are mapped to the logical coordinate system of the virtual display screen, or to the logical coordinate system of the layer where the corresponding augmented reality user interface component is located, according to the proportional relationship between the virtual display screen and the main physical display screen. The window task management module listens for window state change events and physical button trigger events, and maintains the input focus between the virtual display screen, the target application, and the augmented reality user interface components.

Images