A virtual-real display synthesis method and system based on a depth mask
Patent Information
- Application Number
- CN202611115317.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-27
- Publication Date
- 2026-10-23
AI Technical Summary
[0005]本申请提供一种基于深度掩模的虚实显示合成方法及系统，旨在解决背景技术中提出的现有的虚实显示合成技术因缺乏屏幕边界的精确几何描述与独立深度参照，导致虚拟物体易穿透屏幕边界或异常显示，严重影响虚实结合准确性与真实感的问题
[0015]本申请通过在三维场景中构建与屏面对齐的薄层掩模几何体，并将其分配至独立的掩模图层，配合分通道渲染生成掩模深度纹理和前景物体深度纹理，使得系统能够以屏幕物理边界为精确参照，利用深度比较准确判定虚拟物体与屏幕的前后空间位置关系，有效避免了虚拟物体错误穿透屏幕边界或异常显示在屏幕内部的情况，极大地提升了虚实结合显示的准确性与视觉真实感。
Smart Images
Figure CN122970237A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of augmented reality display and computer graphics rendering technology, specifically a method and system for compositing virtual and real displays based on depth masks. Background Technology
[0002] In computer graphics applications such as 3D display with screen, augmented reality (AR), or naked-eye 3D perspective display, it is often necessary to overlay and render virtual 3D objects onto the physical screen or a virtual "window" area to achieve a high-quality display effect that combines virtual and real elements.
[0003] Existing virtual-real display compositing technologies primarily rely on traditional alpha blending algorithms or single-dimensional depth testing mechanisms. They calculate pixel-level transparency or depth values to fuse and present virtual content with real-world scenes. However, due to the lack of precise geometric descriptions of the screen's physical boundaries and an independent depth reference system, the system cannot accurately determine the spatial relationship between virtual objects and screen windows. This easily leads to virtual objects erroneously penetrating screen boundaries or displaying abnormally within the screen, severely impacting the accuracy and visual realism of the virtual-real display.
[0004] Therefore, this application provides a method and system for compositing virtual and real displays based on depth masks to solve the above problems. Summary of the Invention
[0005] This application provides a method and system for virtual-real display synthesis based on depth masks, aiming to solve the problem mentioned in the background art that the existing virtual-real display synthesis technology lacks accurate geometric description of the screen boundary and independent depth reference, which leads to virtual objects easily penetrating the screen boundary or displaying abnormally, seriously affecting the accuracy and realism of virtual-real combination.
[0006] To achieve the above objectives, this application provides the following technical solution: a method for compositing virtual and real displays based on a depth mask, the method comprising: Construct a thin-layer mask geometry aligned with the display screen in a 3D scene, and assign the thin-layer mask geometry to a preset mask layer; The mask layer is rendered to generate a mask depth texture, and non-environmental objects in the scene other than the environment layer are rendered to generate foreground object depth textures. Render the main color texture of visible objects in the scene other than the mask layer; The depth values in the depth texture of the foreground object are compared with the depth values in the depth texture of the mask, and the main color texture is subjected to depth testing and blending processing to generate the final composite image and output it.
[0007] Preferably, when running the programmable rendering pipeline, the generation and output of the final composite image specifically includes: Configure a full-scene color rendering workflow to generate full-scene color textures; The parameters of the mask depth texture and the non-environment depth texture are attached by the renderer feature component, and the compositing logic is executed in the rendering queue to output the final image.
[0008] Preferably, when the transparency overlay mode is disabled, the step of rendering the mask layer to generate a mask depth texture and rendering non-environment objects in the scene (excluding the environment layer) to generate foreground object depth textures specifically involves: Configure an auxiliary camera and set up a depth-replacement shader; The auxiliary camera renders only the mask layer in the first rendering path to generate a mask depth texture; The non-environment object is rendered in the second rendering pass to generate the depth texture of the foreground object.
[0009] Preferably, when the transparent overlay mode is enabled, the step of rendering the mask layer to generate a mask depth texture and rendering non-environment objects in the scene (excluding the environment layer) to generate foreground object depth textures specifically involves: Configure an auxiliary camera and set up a depth-replacement shader; The auxiliary camera renders only the mask layer in the first rendering path to generate a mask depth texture; In the second rendering pass, the mask layer and the non-environment object are rendered to generate the depth texture of the foreground object.
[0010] Furthermore, the synthesis logic employs an alpha blending shading strategy, determining the stacking order of transparent objects based on depth comparison results.
[0011] Preferably, comparing the depth value in the non-environment depth texture with the depth value in the mask depth texture specifically involves: The depth value in the non-environment depth texture is compared with the depth value in the mask depth texture. If the non-environment depth value is greater than the mask depth value, the corresponding pixel is determined to be located behind the screen and is occluded and culled.
[0012] Preferably, the final output image is specifically: At the end of the rendering frame, the rendering texture corresponding to the final composite image is bound to the display component of the graphical user interface.
[0013] Preferably, the final output image is specifically: At the end of the rendering frame, the native pointer to the rendered texture is returned to the external integration system.
[0014] A depth mask-based virtual-real display compositing system, which performs the aforementioned depth mask-based virtual-real display compositing method, includes: The mask configuration module is used to construct a thin-layer mask geometry aligned with the display screen in a 3D scene and assign the thin-layer mask geometry to a preset mask layer; The depth rendering module is used to render the mask layer to generate a mask depth texture and to render non-environment objects in the scene other than the environment layer to generate non-environment depth textures. The color rendering module is used to render the main color texture of visible objects in the scene other than the mask layer. The composite output module is used to compare the depth value in the non-environment depth texture with the depth value in the mask depth texture, perform depth testing and blending processing on the main color texture, generate the final composite image and output it.
[0015] This application constructs a thin-layer mask geometry aligned with the screen surface in a 3D scene and assigns it to an independent mask layer. Combined with multi-channel rendering to generate mask depth texture and foreground object depth texture, the system can accurately determine the spatial relationship between virtual objects and the screen by using depth comparison with the screen physical boundary as a precise reference. This effectively avoids the situation where virtual objects erroneously penetrate the screen boundary or are abnormally displayed inside the screen, greatly improving the accuracy and visual realism of the virtual-real combination display.
[0016] This application utilizes the renderer feature components of a programmable rendering pipeline (such as URP) to mount depth parameters and execute compositing logic in the rendering queue. This allows for flexible adaptation to different graphics rendering architectures, reduces the complexity of cross-platform development, and facilitates integration into various 3D application development environments.
[0017] When the transparent overlay mode is enabled, this application generates a foreground object depth texture containing information from both the mask layer and the non-environment objects by simultaneously rendering the mask layer and the non-environment objects in the second rendering path. Combined with the alpha blending strategy, this allows the virtual objects to be naturally overlaid with the screen area based on the depth relationship, achieving a smooth and realistic transition between virtual and real.
[0018] This application supports both direct preview within the application and convenient output of processed image data to external SDKs or display devices by returning the native pointer of the rendered texture to the external integration system at the end of the rendering frame or binding the final composite image to UI components, thus meeting the needs of diverse application scenarios. Attached Figure Description
[0019] Figure 1 This is a system architecture diagram of a depth mask-based virtual-real display compositing system. Figure 2 The flowchart shows a method for compositing virtual and real displays based on depth masks. Figure 3 This is a schematic diagram showing the alignment of a planar mask with the display area. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0021] Example 1 This embodiment provides a virtual-real display compositing system based on depth masks, such as... Figure 1 As shown, the system includes a mask configuration module, a depth rendering module, a color rendering module, and a compositing output module.
[0022] The mask configuration module is used to construct a thin-layer mask geometry aligned with the display surface in a 3D scene and assign this geometry to a preset independent layer (mask layer). This mask geometry serves as a digital substitute for the screen boundary, providing accurate depth reference.
[0023] The depth rendering module employs an auxiliary rendering camera to render the mask layer into a mask depth texture and to render non-environment objects (i.e., foreground virtual objects) in the scene, excluding the environment layer (usually the skybox or background), into non-environment depth textures. The mask depth texture records the depth information of the screen boundaries.
[0024] The color rendering module uses a composite camera to render visible objects in the scene other than the mask layer to generate the main color texture. The final image content does not include the color of the mask geometry itself, but only retains the real scene visual information.
[0025] The compositing output module employs a compositing shader to compare the depth values in the non-environment depth texture with the depth values in the mask depth texture, and performs depth testing and blending processing on the main color texture to generate the final composite image and output it to a display device or external system.
[0026] In practical use, such as Figure 2 As shown, the system performs the following steps: S1. Construct a thin-layer mask geometry aligned with the display screen in the 3D scene, and assign the thin-layer mask geometry to a preset mask layer.
[0027] Specifically, in a 3D scene, a thin mask geometry is created that is aligned with the physical or logical display screen.
[0028] like Figure 3 As shown, the thin-layer mask geometry is perfectly aligned with the screen surface in both position and orientation, and its thickness dimension is much smaller than the screen's length and width to ensure the accuracy of depth testing. This mask geometry is assigned to a pre-defined, independent mask layer to facilitate subsequent layered rendering control.
[0029] S2. Render the mask layer to generate a mask depth texture, and render non-environment objects in the scene other than the environment layer to generate foreground object depth textures.
[0030] When disabling the transparent overlay mode, the specific steps are as follows: S21. Configure an auxiliary camera and set a depth replacement shader to ignore color output and focus only on depth information; S22. The auxiliary camera renders only the mask layer in the first rendering path to generate a mask depth texture; the mask depth texture records the depth information of the screen boundary.
[0031] S23. Render the non-environment object in the second rendering path to generate the depth texture of the foreground object; the depth texture of the foreground object records the depth information of the virtual object.
[0032] S3. Using a composite camera, render the visible objects in the scene other than the mask layer to generate the main color texture; ensuring that the final image does not contain the color of the mask geometry itself, but only retains the scene content.
[0033] S4. Compare the depth value in the depth texture of the foreground object with the depth value in the depth texture of the mask, and perform depth testing and blending processing on the main color texture to generate and output the final composite image; Specifically, the following steps are performed in the composition shader: S41. Bind the main color texture to the foreground object depth texture; during the pixel shader stage, set the foreground object depth value... With mask depth value Comparison: If the judgment > If the virtual object is located behind the screen, a culling operation is performed to make the pixel transparent. If the judgment < If the virtual object is in front of the screen, then the color of that pixel is preserved. The final result is a synthetic image with correct depth relationships.
[0034] S42. At the end of the rendering frame, the rendering texture corresponding to the composite image is bound to the display component of the graphical user interface, so that the developer can preview it directly in the editor or at runtime.
[0035] Example 2 This embodiment provides a virtual-real display compositing system based on depth mask. The difference between this system and Embodiment 1 is that when the operating environment is a programmable rendering pipeline, the system is configured with a full-scene color rendering process.
[0036] First, the entire scene color is rendered to the entire scene color texture. Then, by attaching a renderer feature component, the parameters of the mask depth texture and the foreground object depth texture are injected into the rendering queue. The compositing logic is executed using shader variants, thereby efficiently outputting the final image.
[0037] Example 3 This embodiment provides a virtual-real display compositing system based on depth masks. Unlike Embodiment 2, this compositing system, when enabling transparent overlay mode, in step S2, renders the mask layer to generate a mask depth texture and renders non-environment objects (excluding the environment layer) in the scene to generate foreground object depth textures. Specifically: S21. Configure an auxiliary camera and set a depth replacement shader to ignore color output and focus only on depth information; S22. The auxiliary camera renders only the mask layer in the first rendering path to generate a mask depth texture; the mask depth texture records the depth information of the screen boundary.
[0038] S23. Render the mask layer and the non-environment object in the second rendering path to generate the depth texture of the foreground object; the depth texture of the foreground object records the depth information of the screen boundary and the virtual object.
[0039] In this mode, during the compositing stage, the compositing logic of the compositing shader adopts an alpha blending shading strategy. Based on the depth comparison results in the depth texture, since the mask information is embedded in the depth map, the system can accurately calculate the superposition order of transparent objects and screen areas based on the subtle differences in depth values. For areas with similar depth values, an interpolation algorithm is used to calculate the transparency weight, so that virtual objects can present a semi-transparent effect through the screen edge, achieving a natural transition between virtual and real.
[0040] Example 4 This embodiment provides a virtual-real display compositing system based on depth mask. The compositing system differs from Embodiments 1, 2 and 3 in that, in step S4, the depth value in the depth texture of the foreground object is compared with the depth value in the depth texture of the mask, and the main color texture is subjected to depth testing and blending processing to generate and output the final composite image. Specifically, the following steps are performed in the composition shader: S41. Bind the main color texture to the foreground object depth texture; during the pixel shader stage, set the foreground object depth value... With mask depth value Comparison: If the judgment > If the virtual object is located behind the screen, a culling operation is performed to make the pixel transparent. If the judgment < If the virtual object is in front of the screen, then the color of that pixel is preserved. The final result is a synthetic image with correct depth relationships.
[0041] S42. At the end of the rendering frame, return the native pointer of the rendered texture to an external integrated system (such as a video encoder, external display SDK, or operating system window manager) for further processing or full-screen output.
[0042] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and concept of this application, should be included within the scope of protection of this application.
Claims
1. A method for compositing virtual and real displays based on depth masks, characterized in that, The synthesis method includes: Construct a thin-layer mask geometry aligned with the display screen in a 3D scene, and assign the thin-layer mask geometry to a preset mask layer; The mask layer is rendered to generate a mask depth texture, and non-environmental objects in the scene other than the environment layer are rendered to generate foreground object depth textures. Render the main color texture of visible objects in the scene other than the mask layer; The depth values in the depth texture of the foreground object are compared with the depth values in the depth texture of the mask, and the main color texture is subjected to depth testing and blending processing to generate the final composite image and output it.
2. The method for compositing virtual and real displays based on depth masks according to claim 1, characterized in that, When running the programmable rendering pipeline, the generation and output of the final composite image specifically includes: Configure a full-scene color rendering workflow to generate full-scene color textures; The parameters of the mask depth texture and the non-environment depth texture are attached by the renderer feature component, and the compositing logic is executed in the rendering queue to output the final image.
3. The method for compositing virtual and real displays based on depth masks according to claim 2, characterized in that, When the transparent overlay mode is disabled, the process of rendering the mask layer to generate a mask depth texture and rendering non-environment objects in the scene (excluding the environment layer) to generate foreground object depth textures specifically involves: Configure an auxiliary camera and set up a depth-replacement shader; The auxiliary camera renders only the mask layer in the first rendering path to generate a mask depth texture; The non-environment object is rendered in the second rendering pass to generate the depth texture of the foreground object.
4. The method for compositing virtual and real displays based on depth masks according to claim 2, characterized in that, When the transparent overlay mode is enabled, the process of rendering the mask layer to generate a mask depth texture and rendering non-environment objects in the scene (excluding the environment layer) to generate foreground object depth textures specifically involves: Configure an auxiliary camera and set up a depth-replacement shader; The auxiliary camera renders only the mask layer in the first rendering path to generate a mask depth texture; In the second rendering pass, the mask layer and the non-environment object are rendered to generate the depth texture of the foreground object.
5. The method for compositing virtual and real displays based on depth masks according to claim 4, characterized in that, The synthesis logic employs an alpha blending shading strategy, determining the stacking order of transparent objects based on depth comparison results.
6. The method for compositing virtual and real displays based on a depth mask according to claim 3 or 4, characterized in that, The comparison of the depth value in the non-environment depth texture with the depth value in the mask depth texture specifically involves: The depth value in the non-environment depth texture is compared with the depth value in the mask depth texture. If the non-environment depth value is greater than the mask depth value, the corresponding pixel is determined to be located behind the screen and is occluded and culled.
7. The method for compositing virtual and real displays based on depth masks according to claim 1, characterized in that, The final output image is specifically as follows: At the end of the rendering frame, the rendering texture corresponding to the final composite image is bound to the display component of the graphical user interface.
8. The method for compositing virtual and real displays based on depth masks according to claim 1, characterized in that, The final output image is specifically as follows: At the end of the rendering frame, the native pointer to the rendered texture is returned to the external integration system.
9. A virtual-real display compositing system based on depth mask, characterized in that, The system is used to execute the virtual-real display synthesis method based on depth mask as described in any one of claims 1-8, including: The mask configuration module is used to construct a thin-layer mask geometry aligned with the display screen in a 3D scene and assign the thin-layer mask geometry to a preset mask layer; The depth rendering module is used to render the mask layer to generate a mask depth texture and to render non-environment objects in the scene other than the environment layer to generate non-environment depth textures. The color rendering module is used to render the main color texture of visible objects in the scene other than the mask layer. The composite output module is used to compare the depth value in the non-environment depth texture with the depth value in the mask depth texture, perform depth testing and blending processing on the main color texture, generate the final composite image and output it.