Method, device and equipment for shielding virtual object by real object in mixed reality

Abstract

The invention discloses a method, device and equipment for shielding a virtual object by a real object in mixed reality, and the method comprises the steps: scanning a real environment through a three-dimensional perception device, constructing a virtual three-dimensional model matched with the real environment, and enabling the virtual three-dimensional model to be completely aligned with a real environment coordinate system; creating a corresponding virtual proxy object for the target object needing to shield the virtual object in the real environment, and configuring a preset material for the virtual proxy object; the preset material comprises the following steps: starting a depth test function to participate in depth buffer calculation; disabling the color channel output function; and based on a GPU shader program, comparing the depth value of the virtual object with the depth value of the virtual agent model under the virtual-real space alignment coordinate system, and automatically hiding the shielded virtual object when the depth value of the virtual object is greater than the depth value of the virtual agent model. According to the method, accurate shielding of a virtual object by a real object is realized through depth testing and color channel separation control of a graphic rendering pipeline.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality (MR) technology, and more particularly to a method, apparatus, and device for occluding virtual objects with real objects in MR. Background Art

[0002] In a mixed reality (MR) scene, the real environment video is displayed on the outermost layer of the three-dimensional virtual scene, which is equivalent to the background image. This achieves the effect of combining virtual and real, and places virtual objects in the "real environment". However, the video of the real environment has no spatial geometric information, while the virtual environment has an independent three-dimensional space coordinate system. There is no corresponding positional relationship between the video image and the virtual object, and occlusion cannot be achieved. For example, Figure 1 As shown in the head-mounted display screen, when the user puts on the MR head-mounted display and enters a room, a virtual ball is placed. When the user walks out of the room and closes the door, the user can still see the virtual ball, as shown in the figure below. Figure 2 As shown; destroying the immersive feeling of combining virtual and real.

[0003] Traditional methods (such as the "shooting line detection method") use ray detection to detect virtual collision bodies to determine occlusion, and build a set of three-dimensional models based on the real environment (scanners can also be used for scanning), such as Figure 3 As shown. By completely aligning the virtual 3D model with the real environment, the virtual wall can block the virtual ball. To achieve the occlusion of virtual objects by real objects, the "shooting line detection method" is implemented as follows:

[0004] like Figure 4 As shown in , a collision body is added to the virtual object corresponding to the real object, and the visibility of the ball is determined by whether the ray between the MR head display and the ball collides with the collision body. Figure 5 As shown, if there is no collision, the MR headset can see the ball; if there is a collision, the MR headset may not be able to see the ball. However, there are the following problems:

[0005] A. Depends on specific hardware or predefined markers:

[0006] Paste a QR code or AprilTag on the surface of the real object that needs to be blocked to ensure that the marker is always visible during operation.

[0007] B. Unable to achieve high-precision dynamic occlusion:

[0008] Objects are either visible or invisible. Partial occlusion cannot be achieved, which is inconsistent with the real environment experience (such as seeing part of an object through a crack in a door).

[0009] C. High computational complexity, difficult to run on mobile devices:

[0010] When there are many objects, the computational overhead is high, and the program may experience lag. In game engines, collision detection for 3D meshes is a computationally expensive process. The more objects there are, the greater the overhead.

[0011] D. The occlusion effect is unnatural and affects the user experience:

[0012] Ray detection has poor accuracy and, due to high performance overhead, cannot perform occlusion ray detection on every vertex on an object's surface. Due to the irregular shapes of 3D meshes, collision detection accuracy is also limited. It's common for objects to "drill into a wall" without triggering a collision. This also easily leads to "mold penetration" issues (for example, objects embedded in a wall are not obscured). Summary of the Invention

[0013] In view of this, the present invention provides a method, device and equipment for real objects to occlude virtual objects in mixed reality, which at least partially solve the above-mentioned technical problems. Through depth testing and color channel separation control of the graphics rendering pipeline, accurate occlusion of virtual objects by real objects is achieved; this solution breaks through the limitations of traditional marker reliance and physical collision detection.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] First, an embodiment of the present invention provides a method for a real object to occlude a virtual object in mixed reality, comprising the following steps:

[0016] S1. Scan the real environment using a 3D sensing device, construct a virtual 3D model that matches the real environment, and completely align the virtual 3D model with the real environment coordinate system;

[0017] S2. Create a corresponding virtual proxy object for the target object in the real environment that needs to block the virtual object, and configure a preset material for the virtual proxy object; the preset material includes:

[0018] a) Enable depth testing to participate in depth buffer calculations;

[0019] b) disabling the color channel output function so that the virtual proxy object does not write to the color buffer during rendering;

[0020] S3. Based on the GPU shader program, in the virtual and real space aligned coordinate system, compare the depth value of the virtual object with the virtual proxy model, and automatically hide the obscured virtual object when the virtual object is larger than the depth value of the virtual proxy model.

[0021] Furthermore, when the real environment is a static environment, step S1 includes:

[0022] (1) Obtaining point cloud data with millimeter-level accuracy in real space through a three-dimensional sensing device;

[0023] (2) Generate a topology-optimized triangular mesh model using a surface reconstruction algorithm;

[0024] (3) Select key structural parts as reference objects to achieve pixel-level alignment of virtual and real space coordinate systems.

[0025] Furthermore, when the real environment is a dynamic environment, step S1 includes:

[0026] 1) Using the depth sensor integrated into the headset, the SLAM (Simultaneous Localization and Mapping) algorithm tracks the user device’s position and environmental changes in real time, capturing 3D information and updating the geometry and coordinate system of the virtual 3D model.

[0027] 2) An iterative feature algorithm is used to match feature points and calculate the transformation matrix between the virtual model and the real environment coordinate system to achieve sub-millimeter alignment accuracy.

[0028] Furthermore, when there is a moving object in the real environment, step S1 includes:

[0029] a) Obtaining a parametric model of a moving object through multi-angle scanning;

[0030] b) Combine visual tracking with inertial measurement unit data to update pose in real time.

[0031] Furthermore, the accuracy of the depth value comparison in step S3 is calculated pixel by pixel, thereby achieving a local occlusion effect of the virtual object.

[0032] In a second aspect, an embodiment of the present invention further provides a device for occluding a virtual object with a real object in mixed reality, using the method for occluding a virtual object with a real object in mixed reality as described in any one of the embodiments of the first aspect, including:

[0033] The environment modeling and alignment module uses a 3D sensing device to scan the real environment, builds a virtual 3D model that matches the real environment, and fully aligns the virtual 3D model with the real environment coordinate system;

[0034] The virtual proxy object configuration module creates a corresponding virtual proxy object for the target object in the real environment that needs to occlude the virtual object, and configures a preset material for the virtual proxy object; the preset material includes:

[0035] a) Enable depth testing to participate in depth buffer calculations;

[0036] b) disabling the color channel output function so that the virtual proxy object does not write to the color buffer during rendering;

[0037] The occlusion logic execution module compares the depth value of the virtual object and the virtual proxy model based on the GPU shader program in the virtual and real space aligned coordinate system, and automatically hides the occluded virtual object when the virtual object is greater than the depth value of the virtual proxy model.

[0038] In a third aspect, an embodiment of the present invention further provides a device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0039] Memory for storing computer programs;

[0040] The processor, when used to execute the program stored in the memory, can implement a method for a real object to occlude a virtual object in mixed reality as described in any one of the embodiments of the first aspect.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for a real object to occlude a virtual object in a mixed reality as described in any one of the embodiments of the first aspect.

[0042] The description of the second to third aspects of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second to third aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0043] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following technical advantages:

[0044] A. High-precision occlusion

[0045] Based on high-precision environmental modeling, millimeter-level occlusion accuracy is achieved, completely eliminating the traditional marker pose estimation error. There is no need to manually stick markers, only to select reference objects, and pixel-level occlusion accuracy can be maintained on the edges of complex surfaces.

[0046] B. Dynamic scene adaptation

[0047] It supports real-time adaptation of unstructured dynamic scenes, realizes self-calibration capability and parameterized update of moving objects through online SLAM, without the need for pre-set markers or static environment calibration process.

[0048] C.GPU Acceleration Architecture

[0049] Migrating core occlusion calculations from the CPU to GPU shaders, leveraging a parallel architecture to achieve a 10x improvement in computational efficiency, enabling deep analysis of millions of pixels to be completed in milliseconds.

[0050] D. Physical Reality Interaction

[0051] The visual characteristics of real objects are fully maintained, and the occlusion effect that conforms to the laws of physics is achieved through pixel-by-pixel judgment of depth values ​​and color separation rendering technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 Schematic diagram of a virtual ball of an MR head-mounted display in scene 1 in the prior art.

[0054] Figure 2 Schematic diagram of a virtual ball of an MR head-mounted display in scene 2 in the prior art.

[0055] Figure 3 A schematic diagram of a three-dimensional model built with reference to the real environment.

[0056] Figure 4 Schematic diagram of adding collision bodies to virtual objects corresponding to real objects in the shooting line detection method.

[0057] Figure 5 Schematic diagram of ray detection between the MR headset and the virtual ball.

[0058] Figure 6 This is a flow chart of the method for real objects occluding virtual objects in mixed reality provided by the present invention.

[0059] Figure 7 This is a schematic diagram showing the occlusion of a virtual sphere by a virtual wall provided by the present invention.

[0060] Figure 8 This is a schematic diagram provided by the present invention showing the occlusion of a virtual sphere by a real wall.

[0061] Figure 9 This is a block diagram of the device for real objects occluding virtual objects in mixed reality provided by the present invention.

[0062] Figure 10 This is a structural diagram of the device provided by the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] The technical terms involved in the present invention are explained as follows:

[0065] Material: defines the physical and optical properties of an object's surface, including color, glossiness, reflectivity, transparency, etc. It forms a complete surface representation by integrating textures and other parameters (such as metallicity and roughness).

[0066] Texture: An image or pattern applied to the surface of a 3D model.

[0067] Shader: A program code that runs on the GPU and is responsible for converting material properties and textures into the final screen pixel color.

[0068] Depth is a crucial concept in real-time graphics. It refers to the distance from the camera; the closer to the lens, the smaller the depth. When the depth of an object's part is less than or equal to the depth of the wall, meaning it's in front of the wall, it displays normally. When the depth of an object's part is greater than the depth of the wall, meaning it's behind the wall, it doesn't display. This is how object occlusion is achieved.

[0069] Color Channel Masking: Each color channel (Red, Green, Blue, Alpha) can be independently turned on or off. By default, the GPU writes to all channels (RGBA). Sometimes, to achieve certain effects, you need to turn off some color channels. For example, disabling color writing completely will make the model completely transparent.

[0070] Example 1:

[0071] The embodiment of the present invention provides a method for real objects to occlude virtual objects in mixed reality, which is to put the occlusion judgment of real objects on virtual objects into the shader for calculation. Through depth testing and color channel control technology, the natural occlusion of virtual objects by real objects is achieved. Figure 6 As shown, the following steps are included:

[0072] S1. Scan the real environment using a 3D sensing device, construct a virtual 3D model that matches the real environment, and completely align the virtual 3D model with the real environment coordinate system;

[0073] S2. Create a corresponding virtual proxy object for the target object in the real environment that needs to block the virtual object, and configure a preset material for the virtual proxy object; the preset material includes:

[0074] a) Enable depth testing to participate in depth buffer calculations;

[0075] b) disabling the color channel output function so that the virtual proxy object does not write to the color buffer during rendering;

[0076] S3. Based on the GPU shader program, the depth values ​​of the virtual object and the virtual proxy model are compared in a virtual-real space aligned coordinate system. When the virtual object's depth value is greater than the virtual proxy model's, the obscured virtual object is automatically hidden. This step transfers occlusion determination from the CPU's ray detection to the GPU's shader program, leveraging the GPU's parallel computing capabilities to improve efficiency. Furthermore, when the depth value comparison is performed pixel by pixel, when the virtual object's depth value is greater than the virtual proxy model's depth value, the obscured virtual object's pixels are automatically hidden, thereby achieving a partial occlusion effect for the virtual object.

[0077] Among them, step S1, for example, uses a depth camera to scan the real environment, builds a 1:1 virtual environment model, and completes spatial alignment through the combination of virtual and real; in step S2, the occlusion logic is implemented, the spatial relationship of real objects is replicated in the virtual environment, and special materials are created for virtual proxy objects (such as virtual walls). The shader simultaneously executes: a) depth test (enabled), b) color output (disabled); then in step S3, through the rendering pipeline control, the virtual proxy object is involved in the depth buffer calculation, but not in the color buffer writing, to achieve an "invisible" depth occlusion effect.

[0078] This invention shifts occlusion determination from raycasting on the CPU to shader programs on the GPU, leveraging the GPU's parallel computing capabilities to improve efficiency. Through depth testing and color channel masking, real objects (such as walls) can be rendered to "hide" their corresponding virtual models while preserving their occlusion effects on the virtual objects.

[0079] The real-world scenario involves three parts. The following further describes the technical solution of the present invention based on each of the three parts:

[0080] 1. Virtual and real occlusion in static environment:

[0081] Virtual and real occlusion in a static environment is achieved by "scanning first and then occluding". There is a clear order between scanning and modeling. At this time, when the real environment is a static environment, step S1 includes:

[0082] (1) Obtaining point cloud data with millimeter-level accuracy in real space through a three-dimensional sensing device;

[0083] (2) Generate a topology-optimized triangular mesh model using a surface reconstruction algorithm;

[0084] (3) Select key structural parts as reference objects to achieve pixel-level alignment of virtual and real space coordinate systems.

[0085] Specifically, the static environment must first be digitally modeled in 3D. A depth-scanning device (such as a LiDAR or structured light camera) is used to comprehensively scan the real space and acquire geometric data of the environment. This data is processed through point cloud processing and surface reconstruction algorithms to generate a virtual 3D model that is completely consistent with the real environment. During the modeling process, special attention is paid to the detailed restoration of objects that may cause occlusion (such as walls and furniture) to ensure the accuracy of edge contours.

[0086] After completing the environment modeling, structures at key locations or objects of known size (such as corners, edges, stationary furniture, etc.) can be selected as reference objects, and the alignment of the virtual and real coordinate systems can be achieved through the spatial calibration technology in the existing technology.

[0087] In the rendering pipeline, "invisible proxy models" can be created for real-world objects. These proxy models have special material properties: they participate in depth buffer calculations but do not output any pixel information in the color buffer. When a virtual object and these proxy models are in an occluded position during the depth test, the system automatically hides the occluded portion, while the real-world object remains fully visible because it is not covered by the virtual proxy model.

[0088] The advantage of this technical solution lies in its physical accuracy. Because the virtual proxy model has the exact same geometry as the real object, the generated occlusion edges perfectly match the contours of the real object. Whether it is the hard edge of a right-angled wall or the soft transition of a curved furniture surface, the occlusion effect is natural and accurate.

[0089] For example, let's take the occlusion of a virtual ball in a static environment as an example:

[0090] First, an identical 3D model is built (or scanned) based on the real environment. The virtual 3D model is completely aligned with the real environment, effectively fitting the virtual 3D model onto the real object.

[0091] Suppose you want to make a real-world wall block a virtual sphere. Relative to the camera, the sphere should be placed behind the virtual wall. Create a material for the virtual wall that uses a custom shader. Enable depth testing in the shader. This will achieve occlusion of the sphere by the virtual wall. However, the virtual wall also blocks the real-world wall, such as Figure 7 shown.

[0092] To make the real wall block the ball, you need to set the virtual wall to be invisible. From the above explanation, you can know that in the shader of the virtual wall, all the colors in the color channel mask are disabled, such as Figure 8 As shown, the virtual wall can be made invisible and the real wall can be displayed.

[0093] As described above, when the virtual ball is greater than the depth value of the virtual wall, the pixels of the virtual ball that are blocked are automatically hidden, thus achieving the effect of the real wall blocking the virtual object.

[0094] Furthermore, when applying the technical solution of this invention to the fields of architecture and interior design, designers wearing MR devices can view virtual renovation plans. Real walls and beams can accurately block virtual furniture and decorations, making spatial relationships clear at a glance. This effect allows designers to intuitively assess the rationality of design plans in the actual space, avoiding the awkward feeling of virtual objects "penetrating" the real structure.

[0095] Another important application of this invention is cultural heritage preservation. At an ancient site, researchers can use MR equipment to view a virtual reconstruction, while the existing structure automatically obscures the missing parts of the virtual model. This precise obscuration relationship helps researchers more accurately understand the relationship between the site's original appearance and its current state.

[0096] Virtual-reality occlusion technology in static environments has several significant advantages: first, because the environment is static and unchanging, the system can use higher-precision modeling algorithms to obtain finer geometric details than dynamic scenes; second, single modeling can be reused, greatly reducing the computational load at runtime; third, static scenes allow for stricter lighting consistency processing, making the light and shadow interaction between virtual objects and the real environment more natural.

[0097] The mature application of this technology lays a crucial foundation for mixed reality experiences, enabling virtual content to truly blend into, rather than be superimposed on, the real environment, significantly enhancing both immersion and practical value. With continued advancements in 3D scanning technology and spatial computing capabilities, the accuracy and efficiency of virtual-real occlusion in static environments will be further enhanced, opening up even broader application prospects.

[0098] 2. Virtual and real occlusion in dynamic environments:

[0099] When the real environment is a dynamic environment, step S1 includes:

[0100] 1) Using the depth sensor integrated into the headset, the SLAM (Simultaneous Localization and Mapping) algorithm tracks the user device’s position and environmental changes in real time, capturing 3D information and updating the geometry and coordinate system of the virtual 3D model.

[0101] 2) An iterative feature algorithm is used to match feature points and calculate the transformation matrix between the virtual model and the real environment coordinate system to achieve sub-millimeter alignment accuracy.

[0102] In dynamic environments, virtual and real occlusion is achieved using a "scan-while-occlusion" approach. The time difference between scanning and modeling is only tens or tens of milliseconds, giving the user the impression that scanning and modeling are occurring simultaneously. Of course, this requires powerful computing performance from the MR headset.

[0103] In this embodiment, the head-mounted device's integrated depth sensor captures three-dimensional information about the environment in real time, acquiring full-scene depth maps dozens of times per second. This data is processed by existing simultaneous localization and mapping (SLAM) algorithms to continuously update the virtual environment model. By adopting an incremental modeling strategy, only the environmental areas that have changed are recalculated, significantly reducing the computational burden.

[0104] For example, in interior design, when users move real furniture, the virtual model can be updated instantly, ensuring that the newly placed sofa immediately and correctly occludes the virtual carpet pattern. In medical scenarios, even slight changes in the patient's position can quickly adjust the occlusion relationship of the organ model, providing a continuous and accurate spatial reference for surgical navigation.

[0105] 3. Virtual and real occlusion of moving objects:

[0106] When there are moving objects in the real environment, step S1 includes:

[0107] a) Obtaining a parametric model of a moving object through multi-angle scanning;

[0108] b) Combine visual tracking with inertial measurement unit data to update pose in real time.

[0109] For objects moving in real space (such as floating balloons), achieving accurate occlusion requires addressing the dual challenges of dynamic modeling and real-time tracking. A two-stage "modeling + tracking" approach can be used to address this requirement.

[0110] During the initial modeling phase, multi-angle scanning is used to obtain an accurate 3D model of the moving object. For regular objects like balloons, existing parametric modeling methods can be used, requiring only key parameters such as diameter and shape. For irregular objects, 3D scanning is used for modeling. During the tracking phase, real-time pose estimation is performed using visual or sensor data.

[0111] Example 2:

[0112] Based on the same inventive concept, the present invention also provides a device for real objects to block virtual objects in mixed reality, using the method for real objects to block virtual objects in mixed reality as in Example 1, referring to Figure 9 Shown, including:

[0113] The environment modeling and alignment module uses a 3D sensing device to scan the real environment, builds a virtual 3D model that matches the real environment, and fully aligns the virtual 3D model with the real environment coordinate system;

[0114] The virtual proxy object configuration module creates a corresponding virtual proxy object for the target object in the real environment that needs to occlude the virtual object, and configures a preset material for the virtual proxy object; the preset material includes:

[0115] a) Enable depth testing to participate in depth buffer calculations;

[0116] b) disabling the color channel output function so that the virtual proxy object does not write to the color buffer during rendering;

[0117] The occlusion logic execution module compares the depth value of the virtual object and the virtual proxy model based on the GPU shader program in the virtual and real space aligned coordinate system, and automatically hides the occluded virtual object when the virtual object is greater than the depth value of the virtual proxy model.

[0118] In this embodiment, a three-dimensional sensing device is used to construct a virtual model of the real environment, and dynamic spatial alignment can be achieved through the SLAM algorithm. In addition, corresponding virtual proxy objects are created for real objects, and special materials are configured to enable depth testing and disable color output. Finally, occlusion logic is executed: depth values ​​are compared pixel by pixel in the GPU shader to dynamically control the visibility of virtual objects.

[0119] The device for occluding virtual objects with real objects in mixed reality provided by the present invention has the following technical advantages:

[0120] High-precision occlusion: pixel-by-pixel depth calculation to avoid "penetration" problems;

[0121] High-efficiency performance: GPU parallel processing increases computing speed by dozens of times;

[0122] Dynamic adaptation: supports real-time changes in the environment without pre-calibration;

[0123] Natural interaction: Preserve the visual integrity of real objects and enhance immersion.

[0124] The present invention can be applied to interior design, industrial training, military simulation and other fields to improve the authenticity and efficiency of MR interaction.

[0125] Example 3:

[0126] Based on the same inventive concept, the present invention further provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0127] Memory for storing computer programs;

[0128] The processor, when used to execute the program stored in the memory, can implement a method for a real object to occlude a virtual object in mixed reality as described in any one of the embodiments 1.

[0129] like Figure 10 As shown, the electronic device may include: a processor 10, a communication interface 20, a memory 30, and a communication bus 40, wherein the processor 10, the communication interface 20, and the memory 30 communicate with each other via the communication bus 40. The processor 10 may call the logic instructions in the memory 30 to execute a method for real objects occluding virtual objects in mixed reality, the method including:

[0130] S1. Scan the real environment using a 3D sensing device, construct a virtual 3D model that matches the real environment, and completely align the virtual 3D model with the real environment coordinate system;

[0131] S2. Create a corresponding virtual proxy object for the target object in the real environment that needs to block the virtual object, and configure a preset material for the virtual proxy object; the preset material includes:

[0132] a) Enable depth testing to participate in depth buffer calculations;

[0133] b) disabling the color channel output function so that the virtual proxy object does not write to the color buffer during rendering;

[0134] S3. Based on the GPU shader program, in the virtual and real space aligned coordinate system, compare the depth value of the virtual object with the virtual proxy model, and automatically hide the obscured virtual object when the virtual object is larger than the depth value of the virtual proxy model.

[0135] Example 4:

[0136] An embodiment of the present invention further provides a computer-readable storage medium, wherein a program stored in the computer-readable storage medium is used to execute the method for real objects occluding virtual objects in mixed reality of the above-mentioned embodiment 1. The program can be executed on a processor.

[0137] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0138] The program stored in the medium is loaded into the processor memory and executed to complete various functions. The storage medium is connected to the hardware device, enabling the computer to perform the above-mentioned environment modeling and alignment steps, virtual proxy object configuration steps, and occlusion logic execution steps.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0140] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for occluding a virtual object by a real object in mixed reality, characterized in that: The following steps are involved: S1. Scan the real environment using a 3D sensing device, construct a virtual 3D model that matches the real environment, and completely align the virtual 3D model with the real environment coordinate system; S2. Creating a corresponding virtual proxy object for the target object in the real environment that needs to occlude the virtual object, and configuring a preset material for the virtual proxy object; The preset materials include: a) Enable depth testing to participate in depth buffer calculations; b) disabling the color channel output function so that the virtual proxy object does not write into the color buffer during rendering; S3. Based on the GPU shader program, in the virtual and real space aligned coordinate system, compare the depth value of the virtual object with the virtual proxy model, and automatically hide the obscured virtual object when the virtual object is greater than the depth value of the virtual proxy model.

2. The method for occluding a virtual object by a real object in mixed reality according to claim 1, characterized in that: When the real environment is a static environment, step S1 includes: (1) Obtaining point cloud data with millimeter-level accuracy in real space through a three-dimensional sensing device; (2) Generate a topology-optimized triangular mesh model using a surface reconstruction algorithm; (3) Select key structural parts as reference objects to achieve pixel-level alignment of virtual and real space coordinate systems.

3. The method for occluding a virtual object by a real object in mixed reality according to claim 1, characterized in that: When the real environment is a dynamic environment, step S1 includes: 1) Using the depth sensor integrated into the headset, the SLAM (Simultaneous Localization and Mapping) algorithm tracks the user device’s position and environmental changes in real time, capturing 3D information and updating the geometry and coordinate system of the virtual 3D model. 2) An iterative feature algorithm is used to match feature points and calculate the transformation matrix between the virtual model and the real environment coordinate system to achieve sub-millimeter alignment accuracy.

4. The method for occluding a virtual object by a real object in mixed reality according to claim 1, characterized in that: When there is a moving object in the real environment, step S1 includes: a) Obtaining a parametric model of a moving object through multi-angle scanning; b) Combine visual tracking with inertial measurement unit data to update pose in real time.

5. The method for occluding a virtual object by a real object in mixed reality according to claim 1, characterized in that: The accuracy of the depth value comparison in step S3 is calculated pixel by pixel to achieve a local occlusion effect of the virtual object.

6. A device for occluding virtual objects with real objects in mixed reality, characterized in that: The method for occluding a virtual object using a real object in mixed reality according to any one of claims 1 to 5 comprises: The environment modeling and alignment module uses a 3D sensing device to scan the real environment, builds a virtual 3D model that matches the real environment, and fully aligns the virtual 3D model with the real environment coordinate system; The virtual proxy object configuration module creates a corresponding virtual proxy object for the target object in the real environment that needs to occlude the virtual object, and configures a preset material for the virtual proxy object; the preset material includes: a) Enable depth testing to participate in depth buffer calculations; b) disabling the color channel output function so that the virtual proxy object does not write into the color buffer during rendering; The occlusion logic execution module compares the depth value of the virtual object and the virtual proxy model based on the GPU shader program in the virtual and real space aligned coordinate system, and automatically hides the occluded virtual object when the virtual object is greater than the depth value of the virtual proxy model.

7. A device, characterized in that The processor, the communication interface, the memory and the communication bus are connected to each other via the communication bus. Memory for storing computer programs; The processor is configured to implement a method for a real object to occlude a virtual object in mixed reality as described in any one of claims 1 to 5 when executing a program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for a real object to occlude a virtual object in mixed reality is implemented as described in any one of claims 1 to 5.

Images