Virtual-real alignment method and device based on mixed reality, equipment, medium and product

Abstract

The embodiment of the invention provides a virtual-real alignment method and device based on mixed reality, equipment, a medium and a product. The method comprises the following steps: creating a three-dimensional model of a real scene, wherein the physical attribute of the three-dimensional model is consistent with the physical attribute of the real scene; determining anchor points in the three-dimensional model, wherein the anchor points correspond to feature points in the real scene; rendering and presenting the three-dimensional model in a display unit of mixed reality equipment; and aligning the anchor points in the presented three-dimensional model with the feature points in the real scene so as to realize virtual-real alignment of the presented three-dimensional model and the real scene. According to the method, virtual-reality alignment in multiple modes based on mixed reality is realized based on the anchor points, the method does not depend on network access of a shared cloud picture system of a real scene any more, and reliability and convenience are improved.

Description

Technical Field

[0001] The present invention belongs to the field of mixed reality technology, and in particular to a virtual-real alignment method, device, equipment, medium and product based on mixed reality. Background Art

[0002] Mixed reality (MR) devices seamlessly combine virtual elements with the real environment by integrating the technical features of virtual reality (VR) and augmented reality (AR) to create an immersive interactive experience. Its core technologies may include: spatial perception and positioning: for example, through cameras, sensors and depth sensors, the device can scan and reconstruct the physical space around the user in real time; full color perspective (VST) technology: for example, using binocular RGB cameras and depth sensors, the device can overlay virtual objects into the real scene to achieve realistic visual effects; interactive technology: support gesture recognition, eye tracking, voice interaction and other natural interaction methods to enhance the user's sense of immersion; display technology: for example, a high-resolution Micro-OLED display can be used to provide a clear, low-latency visual experience.

[0003] More and more MR devices need to detect and determine their own position and movement in three-dimensional space through their own sensors, and feed back the variable information of three-dimensional space to their own devices. MR devices allow users to correctly perceive real scenes and scenes, while presenting virtual patterns / animations calculated and rendered by the device in three dimensions in existing real scenes and scenes, presenting users with correct virtual-real mixed three-dimensional models. Summary of the invention

[0004] The embodiments of the present invention provide a virtual-reality alignment method, device, equipment, medium and product based on MR.

[0005] The technical solution of the embodiment of the present invention is as follows:

[0006] A virtual-real alignment method based on MR, comprising:

[0007] Creating a three-dimensional model of a real scene, wherein the physical properties of the three-dimensional model are consistent with the physical properties of the real scene;

[0008] Determining an anchor point in the three-dimensional model, wherein the anchor point corresponds to a feature point in the real scene;

[0009] Rendering and presenting the three-dimensional model in a display unit of the MR device;

[0010] Anchor points in the presented three-dimensional model are aligned with feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene.

[0011] In one embodiment, the creating of the three-dimensional model of the real scene includes at least one of the following:

[0012] Using a three-dimensional laser scanner to obtain three-dimensional coordinate data of the real scene; generating the three-dimensional model based on the three-dimensional coordinate data;

[0013] Taking and measuring photos of the real scene; converting the photos into three-dimensional coordinate data; and generating the three-dimensional model based on the three-dimensional coordinate data;

[0014] Based on the built-in sensor device of the MR device, the real scene is scanned to generate the three-dimensional model.

[0015] In one embodiment, aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene includes:

[0016] When the MR device moves to the feature point of the real scene, receiving an alignment instruction issued by a user via a handle of the MR device;

[0017] Based on the alignment instruction, the three-dimensional model aligned with the real scene is re-rendered and re-presented in a manner that the anchor point of the three-dimensional model is aligned with the feature point.

[0018] In one embodiment, aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene includes:

[0019] Arranging a ranging system in the real scene;

[0020] Based on the ranging system, determining the current position of the MR device in the real scene coordinate system;

[0021] Determining a position offset between the current position and the feature point in the real scene;

[0022] Based on the position offset, the three-dimensional model aligned with the real scene is re-rendered and re-presented in a manner that the anchor point of the three-dimensional model is aligned with the feature point.

[0023] In one embodiment, the anchor point includes at least one of the following:

[0024] Single anchor point;

[0025] At least two anchor points that are not in the same straight line;

[0026] At least three anchor points that are not in the same line and plane.

[0027] In one embodiment, the number of the MR devices is multiple;

[0028] The rendering and presenting the three-dimensional model in the display unit of the MR device comprises: rendering and presenting a respective three-dimensional model in the display unit of each MR device;

[0029] The step of aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-reality alignment of the presented three-dimensional model with the real scene includes: aligning the anchor points in the three-dimensional model presented in each MR device with the feature points in the real scene to achieve virtual-reality alignment of the three-dimensional model presented in each MR device with the real scene.

[0030] A virtual-real alignment device based on MR, comprising:

[0031] A creation module, used to create a three-dimensional model of a real scene, wherein the physical properties of the three-dimensional model are consistent with the physical properties of the real scene;

[0032] A determination module, configured to determine an anchor point in the three-dimensional model, wherein the anchor point corresponds to a feature point in the real scene;

[0033] A rendering module, used for rendering and presenting the three-dimensional model in a display unit of the MR device;

[0034] The alignment module is used to align the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene.

[0035] An electronic device, comprising:

[0036] processor;

[0037] A memory, configured to store executable instructions of the processor;

[0038] The processor is used to read the executable instructions from the memory, and execute the executable instructions to implement any of the MR-based virtual-real alignment methods described above.

[0039] A computer-readable storage medium stores computer instructions, wherein the computer instructions, when executed by a processor, implement any of the above-mentioned virtual-real alignment methods based on MR.

[0040] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the virtual-real alignment method based on MR as described above is implemented.

[0041] It can be seen from the above technical solution that in the implementation mode of the present invention, a three-dimensional model of the real scene is created, wherein the physical properties of the three-dimensional model are consistent with the physical properties of the real scene; anchor points are determined in the three-dimensional model, and the anchor points correspond to feature points in the real scene; the three-dimensional model is rendered and presented in the display unit of the MR device; the anchor points in the presented three-dimensional model are aligned with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene. The virtual-real alignment in the MR device is achieved based on the anchor points, which no longer depends on the network access to the shared cloud map system of the real scene, thereby improving reliability. Moreover, multiple alignment methods are implemented, which improves the convenience of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flow chart of a virtual-real alignment method based on MR according to an embodiment of the present invention.

[0043] Figure 2 Schematic diagram of anchor points in a three-dimensional model according to an embodiment of the present invention.

[0044] Figure 3 It is a schematic diagram of realizing virtual-real alignment of multiple MR devices in the same real scene according to an embodiment of the present invention.

[0045] Figure 4 4 is a structural diagram of a virtual-real alignment device based on MR according to an embodiment of the present invention.

[0046] Figure 5 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0048] For the sake of brevity and intuitiveness in description, the scheme of the present invention is explained below by describing several representative implementations. A large number of details in the implementations are only used to help understand the scheme of the present invention. However, it is obvious that the technical scheme of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the scheme of the present invention, some implementations are not described in detail, but only a framework is given. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically specified below, it means that the component can be one or more, or can be understood as at least one.

[0049] In MR, virtual-reality alignment refers to the precise alignment of virtual objects with objects or scenes in the real environment, so that the virtual content maintains the same position, size, and orientation as the actual objects in the user's field of view.

[0050] In the prior art, virtual-real alignment is achieved by relying on network access to a shared cloud image system of real scenes. However, in many cases, such network access is not achievable, unstable or unsustainable, making it difficult to achieve consistency in virtual graphics / image display, interaction and collaboration.

[0051] For example: In the prior art, the real space and scene are scanned by MR head display devices or third-party scanning devices to establish a spatial point cloud coordinate system (called system A). When the MR head display device is in use, it scans the surrounding point cloud features in real time and compares the scanned point cloud features with system A to determine the correct three-dimensional spatial position of the MR head display device. Usually, system A is placed in the server. After multiple MR head display devices in the surrounding download and share the data of system A, multiple MR head display devices can achieve consistency in virtual graphics / image display, interaction and collaboration in the same real space. However, in actual applications, taking the Quest 3 head display device as an example, the network server used to store and share system A is usually located outside the country, which is difficult for domestic users to access and use, resulting in difficulty for the Quest 3 head display device to achieve consistency in virtual graphics / image display, interaction and collaboration.

[0052] Figure 1 FIG. 1 is a flow chart of a virtual-real alignment method based on MR according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0053] Step 101: Create a three-dimensional model of a real scene, wherein the physical properties of the three-dimensional model are consistent with the physical properties of the real scene.

[0054] For example, creating a 3D model of a real-world scene can be achieved by:

[0055] (1) Photogrammetry-based methods:

[0056] Data collection: Use a camera to take photos of the scene from multiple angles. Make sure the photos cover every detail of the scene and have enough overlap for subsequent processing. Data processing: Use photogrammetry software (such as Regard3D, VisualSFM, etc.) to process the photos. These software reconstruct the 3D point cloud of the scene through the structure from motion (SfM) technology. Model optimization: The generated point cloud data is further processed to generate a 3D mesh model, and then optimized and detailed adjustments are made.

[0057] (2) Laser scanning based method:

[0058] Data collection: Use a laser scanner to scan the scene and obtain high-precision point cloud data. The laser scanner can quickly obtain the three-dimensional geometric information of the scene. Data processing: Import the scanned point cloud data into professional three-dimensional modeling software, such as AutoCAD or 3D Studio Max, for geometric modeling. Texture mapping: Use the captured photos as texture maps and apply them to the three-dimensional model to make the model more realistic.

[0059] (3) Generate using AI tools:

[0060] Image to 3D model: Use AI tools such as Aiuni, GENIE, etc. to upload a single or multiple photos, and the AI ​​tools will automatically generate 3D models. These tools are based on advanced AI algorithms and can quickly generate high-quality 3D models. Text to 3D model: Some AI tools (such as GENIE, Sudo AI) also support generating 3D models through text descriptions. Users only need to enter a detailed description, and the AI ​​tool can generate the corresponding 3D scene. Editing and optimization: The generated model can be further optimized and adjusted through the tool's own editing function or imported into other 3D modeling software.

[0061] (4) Manual modeling:

[0062] Concept design: Draw a concept map or sketch to clarify the theme, style and color of the scene. Modeling: Use 3D modeling software (such as Blender, 3D Studio Max, etc.) to model the objects in the scene one by one according to the concept design. Texture and material: Add appropriate texture and material to the model to make the model look more realistic. Lighting and rendering: Set lighting and rendering parameters to finally generate a realistic 3D scene.

[0063] It is also possible to combine multiple methods to achieve hybrid modeling: combining photogrammetry, laser scanning and manual modeling methods, first obtaining the basic model through photogrammetry or laser scanning, and then refining the details through manual modeling.

[0064] Specifically, in one embodiment, creating a three-dimensional model of a real scene includes at least one of the following: using a three-dimensional laser scanner to obtain three-dimensional coordinate data of the real scene; generating a three-dimensional model based on the three-dimensional coordinate data; taking and measuring photos of the real scene; converting the photos into three-dimensional coordinate data; generating a three-dimensional model based on the three-dimensional coordinate data; scanning the real scene based on the built-in sensing device of the MR device to generate a three-dimensional model.

[0065] The physical properties of a 3D model refer to the characteristics of the model in the virtual environment that are similar to the physical world. These properties determine the behavior and performance of the model in the scene. In addition to the occlusion property (i.e., the model occupies a volume in space and can block the line of sight or other objects), a 3D model can also contain the following physical properties: (1) Collision properties: Collision detection: Collision detection between models or between models and the environment is a common physical property in 3D scenes. It determines the behavior of objects when they come into contact, such as rebound, stop, or penetrate; Collision shape: Different collision shapes (such as spheres, cubes, capsules, or custom grids) can be set for the model to more accurately simulate the physical interaction of objects. (2) Mass and inertia properties, including: Mass: defines the weight of the model and affects its movement and force in physical simulation. The larger the mass, the harder it is to move the object; Inertia tensor: describes the inertial characteristics of the model in rotational motion, affecting the rotational behavior and stability of the object. (3) Gravity properties and force field properties, including gravity: whether the model is affected by gravity, and the magnitude and direction of gravity. In most real-life scenarios, the model will be subject to the downward force of gravity; force field: other force fields (such as wind force, magnetic field, etc.) can be added to the model to make it subject to additional forces. (4) Elasticity and friction, where the elastic coefficient defines the model's ability to rebound when it collides. The higher the elastic coefficient, the more obvious the rebound of the object after collision; friction coefficient describes the roughness of the model surface and affects the sliding and rolling behavior between objects. The higher the friction coefficient, the harder it is for the object to slide; (5) Rigid body and soft body, where the rigid body property is whether the model is a rigid body, that is, the shape remains unchanged in the physical simulation. The rigid body model is suitable for most hard objects; soft body property is whether the model is a soft body, that is, it can deform when subjected to force. The soft body model is suitable for cloth, ropes, mollusks, etc. (6) Fluid properties, where buoyancy is the buoyancy property of the model in a fluid (such as water), which affects its floating or sinking behavior in the fluid; resistance is the resistance encountered by the model when it moves in the fluid, which affects its movement speed and trajectory. (7) Thermal properties, including: thermal conductivity: the thermal conductivity of the model, which affects its temperature change in a thermal environment; thermal expansion: the volume change characteristics of the model when the temperature changes. (8) Optical properties, including: transparency: the transparency of the model, which affects its ability to transmit light; refractive index: the refractive index of the model, which affects the refraction behavior of light on the surface of the model; reflectivity: the reflective characteristics of the model surface, which affects its ability to reflect light. (9) Destruction and deformation, including fragmentation properties: whether the model will break when subjected to a sufficiently large force, as well as the way it breaks and the rules for the generation of fragments; deformation properties: whether the model will deform when subjected to force, as well as the degree and method of deformation.

[0066] These physical properties can be set and adjusted according to specific application scenarios and requirements to make the 3D model more realistic and natural in the virtual environment.

[0067] Step 102: Determine anchor points in the three-dimensional model, where the anchor points correspond to feature points in the real scene.

[0068] In a specific implementation, an anchor point may be first determined in the three-dimensional model, and then a feature point corresponding to the anchor point in the real scene may be determined based on the anchor point. Alternatively, a feature point may be first set in the real scene, and then an anchor point corresponding to the feature point in the three-dimensional model may be determined based on the feature point.

[0069] Feature points in real scenes can be points on a plane, edges or corners of objects, etc. The anchor points set in a 3D model are usually specific points or parts of the model, which are used to align with feature points in real scenes. The choice of anchor points depends on the structure of the model and the application scenario. For example, the center point of the model can be used as the anchor point (for example, for a symmetrical model, the geometric center of the model can be used as the anchor point), or a specific part of the model can be used as the anchor point (for example, for a virtual furniture model, the bottom center point of the model can be selected as the anchor point).

[0070] In one embodiment, the anchor points include at least one of the following: a single anchor point; at least two anchor points that are not located in the same straight line; at least three anchor points that are not located in the same straight line and in the same plane. The position and direction of the anchor points are both known.

[0071] Figure 2 Schematic diagram of anchor points in a three-dimensional model according to an embodiment of the present invention. It can be seen that in the three-dimensional model of the city wall, anchor points 11 and 12 are set at the corners.

[0072] Step 103: Render and present the three-dimensional model in the display unit of the MR device.

[0073] The display units of MR devices usually include waveguide displays, Micro OLED screens, etc. These hardware can support high-quality 3D model rendering. For example, Apple Vision Pro uses high-resolution image sensors and advanced display technology to render complex 3D models in real time. Real-time rendering is a commonly used technology in MR devices, which can generate images instantly during user interaction. The real-time rendering process includes geometry processing, shading, and rasterization, which can quickly render 3D models. In order to optimize performance, the following strategies can be adopted: Reduce draw calls: Reduce the load on the GPU by merging meshes, using instantiation rendering and other techniques. Use efficient model formats: such as GLTF, which is optimized for WebGL and suitable for real-time rendering. Level of detail (LOD) technology: Dynamically adjust the details of the model according to the distance between the user and the model to reduce the amount of calculation.

[0074] For complex scenes or high-polygon models, the large computing power of cloud servers can be used for rendering, and then the rendering results can be transmitted to the MR device in real time. This method can effectively reduce the computing pressure of local MR devices.

[0075] Step 104: Align the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene.

[0076] Here, after the anchor point of the 3D model and the feature points in the real scene are determined, they can be aligned through the following steps: (1) In the MR framework, create an anchor object and bind it to the detected feature points. For example, in Unity, you can use the ARAnchor component to create an anchor object; in visionOS, you can use AnchorEntity. (2) Adjust the position and orientation of the 3D model: According to the position and orientation of the feature points, adjust the position and rotation angle of the 3D model to align it with the feature points. (3) Real-time update: In MR applications, real-time images are continuously collected, and the images are processed and calibrated using the same method to achieve real-time updates and alignment.

[0077] In one embodiment, aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment of the presented three-dimensional model with the real scene includes: when the MR device moves to the feature point of the real scene, receiving an alignment instruction issued by the user via the handle of the MR device; based on the alignment instruction, re-rendering and re-presenting the three-dimensional model aligned with the real scene in a manner that the anchor points of the three-dimensional model are aligned with the feature points.

[0078] For example: Assume that the real scene is a city wall, and the feature point is the corner of the city wall. When the MR device that loads and presents the 3D model (at this time, the 3D model and the real scene are likely not aligned) moves to the corner of the city wall, the user issues an alignment command through the handle of the MR device. The MR device receives the alignment command, reloads, re-renders, and re-presents the 3D model in the MR device in a way that the anchor point of the 3D model is aligned with the corner of the city wall. At this time, the 3D model presented in the MR device is bound and aligned with the real scene.

[0079] Therefore, an alignment method based on manual movement and manual triggering is realized, which is easy to implement.

[0080] In one embodiment, aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment of the presented three-dimensional model with the real scene includes: arranging a ranging system in the real scene; determining the current position of the MR device in the real scene coordinate system based on the ranging system; determining the position offset between the current position and the feature points in the real scene; based on the position offset, re-rendering and re-presenting the three-dimensional model aligned with the real scene in a manner that the anchor points of the three-dimensional model are aligned with the feature points.

[0081] For example, the ranging system can be a three-point ranging system or a four-point ranging system. The three-point ranging system is a positioning technology based on trilateration, which determines the position of the target point by measuring the distance from the target point to three reference points (anchor points) at known positions. The four-point ranging system is a positioning technology based on quadrilateral measurement, which determines the position of the target point by measuring the distance from the target point to four anchor points at known positions.

[0082] For example: Assume that the real scene is a city wall, and the feature point is the corner of the city wall. Based on the ranging system, the current position (for example, coordinates) of the MR device that loads and presents the three-dimensional model (at this time, the three-dimensional model and the real scene are likely not aligned) can be determined. Then, determine the position offset between the current position of the MR device and the corner of the city wall in the real scene. For example, the position offset between the current position and the corner of the city wall is: the current position is 15 meters to the left of the corner of the city wall. Then, based on the position offset, the three-dimensional model is reloaded, re-rendered, and re-presented in a manner that the anchor point of the three-dimensional model is aligned with the corner of the city wall. For example, the current position in the re-rendered three-dimensional model is 15 meters to the left of the anchor point. At this point, the three-dimensional model presented in the MR device is aligned with the real scene.

[0083] Therefore, the 3D model can be aligned with the real scene without manually moving to the feature points and without manual triggering.

[0084] In one embodiment, the number of MR devices is multiple; in the display unit of the MR device, rendering and presenting the three-dimensional model includes: in the display unit of each MR device, rendering and presenting the respective three-dimensional models; aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment of the presented three-dimensional model with the real scene includes: aligning the anchor points in the three-dimensional model presented in each MR device with the feature points in the real scene to achieve virtual-real alignment of the three-dimensional model presented in each MR device with the real scene. Wherein: the anchor point positions in the three-dimensional model of each MR device are the same. The three-dimensional model of each MR device can be determined by different modeling methods. For example, the modeling method may include: using a three-dimensional laser scanner to obtain three-dimensional coordinate data of the real scene; generating a three-dimensional model based on the three-dimensional coordinate data; taking and measuring photos of the real scene; converting the photos into three-dimensional coordinate data; generating a three-dimensional model based on the three-dimensional coordinate data; scanning the real scene based on the built-in sensor device of the mixed reality device to generate a three-dimensional model, and so on. Therefore, for three-dimensional models established by different modeling methods, through a unified anchor point, the consistency of virtual object display, interaction and coordination of multiple MR devices in the same real space scene can be achieved.

[0085] When multiple MR devices that use different modeling methods to build three-dimensional models are in the same real space scene, multiple MR devices often have simultaneous transactions (for example, simultaneous insertion (INSERT) of the same data, simultaneous update (UPDATE) of the same data, simultaneous deletion (DELETE) of the same data, simultaneous query (SELECT) operations on the same data, etc.). Handling conflicts between transactions to avoid confusion is the key to ensuring user experience and data consistency. When the probability of transaction conflicts is low (for example, below a predetermined first threshold), the conflict is checked when the transaction of each MR device is committed (that is, when the transaction data is sent to the server). If a transaction conflict is detected, the transaction of the MR device is rolled back and retried. Assuming that the probability of conflict is high (for example, above a predetermined second threshold, which is usually greater than the first threshold), at the beginning of the transaction of each MR device, the data is locked to ensure that the locked data can only be accessed by the current transaction of other MR devices, and the transaction of this MR device needs to wait until the current transaction of other MR devices ends before releasing the lock.

[0086] In one embodiment, the method further includes: arranging a model of an interactive object at a predetermined position in the three-dimensional model, wherein the model of the interactive object includes predetermined interaction logic; based on a head-mounted MR device worn by the observer, acquiring in real time an actual scene image of the city wall observed by the observer; determining rendering parameters based on a field of view of the actual scene image; based on the rendering parameters, rendering the model of the interactive object to form a video stream of the interactive object; and superimposing the video stream on the actual scene image.

[0087] Figure 3 It is a schematic diagram of realizing virtual-real alignment of multiple MR devices in the same real scene according to an embodiment of the present invention. First, through the three-dimensional modeling method, an area of ​​a horse face (ancient fortifications, the protruding part of the ancient city wall) of the ancient city wall is accurately drawn, and its physical size is consistent with the actual ancient city wall. Then, the corner of the wall on the east side of the ancient city wall and the lower edge of the first battlement are designated as "anchor points". After any MR head display device is started, the three-dimensional model space of the ancient city wall containing the anchor point position is loaded, and then the anchor point position in the three-dimensional model space is moved to the corner of the east side of the city wall and the lower edge of the first battlement in the actual space scene through the wearer's visual manual method to complete the overlap of the three-dimensional model space with the actual space scene. From then on, the virtual graphic images superimposed by the occlusion effect and collision volume in the three-dimensional model space calculated by the MR head display device can be matched with the real space scene very realistically, so that the virtual rendered graphic images are perfectly integrated with the real space. Moreover, according to multiple MR head display devices (for example, Figure 3 After repeating the above anchor point calibration, the consistency of virtual object display, interaction and collaboration of multiple MR head-mounted display devices in the same real space scene can be finally obtained.

[0088] Figure 4 FIG. 1 is a structural diagram of a virtual-real alignment device based on MR according to an embodiment of the present invention. Figure 4 As shown, the MR-based virtual-reality alignment device includes: a creation module, used to create a three-dimensional model of a real scene, wherein the physical properties of the three-dimensional model are consistent with the physical properties of the real scene; a determination module, used to determine anchor points in the three-dimensional model, wherein the anchor points correspond to feature points in the real scene; a presentation module, used to render and present the three-dimensional model in the display unit of the MR device; an alignment module, used to align the anchor points in the presented three-dimensional model with the feature points in the real scene, so as to achieve virtual-reality alignment of the presented three-dimensional model with the real scene.

[0089] In one embodiment, a module is created to perform at least one of the following: using a three-dimensional laser scanner to obtain point cloud data of a real scene; generating a three-dimensional model based on the point cloud data; taking photos of the real scene; converting the photos into point cloud data; generating a three-dimensional model based on the point cloud data; scanning the real scene based on the built-in sensor device of the MR device to generate a three-dimensional model.

[0090] In one embodiment, the alignment module is used to receive an alignment instruction issued by a user via a handle of the MR device when the MR device moves to a feature point of a real scene; based on the alignment instruction, the three-dimensional model is re-rendered and re-presented in a manner that the anchor point of the three-dimensional model is aligned with the feature point so as to align the anchor point of the three-dimensional model with the feature point.

[0091] In one embodiment, an alignment module is used to arrange a ranging system in a real scene; determine the current position of the MR device in the real scene coordinate system based on the ranging system; determine the position offset between the current position and the feature point in the real scene; based on the position offset, re-render and re-present the three-dimensional model aligned with the real scene in a manner that the anchor point of the three-dimensional model is aligned with the feature point.

[0092] In one embodiment, the anchor points include at least one of the following: a single anchor point whose position and orientation are known; at least two anchor points that are not located in the same straight line; at least three anchor points that are not located in the same straight line and in the same plane.

[0093] In one embodiment, there are multiple MR devices; a rendering module is used to render and present the respective three-dimensional models in the display unit of each MR device; and an alignment module is used to align the anchor points in the three-dimensional model presented in each MR device with the feature points in the real scene, so as to achieve virtual-real alignment of the three-dimensional model presented in each mixed reality device with the real scene.

[0094] The embodiment of the present invention further provides an electronic device having a processor-memory architecture. Figure 5 is a structural diagram of an electronic device according to an embodiment of the present invention. Figure 5 As shown, the electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, any of the above MR-based virtual-real alignment methods is implemented. Among them, the memory can be specifically implemented as a variety of storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash memory), and a programmable program read-only memory (PROM). The processor can be implemented as including one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core can be implemented as a CPU, an MCU, or a DSP, and so on.

[0095] It should be noted that not all steps and modules in the above-mentioned processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices. The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module can include a specially designed permanent circuit or logic device (such as a dedicated processor, such as FPGA or ASIC) for completing a specific operation. The hardware module can also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software for performing a specific operation. As for the specific mechanical method, or the use of a dedicated permanent circuit, or the use of a temporarily configured circuit (such as configured by software) to implement the hardware module, it can be determined based on cost and time considerations.

[0096] The present invention also provides a machine-readable storage medium, storing instructions for making a machine perform a method as described in the present application. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for realizing the function of any of the embodiments in the above-mentioned embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system etc. operated on the computer can also be completed part or all of the actual operation by instructions based on the program code. The program code read out from the storage medium can also be written to the memory provided in the expansion board inserted into the computer or to the memory provided in the expansion unit connected to the computer, and then the CPU etc. installed on the expansion board or the expansion unit are made to perform part and all of the actual operation based on the instructions of the program code, so as to realize the function of any of the embodiments in the above-mentioned embodiments. The storage medium implementation for providing the program code includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card and a ROM. Alternatively, the program code can be downloaded from a server computer or a cloud by a communication network.

[0097] In this article, "schematic" means "serving as an example, instance or explanation", and any diagram or implementation method described as "schematic" in this article should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings concise, only the parts related to the present invention are schematically shown in each figure, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some figures, only one of the parts with the same structure or function is schematically drawn, or only one of them is marked. In this article, "one" does not mean that the number of the relevant parts of the present invention is limited to "only one", and "one" does not mean that the number of the relevant parts of the present invention is "more than one". In this article, "upper", "lower", "front", "back", "left", "right", "inside", "outside", etc. are only used to indicate the relative position relationship between the relevant parts, rather than to limit the absolute position of these relevant parts.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A virtual-real alignment method based on mixed reality, characterized in that: include: Creating a three-dimensional model of a real scene, wherein the physical properties of the three-dimensional model are consistent with the physical properties of the real scene; Determining an anchor point in the three-dimensional model, wherein the anchor point corresponds to a feature point in the real scene; Rendering and presenting the three-dimensional model in a display unit of a mixed reality device; Anchor points in the presented three-dimensional model are aligned with feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene.

2. The method according to claim 1, wherein the step of creating a three-dimensional model of a real scene comprises at least one of the following: Using a three-dimensional laser scanner to obtain three-dimensional coordinate data of the real scene; generating the three-dimensional model based on the three-dimensional coordinate data; Taking and measuring photos of the real scene; converting the photos into three-dimensional coordinate data; and generating the three-dimensional model based on the three-dimensional coordinate data; Based on the built-in sensor device of the mixed reality device, the real scene is scanned to generate the three-dimensional model.

3. According to the method of claim 1, aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene comprises: When the mixed reality device moves to the feature point of the real scene, receiving an alignment instruction issued by a user via a handle of the mixed reality device; Based on the alignment instruction, the three-dimensional model aligned with the real scene is re-rendered and re-presented in a manner that the anchor point of the three-dimensional model is aligned with the feature point.

4. According to the method of claim 1, aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene comprises: Arranging a ranging system in the real scene; Based on the ranging system, determining the current position of the mixed reality device in the real scene coordinate system; Determining a position offset between the current position and the feature point in the real scene; Based on the position offset, the three-dimensional model aligned with the real scene is re-rendered and re-presented in a manner that the anchor point of the three-dimensional model is aligned with the feature point.

5. The method according to any one of claims 1 to 4, characterized in that The anchor point includes at least one of the following: Single anchor point; At least two anchor points that are not in the same straight line; At least three anchor points that are not in the same line and plane.

6. The method according to any one of claims 1 to 4, characterized in that The number of the mixed reality devices is multiple; The rendering and presenting of the three-dimensional model in the display unit of the mixed reality device comprises: rendering and presenting a respective three-dimensional model in the display unit of each mixed reality device; The step of aligning the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-reality alignment of the presented three-dimensional model with the real scene includes: aligning the anchor points in the three-dimensional model presented in each mixed reality device with the feature points in the real scene to achieve virtual-reality alignment of the three-dimensional model presented in each mixed reality device with the real scene.

7. A virtual-real alignment device based on mixed reality, characterized in that: include: A creation module, used to create a three-dimensional model of a real scene, wherein the physical properties of the three-dimensional model are consistent with the physical properties of the real scene; A determination module, configured to determine an anchor point in the three-dimensional model, wherein the anchor point corresponds to a feature point in the real scene; A rendering module, used for rendering and presenting the three-dimensional model in a display unit of a mixed reality device; The alignment module is used to align the anchor points in the presented three-dimensional model with the feature points in the real scene to achieve virtual-real alignment between the presented three-dimensional model and the real scene.

8. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the virtual-reality alignment method based on mixed reality according to any one of claims 1-6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the virtual-real alignment method based on mixed reality described in any one of claims 1-6 is implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the virtual-real alignment method based on mixed reality according to any one of claims 1 to 6.

Images