Mixed reality accurate fusion method, system and equipment
Through multi-sensor collaborative data acquisition and nonlinear optimization methods, the problems of incomplete environmental information and low user experience in the existing technology are solved, and high-precision fusion and dynamic optimization in mixed reality are achieved, which improves the user experience.
Patent Information
- Application Number
- CN202411965416.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-15
AI Technical Summary
The existing mixed reality fusion technology has problems such as incomplete acquisition of environmental information, sensitive to environmental changes and lack of user feedback mechanisms, resulting in unstable fusion effect and reduced user experience.
Multi-sensors are used to collect environmental data in a coordinated manner, and multi-dimensional features are extracted in combination with the scale invariant feature transformation method. Feature matching and pose estimation are performed through nonlinear optimization methods, so as to achieve high-precision fusion of virtual content and real environment, and dynamic optimization is performed through user feedback mechanism.
It improves the integrity of environmental information collection and the robustness of the system in complex environments, realizes the precise integration of virtual content and real environments, and improves user experience and interactivity.
Smart Images
Figure CN119832192A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of mixed reality technology, and in particular to a mixed reality precise fusion method, system and device. Background Art
[0002] Mixed reality technology provides users with an immersive interactive experience by seamlessly integrating virtual content with the real environment. With the rapid development of computer vision, sensor technology, and artificial intelligence, mixed reality technology has been widely used in education and training, industrial manufacturing, medical surgery, and other fields. Achieving accurate integration of virtual content and the real environment is one of the core challenges of mixed reality technology.
[0003] The current mixed reality fusion technology has the following main problems: First, the existing technology mainly relies on a single sensor to collect environmental data, resulting in incomplete environmental information and affecting the stability of the fusion effect; second, the traditional feature extraction method is sensitive to environmental changes, and feature matching failures are prone to occur in complex scenarios such as lighting changes and occlusions; third, the existing fusion method lacks an effective user feedback mechanism and cannot dynamically adjust the fusion effect according to user needs, which reduces the user experience.
[0004] The present invention aims to solve the above-mentioned problems existing in the existing mixed reality fusion technology, and provide a mixed reality precise fusion method based on multi-sensor data fusion and nonlinear optimization. Through technical means such as multi-dimensional data collection, feature extraction and matching, and dynamic optimization, high-precision fusion of virtual content and real environment is achieved, thereby improving the user experience of mixed reality applications. Summary of the invention
[0005] The embodiments of the present invention provide a method, system and device for accurate fusion of mixed reality, which are used to solve the problems of low fusion accuracy and unnatural images in the prior art.
[0006] A first aspect of the present invention provides a mixed reality precise fusion method, the precise fusion method comprising:
[0007] The multi-dimensional data of the real environment is collected through multiple sensors, normalized, and the visual features, depth features and motion features are extracted using a scale-invariant feature transformation method to obtain key feature points of the real environment; the multiple sensors include a high-definition camera, a depth sensor, a temperature and humidity sensor, a sound sensor and an inertial measurement unit;
[0008] Determine the function of the virtual model in mixed reality according to the application scenario of mixed reality, build the virtual model, and mark the key feature points of the virtual model;
[0009] Match the key feature points of the virtual model with the key feature points of the real environment, minimize the error between the virtual model and the real environment using a nonlinear optimization method, and obtain the initial optimal position and posture of the virtual model in the real environment;
[0010] Based on the position deviation of the object in the real environment and the virtual model, the initial optimal position and posture of the virtual model in the real environment are accurately adjusted.
[0011] Furthermore, the scale-invariant feature transformation method is used to extract visual features, depth features and motion features to obtain key feature points of the real environment, including:
[0012] Extract image features from image data, extract the surface shape and edge features of objects from depth data, and extract the motion trajectory, speed and acceleration features of objects from motion data;
[0013] Detecting key points of the image data, the depth data, and the motion data, calculating feature descriptors corresponding to the key points to screen visual features, depth features, and motion features, and removing redundant and / or irrelevant features;
[0014] Match the filtered features with the feature template to identify the key feature points in the real environment.
[0015] Furthermore, determining the function of the virtual model in the mixed reality according to the application scenario of the mixed reality, constructing the virtual model, and marking the key feature points of the virtual model include:
[0016] Determine the function of the virtual model in mixed reality according to the application scenario of mixed reality, and set the performance requirements of the virtual model according to the function;
[0017] The visual features, spatial features and semantic features of the virtual model are marked as key feature points of the virtual model by using color marking, label marking and / or geometric shape marking.
[0018] Furthermore, minimizing the error between the virtual model and the real environment by a nonlinear optimization method includes:
[0019] By calculating the distance between the key feature point P(x1, y1, z1) of the virtual model and the corresponding key feature point Q(x2, y2, z2) of the real environment, the error between the virtual model and the real environment is characterized:
[0020]
[0021] Among them, d(P,Q) represents the distance between the key feature point P(x1,y1,z1) and the key feature point Q(x2,y2,z2),
[0022] (x1,y 1, z1) represents the X / Y / Z coordinates of the key feature point P, (x2, y 2, z2) represents the X / Y / Z coordinates of the key feature point Q.
[0023] Furthermore, the precise adjustment includes:
[0024] Obtain the position information of objects in the real environment, compare it with the current position of the virtual model, calculate the position deviation, re-extract the key feature points in the real environment and the virtual model, perform feature matching, and calculate the posture deviation;
[0025] With the position deviation and the posture deviation becoming smaller as the adjustment direction, the initial optimal position and posture of the virtual model in the real environment are accurately adjusted.
[0026] Furthermore, the position and posture of the precisely adjusted virtual model in the real environment are dynamically adjusted online, including:
[0027] A user interaction interface is set up, and users can dynamically optimize the position and posture of the virtual model in the real environment online according to their operations and changes in the environment.
[0028] Furthermore, the dynamic optimization includes:
[0029] Detect the position and posture deviation of the virtual model fed back by the user through the interactive interface, and analyze the cause of the deviation;
[0030] Based on the analysis results of the causes of deviation, the position and posture parameters of the virtual model are adjusted in real time to achieve dynamic optimization of the position and posture of the online simulation model in the real environment.
[0031] A second aspect of an embodiment of the present invention provides a mixed reality precise fusion system for implementing the mixed reality precise fusion method described above. The precise fusion system includes:
[0032] The data acquisition and processing module is configured to collect multi-dimensional data of the real environment through multiple sensors, perform normalization processing, and extract visual features, depth features and motion features using a scale-invariant feature transformation method to obtain key feature points of the real environment; the multiple sensors include a high-definition camera, a depth sensor, a temperature and humidity sensor, a sound sensor and an inertial measurement unit;
[0033] A virtual model building module, configured to determine the function of the virtual model in the mixed reality according to the application scenario of the mixed reality, build the virtual model, and mark the key feature points of the virtual model;
[0034] A mixed reality fusion module is configured to match key feature points of the virtual model with key feature points of the real environment, minimize the error between the virtual model and the real environment by a nonlinear optimization method, and obtain an initial optimal position and posture of the virtual model in the real environment;
[0035] The fusion adjustment module is configured to accurately adjust the initial optimal position and posture of the virtual model in the real environment based on the position deviation of the object in the real environment and the virtual model.
[0036] According to a third aspect of the present invention, there is provided an electronic device, comprising:
[0037] at least one processor;
[0038] and a memory communicatively coupled to at least one of the processors;
[0039] Among them, the memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned mixed reality precise fusion method.
[0040] According to a fourth aspect of the present invention, the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned mixed reality precise fusion method.
[0041] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:
[0042] (1) The present invention uses multiple sensors to collaboratively collect environmental data and extracts multidimensional features in combination with a scale-invariant feature transformation method, thereby achieving comprehensive perception of the real environment and accurate feature extraction. The introduction of multiple sensors not only improves the integrity of environmental information collection, but also enhances the robustness of the system in complex environments, effectively overcoming the limitations of a single sensor in different scenarios.
[0043] (2) The present invention uses a nonlinear optimization algorithm for feature matching and pose estimation, and combines the visual features, spatial features, and semantic features of the virtual model to achieve accurate integration of virtual content and the real environment. This method can effectively handle the nonlinear relationship between feature points and improve the positioning accuracy and pose accuracy of the virtual model in the real scene.
[0044] (3) By establishing a user feedback mechanism and a dynamic optimization strategy, the present invention enables the system to adjust the position and posture of the virtual model in real time according to the user's interaction needs and environmental changes, providing a more natural and smooth mixed reality experience. This dynamic optimization mechanism significantly improves the interactivity and user experience of the system, making the integration of virtual content and the real environment more harmonious and natural. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 The present invention is a flowchart of a method for accurate fusion of mixed reality according to an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0048] Figure 1 FIG. 1 is a flow chart of a method for accurate fusion of mixed reality according to an embodiment of the present invention. Figure 1 As shown, the precise fusion method includes:
[0049] Step S1, collect multi-dimensional data of the real environment through multiple sensors, perform normalization processing, and use a scale-invariant feature transformation method to extract visual features, depth features and motion features to obtain key feature points of the real environment.
[0050] The multiple sensors include high-definition camera, depth sensor, temperature and humidity sensor, sound sensor and inertial measurement unit.
[0051] High-definition cameras are used to capture images of the real environment;
[0052] Depth sensors are used to measure the distance between objects and sensors in real environments and generate depth maps. The acquisition of depth information is performed simultaneously with image acquisition.
[0053] The temperature and humidity sensor is used to collect temperature and humidity data in the real environment;
[0054] Sound sensors are used to collect sounds in real environments;
[0055] The inertial measurement unit is used to measure the acceleration, angular velocity and orientation information of the sensor to determine the position and attitude of the sensor.
[0056] It should be added that sensors can collect data in real time, allowing the mixed reality system to keep abreast of changes in the real environment. This is very important for applications that require dynamic interaction with the real environment. For example, in augmented reality navigation, real-time updated environmental data can ensure that virtual navigation instructions always maintain accurate correspondence with roads and buildings in the real environment.
[0057] The data collected by sensors can provide contextual information about the real environment, allowing virtual elements to better adapt to different scenes and situations. For example, in an augmented reality tour of a museum, sensors can identify the user's location in the exhibition hall and the exhibits around them, and provide users with corresponding virtual explanations and interactive content.
[0058] Sensors can collect users' biometric data, such as eye movements, gestures, and voice, to provide personalized interaction methods for mixed reality applications. For example, through eye tracking technology, the system can automatically adjust the display position and size of virtual elements according to the user's line of sight; through gesture recognition, users can naturally interact with virtual elements without using traditional input devices.
[0059] The collected data are cleaned and different types of data are normalized to have the same scale and range.
[0060] The scale-invariant feature transformation method is used to extract visual features, depth features and motion features to obtain the key feature points of the real environment, including:
[0061] Step S11, extracting image features from the image data, extracting the surface shape and edge features of the object from the depth data, and extracting the motion trajectory, speed and acceleration features of the object from the motion data.
[0062] Step S12, detecting key points of the image data, the depth data and the motion data, calculating feature descriptors corresponding to the key points, screening visual features, depth features and motion features, and removing redundant and / or irrelevant features.
[0063] Step S13, matching the filtered features with the feature template to identify key feature points in the real environment.
[0064] It should be added that by extracting key feature points, a large amount of raw data can be compressed into a small amount of feature descriptions, thereby reducing the need for data storage and transmission. This is very important for mixed reality devices and network environments with limited resources, and can improve the operating efficiency and response speed of the system.
[0065] Key feature points can effectively describe the key features and structures of the real environment, enabling the mixed reality system to better understand and perceive the surrounding environment. For example, in an indoor environment, extracting feature points such as wall corners, door and window edges can help the system determine the layout and size of the room; in an outdoor environment, extracting feature points such as building corners and landmarks can help the system locate and navigate.
[0066] The extracted key feature points can be used for feature matching and alignment between the virtual model and the real environment, thereby determining the exact position and posture of the virtual model in the real environment. This is a key step in achieving precise fusion, enabling the virtual elements to blend seamlessly with the real environment, enhancing the user's sense of immersion and reality.
[0067] By extracting a large number of key feature points, 3D reconstruction and modeling of the real environment can be performed, which provides richer and more realistic virtual scenes for mixed reality applications, and also provides valuable data and tools for other fields such as virtual reality, architectural design and geographic information systems.
[0068] Step S2, determining the function of the virtual model in the mixed reality according to the application scenario of the mixed reality, constructing the virtual model, and marking the key feature points of the virtual model.
[0069] Determine the function of the virtual model in mixed reality according to the application scenario of mixed reality, build the virtual model, and mark the key feature points of the virtual model, including:
[0070] Step S21, determining the function of the virtual model in the mixed reality according to the application scenario of the mixed reality, and setting the performance requirements of the virtual model according to the function.
[0071] Step S22: Use color marking, label marking and / or geometric shape marking to mark the visual features, spatial features and semantic features of the virtual model as key feature points of the virtual model.
[0072] In the implementation of the present invention, the information of key feature point markings is stored together with the virtual model.
[0073] It should be added that building virtual models according to specific application scenarios and user needs can achieve highly customized designs. Whether it is education, medical care, entertainment or industry, virtual models can be built according to specific teaching content, medical cases, game plots or industrial processes to meet the professional needs of different fields.
[0074] Marking the virtual model with features can better integrate the virtual model with the real environment. Feature marking can serve as a basis for matching and aligning the virtual model with the real environment, helping to determine the position and posture of the virtual model in the real environment. For example, in augmented reality applications, key parts of the virtual model can be marked with features to accurately align and integrate them with objects in the real environment.
[0075] By marking virtual models with features, the user's operating efficiency on the virtual models can be improved. For example, in virtual assembly applications, the assembly interfaces of virtual parts can be marked with features, allowing users to quickly and accurately assemble parts; in virtual training applications, the operation buttons of virtual devices can be marked with features, allowing users to operate and control the devices more intuitively.
[0076] Step S3, matching the key feature points of the virtual model with the key feature points of the real environment, minimizing the error between the virtual model and the real environment by a nonlinear optimization method, and obtaining the initial optimal position and posture of the virtual model in the real environment.
[0077] Match the visual features, spatial features, and semantic features in the virtual model with the features of the real environment, including:
[0078] Step S311, extracting key feature points in the real environment and the virtual model, and establishing a matching relationship by calculating the similarity between the key feature points;
[0079] Step S312, dividing the real environment and the virtual model into different areas, and matching them by comparing the features of the areas; using the known geometric shape and structural information of the virtual model, matching them with the objects in the real environment.
[0080] Step S313, after the initial corresponding relationship is obtained through feature matching, the corresponding relationship can be used to estimate the initial position and posture of the virtual model in the real environment.
[0081] Minimize the error between the virtual model and the real environment using nonlinear optimization methods, including:
[0082] Step S321, characterizes the error between the virtual model and the real environment by calculating the distance between the key feature points P (x1, y1, z1) of the virtual model and the corresponding key feature points Q (x2, y2, z2) of the real environment, that is, the matching relationship is established by calculating the similarity between the key feature points in step S311.
[0083] Step S322: The distance between the key feature point P (x1, y1, z1) of the virtual model and the key feature point Q (x2, y2, z2) of the corresponding real environment is calculated by the following formula:
[0084]
[0085] Among them, d(P,Q) represents the distance between the key feature point P(x1,y1,z1) and the key feature point Q(x2,y2,z2),
[0086] (x1,y 1, z1) represents the X / Y / Z coordinates of the key feature point P, (x2, y 2, z2) represents the X / Y / Z coordinates of the key feature point Q.
[0087] It should be added that by accurately matching the characteristics of the virtual model with the real environment, the virtual model can be seamlessly integrated into the real environment and look like a part of the real world. This natural integration can greatly enhance the user's sense of reality and immersion, allowing users to be more engaged in the mixed reality experience.
[0088] In some applications that require high-precision positioning, such as industrial design and medical surgery, it is crucial to accurately determine the position and posture of the virtual model in the real environment. Through feature matching, the virtual model can be accurately positioned to ensure that it is accurately aligned with objects or scenes in the real environment, providing reliable support for professional applications.
[0089] After determining the position and posture of the virtual model, the user can operate the virtual model more conveniently. For example, the user can directly click virtual buttons and drag virtual objects in the real environment, which is more natural and convenient.
[0090] Step S4, accurately adjusting the initial optimal position and posture of the virtual model in the real environment based on the position deviation of the object in the real environment and the virtual model, including:
[0091] Step S41, obtaining the position information of the object in the real environment, comparing it with the current position of the virtual model, calculating the position deviation, re-extracting the key feature points in the real environment and the virtual model and performing feature matching, and calculating the posture deviation;
[0092] Step S42, with the position deviation and the posture deviation becoming smaller as the adjustment direction, accurately adjusting the initial optimal position and posture of the virtual model in the real environment.
[0093] It should be added that through precise adjustment, the virtual model can be integrated more naturally with the real environment, reducing the visual disharmony. For example, the lighting, material, and color of the virtual model can be adjusted to match the lighting conditions and object materials in the real environment, thereby enhancing the sense of reality.
[0094] Step S5, performing online dynamic adjustment on the position and posture of the precisely adjusted virtual model in the real environment, including:
[0095] A user interaction interface is set up, and users can dynamically optimize the position and posture of the virtual model in the real environment online according to their operations and environmental changes.
[0096] Among them, dynamic optimization includes:
[0097] Detect the position and posture deviation of the virtual model fed back by the user through the interactive interface, and analyze the cause of the deviation;
[0098] Based on the analysis results of the causes of deviation, the position and posture parameters of the virtual model are adjusted in real time to achieve dynamic optimization of the position and posture of the online simulation model in the real environment.
[0099] It should be added that by precisely adjusting the position, posture, size and other parameters of the virtual model, the interaction between it and the user can be more accurate and natural. For example, in an augmented reality game, adjusting the position and posture of the virtual character can make it more compatible with the user's operation, thus improving the interactivity and fun of the game.
[0100] Different users may have different needs and preferences for mixed reality scenes. By precisely adjusting the virtual model, the personalized needs of users can be met, providing a more customized mixed reality experience. For example, users can adjust the appearance, color, and function of the virtual model according to their preferences to make it conform to their own aesthetics and usage habits.
[0101] If the user feedback shows that the position of the virtual model is inaccurate or the posture is unnatural, it is necessary to further analyze the specific deviation and further adjust the fusion effect.
[0102] It should be added that each user may have different needs and expectations for the mixed reality experience. Through user feedback, developers can understand the specific needs of users and make personalized adjustments to the fusion effect, so as to better meet user expectations and improve user satisfaction.
[0103] Different user groups may have different usage habits, device configurations, and environmental conditions. Through user feedback, developers can understand the needs and problems of different user groups, make targeted adjustments to the integration effect, and improve the adaptability of the product to different user groups.
[0104] Mixed reality products may be used in different scenarios, such as indoors, outdoors, for education, and for entertainment. User feedback can help developers understand the needs and problems in different scenarios, make corresponding adjustments to the fusion effects, and improve the adaptability of products in different scenarios.
[0105] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.
[0106] A second embodiment of the present invention provides a mixed reality precise fusion system, which is used to implement the mixed reality precise fusion method described above. The precise fusion system includes:
[0107] The data acquisition and processing module is configured to collect multi-dimensional data of the real environment through multiple sensors, perform normalization processing, and extract visual features, depth features and motion features using a scale-invariant feature transformation method to obtain key feature points of the real environment; the multiple sensors include a high-definition camera, a depth sensor, a temperature and humidity sensor, a sound sensor and an inertial measurement unit;
[0108] A virtual model building module, configured to determine the function of the virtual model in the mixed reality according to the application scenario of the mixed reality, build the virtual model, and mark the key feature points of the virtual model;
[0109] A mixed reality fusion module is configured to match key feature points of the virtual model with key feature points of the real environment, minimize the error between the virtual model and the real environment by a nonlinear optimization method, and obtain an initial optimal position and posture of the virtual model in the real environment;
[0110] The fusion adjustment module is configured to accurately adjust the initial optimal position and posture of the virtual model in the real environment based on the position deviation of the object in the real environment and the virtual model.
[0111] A third embodiment of the present invention provides an electronic device, including:
[0112] at least one processor;
[0113] and a memory communicatively coupled to at least one of the processors;
[0114] Among them, the memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned mixed reality precise fusion method.
[0115] A fourth embodiment of the present invention provides that the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned mixed reality precise fusion method.
[0116] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A mixed reality precise fusion method, characterized in that: The precise fusion method includes: The multi-dimensional data of the real environment is collected through multiple sensors, normalized, and the visual features, depth features and motion features are extracted using a scale-invariant feature transformation method to obtain key feature points of the real environment; the multiple sensors include a high-definition camera, a depth sensor, a temperature and humidity sensor, a sound sensor and an inertial measurement unit; Determine the function of the virtual model in mixed reality according to the application scenario of mixed reality, build the virtual model, and mark the key feature points of the virtual model; Match the key feature points of the virtual model with the key feature points of the real environment, minimize the error between the virtual model and the real environment using a nonlinear optimization method, and obtain the initial optimal position and posture of the virtual model in the real environment; Based on the position deviation of the object in the real environment and the virtual model, the initial optimal position and posture of the virtual model in the real environment are accurately adjusted.
2. The mixed reality precise fusion method according to claim 1, characterized in that: The scale-invariant feature transformation method is used to extract visual features, depth features and motion features to obtain key feature points of the real environment, including: Extract image features from image data, extract the surface shape and edge features of objects from depth data, and extract the motion trajectory, speed and acceleration features of objects from motion data; Detecting key points of the image data, the depth data, and the motion data, calculating feature descriptors corresponding to the key points to screen visual features, depth features, and motion features, and removing redundant and / or irrelevant features; Match the filtered features with the feature template to identify the key feature points in the real environment.
3. The mixed reality precise fusion method according to claim 1, characterized in that: The step of determining the function of the virtual model in the mixed reality according to the application scenario of the mixed reality, constructing the virtual model, and marking the key feature points of the virtual model includes: Determine the function of the virtual model in mixed reality according to the application scenario of mixed reality, and set the performance requirements of the virtual model according to the function; The visual features, spatial features and semantic features of the virtual model are marked as key feature points of the virtual model by using color marking, label marking and / or geometric shape marking.
4. The mixed reality precise fusion method according to claim 1, characterized in that: The method of minimizing the error between the virtual model and the real environment by using a nonlinear optimization method comprises: By calculating the distance between the key feature point P(x1, y1, z1) of the virtual model and the corresponding key feature point Q(x2, y2, z2) of the real environment, the error between the virtual model and the real environment is characterized: Among them, d(P,Q) represents the distance between the key feature point P(x1,y1,z1) and the key feature point Q(x2,y2,z2), (x1,y 1, z1) represents the X / Y / Z coordinates of the key feature point P, (x2, y 2, z2) represents the X / Y / Z coordinates of the key feature point Q.
5. The mixed reality precise fusion method according to claim 1, characterized in that: The precise adjustment includes: Obtain the position information of objects in the real environment, compare it with the current position of the virtual model, calculate the position deviation, re-extract the key feature points in the real environment and the virtual model, perform feature matching, and calculate the posture deviation; With the position deviation and the posture deviation becoming smaller as the adjustment direction, the initial optimal position and posture of the virtual model in the real environment are accurately adjusted.
6. The mixed reality precise fusion method according to claim 1, characterized in that: For the position and posture of the precisely adjusted virtual model in the real environment, online dynamic adjustment is performed, including: A user interaction interface is set up, and users can dynamically optimize the position and posture of the virtual model in the real environment online according to their operations and changes in the environment.
7. The mixed reality precise fusion method according to claim 6, characterized in that: The dynamic optimization includes: Detect the position and posture deviation of the virtual model fed back by the user through the interactive interface, and analyze the cause of the deviation; Based on the analysis results of the causes of deviation, the position and posture parameters of the virtual model are adjusted in real time to achieve dynamic optimization of the position and posture of the online simulation model in the real environment.
8. A mixed reality precision fusion system, used to implement the mixed reality precision fusion method according to any one of claims 1 to 7, characterized in that: The precision fusion system includes: The data acquisition and processing module is configured to collect multi-dimensional data of the real environment through multiple sensors, perform normalization processing, and extract visual features, depth features and motion features using a scale-invariant feature transformation method to obtain key feature points of the real environment; the multiple sensors include a high-definition camera, a depth sensor, a temperature and humidity sensor, a sound sensor and an inertial measurement unit; A virtual model building module, configured to determine the function of the virtual model in the mixed reality according to the application scenario of the mixed reality, build the virtual model, and mark the key feature points of the virtual model; A mixed reality fusion module is configured to match key feature points of the virtual model with key feature points of the real environment, minimize the error between the virtual model and the real environment by a nonlinear optimization method, and obtain an initial optimal position and posture of the virtual model in the real environment; The fusion adjustment module is configured to accurately adjust the initial optimal position and posture of the virtual model in the real environment based on the position deviation of the object in the real environment and the virtual model.
9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to at least one of the processors; The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the mixed reality precise fusion method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the mixed reality precise fusion method described in any one of claims 1-7.