Virtual-real fusion method, system and device based on spatial positioning and storage medium

Abstract

This invention discloses a method, system, device, and storage medium for virtual-real fusion based on spatial positioning. The method includes: determining multiple spatial positioning base points in a real-world scene and determining the relative positional relationship between each spatial positioning base point and the physical object to be interacted with; constructing a first coordinate system in a virtual digital space and determining the first coordinate of each spatial positioning base point in the first coordinate system; modeling the real-world scene according to the first coordinate system to obtain a virtual digital model, and then determining the model area to be fused in the virtual digital model according to the relative positional relationship and the first coordinate; and performing virtual-real fusion between the model area to be fused and the physical object to be interacted with based on the spatial positioning base points. This invention reduces the number of comparisons and identifications between physical objects and digital models, improving the efficiency of mixed reality display and interaction; it also ensures the real-time and preparedness of virtual-real fusion, greatly enhancing the user's interactive experience, and can be widely applied in the field of mixed reality technology.

Description

Technical Field

[0001] This invention relates to the field of mixed reality technology, and in particular to a method, system, device, and storage medium for virtual-real fusion based on spatial positioning. Background Technology

[0002] Mixed Reality (MR) is a further development of Virtual Reality (VR). It can utilize computer graphics technology, sensing technology, and related technologies and equipment such as visual wearable devices to create a visual environment where digital virtual objects and real-world objects coexist. It also enables users to build an interactive feedback loop between the virtual and real worlds based on their normal perception of the real world, achieving timely and in-depth interaction between the virtual and real worlds.

[0003] Mixed reality can overlay digital objects onto the real world, or virtualize real objects onto a virtual environment. However, it is not a simple overlay, but a deep fusion of the virtual and the real, thus forming an organic unity.

[0004] Currently, the mainstream hardware supporting MR technology mainly includes the Microsoft HoloLens 2 headset. When using MR headsets, the process of spatial positioning that combines virtual models with the real world is often used. The mainstream positioning methods include local spatial positioning points of the headset, Microsoft Azure cloud spatial points, and VuForia Engine spatial positioning.

[0005] Spatial positioning points represent important points tracked by the system over a period of time. Each positioning point has an adjustable coordinate system (based on other positioning points or reference frames) to ensure the accuracy of the holographic imagery. Rendering a hologram within the coordinate system of the positioning point allows for precise localization of that hologram at any given time. This represents a small adjustment cost over time as the system continuously returns it to a location based on the real world. These adjustment costs include device latency in the helmet itself, comparison and analysis of physical world photo data and cloud-based model data, and the process of determining positioning points and finally merging the virtual model with the real object. These processes introduce data delays and errors, ultimately leading to errors in the actual integration of the virtual model with the physical object, as well as display delays caused by asynchrony during movement. This negatively impacts the final customer experience, especially in complex interactive scenarios with multiple interaction points and multiple models, such as an aircraft cockpit involving many dashboards, buttons, levers, etc. If all operations are performed by constantly comparing and confirming spatial points between physical and virtual models and then visually merging them, significant delays and errors will occur, ultimately affecting the customer experience. Summary of the Invention

[0006] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0007] Therefore, one objective of this invention is to provide a virtual-real fusion method based on spatial positioning. This method reduces the number and process of comparing and recognizing physical objects with digital models, greatly reducing the complexity of the system's computational process in complex scene interactions, improving the efficiency of mixed reality display and interaction, and reducing the requirements for system computing power. On the other hand, it avoids the delay and stuttering phenomenon in the process of aligning and displaying digital content with physical objects due to network latency and the influence of device computing resources, ensuring the real-time and preparedness of virtual-real fusion, and greatly improving the user's interactive experience.

[0008] Another objective of this invention is to provide a virtual-real fusion system based on spatial positioning.

[0009] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include:

[0010] In a first aspect, embodiments of the present invention provide a virtual-real fusion method based on spatial positioning, comprising the following steps:

[0011] Determine multiple spatial positioning base points in the real-world scene, and determine the relative positional relationship between each of the spatial positioning base points and the object to be interacted with;

[0012] Construct a first coordinate system for the virtual digital space, and determine the first coordinates of each of the spatial positioning base points in the first coordinate system;

[0013] The real scene is modeled according to the first coordinate system to obtain a virtual digital model, and then the model area to be fused is determined in the virtual digital model according to the relative position relationship and the first coordinate.

[0014] The virtual-real fusion is performed on the model region to be fused and the physical object to be interacted with based on the spatial positioning base point.

[0015] Furthermore, in one embodiment of the present invention, the step of determining multiple spatial positioning base points in the real-world scene and determining the relative positional relationship between each spatial positioning base point and the object to be interacted with specifically includes:

[0016] Acquire first-image data of the real-world scene using a head-mounted MR device;

[0017] Based on the first image data, multiple spatial positioning base points are determined, and the physical object to be interacted with is identified.

[0018] The relative positional relationship is determined based on the spatial positioning base point and the spatial coordinates of the object to be interacted with.

[0019] Furthermore, in one embodiment of the present invention, the spatial positioning base point and the spatial position coordinates of the interactive object are both obtained through a binocular vision ranging algorithm.

[0020] Furthermore, in one embodiment of the present invention, the head-mounted MR device has a built-in image capture device, which is a camera or an infrared image capture device.

[0021] Furthermore, in one embodiment of the present invention, the step of constructing a first coordinate system for the virtual digital space and determining the first coordinates of each of the spatial positioning base points in the first coordinate system specifically includes:

[0022] A first coordinate system for constructing a virtual digital space is established with the aforementioned head-mounted MR device as the origin;

[0023] The first coordinates of the spatial positioning base point in the first coordinate system are determined based on the first image data.

[0024] Furthermore, in one embodiment of the present invention, the step of determining the model region to be fused in the virtual digital model based on the relative positional relationship and the first coordinates specifically includes:

[0025] The second coordinates of the object to be interacted with in the first coordinate system are determined based on the relative positional relationship and the first coordinates.

[0026] The region of the model to be fused is determined based on the second coordinates and the virtual digital model.

[0027] Furthermore, in one embodiment of the present invention, the step of performing virtual-real fusion of the model region to be fused and the physical object to be interacted with based on the spatial positioning base point specifically includes:

[0028] The second image data of the object to be interacted with is determined based on the spatial positioning base point and the first image data.

[0029] The model region to be fused is then fitted with the second image data to obtain a virtual-real fused image;

[0030] The virtual-real fusion image is displayed using the head-mounted MR device.

[0031] Secondly, embodiments of the present invention provide a virtual-real fusion system based on spatial positioning, comprising:

[0032] The spatial positioning base point determination module is used to determine multiple spatial positioning base points in the real scene and to determine the relative positional relationship between each spatial positioning base point and the object to be interacted with.

[0033] The first coordinate system construction module is used to construct the first coordinate system of the virtual digital space and determine the first coordinates of each of the spatial positioning base points in the first coordinate system;

[0034] The module for determining the model region to be merged is used to model the real scene according to the first coordinate system to obtain a virtual digital model, and then determine the model region to be merged in the virtual digital model according to the relative positional relationship and the first coordinate.

[0035] The virtual-real fusion module is used to perform virtual-real fusion of the model area to be fused and the physical object to be interacted with based on the spatial positioning base point.

[0036] Thirdly, embodiments of the present invention provide a virtual-real fusion device based on spatial positioning, comprising:

[0037] At least one processor;

[0038] At least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described spatial positioning-based virtual-real fusion method.

[0040] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described spatial positioning-based virtual-real fusion method.

[0041] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0042] This invention first determines multiple spatial positioning base points in the real-world scene and their relative positions to the objects to be interacted with. Then, it constructs a first coordinate system in a virtual digital space and determines the first coordinates of the spatial positioning base points within this system. Next, it models the real-world scene using this first coordinate system to obtain a virtual digital model. Finally, based on the relative positions and the first coordinates, it determines the area to be fused within the virtual digital model. This allows for virtual-real fusion of the area to be fused and the objects to be interacted with, based on the spatial positioning base points. This invention reduces the number and process of comparing and recognizing objects with digital models, significantly reducing the complexity of system calculations in complex scene interactions, improving the efficiency of mixed reality display and interaction, and reducing the computational demands on the system. Furthermore, it avoids delays and stutters in the integration and display of digital content and objects due to network latency and device computing resources, ensuring the real-time and preparedness of virtual-real fusion and greatly enhancing the user's interactive experience. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating the steps of a virtual-real fusion method based on spatial positioning provided in an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the parameters of the binocular vision ranging algorithm provided in an embodiment of the present invention;

[0046] Figure 3 A structural block diagram of a virtual-real fusion system based on spatial positioning provided in an embodiment of the present invention;

[0047] Figure 4 This is a structural block diagram of a virtual-real fusion device based on spatial positioning, provided in an embodiment of the present invention. Detailed Implementation

[0048] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0049] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0050] Reference Figure 1 This invention provides a virtual-real fusion method based on spatial positioning, specifically including the following steps:

[0051] S101. Determine multiple spatial positioning base points in the real scene, and determine the relative positional relationship between each spatial positioning base point and the object to be interacted with.

[0052] Specifically, in this embodiment of the invention, multiple physical objects in a real-world scenario are selected as spatial positioning base points to facilitate subsequent positioning and virtual-real fusion based on these base points. Step S101 specifically includes the following steps:

[0053] S1011. Acquire first image data of the real scene through a head-mounted MR device;

[0054] S1012. Determine multiple spatial positioning base points based on the first image data, and determine the physical object to be interacted with;

[0055] S1013. Determine the relative positional relationship based on the spatial positioning base point and the spatial position coordinates of the object to be interacted with.

[0056] Specifically, in this embodiment of the invention, a physical object in a real-world scene that is relatively easy to anchor is selected as the initial spatial positioning base point. Image data of the real-world scene can be acquired through a head-mounted MR device, and the spatial positioning base point is selected based on this image data. Simultaneously, the physical object to be interacted with can be identified through image recognition, and the relative positional relationship between the two can be determined. Subsequently, mixed reality spatial positioning technology can be used to compare and locate the spatial positioning base point with the virtual digital model to determine the overlap point between the digital space and the real-world scene, thus obtaining the position of the initial spatial positioning base point.

[0057] As an optional implementation, the spatial positioning base point and the spatial position coordinates of the interactive object are both obtained through a binocular vision ranging algorithm.

[0058] Specifically, the binocular vision ranging algorithm is based on the parallax of two cameras and uses the triangulation principle to acquire 3D information. Knowing the positional relationship between the two cameras, the 3D dimensions of objects within the common field of view of the two cameras and the 3D coordinates of spatial object feature points can be obtained.

[0059] like Figure 2 The figure shows a schematic diagram of the parameters of the binocular vision ranging algorithm provided in an embodiment of the present invention. In the figure, P(x c -y c -z c ) represents the spatial coordinates of the interaction point of the target object, X left X represents the x-coordinate of the imaging point of the interaction point of the target object in the left image acquired by the left camera. right Y represents the x-coordinate of the image point of the interaction point of the target object in the right image acquired by the right camera, Y represents the y-coordinate of the two aforementioned image points, f represents the focal length of the camera, and B represents the baseline distance between the left and right cameras.

[0060] Based on the principle of similar triangles, we can obtain the following formula:

[0061]

[0062] The spatial coordinates of the interaction point of the target object can be calculated using this formula.

[0063] As a further optional implementation, the head-mounted MR device has a built-in image capture device, which can be a camera or an infrared image capture device.

[0064] S102. Construct the first coordinate system of the virtual digital space and determine the first coordinates of each spatial positioning base point in the first coordinate system.

[0065] Step S102 specifically includes the following steps:

[0066] S1021. Construct the first coordinate system of the virtual digital space with the head-mounted MR device as the origin;

[0067] S1022. Determine the first coordinate of the spatial positioning base point in the first coordinate system based on the first image data.

[0068] Specifically, in this embodiment of the invention, a spatial coordinate system of a virtual scene is constructed with a head-mounted MR device as the origin. Then, through the transformation of different spatial coordinate systems, the spatial unit of this origin is made to coincide with the spatial unit of the initial spatial positioning base point determined above, and the first coordinate of the spatial positioning base point in the spatial coordinate system is determined.

[0069] S103. Model the real scene according to the first coordinate system to obtain a virtual digital model, and then determine the model area to be fused in the virtual digital model according to the relative position relationship and the first coordinate.

[0070] Specifically, in this embodiment of the invention, the 1:1 model obtained after modeling is converted into a transparent 3D line drawing. During operation, by adjusting the user's head posture, the 3D line drawing is superimposed on the influence of the real object. At this time, the head-mounted MR device will perform real-time logical algorithm calculation and recognition, and present and lock the final stable virtual fusion image based on the calculation and recognition results.

[0071] As a further optional implementation, the step of determining the region of the model to be fused in the virtual digital model based on the relative positional relationship and the first coordinates specifically includes:

[0072] A1. Determine the second coordinates of the object to be interacted with in the first coordinate system based on the relative positional relationship and the first coordinate.

[0073] A2. Determine the area of ​​the model to be fused based on the second coordinate and the virtual digital model.

[0074] Specifically, in this embodiment of the invention, the spatial coordinates of the object to be interacted with in the virtual digital model are obtained by calculating the relative position of the spatial base point and the object to be interacted with. This results in an array of spatial coordinates of the object to be interacted with and the virtual digital model. Then, according to the business scenario operation process, the specific object and interaction point are mapped through the input steps of the spatial interface operation to determine the spatial point of the object to be interacted with in the virtual digital model. This allows the digital model area (i.e., the model area to be merged) corresponding to the object to be obtained.

[0075] S104. Perform virtual-real fusion between the area of ​​the model to be fused and the physical object to be interacted with, based on the spatial positioning reference point.

[0076] Step S104 specifically includes the following steps:

[0077] S1041. Determine the second image data of the object to be interacted with based on the spatial positioning base point and the first image data;

[0078] S1042. The model region to be fused is fitted with the second image data to obtain a virtual-real fused image.

[0079] S1043. Displaying virtual-real fusion images through a head-mounted MR device.

[0080] Specifically, based on the spatial positioning points, the area of ​​the model to be fused is aligned with the corresponding position of the physical object according to the virtual-real fit parameter requirements, thus completing the alignment and visualization of the physical object and the digital model.

[0081] Optionally, in different mixed reality interaction scenarios, the fitting requirements between physical objects and digital models vary. In this embodiment of the invention, an overlap parameter calibration mechanism can be formulated. By adjusting the fitting deviation, the optimal correction value for operation can be found, and finally, the correction calculation can be completed and debugged in a real-world scenario.

[0082] Specifically, based on the physical objects corresponding to the interaction points, relative coordinate system data is calculated. Through a virtual-real integration calibration mechanism, the spatial positioning coordinates of a series of interaction points are calculated. Then, the determined spatial coordinate system is converted into the camera coordinate system in the Unity environment, and the digital model is displayed by virtual-real integration with the physical objects. This embodiment of the invention greatly facilitates mixed reality operations in real-world scenarios through the adjustment of virtual integration parameters.

[0083] This invention can be applied to MR devices, which have built-in image capture equipment. The computational space positioning coordinate system is determined by the operator's workflow and business processes within the virtual environment. The image capture equipment can be a camera, infrared image capture device, etc.

[0084] The MR device based on the embodiments of the present invention will be described using an aircraft cockpit model as an example.

[0085] 1) By model recognition, the position of the target cockpit model in space is obtained based on the principle of binocular vision ranging.

[0086] 2) After successful recognition, display the coarsely calibrated virtual cockpit model and set it to the position of the acquired target model in space. Turn off the real-time calculation of the binocular vision ranging algorithm (at this time, due to the instability of the MR device's head-mounted screen, although the algorithm is refreshed in real time, the result of taking one frame of the result as the model's posture due to the delay still has a large error).

[0087] 3) By manually fine-tuning the cockpit posture to achieve detailed fit, the virtual cockpit posture is locked after confirming the fit, thereby achieving virtual-real fit and target locking (due to the limitations of the business scenario, the height of the real cockpit and its position relative to the front panel can be adjusted, and the target being identified must be centered directly in front of the camera).

[0088] It should be noted that the spatial rapid positioning method in this embodiment of the invention is only a control technology based on MR devices, and its specific use requires the use of corresponding application software.

[0089] In addition, after locking the spatial location of the virtual target, the virtual target's display status is set to invisible, while the physical triggers and logical responses preset for interaction on the virtual target during development are still retained, thereby realizing the mapping between the digital model and the physical object.

[0090] The method steps of the embodiments of the present invention have been described above. It can be understood that the embodiments of the present invention can provide an excellent interactive experience for the mixed reality technology of three-dimensional digital aircraft systems based on the digital platform for aircraft piloting and maintenance. The embodiments of the present invention can also be applied to other mixed reality business operation scenarios. Compared with existing control technologies, the embodiments of the present invention, on the one hand, reduce the number and process of comparison and recognition between physical objects and digital models, greatly reduce the complexity of the system calculation process under complex scene interaction, improve the efficiency of mixed reality display and interaction, and reduce the requirements for system computing power; on the other hand, it avoids the delay and stuttering phenomenon in the process of fitting and displaying digital content and physical objects due to network latency and the influence of device computing resources, ensuring the real-time and readiness of virtual and real fusion, and greatly improving the user's interactive experience.

[0091] Reference Figure 3 This invention provides a virtual-real fusion system based on spatial positioning, comprising:

[0092] The spatial positioning base point determination module is used to determine multiple spatial positioning base points in the real scene and to determine the relative positional relationship between each spatial positioning base point and the object to be interacted with.

[0093] The first coordinate system construction module is used to construct the first coordinate system of the virtual digital space and determine the first coordinates of each spatial positioning base point in the first coordinate system;

[0094] The module for determining the region of the model to be merged is used to model the real scene according to the first coordinate system to obtain a virtual digital model, and then determine the region of the model to be merged in the virtual digital model according to the relative position relationship and the first coordinate.

[0095] The virtual-real fusion module is used to perform virtual-real fusion between the area of ​​the model to be fused and the physical object to be interacted with, based on the spatial positioning reference point.

[0096] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0097] Reference Figure 4 This invention provides a virtual-real fusion device based on spatial positioning, comprising:

[0098] At least one processor;

[0099] At least one memory for storing at least one program;

[0100] When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned virtual-real fusion method based on spatial positioning.

[0101] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0102] This invention also provides a computer-readable storage medium storing a processor-executable program that, when executed by a processor, performs the aforementioned spatial positioning-based virtual-real fusion method.

[0103] A computer-readable storage medium according to an embodiment of the present invention can execute a virtual-real fusion method based on spatial positioning provided in an embodiment of the present invention, and can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0104] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0105] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0106] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0107] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0109] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0110] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0111] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0113] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A virtual-real fusion method based on spatial positioning, characterized in that, Includes the following steps: Determine multiple spatial positioning base points in the real-world scene, and determine the relative positional relationship between each of the spatial positioning base points and the object to be interacted with, specifically including: Acquire first-image data of the real-world scene using a head-mounted MR device; Based on the first image data, multiple spatial positioning base points are determined, and the physical object to be interacted with is identified. The object to be interacted with is a real object that is relatively easy to anchor in the real scene, and its three-dimensional information is obtained based on the principle of triangulation. The relative positional relationship is determined based on the spatial positioning base point and the spatial coordinates of the object to be interacted with. A calibration mechanism for overlap parameters is established to dynamically correct the relative positional relationship. The adjustment is made based on the deviation of the fit to find the optimal correction value for operation. Finally, the correction calculation is completed and debugged in a real-world scenario. The overlap parameter calibration mechanism specifically includes: calculating relative coordinate system data based on the physical object corresponding to the interaction point, calculating the spatial positioning coordinates of a series of interaction points through the virtual-real bonding calibration mechanism, then converting the determined spatial coordinate system into the camera coordinate system in the Unity environment, and displaying the digital model and the physical object in a virtual-real bonding manner. The spatial positioning base point and the spatial position coordinates of the object to be interacted with are both obtained through a binocular vision ranging algorithm. The head-mounted MR device has a built-in image capture device, which is a camera or an infrared image capture device. Construct a first coordinate system for the virtual digital space, and determine the first coordinates of each of the spatial positioning base points in the first coordinate system; The real-world scene is modeled using the first coordinate system to obtain a virtual digital model. Then, the region to be fused within the virtual digital model is determined based on the relative positional relationships and the first coordinates. This process specifically includes: The second coordinates of the object to be interacted with in the first coordinate system are determined based on the relative positional relationship and the first coordinates. The region of the model to be fused is determined based on the second coordinates and the virtual digital model; Based on the spatial positioning base point, the virtual-real fusion is performed on the model region to be fused and the physical object to be interacted with. The 1:1 model obtained after modeling is converted into a transparent 3D line drawing. During runtime, by adjusting the user's head posture, the 3D line drawing is superimposed on the influence of the real object, thereby improving the efficiency of mixed reality display and interaction and reducing the requirements for system computing power.

2. The virtual-real fusion method based on spatial positioning according to claim 1, characterized in that, The step of constructing a first coordinate system for the virtual digital space and determining the first coordinates of each spatial positioning base point in the first coordinate system specifically includes: A first coordinate system for constructing a virtual digital space is established with the aforementioned head-mounted MR device as the origin; The first coordinates of the spatial positioning base point in the first coordinate system are determined based on the first image data.

3. The virtual-real fusion method based on spatial positioning according to claim 1, characterized in that, The step of performing virtual-real fusion of the model region to be fused and the physical object to be interacted with based on the spatial positioning base point specifically includes: The second image data of the object to be interacted with is determined based on the spatial positioning base point and the first image data. The model region to be fused is then fitted with the second image data to obtain a virtual-real fused image; The virtual-real fusion image is displayed using the head-mounted MR device.

4. A virtual-real fusion system based on spatial positioning, characterized in that, include: The spatial positioning base point determination module is used to determine multiple spatial positioning base points in the real-world scene and to determine the relative positional relationship between each spatial positioning base point and the object to be interacted with. Specifically, it includes: Acquire first-image data of the real-world scene using a head-mounted MR device; Based on the first image data, multiple spatial positioning base points are determined, and the physical object to be interacted with is identified. The object to be interacted with is a real object that is relatively easy to anchor in the real scene, and its three-dimensional information is obtained based on the principle of triangulation. The relative positional relationship is determined based on the spatial positioning base point and the spatial coordinates of the object to be interacted with. An overlap parameter calibration mechanism is established to dynamically correct the relative positional relationship. Adjustments are made based on the deviation in the fit to find the optimal correction value. Finally, the correction is calculated and tested in a real-world scene. Specifically, the overlap parameter calibration mechanism includes: calculating relative coordinate system data based on the physical object corresponding to the interaction point; calculating the spatial positioning coordinates of a series of interaction points through a virtual-real fit calibration mechanism; then converting the determined spatial coordinate system into the camera coordinate system in the Unity environment; and finally, displaying the digital model and the physical object through virtual-real fit. The spatial positioning base point and the spatial position coordinates of the object to be interacted with are both obtained through a binocular vision ranging algorithm. The head-mounted MR device has a built-in image capture device, which is a camera or an infrared image capture device. The first coordinate system construction module is used to construct the first coordinate system of the virtual digital space and determine the first coordinates of each of the spatial positioning base points in the first coordinate system; The module for determining the model region to be merged is used to model the real scene according to the first coordinate system to obtain a virtual digital model, and then determine the model region to be merged in the virtual digital model according to the relative positional relationship and the first coordinates. Specifically, it includes: The second coordinates of the object to be interacted with in the first coordinate system are determined based on the relative positional relationship and the first coordinates. The region of the model to be fused is determined based on the second coordinates and the virtual digital model; The virtual-real fusion module is used to perform virtual-real fusion of the model region to be fused and the physical object to be interacted with based on the spatial positioning base point; The 1:1 model obtained after modeling is converted into a transparent 3D line drawing. During runtime, by adjusting the user's head posture, the 3D line drawing is superimposed on the influence of the real object, thereby improving the efficiency of mixed reality display and interaction and reducing the requirements for system computing power.

5. A virtual-real fusion device based on spatial positioning, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a virtual-real fusion method based on spatial positioning as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform a spatial positioning-based virtual-real fusion method as described in any one of claims 1 to 3.

Images