Mixed reality space registration method and device, equipment and storage medium
By setting reflective markers and feature point cloud coordinates on the probe tool for mixed reality spatial registration, the problems of low efficiency and insufficient accuracy in existing technologies are solved, achieving efficient and accurate registration without the need for pasting markers and real-time CT scanning.
Patent Information
- Application Number
- CN202512016530.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-14
AI Technical Summary
Existing mixed reality spatial registration techniques rely on markers pasted on the target surface and real-time CT scans, resulting in low registration efficiency and inaccurate results.
By setting reflective markers on the probe tool, the first point cloud coordinate set in the optical positioning space is determined, and coarse registration and alignment are performed based on the first feature point cloud coordinates and the second feature point cloud coordinates of preset feature points. Iterative fine registration is then performed by combining the initial fine registration error and error weights, thus achieving spatial registration without the need for pasting markers and real-time CT scanning.
It simplifies the registration process and improves the efficiency and accuracy of registration in mixed reality spaces.
Smart Images
Figure CN121861085A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of mixed reality technology, specifically relating to a mixed reality spatial registration method, apparatus, device, and storage medium. Background Technology
[0002] Mixed reality is a technology that overlays virtual 3D models onto real-world scenes to achieve real-time fusion and interaction between virtual content and the real world. To ensure the real-time performance and accuracy of mixed reality, efficient registration between virtual and real spaces has become a pressing issue in the field.
[0003] In existing technologies, mixed reality spatial registration primarily involves pre-attaching reflective markers or CT-visible metallic markers to the surface of the target object to be spatially registered. Real-time CT scanning acquires image data containing these markers, while an optical locator simultaneously obtains the markers' positions in real time. Registration is then performed between the image space and the optical locator space based on the markers' positions in different spatial environments. However, existing technologies rely on markers attached to the target surface and real-time CT scanning, resulting in cumbersome spatial registration steps and unstable registration processes, leading to low registration efficiency and inaccurate registration results. Summary of the Invention
[0004] The purpose of this application is to provide a mixed reality spatial registration method, apparatus, device, and storage medium, which solves the problems of low registration efficiency and inaccurate registration results in the prior art. It determines the first point cloud coordinate set in the optical positioning space by setting reflective marks on a probe tool. Based on the first and second feature point cloud coordinates of preset feature points, coarse registration and alignment are performed on the optical positioning space and the virtual space. The initial fine registration error and its weight are determined according to the coarse registration transformation result of the first and second point cloud coordinate sets in the coarse registration result. Iterative fine registration and alignment are then performed on the optical positioning space and the virtual space to determine the mixed reality spatial registration result. This achieves the goal of spatial registration without the need for pasted markers or real-time CT scanning, simplifying the registration steps and improving the efficiency and accuracy of mixed reality spatial registration.
[0005] In a first aspect, embodiments of this application provide a mixed reality space registration method, the method comprising: Obtain the coordinates of multiple markers in optical space for the reflective markers set on the probe tool, and determine the first point cloud coordinate set in optical positioning space based on the multiple marker coordinates; Obtain the second point cloud coordinate set in the virtual space, and perform coarse registration and alignment of the optical positioning space and the virtual space based on the first feature point cloud coordinates of the preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set. Based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, the initial fine registration error and the initial fine registration error weight are determined. Then, based on the initial fine registration error and the initial fine registration error weight, the optical positioning space and the virtual space are iteratively fine registered and aligned to obtain the final fine registration transformation matrix. Based on the final fine registration transformation matrix, the registration result of the mixed reality space is determined.
[0006] Furthermore, based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, the initial fine registration error and the initial fine registration error weight are determined, including: Based on the preset initial fine registration transformation relationship, the coordinates of each first coarse registration point cloud in the first point cloud coordinate set coarse registration transformation result are initially fine registered and transformed. Based on the preset nearest neighbor search function, the nearest neighbor point cloud coordinates corresponding to each first initial fine registration point cloud coordinate in the initial fine registration transformation result are determined. Calculate the coordinate deviation between each initial fine-registration point cloud coordinate and its corresponding nearest neighbor point cloud coordinate to obtain the initial fine-registration error; Calculate the first neighborhood covariance matrix of each first point cloud coordinate in the first point cloud coordinate set and the second neighborhood covariance matrix of each second point cloud coordinate in the second point cloud coordinate set. Determine the initial fine registration error weights based on the first neighborhood covariance matrix and the second neighborhood covariance matrix.
[0007] Furthermore, based on the initial fine registration error and its weight, iterative fine registration and alignment are performed between the optical positioning space and the virtual space, including: Based on the initial fine registration error and the initial fine registration error weight, the sum of the initial fine registration Mahalanobis distances is calculated; The initial fine registration transformation relationship update result between the optical positioning space and the virtual space is determined based on the sum of the initial fine registration Mahalanobis distances. The initial fine registration Mahalanobis distance sum is iteratively updated based on the initial fine registration transformation relationship update result and the second point cloud coordinates until the deviation between the update results of the sum of Mahalanobis distances of two adjacent times is less than a preset change threshold, thus obtaining the final fine registration result between the optical positioning space and the virtual space.
[0008] Furthermore, the formula for the sum of the initial fine-registration Mahalanobis distances is as follows: ; in, It is the sum of Mahalanobis distances, and N is the number of first point cloud coordinates in the first point cloud coordinate set. It is the coordinate of the i-th first point cloud. It is the nearest neighbor point cloud coordinate of the i-th point cloud coordinate in the second point cloud coordinate. It is the covariance matrix of the first neighborhood of the i-th point cloud coordinate. It is the second neighborhood covariance matrix of the nearest neighbor point cloud coordinates. It is the transformation matrix corresponding to the initial fine registration and alignment.
[0009] Furthermore, based on the coordinates of the first feature point cloud in the first point cloud coordinate set and the coordinates of the second feature point cloud in the second point cloud coordinate set, coarse registration and alignment are performed between the optical positioning space and the virtual space, including: Determine the first feature point cloud coordinates of the same preset feature points in the first point cloud coordinate set, and the second feature point cloud coordinates in the second point cloud coordinate set; The coarse registration transformation relationship between the optical positioning space and the virtual space is calculated based on the preset coarse registration algorithm, the coordinates of the first feature point cloud, and the coordinates of the second feature point cloud. Based on the coarse registration transformation relationship, the coordinates of the second point cloud in the second point cloud coordinate set are transformed to the optical positioning space, and the coarse registration alignment results of the optical positioning space and the virtual space are obtained.
[0010] Furthermore, the second point cloud coordinate set in the virtual space is obtained, including: Receive the 3D model of the target to be spatially registered in the image space, extract the point cloud from the 3D model to obtain the model point cloud; Import the model point cloud into a preset virtual space to obtain the second point cloud coordinate set under the preset virtual space.
[0011] Furthermore, there are four reflective markers, and the probe tool is equipped with an X-shaped tool for attaching the reflective markers; The first point cloud coordinate set in the optical positioning space is determined based on multiple marker coordinates, including: Based on the positional relationship between the endpoint position of the X-shaped tool and the tip position of the probe tool, and multiple marker coordinates, calculate the coordinates of multiple tip positions collected by the probe tool. By integrating the coordinates of multiple needle tips, the first point cloud coordinate set in the optical positioning space is obtained.
[0012] Secondly, embodiments of this application provide a mixed reality space registration device, the device comprising: The point cloud coordinate set determination module is used to obtain the coordinates of multiple markers of the reflective markers set on the probe tool in the optical space, and determine the first point cloud coordinate set in the optical positioning space based on the multiple marker coordinates. The coarse registration and alignment module is used to obtain the second point cloud coordinate set in the virtual space, and to perform coarse registration and alignment of the optical positioning space and the virtual space based on the first feature point cloud coordinates of the preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set. The fine registration and alignment module is used to determine the initial fine registration error and the initial fine registration error weight based on the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, and to perform iterative fine registration and alignment on the optical positioning space and the virtual space based on the initial fine registration error and the initial fine registration error weight to obtain the final fine registration transformation matrix. The spatial registration module is used to determine the spatial registration result of mixed reality based on the final fine registration transformation matrix.
[0013] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.
[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0015] Fifthly, embodiments of this application also provide a computer program product comprising a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the method described in the first aspect.
[0016] In this embodiment, multiple marker coordinates of reflective markers set on a probe tool in optical space are obtained, and a first point cloud coordinate set in optical positioning space is determined based on the multiple marker coordinates; a second point cloud coordinate set in virtual space is obtained, and coarse registration and alignment of optical positioning space and virtual space are performed based on the first feature point cloud coordinates of preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set; an initial fine registration error and an initial fine registration error weight are determined based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, and iterative fine registration and alignment of optical positioning space and virtual space are performed based on the initial fine registration error and the initial fine registration error weight to obtain a final fine registration transformation matrix; and a mixed reality space registration result is determined based on the final fine registration transformation matrix. The above-described mixed reality spatial registration method solves the problems of low registration efficiency and inaccurate registration results in existing technologies. It determines the first point cloud coordinate set in the optical positioning space by setting reflective markers on the probe tool. Based on the first and second feature point cloud coordinates of preset feature points, coarse registration and alignment are performed between the optical positioning space and the virtual space. The initial fine registration error and its weight are determined based on the coarse registration transformation result of the first and second point cloud coordinate sets in the coarse registration result. Iterative fine registration and alignment are then performed between the optical positioning space and the virtual space to determine the mixed reality spatial registration result. This method achieves spatial registration without the need for pasted markers or real-time CT scanning, simplifying the registration process and improving the efficiency and accuracy of mixed reality spatial registration. Attached Figure Description
[0017] Figure 1 This is a flowchart of a mixed reality spatial registration method provided in an embodiment of this application; Figure 2 This is a flowchart of determining the initial fine registration error provided in an embodiment of this application; Figure 3 This is a flowchart of the coarse registration and alignment optical positioning space and virtual space provided in the embodiments of this application; Figure 4 This is a schematic diagram of the target to be spatially registered provided in an embodiment of this application; Figure 5 This is a structural block diagram of a mixed reality spatial registration device provided in an embodiment of this application; Figure 6 This is a structural block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application are described in detail below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0021] Firstly, this solution can be used in scenarios where virtual models need to be overlaid onto real space to achieve real-time integration and interaction between virtual content and the real world, especially in scenarios where mixed reality devices are used for operational guidance. For example, it can be used with mixed reality devices and optical positioning instruments to overlay 3D medical images, virtual patient models, or treatment pathways onto a real scene for transcranial magnetic stimulation (TMS) surgical navigation. The solution uses reflective markers on a probe tool to determine the first point cloud coordinate set in the optical positioning space. Based on the first and second feature point cloud coordinates of preset feature points, coarse registration and alignment are performed between the optical positioning space and the virtual space. The initial fine registration error and its weight are determined based on the coarse registration transformation result of the first and second point cloud coordinate sets in the coarse registration result. Iterative fine registration and alignment are then performed between the optical positioning space and the virtual space to determine the mixed reality space registration result. This achieves spatial registration without the need for pasted markers or real-time CT scans, simplifying the registration process and improving the efficiency and accuracy of mixed reality space registration. Based on the above usage scenarios, it is understood that the execution entity of each step in this solution can be a computer device. The computer device refers to any electronic device with data computing, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices, or it can be a server or other devices. This application embodiment does not limit this.
[0022] The following description, in conjunction with the accompanying drawings, details a mixed reality spatial registration method, apparatus, device, and storage medium provided in this application through specific embodiments and application scenarios.
[0023] Figure 1 This is a flowchart of a mixed reality spatial registration method provided in an embodiment of this application. Figure 1 As shown, the specific steps include the following: S101, obtain multiple marker coordinates of the reflective markers set on the probe tool in the optical space, and determine the first point cloud coordinate set in the optical positioning space based on the multiple marker coordinates.
[0024] The reflective marker can be a marker that can be recognized by the optical positioning instrument, such as a reflective marker ball. The marker coordinates can be the optical positioning space coordinates of the same reflective marker in the optical positioning space when the probe tool collects point clouds at different locations. The first point cloud coordinate set can be a set of coordinates of multiple surface point clouds of the target to be spatially registered in the optical positioning space collected by the probe tool.
[0025] In one embodiment, the coordinates of multiple reflective markers set on the probe tool in optical space can be obtained in real time by an optical positioning device. Multiple tip coordinates are calculated based on the positional relationship between the setting position of each reflective marker on the probe tool and the position of the probe tool tip, as well as the coordinates of each reflective marker. The tip coordinates are the surface point cloud coordinates of the target to be spatially registered in the optical positioning space collected by the probe tool. The multiple surface point cloud coordinates are integrated to obtain the first point cloud coordinate set in the optical positioning space.
[0026] In one embodiment, the number of reflective markers is four, and an X-shaped tool is fixed on the probe tool for mounting the reflective markers; determining the first point cloud coordinate set in the optical positioning space based on multiple marker coordinates includes: calculating multiple tip position coordinates collected by the probe tool based on the positional relationship between the endpoint position of the X-shaped tool and the tip position of the probe tool and multiple marker coordinates; integrating the multiple tip position coordinates to obtain the first point cloud coordinate set in the optical positioning space.
[0027] The X-shaped tool can be a rigid body tool used to install four reflective markers.
[0028] In one embodiment, to ensure that the tip coordinates of the probe tool can be accurately calculated under different poses, this scheme sets four reflective marks, each of which is set at one of the four endpoints of an X-shaped tool, which is fixed to the probe tool. The coordinates of each reflective mark of the probe tool in a certain pose in the optical positioning space are obtained. Based on the predetermined positional relationship between the endpoints of the X-shaped tool and the tip position of the probe tool, and the coordinates of each reflective mark, the tip position coordinates of the probe tool in that pose are calculated. Multiple tip position coordinates are then integrated to obtain the first point cloud coordinate set in the optical positioning space.
[0029] This scheme involves fixing four reflective markers to an X-shaped tool on the probe tool, and calculating multiple needle tip coordinates based on the positional relationship between the endpoints of the X-shaped tool and the tip of the probe tool, as well as the coordinates of multiple markers. This yields the first point cloud coordinate set in the optical positioning space, avoiding errors in point cloud coordinate calculations due to different probe tool poses and improving the accuracy of the first point cloud coordinate set calculation.
[0030] S102, obtain the second point cloud coordinate set in the virtual space, and perform coarse registration and alignment of the optical positioning space and the virtual space based on the first feature point cloud coordinates of the preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set.
[0031] The virtual space can be the space where the virtual model resides in a mixed reality device. For example, the space in a HoloLens 2 mixed reality device that displays a 3D virtual model of the target to be spatially registered. The second point cloud coordinate set can be the set of coordinates of the surface point cloud of the virtual model in the virtual space coordinate system. Preset feature points can be pre-defined points or regions with prominent features on the target to be spatially registered. For example, if the target to be spatially registered is a human head, the preset feature points can include the tip of the nose, eyes, eyebrows, and mouth of the face. The first feature point cloud coordinates can be the coordinates of the preset feature points in the optical positioning space. The second feature point cloud coordinates can be the coordinates of the preset feature points in the virtual positioning space. Coarse registration alignment can be an operation that roughly aligns the optical positioning space and the virtual space. The coarsely registered optical positioning space and virtual space can ensure that a point in one space corresponds to a small range in another space, avoiding the problem of low efficiency caused by the need for a full-range search when performing fine registration of point clouds in different spaces.
[0032] In one embodiment, a second set of point cloud coordinates of the target to be spatially registered in virtual space can be obtained. Based on the positioning points or ranges of preset feature points on the target to be spatially registered, the first feature point cloud coordinates of the preset feature points in the first set of point cloud coordinates and the second feature point cloud coordinates in the second set of point cloud coordinates can be determined. Based on the first and second feature point cloud coordinates, the coarse registration transformation relationship between the optical positioning space and the virtual space can be calculated. According to the coarse registration transformation relationship, all the second point cloud coordinates in the second set of point cloud coordinates in the virtual space can be transformed to the optical positioning space to obtain the coarse registration alignment result.
[0033] In one embodiment, obtaining a second point cloud coordinate set in a virtual space includes: receiving a target 3D model of the target to be spatially registered in an image space; extracting point clouds from the target 3D model to obtain a model point cloud; and importing the model point cloud into a preset virtual space to obtain a second point cloud coordinate set in the preset virtual space.
[0034] The target to be spatially registered can be a target object whose corresponding virtual model needs to be superimposed onto the real space using mixed reality equipment. The target's 3D model can be obtained by reconstructing multiple 2D images of the target after CT or MRI scans, and is used to demonstrate the 3D characteristics of the target. The model point cloud can be the surface point cloud of the target's 3D model.
[0035] In one embodiment, a pre-constructed 3D model of the target to be spatially registered in image space can be received. Surface point cloud extraction is performed on the target 3D model to obtain a model point cloud. The model point cloud is uploaded to a virtual space, and the point cloud coordinates in image space are transformed according to the configuration parameters of the mixed reality device and the virtual space coordinate system parameters to obtain a second point cloud coordinate set in a preset virtual space.
[0036] This scheme extracts the point cloud of the target 3D model in the image space to obtain the model point cloud, and imports the model point cloud into a preset virtual space to obtain the second point cloud coordinate set in the preset virtual space. This can achieve the purpose of obtaining the point cloud coordinates of the surface point cloud of the virtual model in the virtual space, which is beneficial to improving the efficiency of subsequent spatial registration between the virtual space and the optical positioning space.
[0037] S103. Based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, determine the initial fine registration error and the initial fine registration error weight. Then, based on the initial fine registration error and the initial fine registration error weight, perform iterative fine registration alignment on the optical positioning space and the virtual space to obtain the final fine registration transformation matrix.
[0038] The coarse registration result includes the first point cloud coordinate set, which can be transformed into a virtual space coordinate set according to the coarse registration transformation relationship. The initial fine registration error is the closest distance between the transformed first point cloud coordinates in the optical positioning space and the second point cloud coordinates after transforming them to the virtual space according to the initial fine registration transformation relationship. The initial fine registration transformation relationship can be an identity matrix. The initial fine registration error weight represents the degree of influence of each first point cloud coordinate and its corresponding second point cloud coordinate on the optimization of the fine registration alignment result. Fine registration alignment is the operation of accurately matching each first point cloud in the optical positioning space with the second point cloud in the virtual space.
[0039] In one embodiment, the nearest second point cloud coordinate can be found among the first point cloud coordinate transformation results in the coarse registration result of the second point cloud coordinate set in the virtual space. The distance between the first point cloud coordinate transformation result and the corresponding nearest second point cloud coordinate is used as the initial fine registration error. The local shape similarity between each second point cloud and each first point cloud is determined based on the second and first point cloud coordinate sets, and the initial fine registration error weight corresponding to each coarse registration error is determined based on the local shape similarity. The fine registration transformation relationship between the optical positioning space and the virtual space is iteratively optimized according to the initial fine registration error and the initial fine registration error weight, and the fine registration is updated based on the optimization results until a preset number of iterations is reached, resulting in the final fine registration transformation matrix of the optical positioning space and the virtual space.
[0040] In one embodiment, iterative fine registration and alignment of the optical positioning space and the virtual space based on the initial fine registration error and the initial fine registration error weight includes: calculating the sum of the initial fine registration Mahalanobis distances based on the initial fine registration error and the initial fine registration error weight; determining the update result of the initial fine registration transformation relationship between the optical positioning space and the virtual space based on the sum of the initial fine registration transformation relationship update result and the second point cloud coordinates, and iteratively updating the sum of the initial fine registration Mahalanobis distances based on the update result of the initial fine registration transformation relationship and the second point cloud coordinates until the deviation of the update result of the sum of the Mahalanobis distances of two adjacent times is less than a preset change threshold, thereby obtaining the final fine registration result between the optical positioning space and the virtual space.
[0041] The initial fine-registration Mahalanobis distance sum can be the sum of the Mahalanobis distances between all first point clouds and their corresponding second point clouds after the initial fine-registration alignment of the virtual space and optical positioning space. The preset change threshold can be the maximum difference between the sums of Mahalanobis distances obtained from two adjacent fine-registration optimizations under the condition of precise alignment between the virtual space and optical positioning space.
[0042] In one embodiment, the sum of the initial fine-registration Mahalanobis distances can be calculated according to the Mahalanobis distance calculation formula, the initial fine-registration error, and the initial fine-registration error weight. The initial fine-registration transformation relationship update result between the optical positioning space and the virtual space is then calculated with the goal of minimizing the sum of the initial fine-registration Mahalanobis distances. Based on the updated initial fine-registration transformation relationship result, the first point cloud coordinates are re-transformed to the virtual space. The fine-registration error and fine-registration error weight are recalculated based on the transformation result and the second point cloud coordinates. The initial fine-registration Mahalanobis distance sum is updated based on the calculation result until the deviation between two consecutive updated Mahalanobis distance sums is less than a preset change threshold. This final fine-registration transformation result is then taken as the final fine-registration result between the optical positioning space and the virtual space.
[0043] This scheme calculates the sum of the initial fine registration Mahalanobis distances by using the initial fine registration error and its weight. It then updates the initial fine registration transformation relationship based on the sum of the initial fine registration Mahalanobis distances until the deviation between the updated results of two consecutive Mahalanobis distance sums is less than a preset change threshold. This yields the final fine registration result between the optical positioning space and the virtual space, which can improve the accuracy of fine registration between the optical positioning space and the virtual space.
[0044] In one embodiment, the formula for the sum of the initial Mahalanobis distances is: ; in, It is the sum of Mahalanobis distances, and N is the number of first point cloud coordinates in the first point cloud coordinate set. It is the coordinate of the i-th first point cloud. It is the nearest neighbor point cloud coordinate of the i-th point cloud coordinate in the second point cloud coordinate. It is the covariance matrix of the first neighborhood of the i-th point cloud coordinate. It is the second neighborhood covariance matrix of the nearest neighbor point cloud coordinates. It is the transformation matrix corresponding to the initial fine registration and alignment.
[0045] This scheme represents the local shape of each point cloud by calculating the neighborhood covariance matrix of the point cloud coordinates, and determines the coarse registration error weights corresponding to each point cloud by calculating the difference of the neighborhood covariance matrix of the point cloud in different spaces. This can further improve the rationality of the evaluation of the spatial coarse registration results and the accuracy of the iterative fine registration.
[0046] S104, based on the final fine registration transformation matrix, determines the registration result of the mixed reality space.
[0047] In one embodiment, the first point cloud coordinate set in optical space can be transformed to virtual space according to the final fine registration transformation matrix to obtain the mixed reality space registration result.
[0048] In one embodiment, after obtaining the initial pose of the virtual model in the optical positioning space through coarse registration, to further improve the alignment accuracy between the virtual space and the optical positioning space, the Generalized Iterative Closest Point (GICP) algorithm can be used to perform fine registration between the point cloud in the optical positioning space acquired by the probe tool's sliding point and the point cloud on the virtual model in the virtual space. GICP achieves higher accuracy and more stable registration by jointly considering the covariance structure of the local neighborhood of the point cloud and using Mahalanobis distance as the minimization objective.
[0049] Assume the first point cloud coordinate set of the target to be spatially registered in optical positioning space is: ; The second point cloud coordinate set of the virtual model of the target to be spatially registered in virtual space is as follows: . Construct the objective function of GICP, which is the sum of Mahalanobis distances, and solve the spatial registration transformation relationship by minimizing the Mahalanobis distance: ; Among them, the coordinates of the first point cloud in the optical positioning space are: The second point cloud coordinates are concentrated at... The coordinates of the nearest second point cloud are The index is determined by the nearest neighbor search function, and the corresponding point index function is... , It is the first neighborhood covariance matrix of the first point cloud coordinates. The second neighborhood covariance matrix of the second point cloud coordinates.
[0050] The transformation matrix corresponding to the initial fine registration and alignment is iteratively optimized using the above expression for the sum of Mahalanobis distances. The final fine registration transformation matrix is obtained. : . The final spatial registration transformation matrix is T: . The final spatial registration transformation matrix T can be used to accurately register any second point cloud coordinate in the virtual space to the optical positioning space, thus obtaining the mixed reality spatial registration result.
[0051] The technical solution provided in this application involves obtaining multiple marker coordinates of reflective markers set on a probe tool in optical space, determining a first point cloud coordinate set in optical positioning space based on the multiple marker coordinates, obtaining a second point cloud coordinate set in virtual space, performing coarse registration and alignment of optical positioning space and virtual space based on the first feature point cloud coordinates of preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set, determining the initial fine registration error and the initial fine registration error weight based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, iterating fine registration and alignment of optical positioning space and virtual space based on the initial fine registration error and the initial fine registration error weight to obtain the final fine registration transformation matrix, and determining the mixed reality space registration result based on the final fine registration transformation matrix. The above-described mixed reality spatial registration method solves the problems of low registration efficiency and inaccurate registration results in existing technologies. It determines the first point cloud coordinate set in the optical positioning space by setting reflective markers on the probe tool. Based on the first and second feature point cloud coordinates of preset feature points, coarse registration and alignment are performed between the optical positioning space and the virtual space. The initial fine registration error and its weight are determined based on the coarse registration transformation result of the first and second point cloud coordinate sets in the coarse registration result. Iterative fine registration and alignment are then performed between the optical positioning space and the virtual space to determine the mixed reality spatial registration result. This method achieves spatial registration without the need for pasted markers or real-time CT scanning, simplifying the registration process and improving the efficiency and accuracy of mixed reality spatial registration.
[0052] Figure 2 This is a flowchart illustrating the determination of the initial fine registration error provided in an embodiment of this application. For example... Figure 2 As shown, the specific steps include the following: S201, based on the preset initial fine registration transformation relationship, perform initial fine registration transformation on each first coarse registration point cloud coordinate in the first point cloud coordinate set coarse registration transformation result, and determine the nearest neighbor point cloud coordinates corresponding to each first initial fine registration point cloud coordinate in the initial fine registration transformation result according to the preset nearest neighbor search function.
[0053] The preset nearest neighbor search function can be a pre-set function used to search for the second point cloud coordinates that are closest to the first point cloud coordinate space transformation result.
[0054] In one embodiment, the first initial fine registration point cloud coordinates after initial fine registration transformation of each first coarse registration point cloud coordinate in the first point cloud coordinate set coarse registration transformation result can be calculated according to the preset initial fine registration transformation relationship, so as to obtain the initial fine registration alignment result between the optical positioning space and the virtual space, and the nearest neighbor point cloud coordinates corresponding to each first initial fine registration point cloud coordinate in the second point cloud coordinates can be found according to the preset nearest neighbor search function.
[0055] S202, calculate the coordinate deviation between each initial fine-registration point cloud coordinate and the corresponding nearest neighbor point cloud coordinate to obtain the initial fine-registration error.
[0056] In one embodiment, the coordinate difference between each first initial fine-registration point cloud coordinate and the corresponding nearest neighbor point cloud coordinate can be calculated, and the coordinate difference can be used as the initial fine-registration error corresponding to the first point cloud.
[0057] S203, calculate the first neighborhood covariance matrix of each first point cloud coordinate in the first point cloud coordinate set and the second neighborhood covariance matrix of each second point cloud coordinate in the second point cloud coordinate set respectively, and determine the initial fine registration error weights based on the first neighborhood covariance matrix and the second neighborhood covariance matrix.
[0058] The initial fine registration error weights are composed of the first neighborhood covariance matrix and the second neighborhood covariance matrix. The more similar the local shapes of the corresponding point clouds in the two spaces, that is, the consistent characteristics of the covariance matrices, the larger the corresponding initial fine registration error weights. Otherwise, the point clouds in the two spaces are considered to have low fitness, and the initial fine registration error weights are smaller in order to reduce the impact of the point cloud on the optimization.
[0059] In one embodiment, the first neighborhood covariance matrix of each first point cloud coordinate in the first point cloud coordinate set and the second neighborhood covariance matrix of each second point cloud coordinate in the second point cloud coordinate set can be calculated according to the covariance matrix algorithm, and the initial fine registration error weight can be calculated according to the first neighborhood covariance matrix and the second neighborhood covariance matrix of each group of corresponding point clouds.
[0060] The technical solution provided in this application, by determining the nearest neighbor point cloud coordinates corresponding to each first initial fine registration point cloud coordinate in the initial fine registration transformation result, calculating the coordinate deviation between each first initial fine registration point cloud coordinate and its corresponding nearest neighbor point cloud coordinate, obtains the initial fine registration error, and respectively calculates the first neighborhood covariance matrix of each first point cloud coordinate in the first point cloud coordinate set and the second neighborhood covariance matrix of each second point cloud coordinate in the second point cloud coordinate set, determines the initial fine registration error weight, which can improve the accuracy and rationality of the calculation of the initial fine registration error and the initial fine registration error weight.
[0061] Figure 3 This is a flowchart illustrating the coarse registration and alignment optical positioning space and virtual space provided in the embodiments of this application. For example... Figure 3 As shown, the specific steps include the following: S301, determine the first feature point cloud coordinates of the same preset feature points in the first point cloud coordinate set, and the second feature point cloud coordinates in the second point cloud coordinate set.
[0062] In one embodiment, based on the position of the preset feature point on the target to be spatially registered, the first feature point cloud coordinates and the second feature point cloud coordinates of the same preset feature point location can be found in the first point cloud coordinate set and the second point cloud coordinate set, respectively.
[0063] S302, calculates the coarse registration transformation relationship between optical positioning space and virtual space based on the preset coarse registration algorithm, the coordinates of the first feature point cloud, and the coordinates of the second feature point cloud.
[0064] In one embodiment, a rotation and translation matrix between the first feature point cloud coordinates and the second feature point cloud coordinates is calculated based on a preset coarse registration algorithm, the first feature point cloud coordinates, and the second feature point cloud coordinates. This rotation and translation matrix is used as the coarse registration transformation relationship between the optical positioning space and the virtual space.
[0065] Figure 4 This is a schematic diagram of the target space to be registered, provided in an embodiment of this application. Figure 4 As shown in the figure, the model is the actual model of the target to be spatially registered. Multiple circles mark the locations of preset feature points on the model. These preset feature points can be points with prominent shapes or characteristics that can represent the three-dimensional features of the target. The feature point cloud coordinates corresponding to the locations of each preset feature point can be found on both the virtual model of the target in virtual space and the actual model in optical positioning space. Based on the feature point cloud coordinates of the same preset feature points in different spaces, the coarse registration transformation relationship between optical positioning space and virtual space can be calculated.
[0066] S303, based on the coarse registration transformation relationship, transform the second point cloud coordinates in the second point cloud coordinate set to the optical positioning space, and obtain the coarse registration alignment result between the optical positioning space and the virtual space.
[0067] In one embodiment, the second point cloud coordinates in the second point cloud coordinate set can be converted to point cloud coordinates in the optical positioning space according to the coarse registration transformation relationship, so as to obtain the coarse registration alignment result between the optical positioning space and the virtual space.
[0068] In one embodiment, it is assumed that the second feature point cloud coordinate set, which corresponds to the feature points of the first feature point cloud coordinate set in the virtual space, is... ; The coordinate set of the first feature point cloud acquired by the probe tool in the optical positioning space is as follows: ; To solve the initial rigid body transformation from virtual space to optical positioning space, a singular value decomposition (SVD) algorithm can be used.
[0069] Point set center calculation: ; . Decentralized point set: ; . The covariance matrix is: . The SVD decomposition formula is: . The formula for solving the rotation matrix is: ; ; . The formula for solving the translation vector is: . Based on the results of solving the rotation matrix and the translation vector, determine the coarse registration transformation matrix: . The technical solution provided in this application embodiment determines the coordinates of the first and second feature points of the same preset feature points, calculates the coarse registration transformation relationship between the optical positioning space and the virtual space based on the preset coarse registration algorithm, the coordinates of the first and second feature points, and obtains the coarse registration alignment result between the optical positioning space and the virtual space, which can improve the efficiency of spatial coarse registration alignment.
[0070] Figure 5 This is a structural block diagram of a mixed reality spatial registration device provided in an embodiment of this application. Figure 5 As shown, it specifically includes the following: The point cloud coordinate set determination module 501 is used to obtain multiple marker coordinates of the reflective markers set on the probe tool in the optical space, and determine the first point cloud coordinate set in the optical positioning space based on the multiple marker coordinates. The coarse registration and alignment module 502 is used to obtain the second point cloud coordinate set in the virtual space, and perform coarse registration and alignment of the optical positioning space and the virtual space based on the first feature point cloud coordinates of the preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set. The fine registration and alignment module 503 is used to determine the initial fine registration error and the initial fine registration error weight based on the first point cloud coordinate set coarse registration transformation result and the second point cloud coordinate set in the coarse registration result, and to perform iterative fine registration and alignment on the optical positioning space and the virtual space based on the initial fine registration error and the initial fine registration error weight to obtain the final fine registration transformation matrix. The spatial registration module 504 is used to determine the spatial registration result of mixed reality based on the final fine registration transformation matrix.
[0071] Furthermore, the fine registration and alignment module 503 is specifically used for: Based on the preset initial fine registration transformation relationship, the coordinates of each first coarse registration point cloud in the first point cloud coordinate set coarse registration transformation result are initially fine registered and transformed. Based on the preset nearest neighbor search function, the nearest neighbor point cloud coordinates corresponding to each first initial fine registration point cloud coordinate in the initial fine registration transformation result are determined. Calculate the coordinate deviation between each initial fine-registration point cloud coordinate and its corresponding nearest neighbor point cloud coordinate to obtain the initial fine-registration error; Calculate the first neighborhood covariance matrix of each first point cloud coordinate in the first point cloud coordinate set and the second neighborhood covariance matrix of each second point cloud coordinate in the second point cloud coordinate set. Determine the initial fine registration error weights based on the first neighborhood covariance matrix and the second neighborhood covariance matrix.
[0072] Furthermore, the fine registration and alignment module 503 is specifically used for: Based on the initial fine registration error and the initial fine registration error weight, the sum of the initial fine registration Mahalanobis distances is calculated; The initial fine registration transformation relationship update result between the optical positioning space and the virtual space is determined based on the sum of the initial fine registration Mahalanobis distances. The initial fine registration Mahalanobis distance sum is iteratively updated based on the initial fine registration transformation relationship update result and the second point cloud coordinates until the deviation between the update results of the sum of Mahalanobis distances of two adjacent times is less than a preset change threshold, thus obtaining the final fine registration result between the optical positioning space and the virtual space.
[0073] Furthermore, the formula for the sum of the initial fine-registration Mahalanobis distances is as follows: ; in, It is the sum of Mahalanobis distances, and N is the number of first point cloud coordinates in the first point cloud coordinate set. It is the coordinate of the i-th first point cloud. It is the nearest neighbor point cloud coordinate of the i-th point cloud coordinate in the second point cloud coordinate. It is the covariance matrix of the first neighborhood of the i-th point cloud coordinate. It is the second neighborhood covariance matrix of the nearest neighbor point cloud coordinates. It is the transformation matrix corresponding to the initial fine registration and alignment.
[0074] Furthermore, the coarse registration and alignment module 502 is specifically used for: Determine the first feature point cloud coordinates of the same preset feature points in the first point cloud coordinate set, and the second feature point cloud coordinates in the second point cloud coordinate set; The coarse registration transformation relationship between the optical positioning space and the virtual space is calculated based on the preset coarse registration algorithm, the coordinates of the first feature point cloud, and the coordinates of the second feature point cloud. Based on the coarse registration transformation relationship, the coordinates of the second point cloud in the second point cloud coordinate set are transformed to the optical positioning space, and the coarse registration alignment results of the optical positioning space and the virtual space are obtained.
[0075] Furthermore, the coarse registration and alignment module 502 is specifically used for: Receive the 3D model of the target to be spatially registered in the image space, extract the point cloud from the 3D model to obtain the model point cloud; Import the model point cloud into a preset virtual space to obtain the second point cloud coordinate set under the preset virtual space.
[0076] Furthermore, there are four reflective markers, and the probe tool is equipped with an X-shaped tool for attaching the reflective markers; Point cloud coordinate set determination module 501 is specifically used for: Based on the positional relationship between the endpoint position of the X-shaped tool and the tip position of the probe tool, and multiple marker coordinates, calculate the coordinates of multiple tip positions collected by the probe tool. By integrating the coordinates of multiple needle tips, the first point cloud coordinate set in the optical positioning space is obtained.
[0077] The technical solution provided in this application includes a point cloud coordinate set determination module, used to acquire multiple marker coordinates of reflective markers set on a probe tool in optical space, and determine a first point cloud coordinate set in optical positioning space based on the multiple marker coordinates; a coarse registration and alignment module, used to acquire a second point cloud coordinate set in virtual space, and perform coarse registration and alignment of optical positioning space and virtual space based on the first feature point cloud coordinates of preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set; a fine registration and alignment module, used to determine the initial fine registration error and the initial fine registration error weight based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, and perform iterative fine registration and alignment of optical positioning space and virtual space based on the initial fine registration error and the initial fine registration error weight to obtain the final fine registration transformation matrix; and a spatial registration module, used to determine the mixed reality spatial registration result based on the final fine registration transformation matrix. The aforementioned mixed reality spatial registration device solves the problems of low registration efficiency and inaccurate registration results in existing technologies. It determines the first point cloud coordinate set in the optical positioning space by using reflective markers on a probe tool. Based on the first and second feature point cloud coordinates of preset feature points, coarse registration and alignment are performed between the optical positioning space and the virtual space. The initial fine registration error and its weight are determined based on the coarse registration transformation result of the first and second point cloud coordinate sets in the coarse registration result. Iterative fine registration and alignment are then performed between the optical positioning space and the virtual space to determine the mixed reality spatial registration result. This achieves spatial registration without the need for pasted markers or real-time CT scanning, simplifying the registration process and improving the efficiency and accuracy of mixed reality spatial registration.
[0078] The mixed reality spatial registration system described in this application can be configured in a device, or in a component, integrated circuit, or chip within a terminal. The system can be configured in mobile electronic devices or non-mobile electronic devices. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application does not impose specific limitations.
[0079] The mixed reality spatial registration system in this application embodiment can be an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.
[0080] The mixed reality spatial registration system provided in this application can realize the various processes implemented in the above-described method embodiments, and will not be repeated here to avoid repetition.
[0081] like Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described mixed reality space registration method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0082] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0083] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described mixed reality space registration method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0084] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0085] This application also provides a program product including program code. When the program product is run on a computer device, the program code causes the computer device to perform the steps of the methods described above according to various exemplary embodiments of this application. For example, the computer device can execute a mixed reality spatial registration method described in the embodiments of this application. The program product can be implemented using any combination of one or more readable media.
[0086] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element. Furthermore, it should be noted that the scope of the methods and systems in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0088] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
[0089] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.
Claims
1. A mixed reality spatial registration method, characterized in that, The method includes: Obtain the coordinates of multiple markers in the optical space for the reflective markers set on the probe tool, and determine the first point cloud coordinate set in the optical positioning space based on the multiple marker coordinates; Obtain the second point cloud coordinate set in the virtual space, and perform coarse registration and alignment of the optical positioning space and the virtual space based on the first feature point cloud coordinates of the preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set. Based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, the initial fine registration error and the initial fine registration error weight are determined. Based on the initial fine registration error and the initial fine registration error weight, the optical positioning space and the virtual space are iteratively fine registered and aligned to obtain the final fine registration transformation matrix. Based on the final fine registration transformation matrix, the registration result of the mixed reality space is determined.
2. The mixed reality spatial registration method according to claim 1, characterized in that, The step of determining the initial fine registration error and its weight based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result includes: Based on the preset initial fine registration transformation relationship, the first coarse registration point cloud coordinates in the first point cloud coordinate set coarse registration transformation result are subjected to initial fine registration transformation, and the nearest neighbor point cloud coordinates corresponding to each first initial fine registration point cloud coordinate in the initial fine registration transformation result are determined according to the preset nearest neighbor search function. Calculate the coordinate deviation between each of the first initial fine-registration point cloud coordinates and the corresponding nearest neighbor point cloud coordinates to obtain the initial fine-registration error; Calculate the first neighborhood covariance matrix of each first point cloud coordinate in the first point cloud coordinate set and the second neighborhood covariance matrix of each second point cloud coordinate in the second point cloud coordinate set. Determine the initial fine registration error weights based on the first neighborhood covariance matrix and the second neighborhood covariance matrix.
3. The mixed reality spatial registration method according to claim 2, characterized in that, The iterative fine registration and alignment of the optical positioning space and the virtual space based on the initial fine registration error and the initial fine registration error weight includes: Based on the initial fine registration error and the initial fine registration error weight, calculate the sum of the initial fine registration Mahalanobis distances; The initial fine registration transformation relationship update result between the optical positioning space and the virtual space is determined based on the sum of the initial fine registration Mahalanobis distances. The initial fine registration Mahalanobis distance sum is iteratively updated based on the initial fine registration transformation relationship update result and the second point cloud coordinates until the deviation between two adjacent Mahalanobis distance sum update results is less than a preset change threshold, thereby obtaining the final fine registration result between the optical positioning space and the virtual space.
4. The mixed reality spatial registration method according to claim 3, characterized in that, The formula for the sum of the initial fine-registration Mahalanobis distances is as follows: ； in, It is the sum of Mahalanobis distances, and N is the number of first point cloud coordinates in the first point cloud coordinate set. It is the coordinate of the i-th first point cloud. It is the nearest neighbor point cloud coordinate of the i-th point cloud coordinate in the second point cloud coordinate. It is the covariance matrix of the first neighborhood of the i-th point cloud coordinate. It is the second neighborhood covariance matrix of the nearest neighbor point cloud coordinates. It is the transformation matrix corresponding to the initial fine registration and alignment.
5. The mixed reality spatial registration method according to claim 1, characterized in that, The coarse registration and alignment of the optical positioning space and the virtual space based on the first feature point cloud coordinates in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set includes: Determine the first feature point cloud coordinates of the same preset feature points in the first point cloud coordinate set, and the second feature point cloud coordinates in the second point cloud coordinate set; The coarse registration conversion relationship between the optical positioning space and the virtual space is calculated based on the preset coarse registration algorithm, the coordinates of the first feature point cloud, and the coordinates of the second feature point cloud. Based on the coarse registration transformation relationship, the second point cloud coordinates in the second point cloud coordinate set are transformed to the optical positioning space to obtain the coarse registration alignment result of the optical positioning space and the virtual space.
6. The mixed reality spatial registration method according to claim 1, characterized in that, The acquisition of the second point cloud coordinate set in the virtual space includes: Receive the three-dimensional model of the target to be spatially registered in the image space, and extract the point cloud from the three-dimensional model to obtain the model point cloud; The model point cloud is imported into a preset virtual space to obtain a second point cloud coordinate set under the preset virtual space.
7. The mixed reality spatial registration method according to claim 1, characterized in that, The number of reflective markers is four, and the probe tool is fixed with an X-shaped tool for installing the reflective markers; The determination of the first point cloud coordinate set in the optical positioning space based on the multiple marker coordinates includes: Based on the positional relationship between the endpoint position of the X-shaped tool and the tip position of the probe tool, and the coordinates of the multiple markers, calculate the coordinates of the multiple tip positions collected by the probe tool; By integrating the coordinates of the multiple needle tips, the first point cloud coordinate set in the optical positioning space is obtained.
8. A mixed reality spatial registration device, characterized in that, The device includes: The point cloud coordinate set determination module is used to acquire multiple marker coordinates of reflective markers set on the probe tool in the optical space, and determine the first point cloud coordinate set in the optical positioning space based on the multiple marker coordinates. The coarse registration and alignment module is used to obtain the second point cloud coordinate set in the virtual space, and to perform coarse registration and alignment on the optical positioning space and the virtual space based on the first feature point cloud coordinates of the preset feature points in the first point cloud coordinate set and the second feature point cloud coordinates in the second point cloud coordinate set. The fine registration and alignment module is used to determine the initial fine registration error and the initial fine registration error weight based on the coarse registration transformation result of the first point cloud coordinate set and the second point cloud coordinate set in the coarse registration result, and to perform iterative fine registration and alignment on the optical positioning space and the virtual space based on the initial fine registration error and the initial fine registration error weight to obtain the final fine registration transformation matrix; The spatial registration module is used to determine the mixed reality spatial registration result based on the final fine registration transformation matrix.
9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and running on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the mixed reality spatial registration method as described in any one of claims 1-7.
10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the mixed reality spatial registration method as described in any one of claims 1-7.