A scalable real-time mixed reality spatial registration and navigation system

Abstract

The application discloses a scalable real-time mixed reality space registration and navigation system, comprising: a three-dimensional reconstruction module for extracting a target position CT image and performing image segmentation and three-dimensional reconstruction on the target position CT image; a marker point acquisition module for acquiring a coordinate point set of the same marker points on a virtual target model and a real target; a mixed reality space registration module for integrating a virtual target scaling factor into a singular value decomposition algorithm to calculate a rotation matrix and a translation matrix between the virtual target model and the real target pose; and a surgical navigation module for judging whether the real target pose changes and updating a navigation line pose of a surgical tool in a virtual space U in real time to complete surgical navigation. The application can monitor target pose changes and real-time registration, and realizes interactive surgical navigation.

Description

Technical Field

[0001] This invention relates to the field of mixed reality technology, and in particular to a scalable real-time mixed reality spatial registration and navigation system. Background Technology

[0002] The application of mixed reality (MR) technology in the medical field has brought new technological approaches to surgical navigation, helping doctors to perform precise positioning and operations while reducing visual clutter from monitors. The core processes of MR surgical navigation typically include key technical steps such as 3D reconstruction of medical images, spatial registration and mapping, real-time intraoperative positioning, and virtual-real fusion visualization. Based on different registration methods, existing navigation methods can be divided into two categories: markerless and marker-based approaches. MR technology, through head-mounted displays, enables the real-time overlay of virtual models and real surgical scenes, providing a new paradigm for the integration and innovation of these two technological approaches.

[0003] Existing mixed reality navigation systems still face significant bottlenecks in mixed reality applications: First, traditional marker registration relies on static rigid registration, which is difficult to adapt to dynamic changes caused by intraoperative target positioning or tissue displacement. Studies show that registration errors can increase to 3-5 mm in such scenarios. Second, for delicate surgeries, non-overlapping mixed reality navigation cannot scale virtual 3D objects. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a scalable real-time mixed reality space registration and navigation system that can monitor target pose changes and register in real time, thereby achieving interactive surgical navigation.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a scalable real-time mixed reality spatial registration and navigation system, comprising:

[0006] The 3D reconstruction module is used to extract CT images of the target location and perform image segmentation and 3D reconstruction on the CT images of the target location. The reconstructed 3D model is called the virtual target model in the virtual space U. The virtual target model is moved and scaled according to the requirements, and the scaling factor s of the virtual target model on the x, y, and z axes of the spatial coordinate system U is obtained by the head-mounted glasses. x s y s z ;

[0007] The marker acquisition module collects the coordinates of n markers on the virtual target model after it has been moved and scaled in the 3D reconstruction module using head-mounted glasses, and collects the coordinates of the same n markers on the real target using an optical positioning device.

[0008] The mixed reality space registration module integrates the scaling factor obtained from the 3D reconstruction module into the singular value decomposition algorithm, and uses the results from the marker point acquisition module to calculate the rotation matrix between the virtual target model and the real target pose. Translation matrix Further realize spatial registration of scalable virtual target models;

[0009] The surgical navigation module uses an optical locator to acquire the coordinates of the real target and surgical tools at the marked points in the positioning space O in real time, determines whether the pose of the real target has changed, and then corrects it. It also uses the results of the mixed reality space registration module to update the pose of the navigation line of the surgical tools in the virtual space U in real time to complete the surgical navigation.

[0010] Furthermore, the 3D reconstruction module uses a CT scanner or MRI scanner to acquire CT images of the target location, performs 3D reconstruction using multi-view geometry, moves the scaled virtual target model to the required position and then keeps it stationary, and then obtains the current scaling factor s. x s y s z .

[0011] Furthermore, the marker point acquisition module includes a virtual target marker point acquisition module and a real target marker point acquisition module, wherein:

[0012] The virtual target marker point acquisition module, for the moved and scaled virtual target model in the 3D reconstruction module, uses head-mounted glasses to acquire the coordinate point set of n marker points on the virtual target model in the virtual space U. in This represents the coordinates of the i-th marker point in the point set Q;

[0013] The real target marker point acquisition module uses an optical positioning instrument to acquire the coordinate point set of n identical marker points on the real target in the positioning space O. in This represents the coordinates of the i-th marker point in the point set P;

[0014] coordinate point set and coordinate point set There is a correspondence, that is and These are marker points at the same locations on the virtual target model and the real target, respectively.

[0015] Furthermore, the mixed reality space registration module performs the following steps:

[0016] 1) Utilize the scaling factors s of the current virtual target model on the x, y, and z axes obtained from the 3D reconstruction module. x s y sz The point set Q acquired by the marker point acquisition module is transformed to obtain the transformed point set. in This represents the coordinates of the i-th marked point in the point set Q'; for each point... as well as All of them are:

[0017]

[0018] In the formula, x i y i and z i Points The values ​​on the x, y, and z axes in the virtual space U coordinate system, x' i y' i and z' i Points The values ​​on the x, y, and z axes in the virtual space U coordinate system;

[0019] 2) Use singular value decomposition on the coordinate point sets Q' and P to obtain the transformation matrix between the virtual space U coordinate system and the positioning space O coordinate system, that is, find the optimal 3×3 rotation matrix. And a 3×1 translation matrix Make: Minimize, the specific method is as follows:

[0020] Decentralize two point sets P and Q' to obtain a new point set {m} i} and {n i Simultaneously calculate the covariance matrix. Where, m i For the point set {m i The i-th point in}, n i For the point set {n i The i-th point in H; perform singular value decomposition on H to calculate the rotation and translation matrices between the point sets:

[0021]

[0022] In the formula, and Let U and V be the centroids of point sets Q' and P, respectively, and let U and V be unitary matrices, and let S be a singular value diagonal matrix.

[0023] Furthermore, the surgical navigation module performs the following operations:

[0024] First, an error range Δ is set, and the coordinates of the real target marker points in the positioning space O are acquired in real time and compared with the point set P. If the difference is greater than Δ, the pose of the real target is considered to have changed. The current set of marker point coordinates is then used to replace the point set P, and the mixed reality space registration module is called again to perform space registration and update the rotation matrix. Translation matrix Secondly, the coordinates t of the tip of the surgical instrument in the positioning space O are obtained in real time using an optical positioning device. o1 =(x o1 ,y o1 ,z o1 ) and the coordinates of the other endpoint t o2 =(x o2 ,y o2 ,z o2 ), x o1 y o1 and z o1 Point t oi The values ​​on the x, y, and z axes in the positioning space O coordinate system, x o2 y o2 and z o2 Point t o2 Calculate the values ​​on the x, y, and z axes in the coordinate system O; calculate the corresponding point coordinates t in the virtual space U. u1 =(x u1 ,y u1 ,z u1 ) and t u2 =(x u2 ,y u2 ,z u2 ), x u1 y u1 and z u1 Point t ui The values ​​on the x, y, and z axes in the virtual space U coordinate system, x u2 y u2 and z u2 Point t u2 The values ​​on the x, y, and z axes in the virtual space U-coordinate system; where Subscript j = 1, 2; through the above operations, the pose update of the surgical tool navigation line in the virtual space is finally realized, thereby completing the surgical navigation.

[0025] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0026] 1. This invention utilizes the scaling factor of the acquired virtual target model to improve the registration of mixed reality space, while maintaining the positional relationship between the surgical tool navigation line and the virtual target model consistent with reality, thereby achieving scalable surgical navigation.

[0027] 2. This invention can judge the changes in the target's pose in real time, and realize surgical navigation with a movable target. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the communication process between the glasses (i.e., head-mounted glasses), the computer, and the optical positioning device.

[0029] Figure 2 This is a schematic diagram showing the relationship between the various modules of the present invention.

[0030] Figure 3 A coordinate system transformation diagram registered for mixed reality space.

[0031] Figure 4 A diagram showing the correspondence between surgical tool tracking and mixed reality space registration results. Detailed Implementation

[0032] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0033] This embodiment discloses a scalable real-time mixed reality spatial registration and navigation system, implemented on three devices: glasses, a computer, and an optical positioning device. The communication connection of the three devices is as follows: Figure 1 As shown. The relationships between the various modules of the system are as follows. Figure 2 As shown. It includes:

[0034] The 3D reconstruction module is used to extract CT images of the target location and perform image segmentation and 3D reconstruction on the CT images of the target location. The reconstructed 3D model is called the virtual target model in the virtual space U. The virtual target model is moved and scaled according to the requirements, and the scaling factor s of the virtual target model on the x, y, and z axes of the spatial coordinate system U is obtained by the head-mounted glasses. x s y s z ;

[0035] The marker acquisition module collects the coordinates of n markers on the virtual target model after it has been moved and scaled in the 3D reconstruction module using head-mounted glasses, and collects the coordinates of the same n markers on the real target using an optical positioning device.

[0036] The mixed reality space registration module integrates the scaling factor obtained from the 3D reconstruction module into the singular value decomposition algorithm, and uses the results from the marker point acquisition module to calculate the rotation matrix between the virtual target model and the real target pose. Translation matrix Further realize spatial registration of scalable virtual target models;

[0037] The surgical navigation module uses an optical locator to acquire the coordinates of the real target and surgical tools at the marked points in the positioning space O in real time, determines whether the pose of the real target has changed, and then corrects it. It also uses the results of the mixed reality space registration module to update the pose of the navigation line of the surgical tools in the virtual space U in real time to complete the surgical navigation.

[0038] Specifically, the 3D reconstruction module targets the head and jaw of a person wearing an occlusion positioning tool with marked points. It acquires CT images of the target location using equipment such as a CT scanner or MRI scanner, and performs 3D reconstruction using methods such as multi-view geometry. After moving the scaled virtual target model to the desired position, it must remain stationary, and then the current scaling factor s is obtained. x s y s z .

[0039] Specifically, the marker point acquisition module includes a virtual target marker point acquisition module and a real target marker point acquisition module, wherein:

[0040] The virtual target marker point acquisition module, for the moved and scaled virtual target model in the 3D reconstruction module, uses head-mounted glasses to acquire the coordinate point set of n marker points on the virtual target model in the virtual space U. in This represents the coordinates of the i-th marker point in the point set Q;

[0041] The real target marker point acquisition module uses an optical positioning device to acquire the coordinate point set of n identical marker points on the real target in the positioning space O. in This represents the coordinates of the i-th marked point in the point set P.

[0042] coordinate point set and coordinate point set There is a correspondence, that is and These are marker points at the same locations on the virtual target model and the real target, respectively.

[0043] Specifically, such as Figure 3 As shown, the mixed reality space registration module performs the following steps:

[0044] 1) Utilize the scaling factors s of the current virtual target model on the x, y, and z axes obtained from the 3D reconstruction module. x s y s z The point set Q acquired by the marker point acquisition module is transformed to obtain the transformed point set. in This represents the coordinates of the i-th marked point in the point set Q'; for each point... as well as All of them are:

[0045]

[0046] Where, x i y i and z i Points The values ​​on the x, y, and z axes of the coordinate system in the virtual space U; similarly, x' i y' i and z' i Points The values ​​on the x, y, and z axes in the virtual space U coordinate system.

[0047] 2) Use singular value decomposition on the coordinate point sets Q' and P to obtain the transformation matrix between the virtual space U coordinate system and the positioning space O coordinate system, that is, find the optimal 3×3 rotation matrix. And a 3×1 translation matrix Make

[0048]

[0049] Minimize, the specific method is as follows:

[0050] Decentralize two point sets P and Q' to obtain a new point set {m} i} and {n i Simultaneously calculate the covariance matrix. Where, m i For the point set {m i The i-th point in}, n i For the point set {n i The i-th point in H. Perform singular value decomposition on H to compute the rotation and translation matrices between the point sets:

[0051]

[0052] In the formula, and Let U and V be the centroids of point sets Q' and P, respectively, and let U and V be unitary matrices, and let S be a singular value diagonal matrix.

[0053] Specifically, the surgical navigation module performs the following operations:

[0054] First, an error range Δ is set, and the coordinates of the real target marker points in the positioning space O are acquired in real time and compared with the point set P. If the difference is greater than Δ, the pose of the real target is considered to have changed. The current set of marker point coordinates is then used to replace the point set P, and the mixed reality space registration module is called again to perform space registration and update the rotation matrix. Translation matrix Secondly, the coordinates t of the tip of the surgical instrument in the positioning space O are obtained in real time using an optical positioning device. o1 =(x o1 ,y o1 ,z o1 ) and the coordinates of the other endpoint t o2 =(x o2 ,y o2 ,z o2 ), x o1 y o1 and z o1 Point t oi The values ​​on the x, y, and z axes in the positioning space O coordinate system, x o2 y o2 and z o2 Point t o2 Calculate the values ​​on the x, y, and z axes in the coordinate system O; calculate the corresponding point coordinates t in the virtual space U. u1 =(x u1 ,y u1 ,z u1 ) and t u2 =(x u2 ,y u2 ,z u2 ), x u1 y u1 and z u1 Point t ui The values ​​on the x, y, and z axes in the virtual space U coordinate system, x u2 y u2 and z u2 Point t u2 The values ​​on the x, y, and z axes in the virtual space U-coordinate system; where Subscript j = 1, 2; through the above operations, the pose update of the surgical tool navigation line in the virtual space is finally realized, thereby completing the surgical navigation.

[0055] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A scalable real-time mixed reality spatial registration and navigation system, characterized in that, include: The 3D reconstruction module is used to extract CT images of the target location and perform image segmentation and 3D reconstruction on the CT images of the target location. The reconstructed 3D model is called a virtual space. The virtual target model in; The virtual target model is moved and scaled according to requirements, and the virtual target model in space is captured by the head-mounted glasses. Scaling factors on the x, y, and z axes of the coordinate system , , ; The marker acquisition module, for the moved and scaled virtual target model in the 3D reconstruction module, uses head-mounted glasses to acquire data on the marker points. A set of coordinate points for each marker point is used, and an optical positioning device is employed to collect the coordinates of the same points on the real target. A set of coordinates of a marker point; The mixed reality space registration module performs the following steps: 1) Utilize the scaling factors of the current virtual target model on the x, y, and z axes obtained from the 3D reconstruction module. , , and the point set acquired in the marker point acquisition module Perform a transformation to obtain the transformed point set. ,in Represents a point set The first in The coordinates of each marker point; for each point as well as All of them are: ； In the formula, , and Points In virtual space The values ​​on the x, y, and z axes in the coordinate system. , and Points In virtual space Values ​​on the x, y, and z axes in a coordinate system; 2) For the set of coordinate points and Using singular value decomposition to obtain the virtual space Coordinate system and positioning space The transformation matrix of the coordinate system, i.e., finding the optimal 3×3 rotation matrix. And a 3×1 translation matrix , so that: Minimize, where Represents a point set The first in The coordinates of each marker point are determined using the following method: For two point sets and Decentralization yields new point sets and Simultaneously calculate the covariance matrix. ,in, For point set The Middle One point, For point set The Middle One point; for Perform singular value decomposition to calculate the rotation and translation matrices between the point sets: ； ； In the formula, and Point sets and The center of mass, , It is a unitary matrix. It is a singular value diagonal matrix; The surgical navigation module performs the following operations: First, set the error range. Real-time location space acquisition The set of coordinates of the real target marker points in the data, and the set of points. Compare them; if the difference is greater than If the actual target pose changes, the current set of marker point coordinates is used to replace the original set of marker points. Then, the mixed reality space registration module is invoked again to register the space and update the rotation matrix. Translation matrix Secondly, an optical locator is used to obtain real-time information about the surgical instruments in the positioning space. The coordinates of the tip point and the coordinates of the other endpoint , , and Points In positioning space The values ​​on the x, y, and z axes in the coordinate system. , and Points In positioning space The values ​​on the x, y, and z axes of the coordinate system; calculate the values ​​in the virtual space. The corresponding point coordinates and , , and Points In virtual space The values ​​on the x, y, and z axes in the coordinate system. , and Points In virtual space The values ​​on the x, y, and z axes of the coordinate system; where The subscripts j=1,2; through the above operations, the pose update of the surgical tool navigation line in the virtual space is finally realized, thereby completing the surgical navigation.

2. A scalable real-time mixed reality spatial registration and navigation system according to claim 1, characterized in that, The 3D reconstruction module uses a CT scanner to acquire CT images of the target location, performs 3D reconstruction using multi-view geometry, moves the scaled virtual target model to the required position and then keeps it stationary, and then obtains the current scaling factor. , , .

3. A scalable real-time mixed reality spatial registration and navigation system according to claim 2, characterized in that, The marker point acquisition module includes a virtual target marker point acquisition module and a real target marker point acquisition module, wherein: The virtual target marker acquisition module acquires the virtual target model after it has been moved and scaled in the 3D reconstruction module within the virtual space. In the middle, the data was collected using head-mounted glasses. The set of coordinates of the marked points ,in Represents a point set The first in Coordinates of the marked points; The real target marker acquisition module utilizes an optical positioning device in the positioning space. Obtain the same as the real target in The set of coordinates of the marked points ,in Represents a point set The first in Coordinates of the marked points; coordinate point set and coordinate point set There is a correspondence, that is and These are marker points at the same locations on the virtual target model and the real target, respectively.

Images