Mixed reality navigation registration method and device, computer equipment and storage medium

Abstract

The invention relates to a mixed reality navigation registration method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring first point cloud data of an operation object in a real scene; acquiring second point cloud data by using a depth camera, calculating a first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data under the optical positioning system coordinate system through the first spatial transformation matrix; third point cloud data of the surgical object in the real scene are collected, the second point cloud data and the third point cloud data are registered, and a second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system is obtained; and constructing a surgical instrument virtual model on mixed reality glasses by using the first spatial transformation matrix, the second spatial transformation matrix and the three-dimensional model data. According to the invention, the operation process of mixed reality navigation is simplified, and the precision of mixed reality navigation is improved.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular, relates to a mixed reality navigation registration method, device, computer equipment and storage medium. Background Art

[0002] Mixed Reality (MR) technology is a mixture of the real physical world and the digital world (or virtual scene), and the operator can interact with the three-dimensional digital model in the digital world in 3D. Due to the characteristics of miniaturization, lightness and portability of mixed reality technology, favorable conditions have been created for the application of mixed reality technology in clinical surgical operations or surgical teaching. Mixed reality technology can integrate three-dimensional models, images and other data with real scenes, getting rid of the limitation that operators cannot see the internal structure of the human body or organs during clinical surgery or surgical teaching, greatly promoting the safety and accuracy of surgical operations and a deep understanding of the position of human anatomical structures in a teaching environment. When applying mixed reality technology to the teaching environment of clinical surgery or medical anatomy courses, the three-dimensional models of organs, blood vessels, bones, and lesions segmented and reconstructed from computed tomography (CT) or magnetic resonance imaging (MRI) images can be imported into mixed reality glasses (such as Microsoft's HoloLens), and then the three-dimensional model registration algorithm can be used to align the surgical objects in the real space with the three-dimensional digital models, thereby realizing the fusion of the physical world and the digital world and achieving the goal of "combining the virtual and the real".

[0003] In mixed reality technology, the key technology for the fusion of virtual and real information is to realize the spatial information registration of different coordinate systems. Common registration methods are usually divided into two steps. The first step is to detect the key feature points of model information in different coordinate spaces or manually mark the feature points; the second step is to use the least squares algorithm to calculate the spatial transformation matrix between feature points in different coordinate systems. The defect of the above registration method is that if not enough key feature points are detected in the scene, the registration process will fail, and the process of manually marking feature points is highly subjective and unstable. The spatial information of feature points marked by different operators in the same period or the same operator in different periods may be different, and the marking process of feature information is relatively cumbersome.

[0004] In the prior art, when applying mixed reality technology to mixed reality navigation scenarios such as clinical surgical operations or surgical teaching, it is usually necessary to fix a positioning tool that can be tracked by an optical positioning system on the mixed reality glasses. When it comes to surgical instrument operations, it is necessary to track the positioning tools on the mixed reality glasses and the surgical instruments at the same time. If one of the positioning tools is blocked by light, the tracking process will be interrupted. In actual operation, it is necessary to always ensure that the positioning tools on the mixed reality glasses and the surgical instruments can be tracked by the optical positioning system at the same time. Compared with tracking only one positioning tool, this method of implementing mixed reality navigation is obviously not friendly enough. At the same time, after fixing the positioning tool on the mixed reality glasses, it is necessary to use a calibration algorithm to calculate the spatial transformation of the local coordinate system of the positioning tool and the glasses coordinate system. This process will also introduce new errors, which will ultimately affect the system accuracy of the mixed reality navigation.

[0005] On the other hand, when realizing the "coordinate transformation" in mixed reality navigation, the spatial coordinates of the virtual marker ball and the real marker ball need to be manually selected, which is highly subjective and unstable. It is usually manifested in that the spatial coordinate values ​​selected by the same operator at different time periods or by different operators at the same time period may be different, which ultimately leads to poor traceability and repeatability of the fusion results. Summary of the invention

[0006] The present application provides a mixed reality navigation registration method, apparatus, computer device and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0007] In order to solve the above problems, this application provides the following technical solutions:

[0008] A mixed reality navigation registration method, comprising:

[0009] Acquire a preoperative DICOM image of a surgical object in a real scene, generate three-dimensional model data of the surgical target according to the preoperative DICOM image, and collect first point cloud data according to the three-dimensional model data;

[0010] A second point cloud data of a surgical object in a real scene is acquired by using a depth camera, and spatial position information of a surgical instrument in the real scene is recorded by using an optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera;

[0011] Using a point cloud registration algorithm to register the first point cloud data and the second point cloud data, obtaining a first space transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first space transformation matrix;

[0012] Using the mixed reality glasses to collect third point cloud data of the surgical object in the real scene, and using a point cloud registration algorithm to register the second point cloud data and the third point cloud data, to obtain a second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses;

[0013] The three-dimensional model data in the optical positioning system coordinate system and the spatial posture information of the surgical instrument are transmitted to the mixed reality glasses. The mixed reality glasses use a second spatial transformation matrix to transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system, and construct a virtual model of the surgical instrument based on the transformed three-dimensional model data and the spatial posture information of the surgical instrument, and perform mixed reality navigation through the virtual model of the surgical instrument.

[0014] The technical solution adopted by the embodiment of the present application also includes: obtaining a preoperative DICOM image of a surgical object in a real scene, generating three-dimensional model data of the surgical target according to the preoperative DICOM image, and collecting first point cloud data according to the three-dimensional model data, specifically:

[0015] Performing CT or MRI image scanning on the surgical object to obtain a preoperative DICOM image of the surgical object;

[0016] The preoperative DICOM image is segmented by using a threshold segmentation algorithm to obtain an image of a region of interest of the surgical target, and the image of the region of interest is reconstructed by using a marching cube method to generate three-dimensional model data of the surgical target, and the three-dimensional model data is stored in a file format that can be imported into an operating system embedded in the mixed reality glasses;

[0017] Point cloud data sampling is performed on the three-dimensional model data to obtain first point cloud data of the surgical object.

[0018] The technical solution adopted by the embodiment of the present application also includes: before using the depth camera to obtain the second point cloud data of the surgical object in the real scene, it also includes:

[0019] A positioning tool that can be tracked by the optical positioning system is fixed on the depth camera, and the optical positioning system is fixed at a position to ensure that the positioning tool is within the visual range of the optical positioning system and the positions of the optical positioning system and the depth camera are kept unchanged;

[0020] The optical positioning system is used to obtain the spatial pose information of the positioning tool, and the depth camera is used to collect the depth information of the calibration reference object in the depth camera coordinates, and the least squares method is used to calculate the calibration transformation matrix between the positioning tool coordinate system and the depth camera coordinate system.

[0021] The technical solution adopted by the embodiment of the present application also includes: after the second point cloud data of the surgical object in the real scene is collected by the depth camera, it also includes:

[0022] The RANSAC algorithm is used to eliminate point cloud noise in the second point cloud data, and the calibration transformation matrix is ​​used to transform the second point cloud data into the optical positioning system coordinate system, and the transformed second point cloud data is sent to the server.

[0023] The technical solution adopted in the embodiment of the present application also includes: using the mixed reality glasses to collect the third point cloud data of the surgical object in the real scene, and using the point cloud registration algorithm to register the second point cloud data and the third point cloud data, to obtain the second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses, specifically:

[0024] The third point cloud data is transformed into the mixed reality glasses coordinate system to obtain the spatial transformation parameters of the mixed reality glasses, and the third point cloud data and the spatial transformation parameters are sent to the server side, the server side calculates the spatial transformation between the transformed second point cloud data and the third point cloud data according to the point cloud registration algorithm, obtains the second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system according to the operation rule of the spatial transformation, and returns the second spatial transformation matrix to the mixed reality glasses.

[0025] The technical solution adopted in the embodiment of the present application also includes: the mixed reality glasses use the second space transformation matrix to transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, specifically:

[0026] Model MR =M*T2*T1*Model

[0027] Model MR It represents the virtual model of the surgical instrument located in the virtual scene of the mixed reality glasses after transformation, M represents the transformation matrix of the mixed reality glasses coordinate system, T2 and T1 represent the second space transformation matrix and the first space transformation matrix respectively, and Model represents the transformed three-dimensional model data.

[0028] The technical solution adopted by the embodiment of the present application also includes: after constructing the virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, it also includes:

[0029] The spatial position information of the surgical instrument is transmitted to the mixed reality glasses in real time through the optical positioning system, and the mixed reality glasses update the position information of the surgical instrument in the virtual model of the surgical instrument in real time according to the spatial position information:

[0030] P MR =M*T2*T1*P

[0031] Wherein, P represents the spatial posture information of the surgical instrument.

[0032] Another technical solution adopted by the embodiment of the present application is: a mixed reality navigation registration device, comprising:

[0033] A first point cloud acquisition module: used to acquire a preoperative DICOM image of a surgical object in a real scene, generate three-dimensional model data of the surgical object according to the preoperative DICOM image, and collect first point cloud data according to the three-dimensional model data;

[0034] The second point cloud acquisition module is used to acquire the second point cloud data of the surgical object in the real scene by using the depth camera, and to record the spatial posture information of the surgical instrument in the real scene by using the optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera;

[0035] A first registration module: used to register the first point cloud data and the second point cloud data using a point cloud registration algorithm, obtain a first space transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and convert the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first space transformation matrix;

[0036] A second registration module is used to collect third point cloud data of the surgical object in the real scene by using the mixed reality glasses, and to register the second point cloud data with the third point cloud data by using a point cloud registration algorithm to obtain a second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses;

[0037] Virtual model construction module: used to transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial posture information of the surgical instrument to the mixed reality glasses, and the mixed reality glasses use the second space transformation matrix to transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, and perform mixed reality navigation through the virtual model of the surgical instrument.

[0038] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein:

[0039] The memory stores program instructions for implementing the mixed reality navigation registration method;

[0040] The processor is used to execute the program instructions stored in the memory to control a mixed reality navigation registration method.

[0041] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the mixed reality navigation alignment method.

[0042] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the mixed reality navigation registration method, device, computer equipment and storage medium of the embodiments of the present application solve the technical problems of registration and fusion of different coordinate space information and spatial posture tracking and positioning of mixed reality glasses in mixed reality scenes by introducing a depth camera for the registration process of mixed reality navigation, and effectively simplify the registration process of mixed reality navigation; secondly, the embodiments of the present application do not require manual intervention in selecting marker points, overcome the operational inconvenience caused by the need to manually select marker points, and reduce the problems of poor traceability and repeatability caused by the interaction; in addition, the embodiments of the present application do not need to fix the positioning tool on the mixed reality glasses. When tracking surgical instruments, there is no need to track the positioning tools on the surgical instruments and the mixed reality glasses at the same time, which reduces the possibility of failure of real-time update of the spatial position of the virtual model due to light obstruction. At the same time, there is no need to determine the spatial transformation of the local coordinate system of the positioning tool and the glasses coordinate system through a calibration algorithm, which simplifies the operation process of mixed reality navigation, reduces the error introduced by the calibration process of the mixed reality glasses, and improves the accuracy of mixed reality navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a mixed reality navigation registration method according to an embodiment of the present application;

[0044] Figure 2 A spatial transformation relationship diagram between various entity objects in the mixed reality navigation of the embodiment of the present application;

[0045] Figure 3 A schematic diagram of a mixed reality navigation registration phantom experiment according to an embodiment of the present application;

[0046] Figure 4 A schematic diagram of visualization results of navigation evaluation errors of different puncture paths performed on a phantom according to an embodiment of the present application;

[0047] Figure 5 This is a schematic diagram of the structure of a mixed reality navigation registration device according to an embodiment of the present application;

[0048] Figure 6A schematic diagram of the structure of a computer device according to an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The terms "first", "second", "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or computer device that includes a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or computer devices.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] See also Figure 1 , is a flow chart of the mixed reality navigation registration method of the embodiment of the present application. The mixed reality navigation registration method of the second embodiment of the present application comprises the following steps:

[0054] S100: reconstructing three-dimensional model data of the surgical target by using CT or MRI, and sampling to generate first point cloud data of the surgical object;

[0055] In this step, the surgical target refers to the skin, tissue, organ, blood vessel or lesion of the surgical object, which needs to be operated on using mixed reality navigation. The first point cloud data acquisition method of the surgical object is as follows:

[0056] S101: Perform CT or MRI image scanning on the surgical object to obtain a preoperative DICOM (Digital Imaging and Communications in Medicine) image of the surgical object;

[0057] S102: Segment the preoperative DI COM image using a threshold segmentation algorithm to obtain an image of a region of interest of the surgical target, and reconstruct the image of the region of interest using a marching cube method to generate three-dimensional model data of the surgical target;

[0058] Among them, the segmentation method of the preoperative DI COM image is specifically as follows: first, the preoperative DI COM image of the surgical object is imported into the medical image segmentation system. The medical image segmentation system uses medical image segmentation algorithms such as threshold segmentation or level set to segment the preoperative DI COM image to obtain a segmentation mask binary image of the region of interest. Then, the moving cube method is used to reconstruct the region of interest to generate three-dimensional model data of surgical targets such as skin, tissue, organ, blood vessel, lesion, etc., recorded as Model, and the three-dimensional model data Model l is stored in a file format such as obj or st l that can be imported into the embedded operating system of the mixed reality glasses.

[0059] S103: performing point cloud data sampling on the three-dimensional model data to obtain first point cloud data of the surgical object;

[0060] The point cloud data sampling method is as follows: first, the 3D model data is sampled at equal intervals to generate the first point cloud data of the surgical object surface; at the same time, in order to improve the registration accuracy, the point cloud data irrelevant to the surgery in the first point cloud data is cut off. For example, when the surgical object lies flat during the surgery, the point cloud data on the back of the surgical object is cut off and does not need to participate in the subsequent point cloud registration calculation.

[0061] S110: using the depth camera to obtain second point cloud data of the surgical object in the real scene, transforming the second point cloud data into the optical positioning system coordinate system, and using the optical positioning system to obtain spatial posture information of the surgical instrument in the real scene; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera, and the positions of the optical positioning system and the depth camera remain unchanged;

[0062] In this step, the depth camera includes but is not limited to a laser radar, a binocular vision depth camera, a camera based on time of flight (ToF), a depth camera based on structured light, and any other camera that can obtain three-dimensional space depth information. In the embodiment of the present application, the method for obtaining the second point cloud data is specifically as follows:

[0063] S111: fixing a positioning tool that can be tracked by the optical positioning system on the depth camera, and then fixing the optical positioning system at a certain position, ensuring that the positioning tool on the depth camera is within the visual range of the optical positioning system, and keeping the positions of the optical positioning system and the depth camera unchanged;

[0064] S112: using the optical positioning system to obtain the position and posture information of the positioning tool, using the depth camera to collect the depth information of the calibration reference object in the depth camera coordinates, and using the least squares method to calculate the calibration transformation matrix between the positioning tool coordinate system and the depth camera coordinate system;

[0065] Among them, since the position between the depth camera and the positioning tool is relatively fixed, the calibration transformation matrix between the positioning tool coordinate system and the depth camera coordinate system only needs to be calculated once, and the calibration transformation matrix is ​​solidified into a calibration file for subsequent repeated use.

[0066] S113: using the depth camera to collect second point cloud data of the surgical object in the real scene, and using the calibration transformation matrix to transform the second point cloud data into the optical positioning system coordinate system, transmitting the transformed second point cloud data to the server, and synchronously recording the spatial posture information of the surgical instrument in the real scene through the optical positioning system;

[0067] In order to reduce the influence of noise on subsequent registration, after collecting the second point cloud data, the RANSAC (RANdomSAmple Consensus) algorithm is used to eliminate the point cloud noise in the second point cloud data.

[0068] In the above, the second point cloud data collected by the depth camera is transformed into the optical positioning system coordinate system through the calibration transformation matrix, which is used for the registration algorithm of mixed reality navigation.

[0069] S120: aligning the first point cloud data and the transformed second point cloud data using a point cloud registration algorithm to obtain a first space transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first space transformation matrix;

[0070] In this step, a point cloud registration algorithm is used to register the first point cloud data collected through the three-dimensional model data and the second point cloud data collected through the depth camera, and the first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system is obtained, so as to realize the automation of the registration process without introducing marker points. The embodiment of the present application acquires the three-dimensional point cloud data of the surgical object in the real scene by introducing a depth camera, and fixes a positioning tool that can be tracked by the optical positioning system on the depth camera. Before using mixed reality navigation for surgical operation, the first spatial transformation matrix between the positioning tool coordinate system and the depth camera coordinate system is calculated by the point cloud registration algorithm. Since the position between the positioning tool and the depth camera is relatively fixed, the first spatial transformation matrix between the two only needs to be calculated once to be solidified into a calibration file, and no repeated calculation is required for subsequent use.

[0071] S130: using the mixed reality glasses to collect third point cloud data of the surgical object in the real scene, transforming the third point cloud data into the mixed reality glasses coordinate system, obtaining the spatial transformation parameters of the mixed reality glasses, and sending the third point cloud data and the spatial transformation parameters to the server;

[0072] In this step, the mixed reality glasses include all wearable computer devices that can realize mixed reality rendering, such as HoloLens, etc. Before using mixed reality navigation to perform surgical operations, the embodiment of the present application uses mixed reality glasses to collect third point cloud data of surgical objects in real scenes, and after spatially transforming the third point cloud data with the mixed reality glasses coordinate system, obtains the spatial transformation parameters of the mixed reality glasses, and transmits the third point cloud data and the spatial transformation parameters to the server via a wireless network.

[0073] S140: receiving, through the server, second point cloud data in the optical positioning system coordinate system, third point cloud data in the mixed reality glasses coordinate system, and spatial transformation parameters, calculating a second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system according to the second point cloud data, the third point cloud data, and the spatial transformation parameters, and returning the calculation result to the mixed reality glasses;

[0074] In this step, after the server receives the second point cloud data, the third point cloud data and the spatial transformation parameters, it first calculates the spatial transformation between the second point cloud data and the third point cloud data according to the point cloud registration algorithm, and then obtains the second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system according to the spatial transformation operation rule, and transmits the second spatial transformation matrix to the mixed reality glasses via the wireless network.

[0075] S150: transmitting the three-dimensional model data in the optical positioning system coordinate system and the spatial position information of the surgical instrument to the mixed reality glasses, the mixed reality glasses transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second spatial transformation matrix, and constructing a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial position information of the surgical instrument, and performing mixed reality navigation through the virtual model of the surgical instrument;

[0076] In this step, if Figure 2 As shown, it is a spatial transformation relationship diagram between various entity objects in the mixed reality navigation of the embodiment of the present application, and the entity objects include mixed reality glasses, an optical positioning system, a depth camera, a server (personal computer), a surgical object, and a surgical instrument. Specifically, after receiving the three-dimensional model data in the optical positioning system coordinate system, the mixed reality glasses use the second spatial transformation matrix to transform the three-dimensional model data to the mixed reality glasses coordinate system, and after calculating the spatial posture information of the surgical instrument and the second spatial transformation matrix, the transformed three-dimensional model data is rendered in the mixed reality glasses coordinate system to achieve the virtual and real fusion of the virtual scene and the real scene. Specifically, the embodiment of the present application transforms the three-dimensional model data to the mixed reality glasses coordinate system for rendering by calculating the following expression:

[0077] Model MR =M*T2*T1*Model(1)

[0078] Model MR It represents the virtual model of the surgical instrument in the virtual scene of the mixed reality glasses after transformation, M represents the transformation matrix of the mixed reality glasses coordinate system, T2 and T1 represent the second space transformation matrix and the first space transformation matrix respectively, and Model represents the transformed three-dimensional model data.

[0079] S160: During the mixed reality navigation process, the spatial position information of the surgical instrument is transmitted to the mixed reality glasses in real time through the optical positioning system, and the mixed reality glasses update the position information of the surgical instrument in the virtual model of the surgical instrument in real time according to the spatial position information until the mixed reality navigation ends;

[0080] In this step, after the mixed reality glasses receive the spatial posture information of the surgical instrument, they perform operations with the second spatial transformation matrix T2 and the spatial transformation parameters of the mixed reality glasses, transform the spatial posture information into the mixed reality glasses coordinate system, and update the spatial position of the surgical instrument in the virtual model of the surgical instrument, so as to realize real-time tracking of the surgical instrument in the mixed reality navigation process. Specifically, the spatial posture information of the surgical instrument under the optical positioning system is denoted as P, and the embodiment of the present application transforms its spatial posture information into the mixed reality glasses coordinate system through the following expression:

[0081] P MR =M*T2*T1*P(2)

[0082] In the above, the embodiment of the present application uses the spatial transformation matrix between different coordinate systems embedded in the mixed reality glasses for the calculation of the mixed reality navigation alignment process, and based on the solidification characteristics of the embedded coordinate system, it achieves the purpose of dynamically tracking surgical instruments without relying on additional positioning tools.

[0083] It should be noted that the role of the depth camera in the embodiment of the present application is to calculate the first spatial transformation matrix used for alignment. After the first spatial transformation matrix is ​​calculated, the depth camera can be removed, and there is no need to track the depth camera during the mixed reality navigation process. In addition, the embodiment of the present application does not need to fix the positioning tool on the mixed reality glasses. When tracking surgical instruments, there is no need to track the surgical instruments and the positioning tools on the mixed reality glasses at the same time, which reduces the possibility of failure of real-time update of the spatial position of the virtual model of the surgical instrument due to light obstruction. At the same time, there is no need to determine the spatial transformation of the local coordinate system of the positioning tool and the coordinate system of the mixed reality glasses through a calibration algorithm, which is beneficial to improving the accuracy of mixed reality navigation.

[0084] In order to verify the feasibility and effectiveness of the embodiments of the present application, the following embodiments are verified by mixed reality navigation phantom experiments. Figure 3 As shown, in Figure 3 In the figure, (A) and (B) respectively represent the effects of fusion rendering of the virtual model of surgical instruments and the real phantom viewed in mixed reality glasses from different angles after mixed reality registration, (C) is the scene of the operator wearing mixed reality glasses to perform surgery, and (D) is a scene diagram of the mixed reality navigation phantom experiment. Figure 4 This is a schematic diagram of the visualization results of navigation evaluation errors of different puncture paths on a phantom, in which 6 different puncture paths are designed. Under the guidance of the implemented mixed reality navigation system, after tracking and puncturing the surgical instrument, the evaluation errors of the 6 different puncture paths are: 4.3mm, 4.6mm, 3.5mm, 4.5mm, 3.2mm, 4.8mm. The experimental results show that the embodiment of the present application achieves the registration effect of mixed reality navigation without markers by introducing a depth camera. It only needs to track the spatial posture information of the surgical instrument in the real scene, and the corresponding position can be synchronously updated in real time in the virtual scene of the mixed reality glasses, which can improve the accuracy of the mixed reality navigation system and greatly promote the safety and accuracy of surgical operations.

[0085] Based on the above, the mixed reality navigation registration method of the embodiment of the present application solves the technical problems of registration and fusion of different coordinate space information and spatial posture tracking and positioning of mixed reality glasses in mixed reality scenes by introducing a depth camera for the registration process of mixed reality navigation, thereby effectively simplifying the registration process of mixed reality navigation; secondly, the embodiment of the present application does not require manual intervention in selecting marker points, overcomes the operational inconvenience caused by the need to manually select marker points, and reduces the problems of poor traceability and repeatability caused by the interaction; in addition, the embodiment of the present application does not require fixing the positioning tool on the mixed reality glasses. When tracking surgical instruments, there is no need to track the positioning tools on the surgical instruments and the mixed reality glasses at the same time, reducing the possibility of failure of real-time update of the spatial position of the virtual model due to light obstruction. At the same time, there is no need to determine the spatial transformation of the local coordinate system of the positioning tool and the glasses coordinate system through a calibration algorithm, which simplifies the operation process of mixed reality navigation, reduces the errors introduced by the calibration process of the mixed reality glasses, and improves the accuracy of mixed reality navigation.

[0086] See also Figure 5 , is a schematic diagram of the structure of a mixed reality navigation registration device according to an embodiment of the present application. The mixed reality navigation registration device 40 according to an embodiment of the present application comprises:

[0087] The first point cloud acquisition module 41 is used to acquire a preoperative DICOM image of a surgical object in a real scene, generate three-dimensional model data of the surgical object according to the preoperative DICOM image, and collect first point cloud data according to the three-dimensional model data;

[0088] The second point cloud acquisition module 42 is used to acquire the second point cloud data of the surgical object in the real scene by using the depth camera, and to record the spatial position information of the surgical instrument in the real scene by using the optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera;

[0089] The first registration module 43 is used to register the first point cloud data and the second point cloud data by using a point cloud registration algorithm to obtain a first space transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and convert the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first space transformation matrix;

[0090] The second registration module 44 is used to collect the third point cloud data of the surgical object in the real scene by using the mixed reality glasses, and register the second point cloud data with the third point cloud data by using the point cloud registration algorithm to obtain a second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses;

[0091] Virtual model construction module 45: used to transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial posture information of the surgical instrument to the mixed reality glasses, and the mixed reality glasses use the second space transformation matrix to transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, and perform mixed reality navigation through the virtual model of the surgical instrument.

[0092] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0093] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0094] See also Figure 6 , is a schematic diagram of the computer device structure of an embodiment of the present application. The computer device 50 includes:

[0095] A memory 51 storing executable program instructions;

[0096] A processor 52 connected to the memory 51;

[0097] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: obtain a preoperative DICOM image of the surgical object in the real scene, generate three-dimensional model data of the surgical target according to the preoperative DICOM image, and collect first point cloud data according to the three-dimensional model data; use a depth camera to obtain second point cloud data of the surgical object in the real scene, and use an optical positioning system to record the spatial posture information of the surgical instrument in the real scene; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; use a point cloud registration algorithm to register the first point cloud data and the second point cloud data, obtain a first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and convert the three point cloud data into a single point cloud by using the first spatial transformation matrix. The three-dimensional model data is converted into three-dimensional model data in the coordinate system of the optical positioning system; the third point cloud data of the surgical object in the real scene is collected by using the mixed reality glasses, and the second point cloud data and the third point cloud data are registered by using the point cloud registration algorithm to obtain a second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses; the three-dimensional model data in the coordinate system of the optical positioning system and the spatial posture information of the surgical instrument are transmitted to the mixed reality glasses, and the mixed reality glasses use the second space transformation matrix to transform the three-dimensional model data in the coordinate system of the optical positioning system into the coordinate system of the mixed reality glasses, and a virtual model of the surgical instrument is constructed according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, and mixed reality navigation is performed through the virtual model of the surgical instrument.

[0098] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0099] See also Figure 7 , Figure 7Schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 that can implement the following steps: obtaining a preoperative DICOM image of a surgical object in a real scene, generating three-dimensional model data of the surgical target based on the preoperative DICOM image, and collecting first point cloud data based on the three-dimensional model data; using a depth camera to obtain second point cloud data of the surgical object in the real scene, and using an optical positioning system to record the spatial posture information of the surgical instrument in the real scene; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; using a point cloud registration algorithm to register the first point cloud data and the second point cloud data, to obtain a first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and converting the first spatial transformation matrix into the three-dimensional model The method comprises the steps of: converting the three-dimensional model data into three-dimensional model data in the coordinate system of the optical positioning system; using the mixed reality glasses to collect the third point cloud data of the surgical object in the real scene, and using the point cloud registration algorithm to register the second point cloud data and the third point cloud data, so as to obtain the second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses; transmitting the three-dimensional model data in the coordinate system of the optical positioning system and the spatial posture information of the surgical instrument to the mixed reality glasses, and the mixed reality glasses use the second space transformation matrix to transform the three-dimensional model data in the coordinate system of the optical positioning system into the coordinate system of the mixed reality glasses, and constructing a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, and performing mixed reality navigation through the virtual model of the surgical instrument. Among them, the program instruction 61 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network computer device, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program instructions, or terminal computer devices such as computers, servers, mobile phones, tablets, etc. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0100] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0101] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A mixed reality navigation registration method, characterized in that: include: Acquire a preoperative DICOM image of a surgical object in a real scene, generate three-dimensional model data of the surgical target according to the preoperative DICOM image, and collect first point cloud data according to the three-dimensional model data; A second point cloud data of a surgical object in a real scene is acquired by using a depth camera, and spatial position information of a surgical instrument in the real scene is recorded by using an optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; Using a point cloud registration algorithm to register the first point cloud data and the second point cloud data, obtaining a first space transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first space transformation matrix; Using the mixed reality glasses to collect third point cloud data of the surgical object in the real scene, and using a point cloud registration algorithm to register the second point cloud data and the third point cloud data, to obtain a second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses; The three-dimensional model data in the optical positioning system coordinate system and the spatial posture information of the surgical instrument are transmitted to the mixed reality glasses. The mixed reality glasses use a second spatial transformation matrix to transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system, and construct a virtual model of the surgical instrument based on the transformed three-dimensional model data and the spatial posture information of the surgical instrument, and perform mixed reality navigation through the virtual model of the surgical instrument.

2. The mixed reality navigation registration method according to claim 1, characterized in that: The step of acquiring a preoperative DICOM image of a surgical object in a real scene, generating three-dimensional model data of the surgical object according to the preoperative DICOM image, and collecting first point cloud data according to the three-dimensional model data is specifically as follows: Performing CT or MRI image scanning on the surgical object to obtain a preoperative DICOM image of the surgical object; The preoperative DICOM image is segmented by using a threshold segmentation algorithm to obtain an image of a region of interest of the surgical target, and the image of the region of interest is reconstructed by using a marching cube method to generate three-dimensional model data of the surgical target, and the three-dimensional model data is stored in a file format that can be imported into an operating system embedded in the mixed reality glasses; Point cloud data sampling is performed on the three-dimensional model data to obtain first point cloud data of the surgical object.

3. The mixed reality navigation registration method according to claim 2, characterized in that: Before using the depth camera to obtain the second point cloud data of the surgical object in the real scene, the method also includes: A positioning tool that can be tracked by the optical positioning system is fixed on the depth camera, and the optical positioning system is fixed at a position to ensure that the positioning tool is within the visual range of the optical positioning system and the positions of the optical positioning system and the depth camera are kept unchanged; The optical positioning system is used to obtain the spatial pose information of the positioning tool, and the depth camera is used to collect the depth information of the calibration reference object in the depth camera coordinates, and the least squares method is used to calculate the calibration transformation matrix between the positioning tool coordinate system and the depth camera coordinate system.

4. The mixed reality navigation registration method according to claim 3, characterized in that: After the second point cloud data of the surgical object in the real scene is collected by the depth camera, the method further includes: The RANSAC algorithm is used to eliminate point cloud noise in the second point cloud data, and the calibration transformation matrix is ​​used to transform the second point cloud data into the optical positioning system coordinate system, and the transformed second point cloud data is sent to the server.

5. The mixed reality navigation registration method according to claim 4, characterized in that: The third point cloud data of the surgical object in the real scene is collected by using the mixed reality glasses, and the second point cloud data and the third point cloud data are registered by using the point cloud registration algorithm to obtain the second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses, which is specifically: The third point cloud data is transformed into the mixed reality glasses coordinate system to obtain the spatial transformation parameters of the mixed reality glasses, and the third point cloud data and the spatial transformation parameters are sent to the server side, the server side calculates the spatial transformation between the transformed second point cloud data and the third point cloud data according to the point cloud registration algorithm, obtains the second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system according to the operation rule of the spatial transformation, and returns the second spatial transformation matrix to the mixed reality glasses.

6. The mixed reality navigation registration method according to claim 5, characterized in that: The mixed reality glasses transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second space transformation matrix, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, specifically: Model MR =M*T2*T1*Model Model MR It represents the virtual model of the surgical instrument located in the virtual scene of the mixed reality glasses after transformation, M represents the transformation matrix of the mixed reality glasses coordinate system, T2 and T1 represent the second space transformation matrix and the first space transformation matrix respectively, and Model represents the transformed three-dimensional model data.

7. The mixed reality navigation registration method according to any one of claims 1 to 6, characterized in that: After constructing the virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, the method further includes: The spatial position information of the surgical instrument is transmitted to the mixed reality glasses in real time through the optical positioning system, and the mixed reality glasses update the position information of the surgical instrument in the virtual model of the surgical instrument in real time according to the spatial position information: P MR =M*T2*T1*P Wherein, P represents the spatial posture information of the surgical instrument.

8. A mixed reality navigation registration device, characterized in that: include: A first point cloud acquisition module: used to acquire a preoperative DICOM image of a surgical object in a real scene, generate three-dimensional model data of the surgical object according to the preoperative DICOM image, and collect first point cloud data according to the three-dimensional model data; The second point cloud acquisition module is used to acquire the second point cloud data of the surgical object in the real scene by using the depth camera, and to record the spatial posture information of the surgical instrument in the real scene by using the optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; A first registration module: used to register the first point cloud data and the second point cloud data using a point cloud registration algorithm, obtain a first space transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and convert the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first space transformation matrix; A second registration module is used to collect third point cloud data of the surgical object in the real scene by using the mixed reality glasses, and to register the second point cloud data with the third point cloud data by using a point cloud registration algorithm to obtain a second space transformation matrix between the coordinate system of the optical positioning system and the coordinate system of the mixed reality glasses; Virtual model construction module: used to transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial posture information of the surgical instrument to the mixed reality glasses, and the mixed reality glasses use the second space transformation matrix to transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial posture information of the surgical instrument, and perform mixed reality navigation through the virtual model of the surgical instrument.

9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the mixed reality navigation registration method according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control a mixed reality navigation registration method.

10. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the mixed reality navigation alignment method described in any one of claims 1 to 7.

Images