Mixed reality navigation registration method and device, computer device and storage medium

Abstract

The application relates to a mixed reality navigation registration method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring first point cloud data of a surgical object in a real scene; acquiring second point cloud data by using a depth camera; calculating a first space transformation matrix between a preoperative DICOM image coordinate system and an optical positioning system coordinate system; converting three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first space transformation matrix; collecting third point cloud data of the surgical object in the real scene; registering the second point cloud data and the third point cloud data to obtain a second space transformation matrix between the optical positioning system coordinate system and a mixed reality glasses coordinate system; and constructing a virtual surgical instrument model in the mixed reality glasses by using the first space transformation matrix, the second space transformation matrix and the three-dimensional model data. The application simplifies the operation process of the mixed reality navigation and improves the accuracy of the mixed reality navigation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a mixed reality navigation registration method and device, a computer device and a storage medium. BACKGROUND

[0002] Mixed reality (MR) technology is a mixture of the real physical world and the digital world (or virtual scene), and the operator can interact with the three-dimensional digital model in the digital world. Due to the miniaturization, lightness and portability of the mixed reality technology, it creates favorable conditions for the application of the mixed reality technology in clinical operation or surgical teaching scenarios. The mixed reality technology can realize the fusion of three-dimensional models, images and real scenes, and get rid of the limitation that the operator cannot see the internal structure of the human body or organs during clinical operation or surgical teaching, greatly improving the safety and accuracy of the operation and the deep understanding of the position of the human anatomical structure in the teaching environment. When the mixed reality technology is applied to clinical surgical operation or medical anatomy teaching environment, the three-dimensional model of organs, blood vessels, bones and lesions reconstructed from the images of computed tomography (CT) or magnetic resonance imaging (MRI) can be imported into the mixed reality glasses (such as HoloLens of Microsoft Corporation), and then the real space surgical object and the three-dimensional digital model are registered by using a three-dimensional model registration algorithm, so as to realize the fusion of the physical world and the digital world and achieve the purpose of "virtual and real combination".

[0003] In the mixed reality technology, the key technology of virtual and real information fusion is to realize the spatial information registration of different coordinate systems. The common registration method usually includes two steps, the first step is to detect the key feature points of the model information in different coordinate spaces or manually mark the feature points, and the second step is to calculate the spatial transformation matrix of the feature points in different coordinate systems by using the least square algorithm. The defect of the above registration method is that if enough key feature points cannot be detected in the scene, the registration process will fail, and the manual marking of the feature points has strong subjectivity and instability, and the spatial information of the feature points marked by different operators at the same time or the same operator at different times may be different, and the marking process of the feature information is relatively cumbersome.

[0004] In the prior art, when applying mixed reality technology to a mixed reality navigation scene such as a clinical operation or surgical teaching, a positioning tool that can be tracked by an optical positioning system is usually fixed on a mixed reality glasses. When it comes to the operation of a surgical instrument, the positioning tools on the mixed reality glasses and the surgical instrument need to be tracked at the same time, and the tracking process will be interrupted if one of the positioning tools is blocked by light. In actual operation, it is necessary to keep the positioning tools on the mixed reality glasses and the surgical instrument being tracked by the optical positioning system at the same time, which is obviously less friendly than tracking only one positioning tool. At the same time, after fixing the positioning tool on the mixed reality glasses, a calibration algorithm is needed to calculate the spatial transformation between the local coordinate system of the positioning tool and the glasses coordinate system, which will introduce new errors and ultimately affect the system accuracy of the mixed reality navigation.

[0005] On the other hand, when implementing the "coordinate transformation" in the mixed reality navigation, the spatial coordinates of the virtual marker ball and the real marker ball need to be manually selected, which has strong subjectivity and instability. Usually, the spatial coordinate values selected by the same operator at different times or by different operators at the same time may differ, ultimately resulting in poor traceability and repeatability of the fusion results. SUMMARY

[0006] The present application provides a mixed reality navigation registration method, device, computer equipment and storage medium, aiming to at least solve one of the above technical problems in the prior art to some extent.

[0007] In order to solve the above problems, the present application provides the following technical solutions:

[0008] A mixed reality navigation registration method, comprising:

[0009] Obtaining a preoperative DICOM image of a surgical object in a real scene, generating a three-dimensional model data of a surgical target according to the preoperative DICOM image, and collecting first point cloud data according to the three-dimensional model data;

[0010] Obtaining second point cloud data of the surgical object in the real scene by using a depth camera, and recording spatial pose information of a surgical instrument in the real scene by using an optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera;

[0011] Registering the first point cloud data and the second point cloud data by using a point cloud registration algorithm, obtaining a first spatial transformation matrix between a preoperative DICOM image coordinate system and an optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system by using the first spatial transformation matrix;

[0012] acquire third point cloud data of the surgical object in the real scene by using the mixed reality glasses, and register the second point cloud data and the third point cloud data by using a point cloud registration algorithm to obtain a second spatial transformation matrix between the optical positioning system coordinate system and a mixed reality glasses coordinate system;

[0013] transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, transform the three-dimensional model data in the optical positioning system coordinate system to the mixed reality glasses coordinate system by using the second spatial transformation matrix, construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, and perform mixed reality navigation through the virtual model of the surgical instrument.

[0014] The technical scheme adopted by the embodiment of the application further includes: acquiring preoperative DICOM images of the surgical object in the real scene, generating three-dimensional model data of the surgical target according to the preoperative DICOM images, and acquiring first point cloud data according to the three-dimensional model data, specifically:

[0015] performing CT or MRI image scanning on the surgical object to acquire preoperative DICOM images of the surgical object;

[0016] segmenting the preoperative DICOM images by using a threshold segmentation algorithm to acquire an image of a region of interest of the surgical target, reconstructing the image of the region of interest by using a marching cubes method to generate three-dimensional model data of the surgical target, and storing the three-dimensional model data in a file format that can be imported into an embedded operating system of the mixed reality glasses;

[0017] sampling point cloud data from the three-dimensional model data to acquire first point cloud data of the surgical object.

[0018] The technical scheme adopted by the embodiment of the application further includes: before the second point cloud data of the surgical object in the real scene is acquired by using the depth camera, the following steps are further included:

[0019] fix a positioning tool that can be tracked by an optical positioning system on the depth camera, fix the optical positioning system at a position, ensure that the positioning tool is within the visual range of the optical positioning system, and keep the positions of the optical positioning system and the depth camera unchanged;

[0020] acquire spatial pose information of the positioning tool by using the optical positioning system, simultaneously acquire depth information of a calibration reference object in the depth camera coordinate by using the depth camera, and calculate a calibration transformation matrix between the positioning tool coordinate system and the depth camera coordinate system by using a least squares method.

[0021] The technical scheme adopted by the embodiment of the application further includes:

[0022] The RANSAC algorithm is adopted to eliminate point cloud noise in the second point cloud data, and the second point cloud data is transformed to the optical positioning system coordinate system by using the calibration transformation matrix, and the transformed second point cloud data is sent to the server end.

[0023] The technical scheme adopted by the embodiment of the application further includes: the third point cloud data of the surgical object in the real scene is collected by using the mixed reality glasses, and the second point cloud data and the third point cloud data are registered by using a point cloud registration algorithm to obtain a second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system, specifically:

[0024] The third point cloud data is transformed to the mixed reality glasses coordinate system to obtain the spatial transformation parameter of the mixed reality glasses, and the third point cloud data and the spatial transformation parameter are sent to the server end, and the server end calculates the spatial transformation between the transformed second point cloud data and the third point cloud data according to the point cloud registration algorithm, obtains the second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system according to the operation rule of the spatial transformation, and returns the second spatial transformation matrix to the mixed reality glasses.

[0025] The technical scheme adopted by the embodiment of the application further includes: the mixed reality glasses transform the three-dimensional model data under the optical positioning system coordinate system to the mixed reality glasses coordinate system by using the second spatial transformation matrix, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, specifically:

[0026] Model MR =M*T2*T1*Model

[0027] Wherein Model MR represents the virtual model of the surgical instrument in the virtual scene of the mixed reality glasses after transformation, M represents the transformation matrix of the mixed reality glasses coordinate system, T2 and T1 represent the second spatial transformation matrix and the first spatial transformation matrix respectively, and Model represents the transformed three-dimensional model data.

[0028] The technical scheme adopted by the embodiment of the application further includes: after the virtual model of the surgical instrument is constructed according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, the method further includes:

[0029] The spatial pose information of the surgical instrument is transmitted in real time to the mixed reality glasses by the optical positioning system, and the mixed reality glasses updates the position information of the surgical instrument in the virtual model of the surgical instrument in real time according to the spatial pose information.

[0030] P MR =M*T2*T1*P

[0031] Wherein, P represents the spatial pose information of the surgical instrument.

[0032] Another technical solution adopted by the embodiment of the present application is a mixed reality navigation registration device, comprising:

[0033] The first point cloud acquisition module is configured to acquire a preoperative DICOM image of a surgical object in a real scene, generate three-dimensional model data of a surgical target according to the preoperative DICOM image, and collect first point cloud data according to the three-dimensional model data.

[0034] The second point cloud acquisition module is configured to acquire second point cloud data of the surgical object in the real scene by using a depth camera, and record spatial pose information of a surgical instrument in the real scene by using an optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera.

[0035] The first registration module is configured to register the first point cloud data and the second point cloud data by using a point cloud registration algorithm, obtain a first spatial transformation matrix between a preoperative DICOM image coordinate system and an optical positioning system coordinate system, and convert the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system by using the first spatial transformation matrix.

[0036] The second registration module is configured to acquire third point cloud data of the surgical object in the real scene by using a mixed reality glasses, and register the second point cloud data and the third point cloud data by using a point cloud registration algorithm, to obtain a second spatial transformation matrix between the optical positioning system coordinate system and a mixed reality glasses coordinate system.

[0037] The virtual model construction module is configured to transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second spatial transformation matrix, construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, and perform mixed reality navigation by using the virtual model of the surgical instrument.

[0038] Still another technical solution adopted by the embodiment of the present application is a computer device, comprising a processor and a memory coupled to the processor, wherein

[0039] The memory stores program instructions for implementing the mixed reality navigation registration method;

[0040] The processor is configured to execute the program instructions stored in the memory to control the mixed reality navigation registration method.

[0041] Another technical solution adopted by the embodiments of the present application is a storage medium storing program instructions executable by a processor, the program instructions being used to execute the mixed reality navigation registration method.

[0042] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the mixed reality navigation registration method, device, computer equipment and storage medium of the embodiments of the present application introduce a depth camera into the registration process of mixed reality navigation, solve the technical problems of registration fusion of different coordinate space information and tracking and positioning of the spatial pose of mixed reality glasses in a mixed reality scene, and effectively simplify the registration process of mixed reality navigation; secondly, the embodiments of the present application do not need manual intervention to select a marker point, overcome the inconvenience brought by manual selection of a marker point, reduce the problems of traceability and poor repeatability caused by interaction; in addition, the embodiments of the present application do not need to fix a positioning tool on the mixed reality glasses, do not need to track the positioning tool on the mixed reality glasses at the same time when tracking a surgical instrument, reduce the possibility that real-time updating of the spatial position of a virtual model is invalid due to light blocking, and also do not need to determine the spatial transformation between the local coordinate system of the positioning tool and the glasses coordinate system through a calibration algorithm, simplify the operation process of mixed reality navigation, reduce the error introduced by the calibration process of mixed reality glasses, and improve the accuracy of mixed reality navigation. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the mixed reality navigation registration method of the embodiments of the present application;

[0044] Figure 2 is a spatial transformation relationship diagram between various entity objects in the mixed reality navigation of the embodiments of the present application;

[0045] Figure 3 is a phantom experiment schematic diagram of the mixed reality navigation registration of the embodiments of the present application;

[0046] Figure 4 is a navigation evaluation error visualization result schematic diagram of the embodiments of the present application in the navigation of different puncture paths on a phantom;

[0047] Figure 5 is a structure schematic diagram of the mixed reality navigation registration device of the embodiments of the present application;

[0048] Figure 6A structural schematic diagram of a computer device according to an embodiment of the present application;

[0049] Figure 7 A structural schematic diagram of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0051] The terms “first”, “second”, “third” in the present application are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first”, “second”, “third” can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of “plurality” is at least two, for example, two, three, etc., unless otherwise specifically limited. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or computer device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or computer device.

[0052] In this document, the term “embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0053] Please refer to Figure 1 is a flowchart of a mixed reality navigation registration method according to an embodiment of the present application. The mixed reality navigation registration method according to the second embodiment of the present application includes the following steps:

[0054] S100: reconstructing three-dimensional model data of a surgical target by using CT or MRI, and sampling to generate first point cloud data of the surgical object;

[0055] In this step, the surgical target refers to the skin, tissue, organ, blood vessel or lesion of the surgical object, etc. which needs to be operated by using mixed reality navigation, and the first point cloud data of the surgical object is obtained in the following manner:

[0056] S101: CT or MRI image scanning is performed on the surgical object to obtain preoperative DICOM (Digital Imaging and Communications in Medicine) images of the surgical object;

[0057] S102: The preoperative DICOM images are segmented by using a threshold segmentation algorithm to obtain the region of interest images of the surgical target, and the region of interest images are reconstructed by using a moving cube method to generate three-dimensional model data of the surgical target;

[0058] The segmentation method of the preoperative DICOM images is as follows: first, the preoperative DICOM images of the surgical object are imported into a medical image segmentation system, the medical image segmentation system uses a threshold segmentation or level set medical image segmentation algorithm to segment the preoperative DICOM images to obtain a segmentation mask binary image of the region of interest, then the region of interest is reconstructed by using a moving cube method to generate three-dimensional model data of the surgical target such as skin, tissue, organ, blood vessel, lesion, etc., denoted as Model, and the three-dimensional model data Model is stored as an obj or stl file format which can be imported into the mixed reality glasses embedded operating system.

[0059] S103: Point cloud data sampling is performed on the three-dimensional model data to obtain first point cloud data of the surgical object;

[0060] The point cloud data sampling method is as follows: first, the three-dimensional model data is sampled by using an equal interval sampling method to generate first point cloud data of the surface of the surgical object; at the same time, in order to improve the registration accuracy, the point cloud data unrelated to the surgery in the first point cloud data is cut off. Taking the case where the surgical object adopts a lying posture during the surgery as an example, the point cloud data of the back of the surgical object is cut off, which is not needed for subsequent point cloud registration calculation.

[0061] S110: The second point cloud data of the surgical object in the real scene is obtained by using a depth camera, the second point cloud data is transformed to the coordinate system of the optical positioning system, and the spatial pose information of the surgical instrument in the real scene is obtained by using the optical positioning system; wherein a positioning tool which can be tracked by the optical positioning system is fixed on the depth camera, and the position of the optical positioning system and the depth camera remains unchanged;

[0062] In this step, the depth camera includes but is not limited to any one of the following: a laser radar, a binocular vision depth camera, a camera based on time flight method (ToF), a depth camera based on structured light, etc. In the embodiment of the present application, the acquisition method of the second point cloud data is specifically as follows:

[0063] S111: A positioning tool that can be tracked by an optical positioning system is fixed on the depth camera, and then the optical positioning system is fixed at a certain position to ensure that the positioning tool on the depth camera is within the visual range of the optical positioning system, and the positions of the optical positioning system and the depth camera are kept unchanged;

[0064] S112: The pose information of the positioning tool is acquired by using the optical positioning system, and the depth information of the calibration reference in the depth camera coordinates is collected by using the depth camera, and the calibration transformation matrix between the positioning tool coordinate system and the depth camera coordinate system is calculated by using the least square method;

[0065] Since the positions of the depth camera and the positioning tool are relatively fixed, the calibration transformation matrix between the positioning tool coordinate system and the depth camera coordinate system only needs to be calculated once, and the calibration transformation matrix is solidified as a calibration file for subsequent repeated use.

[0066] S113: The second point cloud data of the surgical object in the real scene is collected by using the depth camera, and the second point cloud data is transformed to the optical positioning system coordinate system by using the calibration transformation matrix, the transformed second point cloud data is transmitted to the server side, and the spatial pose information of the surgical instrument in the real scene is recorded synchronously by using the optical positioning system;

[0067] In order to reduce the influence of noise on subsequent registration, the RANSAC (RAndom SAmple Consensus) algorithm is used to eliminate the point cloud noise in the second point cloud data after the second point cloud data is collected.

[0068] In the above, the second point cloud data collected by the depth camera is transformed to the optical positioning system coordinate system by using the calibration transformation matrix, and is used for the registration algorithm of mixed reality navigation.

[0069] S120: The first point cloud data and the transformed second point cloud data are registered by using the point cloud registration algorithm to obtain the first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and the three-dimensional model data is converted to the three-dimensional model data in the optical positioning system coordinate system by using the first spatial transformation matrix;

[0070] In this step, the first point cloud data collected by the three-dimensional model data acquisition and the second point cloud data collected by the depth camera are registered by using a point cloud registration algorithm to obtain a first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and the registration process without introducing a marker point is realized. The three-dimensional point cloud data of the surgical object in the real scene is obtained by introducing the depth camera, and a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera. Before the mixed reality navigation is used for the surgical operation, the first spatial transformation matrix between the positioning tool coordinate system and the depth camera coordinate system is calculated by using the point cloud registration algorithm. Since the positions of the positioning tool and the depth camera are relatively fixed, the first spatial transformation matrix between the two only needs to be calculated once and can be fixed as a calibration file, and does not need to be repeatedly calculated in subsequent use.

[0071] In S130, the third point cloud data of the surgical object in the real scene is collected by using the mixed reality glasses, the third point cloud data is transformed into the mixed reality glasses coordinate system to obtain the spatial transformation parameter of the mixed reality glasses, and the third point cloud data and the spatial transformation parameter are sent to the server end.

[0072] In this step, the mixed reality glasses include all wearable computer devices capable of realizing mixed reality rendering, such as HoloLens and the like. Before the surgical operation is performed by using the mixed reality navigation, the third point cloud data of the surgical object in the real scene is collected by using the mixed reality glasses, the third point cloud data is spatially transformed with the mixed reality glasses coordinate system to obtain the spatial transformation parameter of the mixed reality glasses, and the third point cloud data and the spatial transformation parameter are transmitted to the server end through a wireless network.

[0073] In S140, the second point cloud data in the optical positioning system coordinate system, the third point cloud data in the mixed reality glasses coordinate system, and the spatial transformation parameter are received by the server end, a second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system is calculated according to the second point cloud data, the third point cloud data, and the spatial transformation parameter, and the calculation result is returned to the mixed reality glasses.

[0074] In this step, after the server end receives the second point cloud data, the third point cloud data, and the spatial transformation parameter, the spatial transformation between the second point cloud data and the third point cloud data is first calculated according to the point cloud registration algorithm, then the second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system is obtained according to the operation rule of the spatial transformation, and the second spatial transformation matrix is transmitted to the mixed reality glasses through a wireless network.

[0075] S150: transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, and the mixed reality glasses transform the three-dimensional model data in the optical positioning system coordinate system to the mixed reality glasses coordinate system by using the second spatial transformation matrix, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, and perform mixed reality navigation through the virtual model of the surgical instrument;

[0076] In this step, as shown in the following table, it is a space transformation relationship diagram between various entity objects in the mixed reality navigation of the embodiment of the present application, and the entity objects include the mixed reality glasses, the optical positioning system, the depth camera, the server side (personal computer), the surgical object and the surgical instrument. Specifically, after receiving the three-dimensional model data in the optical positioning system coordinate system, the mixed reality glasses transform the three-dimensional model data to the mixed reality glasses coordinate system by using the second spatial transformation matrix, and render the transformed three-dimensional model data in the mixed reality glasses coordinate system after operation of the spatial pose information of the surgical instrument and the second spatial transformation matrix, to realize the virtual-real fusion of the virtual scene and the real scene. Specifically, the three-dimensional model data is transformed to the mixed reality glasses coordinate system for rendering by operation of the following expression: Figure 2

[0077] Model MR = M * T2 * T1 * Model (1)

[0078] wherein Model MR represents the virtual model of the surgical instrument in the mixed reality glasses virtual scene after transformation, M represents the transformation matrix of the mixed reality glasses coordinate system, T2 and T1 represent the second spatial transformation matrix and the first spatial transformation matrix respectively, and Model represents the transformed three-dimensional model data.

[0079] S160: during the mixed reality navigation, transmit the spatial pose information of the surgical instrument to the mixed reality glasses in real time through the optical positioning system, and the mixed reality glasses update the position information of the surgical instrument in the virtual model of the surgical instrument in real time according to the spatial pose information until the mixed reality navigation ends;

[0080] In this step, after the mixed reality glasses receive the spatial pose information of the surgical instrument, the spatial pose information is transformed to the mixed reality glasses coordinate system after operation of the second spatial transformation matrix T2 and the spatial transformation parameters of the mixed reality glasses, and the spatial position of the surgical instrument in the virtual model of the surgical instrument is updated, to realize real-time tracking of the surgical instrument during the mixed reality navigation. Specifically, let the spatial pose information of the surgical instrument in the optical positioning system be P, and the spatial pose information of the surgical instrument is transformed to the mixed reality glasses coordinate system by the following expression:​

[0081] P MR =M*T2*T1*P(2)

[0082] In the above-described embodiments, the spatial transformation matrix between different coordinate systems embedded in the mixed reality glasses is used for the calculation of the mixed reality navigation registration process. Based on the fixed characteristics of the embedded coordinate system, the purpose of dynamically tracking surgical instruments without relying on additional positioning tools is achieved.

[0083] It should be noted that in this embodiment, the purpose of the depth camera is to calculate the first spatial transformation matrix used for registration. After the first spatial transformation matrix is ​​calculated, the depth camera can be removed. There is no need to track the depth camera during mixed reality navigation. In addition, this embodiment does not require fixing the positioning tool on the mixed reality glasses. When tracking surgical instruments, it is not necessary to track both the surgical instruments and the positioning tool on the mixed reality glasses at the same time. This reduces the possibility of failure in real-time updates of the spatial position of the virtual model of the surgical instruments due to light occlusion. At the same time, it is not necessary to determine the spatial transformation between the local coordinate system of the positioning tool and the coordinate system of the mixed reality glasses through a calibration algorithm, which is beneficial to improving the accuracy of mixed reality navigation.

[0084] To verify the feasibility and effectiveness of the embodiments of this application, the following embodiments were verified through a mixed reality navigation phantom experiment. The mixed reality navigation registration phantom experiment is as follows: Figure 3 As shown, in Figure 3 In the diagram, (A) and (B) represent the effect of the virtual surgical instrument model and the real model being blended and rendered in the mixed reality glasses from different angles after mixed reality registration, (C) is the scene of the operator performing surgery while wearing mixed reality glasses, and (D) is the scene diagram of the mixed reality navigation model experiment. Figure 4 This diagram illustrates the visualization results of navigation evaluation errors for different puncture paths on a phantom. Six different puncture paths were designed, and under the guidance of the implemented mixed reality navigation system, after tracking and puncturing the surgical instrument, the evaluation errors for the six different puncture paths were 4.3mm, 4.6mm, 3.5mm, 4.5mm, 3.2mm, and 4.8mm, respectively. Experimental results show that this embodiment, by introducing a depth camera, achieves the registration effect of markerless mixed reality navigation. It only needs to track the spatial pose information of the surgical instrument in the real scene to synchronously update the corresponding position in the virtual scene of the mixed reality glasses in real time. This improves the accuracy of the mixed reality navigation system and greatly promotes the safety and accuracy of surgical operations.

[0085] Based on the above, the mixed reality navigation registration method of the embodiment of the application introduces a depth camera into the registration process of mixed reality navigation, solves the technical problems of registration fusion of different coordinate space information and tracking and positioning of the spatial pose of the mixed reality glasses in the mixed reality scene, and effectively simplifies the registration process of mixed reality navigation. Secondly, the embodiment of the application does not need manual intervention to select the marker point, overcomes the inconvenience brought by manual selection of the marker point, reduces the problems of traceability and poor repeatability caused by interaction. In addition, the embodiment of the application does not need to fix the positioning tool on the mixed reality glasses. When tracking the surgical instrument, the positioning tool on the mixed reality glasses does not need to be tracked at the same time, which reduces the possibility of real-time updating failure of the spatial position of the virtual model due to light blocking, and also does not need to determine the spatial transformation between the local coordinate system of the positioning tool and the glasses coordinate system through the calibration algorithm, simplifies the operation process of mixed reality navigation, reduces the error introduced by the calibration process of the mixed reality glasses, and improves the accuracy of mixed reality navigation.

[0086] Please refer to Figure 5 The mixed reality navigation registration device 40 of the embodiment of the application includes:

[0087] The first point cloud acquisition module 41 is configured to acquire the preoperative DICOM image of the surgical object in the real scene, generate the three-dimensional model data of the surgical target according to the preoperative DICOM image, and acquire the first point cloud data according to the three-dimensional model data.

[0088] The second point cloud acquisition module 42 is configured to acquire the second point cloud data of the surgical object in the real scene by using the depth camera, and record the spatial pose information of the surgical instrument in the real scene by using the optical positioning system. The depth camera is fixed with a positioning tool that can be tracked by the optical positioning system.

[0089] The first registration module 43 is configured to register the first point cloud data and the second point cloud data by using a point cloud registration algorithm, obtain the first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and convert the three-dimensional model data into the three-dimensional model data in the optical positioning system coordinate system by using the first spatial transformation matrix.

[0090] The second registration module 44 is configured to acquire the third point cloud data of the surgical object in the real scene by using the mixed reality glasses, and register the second point cloud data and the third point cloud data by using a point cloud registration algorithm, to obtain the second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system.

[0091] The virtual model construction module 45 is configured to transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, transform the three-dimensional model data in the optical positioning system coordinate system into a mixed reality glasses coordinate system by using a second spatial transformation matrix, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, so as to perform mixed reality navigation through the virtual model of the surgical instrument.

[0092] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and specific functions and brought technical effects can be referred to the method embodiments part, which will not be repeated here.

[0093] The apparatus provided by the embodiments of the present application can be applied in the foregoing method embodiments, and details can be referred to the description of the method embodiments, which will not be repeated here.

[0094] Please refer to Figure 6 , which is a structural schematic diagram of a computer device of the embodiments of the present application. The computer device 50 comprises:

[0095] a memory 51 storing executable program instructions;

[0096] a processor 52 connected with the memory 51;

[0097] The processor 52 is configured to invoke executable program instructions stored in the memory 51 and perform the following steps: acquiring a preoperative DICOM image of a surgical object in a real scene, generating three-dimensional model data of a surgical target according to the preoperative DICOM image, and collecting first point cloud data according to the three-dimensional model data; acquiring second point cloud data of the surgical object in the real scene by using a depth camera, and recording spatial pose information of a surgical instrument in the real scene by using an optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; registering the first point cloud data and the second point cloud data by using a point cloud registration algorithm, obtaining a first spatial transformation matrix between a preoperative DICOM image coordinate system and an optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first spatial transformation matrix; collecting third point cloud data of the surgical object in the real scene by using a mixed reality glasses, and registering the second point cloud data and the third point cloud data by using a point cloud registration algorithm, obtaining a second spatial transformation matrix between the optical positioning system coordinate system and a mixed reality glasses coordinate system; transmitting the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, and the mixed reality glasses converting the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second spatial transformation matrix, and constructing a virtual model of the surgical instrument according to the converted three-dimensional model data and the spatial pose information of the surgical instrument, and performing mixed reality navigation through the virtual model of the surgical instrument.

[0098] The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 can be an integrated circuit chip having a processing capability of signals. The processor 52 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0099] Please refer to Figure 7 , Figure 7A structural schematic diagram of a storage medium of an embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 capable of implementing the following steps: acquiring a preoperative DICOM image of a surgical object in a real scene, generating three-dimensional model data of a surgical target according to the preoperative DICOM image, and collecting first point cloud data according to the three-dimensional model data; acquiring second point cloud data of the surgical object in the real scene by using a depth camera, and recording spatial pose information of a surgical instrument in the real scene by using an optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; registering the first point cloud data and the second point cloud data by using a point cloud registration algorithm to obtain a first spatial transformation matrix between a preoperative DICOM image coordinate system and an optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first spatial transformation matrix; collecting third point cloud data of the surgical object in the real scene by using a mixed reality glasses, and registering the second point cloud data and the third point cloud data by using a point cloud registration algorithm to obtain a second spatial transformation matrix between the optical positioning system coordinate system and a mixed reality glasses coordinate system; transmitting the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, and the mixed reality glasses converts the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second spatial transformation matrix, and constructs a surgical instrument virtual model according to the converted three-dimensional model data and the spatial pose information of the surgical instrument, and performs mixed reality navigation through the surgical instrument virtual model. Wherein the program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network computer device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program instructions, or a computer, a server, a mobile phone, a tablet, and other terminal computer devices. Wherein the server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, and other basic cloud computing services.

[0100] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the system embodiments is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0101] In addition, each function unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist alone physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software function unit. The above is only an implementation manner of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A mixed reality navigation registration method, characterized by, The method comprises the following steps: acquiring preoperative DICOM images of a surgical object in a real scene, generating three-dimensional model data of a surgical target according to the preoperative DICOM images, and collecting first point cloud data according to the three-dimensional model data; acquiring second point cloud data of the surgical object in the real scene by using a depth camera, and recording spatial pose information of a surgical instrument in the real scene by using an optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; performing registration on the first point cloud data and the second point cloud data by using a point cloud registration algorithm to obtain a first spatial transformation matrix between a preoperative DICOM image coordinate system and an optical positioning system coordinate system, and converting the three-dimensional model data into three-dimensional model data in the optical positioning system coordinate system through the first spatial transformation matrix; acquiring third point cloud data of the surgical object in the real scene by using a mixed reality glasses, and performing registration on the second point cloud data and the third point cloud data by using a point cloud registration algorithm to obtain a second spatial transformation matrix between the optical positioning system coordinate system and a mixed reality glasses coordinate system; transmitting the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, and converting the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second spatial transformation matrix, and constructing a virtual model of the surgical instrument according to the converted three-dimensional model data and the spatial pose information of the surgical instrument, and performing mixed reality navigation through the virtual model of the surgical instrument.

2. The mixed reality navigation registration method of claim 1, wherein, The method comprises the following steps: performing CT or MRI image scanning on the surgical object to acquire preoperative DICOM images of the surgical object; segmenting the preoperative DICOM images by using a threshold segmentation algorithm to acquire an image of a region of interest of the surgical target, and reconstructing the image of the region of interest by using a marching cubes method to generate three-dimensional model data of the surgical target, and storing the three-dimensional model data in a file format that can be imported into an embedded operating system of the mixed reality glasses; sampling point cloud data from the three-dimensional model data to acquire first point cloud data of the surgical object.

3. The mixed reality navigation registration method of claim 2, wherein, Before the step of acquiring second point cloud data of the surgical object in the real scene by using a depth camera, the method further comprises the following steps: fixing a positioning tool that can be tracked by an optical positioning system on the depth camera, and fixing the optical positioning system at a position to ensure that the positioning tool is within the visual range of the optical positioning system, and keeping the positions of the optical positioning system and the depth camera unchanged; acquiring spatial pose information of the positioning tool by using the optical positioning system, collecting depth information of a calibration reference object in the depth camera coordinate system by using the depth camera, and calculating a calibration transformation matrix between a positioning tool coordinate system and a depth camera coordinate system by using a least squares method.

4. The mixed reality navigation registration method of claim 3, wherein, The method further comprises the following steps after acquiring the second point cloud data of the surgical object in the real scene by using the depth camera: The RANSAC algorithm is used to eliminate the point cloud noise in the second point cloud data, and the second point cloud data is transformed into the optical positioning system coordinate system by using the calibration transformation matrix, and the transformed second point cloud data is sent to the server end.

5. The mixed reality navigation registration method of claim 4, wherein, The third point cloud data of the surgical object in the real scene is collected by using the mixed reality glasses, and the second point cloud data and the third point cloud data are registered by using a point cloud registration algorithm to obtain a second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system, specifically comprising the following steps: The third point cloud data is transformed into the mixed reality glasses coordinate system to obtain the spatial transformation parameters of the mixed reality glasses, and the third point cloud data and the spatial transformation parameters are sent to the server end, and the server end calculates the spatial transformation between the transformed second point cloud data and the third point cloud data according to the point cloud registration algorithm, obtains the second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system according to the operation algorithm of the spatial transformation, and returns the second spatial transformation matrix to the mixed reality glasses.

6. The mixed reality navigation registration method of claim 5, wherein, The mixed reality glasses transform the three-dimensional model data under the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second spatial transformation matrix, and construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, specifically comprising the following steps: denotes the virtual model of the surgical instrument after the transformation, M denotes a transformation matrix of the mixed reality glasses coordinate system, and denote a second spatial transformation matrix and a first spatial transformation matrix, respectively, denotes the transformed three-dimensional model data.

7. The mixed reality navigation registration method of claim 6, wherein, After constructing the virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, the method further comprises the following steps: The spatial pose information of the surgical instrument is transmitted to the mixed reality glasses in real time by the optical positioning system, and the mixed reality glasses update the position information of the surgical instrument in the virtual model of the surgical instrument in real time according to the spatial pose information: Wherein, P represents the spatial pose information of the surgical instrument.

8. A mixed reality navigation registration apparatus characterized by comprising: Comprise: The first point cloud acquisition module is used to acquire the preoperative DICOM image of the surgical object in the real scene, generate the three-dimensional model data of the surgical target according to the preoperative DICOM image, and collect the first point cloud data according to the three-dimensional model data; The second point cloud acquisition module is used to acquire the second point cloud data of the surgical object in the real scene by using the depth camera, and record the spatial pose information of the surgical instrument in the real scene by using the optical positioning system; wherein a positioning tool that can be tracked by the optical positioning system is fixed on the depth camera; The first registration module is used to register the first point cloud data and the second point cloud data by using a point cloud registration algorithm to obtain a first spatial transformation matrix between the preoperative DICOM image coordinate system and the optical positioning system coordinate system, and convert the three-dimensional model data into three-dimensional model data under the optical positioning system coordinate system by using the first spatial transformation matrix. The second registration module is configured to acquire third point cloud data of the surgical object in the real scene by using the mixed reality glasses, and to register the second point cloud data and the third point cloud data by using a point cloud registration algorithm, so as to obtain a second spatial transformation matrix between the optical positioning system coordinate system and the mixed reality glasses coordinate system. The virtual model construction module is configured to transmit the three-dimensional model data in the optical positioning system coordinate system and the spatial pose information of the surgical instrument to the mixed reality glasses, to transform the three-dimensional model data in the optical positioning system coordinate system into the mixed reality glasses coordinate system by using the second spatial transformation matrix, and to construct a virtual model of the surgical instrument according to the transformed three-dimensional model data and the spatial pose information of the surgical instrument, so as to perform mixed reality navigation by using the virtual model of the surgical instrument.

9. A computer device, comprising: The computer device comprises a processor and a memory coupled to the processor, wherein The memory stores program instructions for implementing the mixed reality navigation registration method according to any one of claims 1-7. The processor is configured to execute the program instructions stored in the memory to control the mixed reality navigation registration method.

10. A storage medium, characterized by The memory stores processor-executable program instructions for executing the mixed reality navigation registration method according to any one of claims 1-7.

Images