Auxiliary positioning method of minimally invasive spine surgery robot

Abstract

The present invention provides an auxiliary positioning method for a minimally invasive spine surgery robot, comprising the steps of: generating semantic two-dimensional scene information according to two-dimensional scene image information, and inputting the semantic two-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene; generating semantic three-dimensional scene information according to three-dimensional scene point cloud information, and inputting the semantic three-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene; combining magnetic positioning information and two-dimensional ultrasonic images during surgery to generate three-dimensional ultrasonic scene information, and inputting the three-dimensional ultrasonic scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene. The auxiliary positioning method for a minimally invasive spine surgery robot provided by the present invention realizes the real-time surgical area perception of the minimally invasive spine surgery robot.

Description

Technical Field

[0001] The invention relates to the field of medical technology, and in particular to an auxiliary positioning method of a spinal minimally invasive surgical robot. Background Art

[0002] In recent years, spinal surgical robots have been gradually applied in clinical practice, with advantages such as accurate positioning, high precision and strong stability. The development of minimally invasive spinal surgical robot technology is more in line with the current trend of precision and minimally invasive spinal surgery. However, the current surgical robots have weak positioning and directional perception capabilities, cannot quickly and effectively understand surgical scene information and automatically position themselves, and it is difficult to achieve rapid tracking and positioning of the surgical vertebrae.

[0003] To make up for the above shortcomings, the minimally invasive spinal surgery robot requires a precise auxiliary positioning system, including operating room environment positioning and surgical area positioning. The operating room environment positioning is to understand the surgical scene, including many non-structural scene factors, such as the patient's body shape and posture, the layout of the operating room objects, and the light environment, so as to clarify the operating room layout and the patient's position; and the surgical area positioning mainly realizes the calibration of the target vertebrae for surgery. Unlike structures such as the skull and brain, the spinal vertebrae are located deep in the human body, and the relative positions of the vertebrae will change with the changes in the body position. Therefore, the operation of the minimally invasive spinal surgery robot cannot simply rely on preoperative imaging examinations and calibration. It is necessary to build a real-time and non-invasive auxiliary positioning method to achieve real-time surgical area perception. Summary of the invention

[0004] The purpose of the present invention is to provide an auxiliary positioning method for a minimally invasive spinal surgery robot to achieve real-time surgical area perception of the minimally invasive spinal surgery robot.

[0005] The present invention provides an auxiliary positioning method for a minimally invasive spine surgery robot, comprising:

[0006] Generate semantic two-dimensional scene information according to the two-dimensional scene image information, and input the semantic two-dimensional scene information into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene;

[0007] Generate semantic three-dimensional scene information based on the three-dimensional scene point cloud information, and input the semantic three-dimensional scene information into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene; and

[0008] During surgery, magnetic positioning information and two-dimensional ultrasound images are combined to generate three-dimensional ultrasound scene information, which is then input into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene.

[0009] The auxiliary positioning method of the minimally invasive spinal surgery robot provided by the present invention generates semantic two-dimensional scene information and semantic three-dimensional scene information according to the two-dimensional scene image information and three-dimensional scene point cloud information during the operation, and combines the magnetic positioning information and the two-dimensional ultrasonic image to generate the three-dimensional ultrasonic scene information during the operation, and inputs the semantic two-dimensional scene information, the semantic three-dimensional scene information and the three-dimensional ultrasonic scene information into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene, thereby realizing the real-time surgical area perception of the minimally invasive spinal surgery robot.

[0010] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, the step of generating semantic two-dimensional scene information according to the two-dimensional scene image information, and inputting the semantic two-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene includes:

[0011] Use existing clinical scenario datasets to train the Mask R-CNN network model;

[0012] Obtain two-dimensional scene image information;

[0013] The Mask R-CNN network model is used to identify and segment the target objects in the two-dimensional scene image information to obtain primary two-dimensional information;

[0014] Using a pre-trained conditional random field model and superpixel segmentation algorithm, the target object in the primary two-dimensional information is finely identified and segmented to obtain fine two-dimensional information; and

[0015] The refined two-dimensional information is combined into semantic two-dimensional scene information and input into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene.

[0016] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, the step of generating semantic two-dimensional scene information according to the two-dimensional scene image information, and inputting the semantic two-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene, further includes:

[0017] Using a pre-trained attitude estimation model to identify the primary two-dimensional information to obtain attitude estimation information; and

[0018] The posture estimation information is combined into semantic two-dimensional scene information and input into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene.

[0019] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, the step: using an existing clinical scenario dataset to train a Mask R-CNN network model includes:

[0020] Use the transfer learning strategy to pre-train on the COCO dataset and migrate to the Mask R-CNN network model; and

[0021] Fine-tune the Mask R-CNN network model and adjust some top-level network parameters.

[0022] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, semantic three-dimensional scene information is generated according to the three-dimensional scene point cloud information, and the semantic three-dimensional scene information is input into the minimally invasive spine surgery robot to enhance the perception of the surgical scene, including:

[0023] Obtain three-dimensional scene point cloud information;

[0024] Using the multi-view projection method, the projection segmentation point cloud information is obtained according to the three-dimensional scene point cloud information and the refined two-dimensional information;

[0025] The three-dimensional scene point cloud information is segmented using a supervoxel-based region growing algorithm to obtain supervoxel point cloud segmentation information;

[0026] Fusing the projected segmentation point cloud information and the supervoxel point cloud segmentation information, and composing the fusion results into semantic 3D scene information; and

[0027] Input semantic three-dimensional scene information into minimally invasive spinal surgery robots to enhance the perception of surgical scenes.

[0028] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, the step: using a multi-view projection method to obtain projection segmentation point cloud information according to the three-dimensional scene point cloud information and the fine two-dimensional information, includes:

[0029] Project each point in the three-dimensional scene point cloud information into fine two-dimensional information;

[0030] Voting on the object attribution of each point according to each detailed two-dimensional information; and

[0031] The voting value of the object attribution of each point is counted, and the final classification of the point is determined according to the voting value of the object attribution of each point.

[0032] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, the step of counting the voting value of the object attribution of each point and determining the final classification of the point according to the voting value of the object attribution of each point includes:

[0033] Count the voting values ​​of the objects at each point;

[0034] Multiply the voting value of the object attribution of each point by the preset weighting coefficient to obtain the weighted voting value of the object attribution of each point;

[0035] Selecting the maximum weighted voting value of the object attribution of each point, and determining whether the maximum weighted voting value is greater than a preset threshold; and

[0036] If the answer is yes, the final classification of the point is determined based on the object with the maximum weighted voting value.

[0037] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, the step of fusing the projected segmentation point cloud information and the supervoxel point cloud segmentation information to form semantic three-dimensional scene information includes:

[0038] Calculating an object classification overlap ratio of the projected segmentation point cloud information and the supervoxel point cloud segmentation information; and

[0039] The super-voxel point cloud segmentation information whose classification overlap ratio exceeds a preset threshold is retained as the semantic 3D scene information.

[0040] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spine surgery robot, the steps of: combining magnetic positioning information and two-dimensional ultrasound images to generate three-dimensional ultrasound scene information during surgery, and inputting the three-dimensional ultrasound scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene include:

[0041] Perform spatial calibration on ultrasonic devices and magnetic positioning devices;

[0042] During the operation, two-dimensional ultrasound images are acquired through ultrasound devices and reconstructed into three-dimensional ultrasound images;

[0043] Using a magnetic positioning device in combination with three-dimensional ultrasound images and preoperative images to locate the object in space; and

[0044] The three-dimensional ultrasound image after spatial positioning of the object is input into the minimally invasive spinal surgery robot as three-dimensional ultrasound scene information to enhance the perception of the surgical scene.

[0045] In another exemplary embodiment of the auxiliary positioning method of the minimally invasive spinal surgery robot, in the step of acquiring a two-dimensional ultrasonic image by an ultrasonic device during surgery and reconstructing a three-dimensional ultrasonic image, a pixel nearest neighbor method is used to reconstruct the three-dimensional ultrasonic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The following drawings are only used to schematically illustrate and explain the present invention, and do not limit the scope of the present invention.

[0047] Figure 1 The present invention is a flow chart of an illustrative implementation of a method for assisting positioning of a minimally invasive spine surgery robot.

[0048] Figure 2 Used to illustrate the information flow in the assisted positioning method of the spinal minimally invasive surgical robot.

[0049] Figure 3 The present invention is a partial flow chart of the auxiliary positioning method of the minimally invasive spine surgery robot.

[0050] Figure 4 The present invention is a partial flow chart of the auxiliary positioning method of the minimally invasive spine surgery robot.

[0051] Figure 5 The present invention is a partial flow chart of the auxiliary positioning method of the minimally invasive spine surgery robot.

[0052] Figure 6 The present invention is a partial flow chart of the auxiliary positioning method of the minimally invasive spine surgery robot.

[0053] Figure 7 The present invention is a partial flow chart of the auxiliary positioning method of the minimally invasive spine surgery robot.

[0054] Figure 8 The present invention is a partial flow chart of the auxiliary positioning method of the minimally invasive spine surgery robot.

[0055] Fig. 9 The present invention is a partial flow chart of the auxiliary positioning method of the minimally invasive spine surgery robot. DETAILED DESCRIPTION

[0056] In order to have a clearer understanding of the technical features, purposes and effects of the invention, the specific embodiments of the present invention are now described with reference to the accompanying drawings. The same reference numerals in the drawings represent components with the same structure or similar structures but the same functions.

[0057] In this document, “exemplary” means “serving as an example, instance or illustration”, and any diagram or implementation described in this document as “exemplary” should not be interpreted as a more preferred or more advantageous technical solution.

[0058] Figure 1 The flowchart is a schematic implementation of the auxiliary positioning method of the minimally invasive spine surgery robot. Figure 1 The auxiliary positioning method of the minimally invasive spine surgery robot includes the following steps S100 to S300.

[0059] Step S100: Generate semantic two-dimensional scene information based on the two-dimensional scene image information, and input the semantic two-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene. Figure 2 and Figure 3 , step S100 specifically includes the following steps S110 to S117.

[0060] Step S110: Use the existing clinical scenario dataset to train the Mask R-CNN network model. The trained Mask R-CNN network model is used to identify the region of interest (ROI), category, and segmentation probability map of the target object. Figure 4 , Considering that the Mask R-CNN network model is deep and the training data is limited, in order to obtain better training results and prevent network overfitting, step S110 specifically includes the following steps S111 to S112.

[0061] Step S111: Use the transfer learning strategy to pre-train on the COCO dataset and migrate to the Mask R-CNN network model.

[0062] Step S112: Fine-tune the Mask R-CNN network model and adjust some top-level network parameters. During the fine-tuning process, only the top-level network parameters such as recommendation box generation, classification, and mask segmentation are adjusted, and other parameters are fixed.

[0063] Step S120: Acquire two-dimensional scene image information. The two-dimensional scene image information is acquired during surgery by a medical imaging instrument.

[0064] Step S130: using the Mask R-CNN network model to identify and segment the target object in the two-dimensional scene image information to obtain primary two-dimensional information. The primary two-dimensional information includes a basic segmentation probability map given by the Mask R-CNN network model.

[0065] Step S140: Use the pre-trained conditional random field model and superpixel segmentation algorithm to finely identify and segment the target object in the primary two-dimensional information to obtain fine two-dimensional information. Use the conditional random field (CRF) model to finely segment the primary two-dimensional information, and then merge the post-processing results with the superpixel segmentation results to obtain better segmented edges and filter out isolated areas that may be generated by the conditional random field model.

[0066] Step S150: The refined two-dimensional information is combined into semantic two-dimensional scene information, and input into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene.

[0067] Step S160: using the pre-trained posture estimation model to identify the primary two-dimensional information to obtain posture estimation information. The posture estimation information is used to understand the patient's body posture during surgery.

[0068] Step S170: The posture estimation information is combined into semantic two-dimensional scene information, and input into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene.

[0069] Steps S160 and S170 are used to integrate the patient's posture information during surgery into the semantic two-dimensional scene information, further enhancing the minimally invasive spine surgery robot's perception of the surgical area. However, this is not limited to this, and in other exemplary embodiments, step S100 may also not include the above steps S160 and S170.

[0070] Step S200: Generate semantic three-dimensional scene information based on the three-dimensional scene point cloud information, and input the semantic three-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene. Figure 2 and Figure 5 , step S200 specifically includes the following steps S210 to S250.

[0071] Step S210: Acquire three-dimensional scene point cloud information. Two-dimensional scene image information is acquired during surgery through medical imaging equipment.

[0072] Step S220: Use the multi-view projection method to obtain the projection segmentation point cloud information based on the 3D scene point cloud information and the refined 2D information. The segmentation of the target in the 2D image may cause under-segmentation and over-segmentation of the segmentation boundary area, resulting in contradictory classification results of the point cloud at the boundary of the 3D object segmented from different perspectives. For this purpose, this application introduces a voting mechanism, which is specifically referred to in Figure 6 , step S220 includes the following steps S221 to S223.

[0073] Step S221: Project each point in the three-dimensional scene point cloud information into fine two-dimensional information.

[0074] Step S222: Vote for the object attribution of each point based on each piece of detailed two-dimensional information.

[0075] Step S223: Count the voting values ​​of the object attribution of each point, and determine the final classification of the point according to the voting values ​​of the object attribution of each point. Figure 7 , step S223 includes the following steps S2231 to S2234.

[0076] Step S2231: Count the voting value of the object attribution at each point.

[0077] Step S2232: multiply the voting value of the object attribution of each point by a preset weighting coefficient to obtain the weighted voting value of the object attribution of each point.

[0078] Step S2233: Select the maximum weighted voting value of the object attribution of each point, and determine whether the maximum weighted voting value is greater than a preset threshold.

[0079] Step S2234: If the answer is yes, the final classification of the point is determined based on the object with the maximum weighted voting value.

[0080] Since the Mask R-CNN network model has large differences in segmentation and detection performance for target objects of different complexity. The frequency of occurrence of each target object in two-dimensional images with different viewing angles and field of view will also be different. This will lead to unbalanced segmentation performance of different objects in the three-dimensional scene point cloud voting segmentation. In steps S2231 to S2234, a personalized weight mechanism is used to set different weighted voting values ​​according to the network's recognition and segmentation accuracy for different objects, and set personalized weights for different target objects, thereby obtaining a more balanced three-dimensional point cloud segmentation result.

[0081] Step S230: segment the three-dimensional scene point cloud information using a supervoxel-based region growing algorithm to obtain supervoxel point cloud segmentation information. Specifically, a color-based three-dimensional point cloud region growing algorithm is used to segment the scene point cloud into supervoxels according to color and spatial position.

[0082] Step S240: Fusing the projected segmentation point cloud information and the supervoxel point cloud segmentation information, and combining the fusion results into semantic three-dimensional scene information. Figure 2 and Figure 8 , step S240 includes the following steps S241 to S242.

[0083] Step S241: Calculate the object classification overlap ratio of the projected segmentation point cloud information and the supervoxel point cloud segmentation information.

[0084] Step S242: retaining the super-voxel point cloud segmentation information whose classification overlap ratio exceeds a preset threshold as semantic three-dimensional scene information.

[0085] Step S250: inputting semantic three-dimensional scene information into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene.

[0086] Step S300: During the operation, the magnetic positioning information and the two-dimensional ultrasound image are combined to generate three-dimensional ultrasound scene information, and the three-dimensional ultrasound scene information is input into the minimally invasive spine surgery robot to enhance the perception of the surgical scene. Figure 2 and Fig. 9 , step S300 includes the following steps S310 to S340.

[0087] Step S310: perform spatial calibration on the ultrasonic device and the magnetic positioning device.

[0088] Step S320: During the operation, a two-dimensional ultrasound image is obtained through an ultrasound device, and reconstructed into a three-dimensional ultrasound image. In this embodiment, the pixel nearest neighbor method (PNN, Pseudo-nearest neighbors) is used to reconstruct a three-dimensional ultrasound image, and the mean value of pixels mapped to the same voxel position is used as the assignment strategy to reconstruct the three-dimensional ultrasound image, and the Taichi parallel computing framework is used to further accelerate the reconstruction process, so that the reconstruction time reaches the second level, and each pixel point in the two-dimensional ultrasound image can be quickly mapped to the three-dimensional space, which is more in line with the reconstruction requirements during the operation. Due to the large degree of freedom of Freehand ultrasound data acquisition, after all pixel points are mapped to the three-dimensional volume data, there will still be some blank voxel points in the three-dimensional volume data that fail to obtain ultrasound data. Therefore, this embodiment uses a three-dimensional Gaussian filter for interpolation to fill in the blank voxel points to avoid the appearance of more abnormal value points in the subsequent registration link.

[0089] In other implementations, other pixel-based reconstruction methods (PBM), voxel-based three-dimensional reconstruction methods (VBM), and function-based three-dimensional reconstruction methods (FBM) may be used to reconstruct a three-dimensional ultrasound image.

[0090] Step S330: Use the magnetic positioning device to combine the three-dimensional ultrasound image and the preoperative image to perform spatial positioning of the object. Specifically, the three-dimensional ultrasound image is combined with the three-dimensional information of each vertebra of the spine segmented and extracted from the preoperative image to perform rigid registration based on Mattes mutual information to obtain three-dimensional ultrasound scene information containing the patient's spatial registration information.

[0091] Step S340: input the three-dimensional ultrasound image after the object is spatially positioned as three-dimensional ultrasound scene information into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene.

[0092] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0093] The series of detailed descriptions listed above are only specific descriptions of feasible embodiments of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation scheme or changes that do not deviate from the technical spirit of the present invention, such as combination, division or repetition of features, should be included in the scope of protection of the present invention.

Claims

1. An auxiliary positioning method for a minimally invasive spine surgery robot, characterized in that: include: Generating semantic two-dimensional scene information according to the two-dimensional scene image information, and inputting the semantic two-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene, specifically including: Use the existing clinical scenario dataset to train the Mask R-CNN network model. Acquiring the two-dimensional scene image information, The Mask R-CNN network model is used to identify and segment the target object in the two-dimensional scene image information to obtain primary two-dimensional information. Using the pre-trained conditional random field model and superpixel segmentation algorithm, the target object in the primary two-dimensional information is finely identified and segmented to obtain fine two-dimensional information. The fine two-dimensional information is combined into the semantic two-dimensional scene information, and input into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene. Using a pre-trained posture estimation model to identify the primary two-dimensional information to obtain posture estimation information, and The posture estimation information is combined into the semantic two-dimensional scene information, and input into the minimally invasive spinal surgery robot to enhance the perception of the surgical scene; Generating semantic three-dimensional scene information according to the three-dimensional scene point cloud information, and inputting the semantic three-dimensional scene information into the minimally invasive spine surgery robot to enhance the perception of the surgical scene, specifically including: Get 3D scene point cloud information, The projected segmentation point cloud information is obtained according to the three-dimensional scene point cloud information and the fine two-dimensional information by using a multi-view projection method, specifically including: Projecting each point in the three-dimensional scene point cloud information onto the refined two-dimensional information, Voting on the object attribution of each point according to each of the detailed two-dimensional information, and The voting value of the object attribution of each point is counted, and the final classification of the point is determined according to the voting value of the object attribution of each point, which specifically includes: Count the voting values ​​of the objects belonging to each point, Multiply the voting value of the object attribution of each point by the preset weighting coefficient to obtain the weighted voting value of the object attribution of each point. Select the maximum weighted voting value of the object attribution of each point, determine whether the maximum weighted voting value is greater than a preset threshold, and If the answer is yes, the final classification of the point is determined based on the object with the maximum weighted voting value. The three-dimensional scene point cloud information is segmented using a supervoxel-based region growing algorithm to obtain supervoxel point cloud segmentation information. fusing the projected segmentation point cloud information and the supervoxel point cloud segmentation information, and forming semantic three-dimensional scene information from the fusion result, Inputting the semantic three-dimensional scene information into a minimally invasive spinal surgery robot to enhance the perception of the surgical scene; and During surgery, magnetic positioning information and two-dimensional ultrasound images are combined to generate three-dimensional ultrasound scene information, which is then input into a minimally invasive spinal surgery robot to enhance perception of the surgical scene.

2. The auxiliary positioning method of the minimally invasive spine surgery robot according to claim 1, characterized in that: Steps: Use the existing clinical scenario dataset to train the Mask R-CNN network model, including: Use the transfer learning strategy to pre-train on the COCO dataset and migrate to the Mask R-CNN network model; and The Mask R-CNN network model is fine-tuned, and some top-level network parameters are adjusted.

3. The auxiliary positioning method of the minimally invasive spine surgery robot according to claim 1, characterized in that: Step: fusing the projected segmentation point cloud information and the supervoxel point cloud segmentation information to form semantic three-dimensional scene information, including: Calculating an object classification overlap ratio of the projected segmentation point cloud information and the supervoxel point cloud segmentation information; and The supervoxel point cloud segmentation information whose classification overlap ratio exceeds a preset threshold is retained as semantic three-dimensional scene information.

4. The auxiliary positioning method of the minimally invasive spine surgery robot according to claim 1, characterized in that: Step: During surgery, the magnetic positioning information and the two-dimensional ultrasound image are combined to generate three-dimensional ultrasound scene information, and the three-dimensional ultrasound scene information is input into the minimally invasive spine surgery robot to enhance the perception of the surgical scene, including: Perform spatial calibration on ultrasonic devices and magnetic positioning devices; During the operation, two-dimensional ultrasound images are acquired through ultrasound devices and reconstructed into three-dimensional ultrasound images; Using the magnetic positioning device in combination with the three-dimensional ultrasound image and the preoperative image to perform spatial positioning of the object; and The three-dimensional ultrasonic image after the object is spatially positioned is input into the minimally invasive spinal surgery robot as the three-dimensional ultrasonic scene information to enhance the perception of the surgical scene.

5. The auxiliary positioning method of the minimally invasive spine surgery robot according to claim 4, characterized in that: In the step of obtaining a two-dimensional ultrasonic image by an ultrasonic device during surgery and reconstructing the three-dimensional ultrasonic image, a pixel nearest neighbor method is used to reconstruct the three-dimensional ultrasonic image.

Images