Pedicle screw path autonomous planning method and system based on neural network

Abstract

The invention discloses a vertebral pedicle screw path autonomous planning method and system based on a neural network, and belongs to the field of biomedicine volume data set segmentation and clinician interaction.The method comprises the steps that a spine ROI is extracted from a lumbar vertebra CT plain scanning and bone three-dimensional reconstruction image data bone window thin layer image sequence, and a data set is generated; a Bi-Conv-LSTM structure is introduced into a 3D-UNet model, an SIS-3D-UNet model is constructed, a data set is used for training, and a first segmentation planning model is generated and used for spine segmentation and pedicle screw planning. According to the method, less lumbar CT image data is utilized to enable the deep learning model to achieve the spine segmentation and pedicle screw planning effects satisfied by spine surgeons.

Description

Technical Field

[0001] The present invention relates to the field of biomedical volume data set segmentation and clinical physician interaction, and in particular to a method and system for autonomous pedicle screw tract planning based on a neural network. Background Art

[0002] How to improve the safety, accuracy and speed of pedicle screw placement has long been a clinical pain point and research hotspot in spinal surgery. Accurate and rapid preoperative planning of pedicle screws can not only help young physicians successfully complete surgical screw placement, but also has important significance for promoting the practical clinical application of various auxiliary screw placement technologies. In the past decade, artificial intelligence and robotics technologies have evolved rapidly, enabling and driving spinal surgery to develop in the direction of precise minimally invasive surgery and digital surgery. The ability of robots to perform a variety of precision operations has attracted the attention of scholars, and robot-assisted pedicle screw placement has naturally become a new exploration direction for spinal surgeons.

[0003] In clinical practice, it is found that preoperative planning has a considerable degree of restriction on robot-assisted pedicle screw placement. As a key core step in the entire robotic surgery process, pedicle screw planning requires spinal surgeons to determine the entry point, midpoint and end point of the pedicle screw insertion path in the preoperative CT image based on previous surgical experience and subjective judgment, plan key morphological parameters such as pedicle screw length and diameter, and reconfirm them after image registration to ensure the safety and accuracy of pedicle screw placement. This step usually requires the surgeon to measure and draw on the preoperative CT image by hand. The process is cumbersome and time-consuming, and mostly depends on the surgeon's previous surgical experience and subjective judgment. While greatly consuming the surgeon's energy, the planning quality and efficiency cannot be effectively guaranteed, which largely restricts the surgical robot's ability to perform its precise surgical operations. In summary, robot-assisted pedicle screw placement requires the surgeon to perform two manual positioning procedures: preoperative pedicle screw planning and intraoperative image registration confirmation. The degree of automation still has room for further improvement. Robots that can perform full-process intelligent pedicle screw placement surgery are still relatively blank in the research field and application market.

[0004] Early research on pedicle screw-assisted planning was mainly based on the optimization solution of spinal anatomical morphology or the use of template matching methods, first focusing on the detection of the spine and the segmentation and modeling of the spinal anatomical structure. However, it has disadvantages such as poor accuracy and too many application conditions, and cannot fully meet the actual clinical application needs of spinal surgeons.

[0005] With the development of computer science (CS), the new direction of "medical-engineering integration" of artificial intelligence combined with spinal surgery has strongly entered the field of vision of researchers. Under this trend, the latest research on pedicle screw-assisted planning mostly uses machine learning (ML) technology, relying on the powerful computing power of computers to improve the safety, accuracy and timeliness of pedicle screw planning. However, these studies are either based on training based on preoperative unlabeled CT image data, which cannot be easily integrated into real-time image-guided spinal surgical robots and navigation surgery planning systems; or based on intraoperative CBCT image data after pedicle screw placement. Although it is more in line with the application mode of spinal surgical robots and navigation surgery planning systems, it will lose some data compared to the more sophisticated preoperative CT images.

[0006] At present, deep learning (DL) technology represented by neural networks is considered to have research prospects worth exploring in medical image processing. Currently, many related projects have been launched and have achieved certain research results. In the field of spinal surgery, scholars have also carried out research on the use of deep learning technology to achieve full spine detection, automatic segmentation of single spinal structures, bone density calculation and other specialized tasks, which provides a certain research foundation for subsequent research on pedicle screw intelligent planning.

[0007] Currently, young spinal surgeons lack an efficient and intelligent pedicle screw-assisted planning method to help them successfully complete pedicle screw placement surgery. Various pedicle screw placement-assisted technologies involving medical imaging also require accurate and rapid pedicle screw preoperative planning to optimize their clinical application models. At the same time, there has been no report on a pedicle screw-assisted planning system that uses deep learning technology and can be further equipped on a real-time image-guided spinal surgical robot and navigation surgery planning system. Summary of the invention

[0008] In order to solve the above problems, the purpose of the present invention is to use deep learning technology to design a neural network-based pedicle screw assisted planning technology, aiming to solve the problem of full-process intelligent planning of pedicle screws in posterior spinal internal fixation, and to solve the problems of young spinal surgeons' lack of surgical experience and subjective judgment ability and poor consistency in clinical practice, as well as the inability to effectively guarantee the planning quality and planning efficiency of preoperative pedicle screw planning, resulting in difficulties in the actual clinical application of pedicle screw assisted placement technology.

[0009] In order to achieve the above technical objectives, the present application provides a method for autonomous planning of pedicle screw trajectories based on a neural network, comprising the following steps:

[0010] Extract the spine ROI from the bone window thin-layer image sequence of lumbar CT plain scan + bone 3D reconstruction image data to generate a data set;

[0011] By introducing the Bi-Conv-LSTM structure into the 3D-UNet model, the SIS-3D-UNet model was constructed, and the dataset was used for training to generate the first segmentation planning model for spinal segmentation and pedicle screw planning.

[0012] Preferably, in the process of generating the data set, ROI interception is performed based on the selected image through spatial coordinate calculation and projection coordinate calculation of each viewing angle, wherein each viewing angle includes a cross-section, a sagittal plane and a coronal plane.

[0013] Preferably, in the process of constructing the SIS-3D-UNet model, the 3D-UNet model is used as the function of the encoder-decoder structure to learn the anatomical structure characteristics of the lumbar spine and the pedicle screw planning rules of the sample data, wherein the encoder gradually downsamples the input image into smaller feature maps, and the decoder upsamples these feature maps to the original input image size while retaining the spatial information.

[0014] Preferably, in the process of constructing the SIS-3D-UNet model, the temporal information of the input data is processed through the Bi-Conv-LSTM structure, wherein the SIS-3D-UNet model is used to process a serialized image set and maintain the order between images, while using convolution operations and long short-term memory network units to capture long-term dependencies in the image sequence, and maintaining a cell state and a hidden state inside it.

[0015] Preferably, in the process of spinal segmentation and pedicle screw planning, based on the SIS-3D-UNet model, the SK-Net structure is introduced and an attention mechanism module is added on its basis to construct the Att-SIS-3D-UNet model, which is trained through the data set to generate a second segmentation planning model for spinal segmentation and pedicle screw planning.

[0016] Preferably, during the pedicle screw planning process, the "gold standard" sample data set is used to perform deep learning training on the SIS-3D-UNet model or the Att-SIS-3D-UNet model so that it can learn the corresponding pedicle screw planning rules.

[0017] Preferably, in the process of generating the second segmentation planning model, based on the Att-SIS-3D-UNet model, after adding the adversarial network, the second segmentation planning model is generated through training through the data set, wherein the adversarial network consists of a segmentation network and a discriminator network, the segmentation network adopts the up-sampled Residual Blocks of the global aggregation module, and the discriminator network is a binary classification network, which receives the segmentation planning results of the Att-SIS-3D-UNet model and the Ground Truth Data as input, and discriminates their sources.

[0018] Preferably, during the process of spinal segmentation and pedicle screw planning, adversarial training is used to mix the spinal segmentation results and pedicle screw planning results output by the Att-SIS-3D-UNet model and the labeled image data as the "gold standard" and re-input the model, and the model is allowed to judge the input results through the discriminator network, and the judgment results of the Att-SIS-3D-UNet model are corrected through the back propagation algorithm.

[0019] The invention discloses a pedicle screw path autonomous planning system based on a neural network, comprising:

[0020] A data processing module is used to extract the spine ROI from the bone window thin-layer image sequence of lumbar CT plain scan + bone 3D reconstruction image data to generate a data set;

[0021] The planning module is used to build the SIS-3D-UNet model by introducing the Bi-Conv-LSTM structure into the 3D-UNet model, and use the data set for training to generate the first segmentation planning model for spinal segmentation and pedicle screw planning.

[0022] Preferably, the planning module is also used to introduce the SK-Net structure based on the SIS-3D-UNet model and add an attention mechanism module on its basis to construct an Att-SIS-3D-UNet model. The spinal segmentation results and pedicle screw planning results output by the Att-SIS-3D-UNet model and the labeled image data as the "gold standard" are mixed and re-input into the model for training through adversarial training, and the model is used to judge the input results through the discriminator network, and the judgment results of the Att-SIS-3D-UNet model are corrected through the back propagation algorithm.

[0023] The present invention discloses the following technical effects:

[0024] The present invention uses adversarial training to mix the spine segmentation results and pedicle screw planning results output by Att-SIS-3D-UNet and the image data that are labeled as the "gold standard" and re-input them into the model. The discriminator network allows the model to determine whether the input is the Ground Truth Data labeled by the physician or the result generated by Att-SIS-3D-UNet, and the back propagation algorithm is used to continuously correct the judgment results of the Att-SIS-3D-UNet network. The images with obvious anatomical structure characteristics of the lumbar spine and representative pedicle screw planning in the sample database are used to assist in detecting images with unclear characteristics and insufficient representativeness, which effectively improves the accuracy of spine segmentation and pedicle screw planning, and then uses less lumbar CT image data to allow the deep learning model to achieve spine segmentation and pedicle screw planning effects that are satisfactory to spinal surgeons. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0027] Figure 2 is a schematic diagram of lumbar spine segmentation based on Att-SIS-3D-UNet according to the present invention;

[0028] Figure 3 It is a schematic diagram of lumbar pedicle screw planning based on Att-SIS-3D-UNet described in the present invention;

[0029] Figure 4 It is a schematic diagram of a three-dimensional reconstruction of the spine and pedicle screws according to the present invention;

[0030] Figure 5 It is the fitting and calculation of the pedicle screw parameters described in the present invention, wherein A is the three-dimensional reconstruction of the spine segmentation result and the pedicle screw planning result; B is the fitting of the central axis of the pedicle screw; C is the calculation of the pedicle screw parameters; and D is the unit conversion of the pedicle screw parameters. DETAILED DESCRIPTION

[0031] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0032] like Figure 1-5 As shown, the present invention aims to solve the problem of full-process intelligent planning of pedicle screws in posterior spinal fixation. In view of the problems in clinical practice, young spinal surgeons lack surgical experience, subjective judgment ability and poor consistency, and the planning quality and planning efficiency of preoperative pedicle screw planning cannot be effectively guaranteed, resulting in difficulties in the actual clinical application of pedicle screw-assisted screw placement technology, etc., the present invention uses deep learning technology to explore and develop a pedicle screw-assisted planning algorithm model based on a neural network, and quantitatively verifies the accuracy and planning efficiency of its spinal segmentation and pedicle screw planning, realizing the intelligent segmentation of spinal structure and intelligent planning of pedicle screws. The specific technical process is as follows:

[0033] 1. Semi-automatic lumbar CT image sequence ROI extraction algorithm based on human-computer interaction:

[0034] Extracting the vertebral ROI from the bone window thin layer image sequence of lumbar CT plain scan + bone three-dimensional reconstruction image data can further improve the information ratio of the spine to the image background, thereby improving the accuracy of spine detection. Generally speaking, in CT images, the CT value range of bone tissue is [300, 3000] Hu. The CT value of the lumbar spine bone tissue is greater than 400 Hu, and the contrast with other tissues such as paravertebral soft tissue and abdominal organs is relatively large. The lumbar spine is presented as grayish white representing high density in the image, and other tissues such as paravertebral soft tissue and abdominal organs are presented as low-density black in the image, which can effectively obtain the anatomical structure information of the lumbar spine from the image more directly. Therefore, in the previous deep learning research on lumbar spine segmentation, many researchers believed that the ROI extraction algorithm was not necessary. However, in the present invention, the ROI is extracted from the lumbar CT sequence image, which can further reduce the interference of other tissues such as paravertebral soft tissue and abdominal organs, reduce redundant calculations, and improve the accuracy of lumbar spine detection, while also assuming another important function.

[0035] In actual clinical work, patients often only need to undergo lumbar pedicle screw fixation of 2 to 3 vertebrae according to the needs of the disease. Therefore, the present invention hopes to allow surgeons to freely choose the lumbar spine segmented by the deep learning model of the present invention and the planning of the corresponding pedicle screws through the semi-automatic lumbar CT image sequence ROI extraction algorithm based on human-computer interaction, avoiding unnecessary feature extraction of all lumbar CT sequence images in actual application, thereby reducing irrelevant parts and feature vectors in the image sequence, which is conducive to the accurate segmentation of the spine and further accurate planning of pedicle screws, and improving the efficiency of spine segmentation and pedicle screw planning of the deep learning model.

[0036] Extracting ROI from lumbar CT sequence images, while ensuring the accuracy of ROI extraction, it is necessary to make some slices after ROI extraction still retain the original sequence association. However, the application conditions of existing algorithms are mostly applicable to single-frame slices, such as selection search, multiscale combinatorial grouping (MCG) and region proposal network (RPN) and other methods, which can only process the information of the input slice itself, and the sequence information cannot be guaranteed to be retained in the processed slice. Different from two-dimensional image data such as X-ray, lumbar CT image data is a volume data set, and each slice in its image sequence has sequence. Based on this, the present invention proposes a non-learning human-computer interactive semi-automatic ROI extraction algorithm suitable for lumbar CT images, combining physician interaction with slice sequence association, accurately extracting ROI and retaining slice sequence at the same time, so as to facilitate surgeons to select the spinal segment to be processed according to the actual needs of the patient's condition. It mainly includes four parts: doctor selection of image, spatial coordinate calculation, projection coordinate calculation of each perspective (cross section, sagittal plane and coronal plane) and ROI interception.

[0037] 2. SIS-3D-UNet algorithm construction:

[0038] In clinical practice, spinal surgeons need to observe the complex anatomical structure of the spine and confirm and review the planning of pedicle screws in combination with the placement principles of pedicle screws. In this process, three-dimensional volume data sets show obvious advantages over two-dimensional images. Therefore, compared with two-dimensional images, the present invention prefers to input and output the research model in the form of three-dimensional image data. At present, the three-dimensional image data processing method based on convolutional neural network is usually subject to the following three major limitations. First, researchers need to adjust the data volume size of the volume data set from the original resolution size to a lower resolution reference size for neural network calculations. Second, due to the limitation of memory performance, the data capacity of this data processing method is very limited and the processing efficiency is low. Finally, the current three-dimensional image data processing method based on convolutional neural network requires that any single frame slice of the volume data set must be in a callable state at any given training or testing time during calculation. In order to solve the above problems, the present invention proposes a method based on sequential image segmentation combined with a 3D-UNet model, namely, SIS-3D-UNet. By introducing the Bi-Conv-LSTM structure into the 3D-UNet model, SIS-3D-UNet can not only process the entire volume data set according to the lumbar CT image sequence, but also process a selected partial single-frame slice set according to the actual needs of the patient's condition.

[0039] Based on SIS-3D-UNet, the present invention proposes a method for spine segmentation and pedicle screw planning using lumbar CT plain scan + bone 3D reconstruction image data bone window thin layer image sequence, and the specific method mainly includes the following steps:

[0040] Preprocessing: Data format conversion, data sampling and other data preprocessing are performed on the bone window thin-layer image sequence of lumbar CT plain scan + bone 3D reconstruction image data to reduce the burden on the equipment, improve the feasibility of the research, and avoid excessive redundant calculations, making the feature differences between adjacent slices of the input data more significant.

[0041] ROI extraction: The proposed non-learning human-computer interactive semi-automatic ROI extraction algorithm suitable for lumbar CT images is used to extract the spinal region of interest in the bone window thin-layer image sequence of lumbar CT plain scan + bone three-dimensional reconstruction image data, select the spinal segments that need deep learning operations and generate ROI sequences to improve the information ratio of the spine and background images, and select the spinal segments to be processed according to the actual needs of the patient's condition.

[0042] Vertebral pre-detection (Detection): The one-stage target detection network YOLO v5 (You Only LookOnce) is used to build a vertebral pre-detection network, and the lumbar CT plain scan + bone 3D reconstruction image data bone window thin layer image sequence after preprocessing and ROI extraction is used for vertebral pre-detection. The vertebral anatomical structure in the ROI sequence is extracted by the vertebral pre-detection network to generate a feature map, which can further improve the accuracy and efficiency of vertebral segmentation and pedicle screw planning.

[0043] Spine segmentation: A method based on sequential image segmentation combined with a 3D-UNet model, namely SIS-3D-UNet, introduces a Bi-Conv-LSTM structure into the 3D-UNet model, so that SIS-3D-UNet can process the entire volume data set according to the lumbar CT image sequence, and can also process a selected set of single-frame slices according to the actual needs of the patient's condition. It can fuse the feature map generated by the spine pre-detection network with the ROI sequence as the input of the Bi-Conv-LSTM network, and output the feature map sequence after the fusion of spatiotemporal information and convert it into a spine segmentation result.

[0044] Pedicle screw planning: The network model is trained with deep learning using the "gold standard" sample data set to learn the corresponding pedicle screw planning rules. Based on the spine segmentation results in the previous step, the pedicle screw planning results are output through the feature extraction module and the post-processing module. The obtained spine segmentation results and pedicle screw planning results are then upsampled to achieve the final data output.

[0045] In order to solve the three main limitations that the three-dimensional image data processing method based on convolutional neural network is usually subject to, the present invention integrates the Bi-Conv-LSTM structure into the 3D-UNet architecture and proposes the SIS-3D-UNet algorithm model. The original design intention of SIS-3D-UNet is to process 3D image data, especially for the spine segmentation and pedicle screw planning tasks of the bone window thin layer image sequence of lumbar CT plain scan + bone three-dimensional reconstruction image data in the present invention. The system architecture includes Bi-Conv-LSTM structure, convolution, pooling, upsampling and connection layer. 3D-UNet mainly assumes the function of encoder-decoder structure to learn the lumbar vertebrae anatomical structure characteristics of sample data and pedicle screw planning rules. The encoder gradually downsamples the input image into a smaller feature map, and the decoder upsamples these feature maps to the original input image size while retaining rich spatial information. The Bi-Conv-LSTM structure is introduced to process the temporal information of the input data. Through the Bi-Conv-LSTM structure, SIS-3D-UNet can process a serialized image set and maintain the sequential association between images. It uses convolution operations and long short-term memory network units to capture long-term dependencies in image sequences, and maintains a cell state and hidden state internally.

[0046] By extracting the three-dimensional background of a single-frame slice in a sequential manner, the SIS-3D-UNet model can learn the correlation between single-frame slices based on spatial levels in the bone window thin-layer image sequence of lumbar CT plain scan + bone three-dimensional reconstruction image data. Because it can only process a small part of the CT image data at any given time point, it is no longer necessary to downsample the input data to a greater extent, retaining more details of the original data. At the same time, the training of SIS-3D-UNet does not require a large amount of computer memory space, which solves another known limitation of the current deep learning network model research. This fully integrated sequential method is of great significance to the real-time application explored and developed by the present invention, because it can synchronously realize spinal segmentation and pedicle screw planning during data acquisition or when loading data, maximizing feature information sharing. During the training process, the spinal segmentation and pedicle screw planning results continuously assist in the extraction of subsequent feature points, and the error gradient information during feature point prediction can also be back-propagated along the neural network to the encoder of image feature extraction, continuously optimizing the decoder, and improving the accuracy of spinal segmentation results and pedicle screw feature extraction.

[0047] 3. Att-SIS-3D-UNet algorithm construction:

[0048] The present invention proposes a SIS-3D-UNet algorithm model, which can process the entire volume data set according to the lumbar CT image sequence, and can also process the selected partial single-frame slice set according to the actual needs of the patient's condition, which solves the three main limitations of the three-dimensional image data processing method based on the convolutional neural network to a certain extent. However, when the number of slice frames to be processed increases, the redundant information attached will also increase with the growth of the sequence. SIS-3D-UNet needs to face more interference from redundant information when operating on a long sequence of single-frame slice sets, resulting in a long-term phenomenon of insufficient reliance on information. Therefore, the performance of SIS-3D-UNet will decrease due to the gradual increase in the number of slice frames to be processed, which poses a challenge to further improve the accuracy of model image processing. Therefore, the present invention further optimizes SIS-3D-UNet and proposes a 3D-UNet model based on attention sequence image segmentation, namely Att-SIS-3D-UNet. The present invention introduces a SK-Net based on a self-attention mechanism and adds an Attention Mechanism module on its basis to further enhance the model's ability to mine and utilize the features of the spine and pedicle screws, and suppress the interference of redundant information.

[0049] The present invention first analyzes the factors that lead to the performance degradation of SIS-3D-UNet. By consulting the literature and referring to other studies, we believe that the possible reason is that when the number of reviews R is large, more single-frame slices need to be inserted into the sequence. On the one hand, excessive single-frame slices will generate a large amount of redundant information, and excessive redundant information will reduce the promotion effect of the Skip-Review attention mechanism; on the other hand, when the ROI sequence is too long, it will be insufficient to rely on long-term information when input into the Bi-Conv-LSTM structure. Inspired by the fact that the existing attention mechanism can adaptively obtain the key features of the image and suppress interference information, the present invention introduces the SK-Net structure into SIS-3D-UNet and adds an attention mechanism module on its basis, and proposes Att-SIS-3D-UNet, which further enhances the model's ability to mine and utilize the features of the spine and pedicle screws, and suppresses the interference of redundant information.

[0050] The significance of adding the attention mechanism module on the basis of the SK-Net network structure in the present invention is to further enhance the mining and utilization capabilities of the Att-SIS-3D-UNet algorithm model for the features of the spinal anatomical structure and pedicle screw planning rules. SK-Net has achieved good results in processing feature extraction of different scales, and after adding the attention mechanism, the weights of different features of the spinal anatomical structure and pedicle screw planning rules can be dynamically adjusted according to their importance, further enhancing the representation ability of the Att-SIS-3D-UNet network. By introducing the attention mechanism, the network can automatically learn the correlation and importance between the different features of the spinal anatomical structure and pedicle screw planning rules, so that the Att-SIS-3D-UNet network is more focused on those features that are more helpful to the final task of improving the safety, accuracy and efficiency of spinal segmentation and pedicle screw planning. This helps to improve the generalization ability of the Att-SIS-3D-UNet network, so that it can perform well in various complex scenarios.

[0051] The specific method of Att-SIS-3D-UNet mainly includes the following steps:

[0052] Preprocessing: Data format conversion, data sampling and other data preprocessing are performed on the bone window thin-layer image sequence of lumbar CT plain scan + bone 3D reconstruction image data to reduce the burden on the equipment, improve the feasibility of the research, and avoid excessive redundant calculations, making the feature differences between adjacent slices of the input data more significant.

[0053] ROI extraction: The proposed non-learning human-computer interactive semi-automatic ROI extraction algorithm suitable for lumbar CT images is used to extract the spinal region of interest in the bone window thin-layer image sequence of lumbar CT plain scan + bone three-dimensional reconstruction image data, select the spinal segments that need deep learning operations and generate ROI sequences to improve the information ratio of the spine and background images, and select the spinal segments to be processed according to the actual needs of the patient's condition.

[0054] Vertebral pre-detection: The One Stage object detection network YOLO is used to build a vertebral pre-detection network, and the lumbar CT plain scan + bone 3D reconstruction image data bone window thin layer image sequence after preprocessing and ROI extraction is used for vertebral pre-detection. The vertebral anatomical structure in the ROI sequence is extracted by the vertebral pre-detection network to generate a feature map, which can further improve the accuracy and efficiency of vertebral segmentation and pedicle screw planning.

[0055] Spine segmentation: The feature map generated by the spine pre-detection network is input into SK-Net for feature aggregation, and then the Skip-Review attention mechanism is used to fuse the aggregated feature map with the original ROI sequence. The fused feature map sequence is input into the closed-loop structure of the feature map sequence for reorganization, and then input into the Bi-Conv-LSTM network for temporal information association, and the feature map sequence after spatiotemporal information fusion is output and converted into the spine segmentation result.

[0056] Pedicle screw planning: The network model is trained with deep learning using the "gold standard" sample data set to learn the corresponding pedicle screw planning rules. Based on the spine segmentation results in the previous step, the pedicle screw planning results are output through the feature extraction module and the post-processing module. The obtained spine segmentation results and pedicle screw planning results are then upsampled to achieve the final data output.

[0057] 4. Att-SIS-3D-UNet algorithm improved based on adversarial training mechanism:

[0058] As we all know, the amount of sample data is an important factor affecting model performance in deep learning. However, compared with traditional computer vision datasets, the acquisition cost of lumbar CT image data in the present invention is higher and the quantity is more scarce. In order to enable Att-SIS-3D-UNet to obtain spinal segmentation results and pedicle screw planning results that are satisfactory to spinal surgeons using limited lumbar CT image data, the present invention uses an adversarial training mechanism to improve Att-SIS-3D-UNet.

[0059] Generative Adversarial Networks (GANs) have been widely used in the field of medical image segmentation since they were proposed, especially when dealing with scarce medical image data. GANs are composed of a generator and a discriminator. The generator generates pseudo data, and the discriminator determines whether the input data is real. The two learn the distribution of data through adversarial games. The present invention introduces the idea of ​​adversarial learning to Att-SIS-3D-UNet, and designs a segmentation network and a discriminator network. The segmentation network adopts the Att-SIS-3D-UNet network structure of upsampling Residual Blocks of the global aggregation module. The discriminator network is a binary classification network that receives the segmentation planning results and GroundTruth Data of the Att-SIS-3D-UNet model as input and discriminates their sources. Through the adversarial learning mechanism, the spinal segmentation and pedicle screw planning results of the Att-SIS-3D-UNet network are made as close to the "gold standard" as possible, thereby achieving excellent spinal segmentation and pedicle screw planning effects on a smaller data set.

[0060] More specifically, the present invention uses adversarial training to mix the spine segmentation results and pedicle screw planning results output by Att-SIS-3D-UNet and the labeled image data as the "gold standard" and re-input them into the model. The discriminator network allows the model to determine whether the input is the Ground Truth Data labeled by the physician or the result generated by Att-SIS-3D-UNet, and the back propagation algorithm is used to continuously correct the judgment results of the Att-SIS-3D-UNet network. The images with obvious anatomical features of the lumbar spine and representative pedicle screw planning in the sample database are used to assist in detecting images with unclear features and insufficient representativeness, effectively improving the accuracy of spine segmentation and pedicle screw planning, and then using less lumbar CT image data to allow the deep learning model to achieve spine segmentation and pedicle screw planning effects that are satisfactory to spinal surgeons.

[0061] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0063] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for autonomous planning of pedicle screw trajectories based on a neural network, characterized in that: The following steps are involved: Extract the spine ROI from the bone window thin-layer image sequence of lumbar CT plain scan + bone 3D reconstruction image data to generate a data set; By introducing the Bi-Conv-LSTM structure into the 3D-UNet model, a SIS-3D-UNet model is constructed, and the data set is used for training to generate a first segmentation planning model for spine segmentation and pedicle screw planning.

2. The method for autonomous pedicle screw trajectories planning based on a neural network according to claim 1, characterized in that: In the process of generating the data set, ROI is intercepted based on the selected images through spatial coordinate calculation and projection coordinate calculation of each perspective, wherein each perspective includes a cross-sectional plane, a sagittal plane, and a coronal plane.

3. A method for autonomous pedicle screw trajectories planning based on a neural network according to claim 2, characterized in that: In the process of building the SIS-3D-UNet model, the 3D-UNet model is used as an encoder-decoder structure to learn the anatomical structural characteristics of the lumbar spine and the pedicle screw planning rules of the sample data. The encoder gradually downsamples the input image into smaller feature maps, and the decoder upsamples these feature maps to the original input image size while retaining the spatial information.

4. The method for autonomous planning of pedicle screw trajectories based on a neural network according to claim 3, characterized in that: In the process of constructing the SIS-3D-UNet model, the temporal information of the input data is processed through the Bi-Conv-LSTM structure, wherein the SIS-3D-UNet model is used to process a serialized image set and maintain the order between images. At the same time, convolution operations and long short-term memory network units are used to capture long-term dependencies in the image sequence, and a cell state and a hidden state are maintained inside it.

5. The method for autonomous planning of pedicle screw trajectories based on a neural network according to claim 4, characterized in that: In the process of spinal segmentation and pedicle screw planning, based on the SIS-3D-UNet model, the SK-Net structure is introduced and an attention mechanism module is added on its basis to construct the Att-SIS-3D-UNet model. The model is trained through the data set to generate a second segmentation planning model for spinal segmentation and pedicle screw planning.

6. The method for autonomous planning of pedicle screw trajectories based on a neural network according to claim 5, characterized in that: In the process of pedicle screw planning, the "gold standard" sample data set is used to perform deep learning training on the SIS-3D-UNet model or the Att-SIS-3D-UNet model so that it can learn the corresponding pedicle screw planning rules.

7. The method for autonomous planning of pedicle screw trajectories based on a neural network according to claim 6, characterized in that: In the process of generating the second segmentation planning model, based on the Att-SIS-3D-UNet model, after adding the adversarial network, the data set is used for training to generate the second segmentation planning model, wherein the adversarial network consists of a segmentation network and a discriminator network, the segmentation network uses the upsampled Residual Blocks of the global aggregation module, and the discriminator network is a binary classification network, which receives the segmentation planning results of the Att-SIS-3D-UNet model and GroundTruth Data as input, and identifies their sources.

8. The method for autonomous planning of pedicle screw trajectories based on a neural network according to claim 7, characterized in that: In the process of spine segmentation and pedicle screw planning, adversarial training is used to mix the spine segmentation results and pedicle screw planning results output by the Att-SIS-3D-UNet model and the labeled image data as the "gold standard" and re-input the model. The model is judged by the discriminator network, and the judgment results of the Att-SIS-3D-UNet model are corrected by the back propagation algorithm.

9. A pedicle screw trajectories autonomous planning system based on neural network, characterized in that: include: A data processing module is used to extract the spine ROI from the bone window thin-layer image sequence of lumbar CT plain scan + bone 3D reconstruction image data to generate a data set; The planning module is used to construct a SIS-3D-UNet model by introducing a Bi-Conv-LSTM structure into the 3D-UNet model, and use the data set for training to generate a first segmentation planning model for spine segmentation and pedicle screw planning.

10. The neural network-based pedicle screw trajectories autonomous planning system according to claim 9, characterized in that: The planning module is also used to introduce the SK-Net structure based on the SIS-3D-UNet model and add an attention mechanism module on its basis to construct an Att-SIS-3D-UNet model. The spinal segmentation results and pedicle screw planning results output by the Att-SIS-3D-UNet model and the labeled image data as the "gold standard" are mixed and re-input into the model for training by adversarial training, and the model is used to judge the input results through the discriminator network, and the judgment results of the Att-SIS-3D-UNet model are corrected by the back propagation algorithm.

Images