An interactive and automatic spine segmentation refinement method based on graph neural network
Through the interactive and automatic segmentation refinement methods of graph neural network, the problems of excessive segmentation, insufficient detection and edge blur in the spinal segmentation in the existing technology are solved, and more accurate and flexible vertebrae segmentation is achieved, reducing annotation work, and improving the The ability of artificial intelligence to apply vertebrae segmentation.
Patent Information
- Application Number
- CN202111621219.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-12-28
AI Technical Summary
The existing automatic spinal segmentation method has problems with oversegment, insufficient detection and edge fuzziness, which affects the accuracy of segmentation and quantitative evaluation of clinical applications.
Adopting interactive and automatic segmentation refinement methods based on graph neural networks, we can improve the segmentation results by enhancing data preprocessing, coarse segmentation of full convolutional neural networks and refinement of gated graph neural networks, combining user interaction and automatic correction. accuracy.
It reduces annotation work, improves the accuracy and flexibility of vertebrae segmentation, enhances the application of artificial intelligence in vertebrae segmentation, and is suitable for three-dimensional image slice data modeling in the field of digital orthopedics.
Smart Images

Figure CN114549396B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a digital orthopedic technology, in particular to a spine interactive and automatic segmentation refinement method based on a graph neural network, and the present invention belongs to the technical field of medical image processing. Background Art
[0002] With the continuous development of imaging, the application of CT technology in clinical practice has become more and more common. Among them, in the process of using CT images to obtain the patient's spinal condition, accurately segmenting the spine and individual vertebrae from CT data plays a very important role in many clinical scenarios. After segmenting the vertebrae, on the one hand, the shape and health status of each vertebra can be determined, and on the other hand, it is helpful for the diagnosis of early spinal diseases, surgical planning and localization of spinal lesions, such as degenerative spinal diseases, spinal deformities, trauma, tumors and fractures. At present, the spinal models used in most computer-aided diagnosis and surgical planning systems are based on manual segmentation by doctors. This method is not only time-consuming and labor-intensive, but also has poor segmentation repeatability. Among them, the segmentation of regions of interest or targets in the image is the basis for image analysis and target recognition. For example, if the segmentation of various tissues and organs in medical images and the localization of lesions rely on manual delineation, there are many shortcomings such as strong subjectivity, poor reliability and repeatability, and cumbersome and time-consuming. Therefore, people are looking for automatic segmentation algorithms. Before the rise of deep learning technology, researchers tried many traditional image segmentation methods such as threshold method, region growing method, geometric deformation model, clustering, statistical decision method, and machine learning methods such as artificial neural network and support vector machine by extracting image features. However, these automatic segmentation methods generally have the disadvantages of complex models, poor robustness, sensitivity to noise, cumbersome feature extraction and selection process, and strong subjectivity, which affect segmentation performance and application.
[0003] In recent years, deep learning models represented by convolutional neural networks have adopted a completely "end-to-end" multi-level segmentation process. Not only does it integrate automatic feature extraction and selection during training, but it also establishes a multi-layer feature system from concrete to abstract, similar to the cognitive process and characteristics of the human brain. In addition, it constructs a deeper network structure and adopts a greedy unsupervised layer-by-layer training method to avoid the serious loss of errors when propagating forward and backward, which leads to gradient disappearance or diffusion, and is prone to falling into local minima or non-convergence problems. Therefore, the application of deep learning technology in image segmentation, registration, classification and recognition is increasing. Among them, in the segmentation of medical images, the most representative method is the full convolutional neural network U-Net with an encoding-decoding structure and jump connections. Since then, many researchers have made improvements on this basis. Although these improvement strategies can improve the performance of the segmentation model, the accuracy of segmentation is crucial for quantitative evaluation in many clinical applications. Automatic segmentation methods often have various errors, such as over-segmentation, insufficient detection and blurred edges, which usually require manual correction and refinement of the results generated by the algorithm. Summary of the invention
[0004] The present invention proposes a spine interactive and automatic segmentation refinement method based on graph neural network, which can solve the problems of over-segmentation, insufficient detection and blurred edges in existing automatic segmentation. In order to achieve the above objectives, the present invention adopts the following technical solutions:
[0005] A spine interactive and automatic segmentation refinement method based on graph neural network, comprising the following steps:
[0006] Step 1: Obtain the medical images to be processed and perform preprocessing;
[0007] The medical image to be processed includes at least one segment of the target spine; and the preprocessing here refers to data enhancement processing, that is, for a training set with a relatively small number of initial samples, the purpose of expanding the sample data set is achieved by randomly performing the following one or more operations:
[0008] Preprocessing methods include:
[0009] (a) Perform random front-to-back and left-to-right mirror flipping on the vertebral CT image sequence;
[0010] (b) Perform random cropping on the vertebral CT image sequence;
[0011] (c) Gaussian noise perturbation is added to the vertebral CT image sequence.
[0012] Step 2: Automatic rough segmentation of the spine;
[0013] Step 2a. Select a fully convolutional neural network for spine segmentation;
[0014] Step 2b. Use the training set to train the neural network to obtain a network model;
[0015] Step 2c. Use the obtained network model to segment the spinal CT data in the test set to obtain a rough segmentation result;
[0016] Step 3: Vertebrae interactive and automatic segmentation refinement;
[0017] Use graph neural network in interactive mode to refine the segmentation result obtained from the rough segmentation result in step 2;
[0018] Step 3a: Convert the 3D segmentation mask into polygons according to the result of the rough segmentation;
[0019] In step 3a: we use a gated graph neural network model to propagate the annotator’s edits to neighboring nodes in the spatial neighborhood and adjacent slices.
[0020] In step 3a: In order to satisfy the requirement of moving all vertices closer to the true object boundary while retaining vertices already on the boundary, the following three strategies are considered: (1) having a separate output branch to classify if the moved vertex is on the boundary, in addition to the motion prediction branch; (2) applying refinement sequentially, prioritizing vertices far from the boundary to converge to the true contour in fewer steps; and (3) designing a controlled training strategy that provides mixed training samples with different motion distances and contains a relatively large proportion of no-motion samples.
[0021] In step 3a: Here we consider two main types of interactive editing of polygons: (1) the user moves a vertex in the polygon to a point on the real boundary; (2) the refined segmentation is automatically obtained by clicking a button. It should be noted that for both methods, the vertices of the polygon are only adjusted on the current slice.
[0022] Step 3b: Use the obtained graph neural network model to obtain a refined segmentation result;
[0023] In step 3b: Unlike convolutional neural networks, gated graph neural network modeling includes two stages for mapping the graph to the output, namely the propagation model and the output model.
[0024] In step 3b: the output model has two parts, one is whether the predicted point has reached the edge, and the other is the direction in which the point should move to the next point.
[0025] In step 3b: Since the movement of each vertex is limited to the mesh centered at the target vertex position, if some vertices are far away from the true contour, further corrections may be required, that is, re-running the algorithm.
[0026] The benefits of the present invention are as follows: the present invention is applied to the field of digital orthopedics, and the vertebrae can be propagated to their three-dimensional neighbors through user editing while retaining the correct contour parts to reduce annotation work. The method is more flexible in modeling three-dimensional image slice data of vertebrae, and is a method that takes into account both interactivity and full automation, which is of great significance to the application of artificial intelligence in vertebrae segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 Shown is a flow chart of the vertebra segmentation editing and refinement method of the present invention.
[0028] Figure 2 Shown is a schematic diagram of the interactive segmentation refinement according to the present invention.
[0029] Figure 3 Shown is the graph neural network framework diagram described in the present invention. DETAILED DESCRIPTION
[0030] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0031] like Figure 1 As shown, a spine interactive segmentation editing and refinement method based on graph neural network includes the following steps:
[0032] Step 1: Obtain the medical images to be processed and perform preprocessing;
[0033] The medical image to be processed includes at least one segment of the target spine; the preprocessing here mainly refers to data enhancement processing, that is, for a training set with a relatively small number of initial samples, one or more of the following operations are randomly performed to achieve the purpose of expanding the sample data set:
[0034] (a) Perform random front-to-back and left-to-right mirror flipping on the vertebral CT image sequence;
[0035] (b) Perform random cropping on the vertebral CT image sequence;
[0036] (c) Gaussian noise perturbation is added to the vertebral CT image sequence.
[0037] Step 2: Rough segmentation of vertebrae;
[0038] Step 2a. Select a fully convolutional neural network for spine segmentation, which can be a U-net network;
[0039] Step 2b. Use the training set to train the neural network to obtain a network model;
[0040] Step 2c. Use the obtained network model to segment the spinal CT data in the test set to obtain a rough segmentation result;
[0041] Step 3: Vertebrae interactive segmentation and refinement;
[0042] The rough segmentation result obtained from step 2 is refined by using graph neural network. The process is mainly divided into two parts: one is the measurement of the polygon of the rough segmentation vertebral contour; the other is to obtain the refined segmentation result using the obtained graph neural network model.
[0043] Step 3a: Convert the 3D segmentation mask into polygons based on the result of the rough segmentation of the vertebrae;
[0044] Before the user edits the segmentation mask, the 3D segmentation mask is converted into the corresponding polygon one by one. In this work, two main methods of editing polygons are considered: (1) the user moves a vertex in the polygon to a point on the real boundary such as Figure 2 As shown; (2) automatically obtained. Figure 3 The latter method is used to obtain the polygon. In both cases, the vertices on the polygon in each slice will be adjusted only in the current slice. The outline of the rough segmentation result in step 2 is a polygon, and the measurement of the polygon is calculated for each slice of the 3D image.
[0045] Step 3b: Establish a graph neural network model to obtain refined segmentation results;
[0046] The graph neural network modeling process for vertebral segmentation is as follows Figure 3 As shown in the figure: the example is a slice with 5 vertices, plus its propagated adjacent slices, a total of 7 nodes. In the construction graph, each vertex of the polygon is used as a node on the graph. Then three types of connection relationships are considered here. The solid line represents the connection between two closely connected nodes, such as nodes 0-1 and 0-3; the long dashed line represents the connection between nodes that are far apart, such as 0-2 and 0-4; the short dashed line represents the connection between two relatively close nodes between two adjacent slices, such as: 0-5 and 0-6. The graph in the example is a directed graph, and then from Figure 2 It can be seen that the adjacency matrix is divided into two parts: input and output. The input represents pointing to the node, and the output represents pointing from the node to other nodes. The features of each node in the graph are obtained using ResNet-50. Finally, the obtained graph structure is input into the graph neural network using GRU. The output model has two parts, one is whether the predicted point has reached the edge, and the other is the direction in which the point is to move to the next point. The movement prediction part in the method is an M*M matrix, which means the range of movement centered on the point. The movement of each vertex is limited to the grid centered on the target vertex position. Figure 3 The size is 15*15. If some vertices are far away from the true contour, Figure 3 If there are more than 7 grids in the algorithm, further corrections may be needed, that is, re-running the algorithm.
[0047] Anything not further described in the present invention is prior art.
[0048] The present invention provides a method for interactive and automatic segmentation and refinement of the spine based on a graph neural network. First, the spinal CT image data is acquired and preprocessed; then, the vertebral data is automatically coarsely segmented through a fully convolutional neural network model; finally, a graph neural network model is established to interactively or automatically refine the coarse segmentation results of the vertebrae to achieve a better segmentation effect. The present invention can not only improve the existing automatic segmentation results, reduce annotation work, and assist doctors in the next step of work, but also, due to its more flexible vertebral slice modeling method, it has certain guidance for fully automatic precise segmentation and is of great significance for in-depth research.
[0049] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An interactive and automatic spine segmentation refinement method based on graph neural network, It is characterized in that The steps include: Step 1: Acquisition and preprocessing of vertebral CT image data; The vertebral CT image data includes at least one target vertebral segment; preprocessing refers to enhancing the data, that is, for a training set with a relatively small number of initial samples, one or more operations are randomly performed to expand the sample data set; The pretreatment method used in step 1 includes: (a) Perform random front-to-back and left-to-right mirror flipping on the vertebral CT image sequence; (b) Perform random cropping on the vertebral CT image sequence; (c) Adding Gaussian noise perturbation to the vertebral CT image sequence; Step 2: Automatic rough segmentation of the spine; For the data processed in step 1, the existing fully convolutional neural network is used to train the network using the training set to obtain the corresponding model, and then the model is used to obtain the rough segmentation result through the test set; The method of automatic rough segmentation of the vertebrae in step 2 is: Step 2a. Build a 3D fully convolutional neural network for spine segmentation; Step 2b. Use the training set to train the neural network to obtain a network model; Step 2c. Use the obtained network model to segment the spinal CT data in the test set to obtain a rough segmentation result; Step 3: Subdivide the rough segmentation result of the vertebrae; Obtain the rough segmentation result from step 2 to perform segmentation refinement; It is divided into two parts: one is the measurement of the roughly segmented vertebral contour polygons, converting the three-dimensional segmentation mask into polygons; the other is the establishment of the graph neural network model, and using the graph neural network model to obtain refined segmentation results.
2. The method for interactive and automatic segmentation and refinement of spine based on graph neural network according to claim 1, It is characterized in that In the step 3, the three-dimensional segmentation mask is converted into a polygon according to the result of the rough segmentation of the vertebrae, in either of two ways, namely, the user moves a vertex of the polygon to a point on the real boundary or obtains it automatically.
3. The method for interactive and automatic segmentation and refinement of spine based on graph neural network according to claim 1, It is characterized in that In the step three, the graph neural network model consists of two parts: a propagation model and an output model; the output model has two parts, one is to predict whether the point has reached the edge, and the other is the direction in which the point should move to the next point.
