A deep learning-based method for intervertebral disc scanning and vertebral identification in MRI spine images

Abstract

The present invention discloses a method for intervertebral disc scanning positioning and vertebra identification of MRI spine images based on deep learning, comprising the following steps: step s1: using a convolutional neural network model to perform semantic segmentation on a pre-prepared MRI spine image; step s2: fitting foreground pixels on the image obtained by semantic segmentation in step s1 to obtain multiple polygons; step s3: performing polygon screening; step s4: performing polygon sorting and adding labels to complete vertebra identification; step s5: performing polygon edge approximation to reduce the number of vertices so that the polygon is closer to a quadrilateral; step s6: calculating and finding the lower left vertex, upper left vertex, upper right vertex and lower right vertex of the polygon; step s7: calculating the intervertebral disc scanning positioning line to complete the intervertebral disc scanning positioning. The present invention can improve the efficiency of MRI spine examination, reduce manual workload, and make the scanning positions of multiple examinations of patients have a high degree of repeatability.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for scanning and positioning intervertebral discs and identifying vertebrae in MRI spinal images based on deep learning. Background Art

[0002] With the improvement of economic level and the development of technology, people have put forward higher requirements for medical health. Magnetic resonance imaging (MRI) as an advanced imaging diagnostic technology is increasingly widely used in patient examinations. The use of MRI for the diagnosis of spinal diseases has obvious advantages over other imaging technologies, such as more obvious tissue contrast and imaging in any direction.

[0003] During an MRI examination of the spine, there are two key operations: disc location and vertebral identification:

[0004] Intervertebral disc positioning refers to the need to further scan the cross-sectional layer of the spinal intervertebral disc based on the positioning image after the MRI scanning positioning phase is completed. The cross-sectional layer to be scanned is usually determined based on the center point of the intervertebral disc and the distribution direction of the vertebrae. Usually, scanning a part requires multiple positioning adjustments, which is time-consuming and labor-intensive, including adjustment of the scanning center and scanning angle.

[0005] Vertebral identification refers to the identification of each vertebra on the image after the scan is completed, such as cervical vertebrae C2 to C7, thoracic vertebrae T1 to T12, and lumbar vertebrae L1 to L5. These identifications are used to determine the location of the disease and generate a diagnostic report.

[0006] In existing MRI products, when scanning the spine, the intervertebral disc is positioned manually. First, the center point of the intervertebral disc is determined, and then the scanning direction is adjusted by rotating the positioning line according to the distribution direction of the vertebrae. In addition, the labels of some vertebrae need to be determined manually in advance. This process generally takes several minutes depending on the proficiency of the operator.

[0007] Vertebral labels are also manually labeled. Some software products provide tools to manually add labels, while some software do not, and require human brain memorization.

[0008] The existing technology mainly has the following two defects:

[0009] 1. Manual operation takes a long time. It usually takes several minutes to perform manual positioning for scanning a disc sequence.

[0010] 2. In the film reading diagnosis, manual determination of vertebral labels is also very time-consuming, and a large amount of manual repetitive work is very easy to cause fatigue, leading to errors and misdiagnosis. Summary of the invention

[0011] The present invention aims to provide a deep learning-based MRI spinal image intervertebral disc scanning positioning and vertebral identification method, which can reduce the workload of clinical staff and its automatic scanning positioning can improve efficiency and make the scanning positions of multiple examinations of patients have high repeatability for easy review.

[0012] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0013] A method for scanning and locating intervertebral discs and identifying vertebrae in MRI spine images based on deep learning, comprising the following steps:

[0014] Step s1: Using a convolutional neural network model to perform semantic segmentation on the pre-prepared MRI spine image;

[0015] Step s2: fitting foreground pixels on the image obtained by semantic segmentation in step s1 to obtain multiple polygons;

[0016] Step s3: perform polygon screening;

[0017] Step s4: sort polygons and add labels to complete vertebrae identification;

[0018] Step s5: Approximate the polygon edge to reduce the number of vertices so that the polygon is closer to a quadrilateral;

[0019] Step s6: Calculate and find the lower left vertex, upper left vertex, upper right vertex and lower right vertex of the polygon;

[0020] Step s7: Calculate the intervertebral disc scanning positioning line to complete the intervertebral disc scanning positioning.

[0021] Preferably, in step s1, the convolutional neural network model is a DUnet model, the output of the DUnet model is a matrix, the matrix has the same size as the MRI spine image, and the matrix is ​​the Mask image;

[0022] The DUnet model includes 4 layers of Encoder modules, 1 layer of dilated convolution module and 2 layers of Decoder modules. The Encoder module includes 2 layers of 3x3 convolution. The Encoder module is activated by Leaky Relu function and normalized by BatchNormalization. The dilated convolution module includes 4 3x3 dilated convolution layers with dilated convolution rates of 1, 2, 3 and 5 respectively. The dilated convolution module also includes a 1x1 convolution layer.

[0023] Furthermore, in step s2, all foreground pixels in the Mask image are fitted to obtain a plurality of mutually independent polygons composed of these foreground pixels.

[0024] Furthermore, in step s3, polygon screening includes the following steps:

[0025] Step s301: traverse each polygon and filter it in the following way: first filter by the number of polygon vertices. If the number of its vertices is less than or equal to 2, delete it directly. If the number of its vertices is greater than 2, calculate the area of ​​the polygon; set the minimum area threshold to S min , if the area of ​​the polygon is less than S min If the area of ​​the polygon is not less than S, then delete it. min then retain;

[0026] Step 302: Calculate the center point of each polygon selected in step s301, and calculate the distance from the center point of the polygon to the center point of the nearest polygon. Calculate the average value davg of the nearest distances of multiple polygons in the Mask image by the following formula:

[0027]

[0028] Among them, d i represents the distance from the center point of the i-th polygon to the center point of the nearest polygon in the Mask image, and N represents the number of polygons; traverse the polygons filtered in step s301, if polygon i satisfies d i >2×d avg , then polygon i is discarded.

[0029] Furthermore, in step s4, the polygons filtered in step s3 are sorted from small to large according to the y-axis coordinate values ​​of the image reference coordinate system specified by the industry standard DICOM. After the sorting is completed, polygon labels can be directly added at the center points of the polygons in combination with the scanning site and manual assistance, and the single polygon is regarded as a single vertebra to complete the vertebral identification.

[0030] Furthermore, in step s5, each sorted polygon is traversed, the perimeter C of the polygon is calculated, and the polygon curve is approximated with an accuracy P, so that the polygon is close to a quadrilateral, where η represents an empirical value, P=η×C.

[0031] Furthermore, in step s6, the following steps are included:

[0032] Step s601: Determine the coordinates of the center point (x0, y0) and the vertex (x i ,y i), and divide the vertices into two sets with x0 as the boundary: those less than x0 are divided into one set, in which the upper left vertex and the lower left vertex are found; those greater than x0 are divided into another set, in which the upper right vertex and the lower right vertex are found;

[0033] Step s602: Calculate the distance d from the polygon vertex set to the center point using the following formula:

[0034]

[0035] Furthermore, in step s7, the calculation of the intervertebral disc scanning positioning line is to calculate the midline between two adjacent polygons.

[0036] The present invention has the following beneficial effects:

[0037] 1. The vertebral markers involved in the present invention can effectively assist imaging doctors in reading films, generate diagnostic reports, reduce workload and work fatigue, and improve the efficiency and accuracy of film reading;

[0038] 2. The intervertebral disc scanning positioning involved in the present invention can greatly reduce the operation steps and time of the scanning operator, serve more patients, thereby improving the efficiency of the nuclear magnetic resonance system and increasing the repeatability of the review;

[0039] 3. The intervertebral disc scanning and positioning involved in the present invention can also realize advanced applications such as fully automatic scanning of nuclear magnetic resonance spine sequences and scanning of arbitrary segments of the entire spine. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of the DUnet model of the present invention;

[0041] Figure 2 This is a schematic diagram of the Encoder module;

[0042] Figure 3 This is a schematic diagram of the Decoder module;

[0043] Figure 4 Schematic diagram of the hole convolution module;

[0044] Figure 5 This is a schematic diagram of the intervertebral disc positioning of the present invention;

[0045] Figure 6 It is a graph showing the convergence trend of the Loss function during the model training process involved in the present invention;

[0046] Figure 7 This is a diagram showing the implementation results of the intervertebral disc scanning positioning and vertebral identification of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0048] A method for scanning and locating intervertebral discs and identifying vertebrae in MRI spine images based on deep learning, comprising the following steps:

[0049] Step s1: Use the convolutional neural network model to perform semantic segmentation on the pre-prepared MRI spine images.

[0050] In step s1, the convolutional neural network model is a DUnet model, and the output of the DUnet model is a matrix, and the matrix has the same size as the MRI spine image, and the matrix is ​​the Mask image.

[0051] The DUnet model includes 4 layers of Encoder modules, 1 layer of dilated convolution module and 2 layers of Decoder modules. The Encoder module includes 2 layers of 3x3 convolution. The Encoder module is activated by Leaky Relu function and normalized by BatchNormalization. The dilated convolution module includes 4 3x3 dilated convolution layers with dilated convolution rates of 1, 2, 3 and 5 respectively. The dilated convolution module also includes a 1x1 convolution layer.

[0052] First, use a 7x7 2D convolution layer to extract features from the image, reduce the size of the feature map, increase the number of feature map channels, and then continue to extract image features using the Unet Encoder-Decoder method. Figure 2 As shown in the figure, it contains 2 layers of 3x3 convolution, activated by Leaky Relu function, and normalized by Batch Normalization. The Encoder uses 3x3 convolution layer stride to downsample 2 pairs of features before output.

[0053] See the dilated convolution module Figure 4 As shown in the figure, there are four 3x3 dilated convolutional layers with dilation rates of 1, 2, 3, and 5 respectively. The four convolutional layers perform convolution operations on the input respectively. After the outputs of the four layers are connected, a 1x1 convolutional layer is used for feature fusion, and finally added to the input.

[0054] In the decoding process, the image size is restored directly using upsize_bilinear instead of deconvolution, which can save computational effort without losing image features. This method is implemented as tf.image.resize_bilinear in tensorflow, which adjusts the image size by bilinear interpolation. Compared with the standard Unet model, Decode only uses two Decoder modules and two upsize_bilinear processes, each upsize_bilinear expands the image size by 4 times. In the Decoder module, Figure 3 As shown in the figure, the layers corresponding to the Encoder module process are first connected, and then features are extracted through two 3x3 convolutional layers, activated with Leaky Relu, and then normalized with BatchNormalization before output.

[0055] After the Decoder layer, a layer of upsize_bilinear is used to expand the features Figure 2 times, so that the feature map and the input image have the same size. Then use a 1x1 convolution layer to fuse the feature map channels to 2. After passing through a softmax classifier, the two channels in the feature map represent the probability that the image element at the corresponding position is divided into background and foreground respectively. Finally, an argmax is used to obtain the channel number with the highest probability for each pixel in the feature map, and finally the Mask is obtained. The output Mask only contains 0 and 1 values, indicating that the input image element at the corresponding position belongs to the background or foreground, 1 represents foreground, and 0 represents background.

[0056] Step s2: Fit foreground pixels on the image obtained by semantic segmentation in step s1 to obtain multiple polygons.

[0057] In step s2, all foreground pixels in the Mask image are fitted to obtain a plurality of mutually independent polygons composed of these foreground pixels. One Mask image contains N polygons.

[0058] Step s3: Perform polygon screening.

[0059] In step s3, polygon screening includes the following steps:

[0060] Step s301: traverse each polygon and filter it in the following way: first filter by the number of polygon vertices. If the number of its vertices is less than or equal to 2, delete it directly. If the number of its vertices is greater than 2, calculate the area of ​​the polygon; set the minimum area threshold to S min , if the area of ​​the polygon is less than S min If the area of ​​the polygon is not less than S, then delete it. minthen retain;

[0061] Step 302: Calculate the center point of each polygon selected in step s301, and calculate the distance from the center point of the polygon to the center point of the nearest polygon. Calculate the average value d of the nearest distances of multiple polygons in the Mask image by the following formula: avg :

[0062]

[0063] Among them, d i represents the distance from the center point of the i-th polygon to the center point of the nearest polygon in the Mask image, and N represents the number of polygons; traverse the polygons filtered in step s301, if polygon i satisfies d i >2×d avg , then polygon i is discarded.

[0064] Step s4: Sort polygons and add labels to complete vertebrae identification.

[0065] In step s4, the polygons filtered in step s3 are sorted from small to large according to the y-axis coordinate values ​​of the image reference coordinate system specified by the industry standard DICOM. After the sorting is completed, polygon labels can be directly added at the center points of the polygons in combination with the scanning position and manual assistance, and the single polygon is regarded as a single vertebra to complete the vertebral identification.

[0066] The image reference coordinate system specified by the industry standard DICOM takes the patient lying on his back with his head first as an example, with the right hand to the left hand as the X-axis, the front to the back as the Y-axis, and the feet to the head as the Z-axis. The center of the scan is the center point of the coordinate system.

[0067] Step s5: Approximate the polygon edges to reduce the number of vertices and make the polygon closer to a quadrilateral.

[0068] In step s5, each sorted polygon is traversed, the perimeter C of the polygon is calculated, and the polygon curve is approximated with an accuracy P so that the polygon is close to a quadrilateral, where η represents an empirical value, P=η×C.

[0069] Step s6: Calculate and find the lower left vertex, upper left vertex, upper right vertex and lower right vertex of the polygon.

[0070] In step s6, the following steps are included:

[0071] Step s601: Determine the coordinates of the center point (x0, y0) and the vertex (x i ,y i), and divide the vertices into two sets with x0 as the boundary: those less than x0 are divided into one set, in which the upper left vertex and the lower left vertex are found; those greater than x0 are divided into another set, in which the upper right vertex and the lower right vertex are found;

[0072] Step s602: Calculate the distance d from the polygon vertex set to the center point using the following formula:

[0073]

[0074] Step s7: Calculate the intervertebral disc scanning positioning line to complete the intervertebral disc scanning positioning.

[0075] In step s7, the calculation of the intervertebral disc scanning positioning line is to calculate the midline between two adjacent polygons. Figure 5 As shown in the left half of the figure, if we need to calculate the intervertebral disc positioning line between the P1 and P2 vertebrae, we can directly convert it into calculating the midline of the two polygons. Calculate the midpoint of the vertex v3 and v1 to get v 31 , calculate the midpoint of vertices v4 and v2 to get v 42 , a straight line connecting two midpoints This is what we are looking for. In some clinical application scenarios, the operator needs to drag and rotate the scan line, so the straight line needs to be calculated. The center point of the intervertebral disc is the center point of the intervertebral disc. Figure 5 As shown in the right half, the complete scan plane center and angle are then obtained.

[0076] The following is a description of the steps in the specific embodiment in sequence:

[0077] Prepare data. Collect MRI spinal images, including lumbar, cervical and thoracic vertebrae. In this implementation, 780 spinal Dicom images were collected. Use artificial intelligence image labeling tools to mark the spinal polygons of the image. Divide the image into training sets, validation sets and test sets. Preprocess the images, including normalization and resizing. In this implementation, large and small images are used. Data augmentation. In this implementation, the images are rotated 90, 180 and 270 degrees.

[0078] Create and train a convolutional neural network model. In this implementation, tensorflow-1.14 is used to create and train the model. The model is trained on a GPU, with a GPU model of NVIDIA 1080Ti and a video memory of 12G. The training BatchSize is equal to 2. The initial learning rate is 1e-3, and the learning rate is gradually reduced by exponential descent during training. The training uses softmax cross entropy as the Loss function. The output of the model is a Mask image. The training epoch is set to 2000, and every 10 steps of training, the validation set is used for validation to observe the generalization ability of the model. The convergence trend of the Loss function during training is as follows: Figure 6 As shown, red indicates the downward trend of the training Loss function, and blue indicates the downward trend of the Loss during verification. Through observation, the model converges well and there is no overfitting phenomenon.

[0079] The trained neural network model is used to segment the image, and the segmentation result is a Mask image.

[0080] Fit polygons on the mask image. This implementation uses a third-party image processing library to obtain independent polygons composed of foreground pixels of the mask image, and the polygons are represented in the form of vertices.

[0081] Filter polygons. The filtering methods include vertex filtering, area filtering, and nearest neighbor distance filtering. The purpose of vertex filtering and area filtering is to exclude small noise points and areas near the vertebrae caused by the segmentation error of the neural network model. This error may be one point, two points, or a small area. The nearest neighbor distance filtering is to exclude segmentation errors far away from the vertebrae, while being compatible with the situation where the vertebrae distribution in some images in the application scenario is incomplete.

[0082] Sorting and marking. Calculate the center points of the existing polygons, and then sort the polygons from top to bottom. Combined with the scan location and manual assistance, the vertebrae can be marked directly at the center point. For example, the cervical, lumbar and full spine images can be marked automatically, while the thoracic spine requires manual assistance to determine the first or last vertebra, and the rest are automatically derived.

[0083] Polygon edge approximation. Traverse each polygon, calculate the polygon perimeter C, and approximate the polygon curve with precision P, so that the polygon is close to a quadrilateral, where η represents an empirical value, η is 0.02, P = η × C. The purpose of polygon approximation is to reduce the number of vertices and make the polygon as close to a quadrilateral as possible.

[0084] Find four vertices in the polygon vertex set: the upper left vertex, the lower left vertex, the upper right vertex, and the lower right vertex.

[0085] Calculate the intervertebral disc positioning line, that is, calculate the midline of every two adjacent polygons, calculate according to the four vertices of the polygon, and calculate the midpoint of the midline at the same time. Finally, use three points to represent a straight line, and the result is presented in the form of a straight line. The results of this implementation are shown as follows Figure 7 .

[0086] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, technicians familiar with the field may make various corresponding changes and deformations based on the present invention, but these corresponding changes and deformations should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for scanning and locating intervertebral discs and identifying vertebrae in MRI spinal images based on deep learning, characterized in that: The following steps are involved: Step s1: Using a convolutional neural network model to perform semantic segmentation on the pre-prepared MRI spine image; Step s2: fitting foreground pixels on the image obtained by semantic segmentation in step s1 to obtain multiple polygons; Step s3: polygon screening is performed; polygon screening includes the following steps: Step s301: traverse each polygon and filter it in the following way: first filter by the number of polygon vertices. If the number of its vertices is less than or equal to 2, delete it directly. If the number of its vertices is greater than 2, calculate the area of ​​the polygon; set the minimum area threshold to S min , if the area of ​​the polygon is less than S min If the area of ​​the polygon is not less than S, then delete it. min then retain; Step 302: Calculate the center point of each polygon selected in step s301, and calculate the distance from the center point of the polygon to the center point of the nearest polygon. Calculate the average value davg of the nearest distances of multiple polygons in the Mask image by the following formula: Among them, d i represents the distance from the center point of the i-th polygon to the center point of the nearest polygon in the Mask image, and N represents the number of polygons; traverse the polygons filtered in step s301, if polygon i satisfies d i >2×d avg , then discard polygon i; Step s4: sort polygons and add labels to complete vertebrae identification; Step s5: Approximate the polygon edge to reduce the number of vertices so that the polygon is closer to a quadrilateral; Step s6: Calculate and find the lower left vertex, upper left vertex, upper right vertex and lower right vertex of the polygon; Step s7: Calculate the intervertebral disc scanning positioning line and the center point of the intervertebral disc to complete the intervertebral disc scanning positioning; the calculation of the intervertebral disc scanning positioning line is to calculate the midline between two adjacent polygons, and the center point of the intervertebral disc is the center point of the midline between two adjacent polygons.

2. The method for intervertebral disc scanning positioning and vertebral identification based on deep learning in MRI spine images according to claim 1, characterized in that: In step s1, the convolutional neural network model is a DUnet model, and the output of the DUnet model is a matrix, the matrix has the same size as the MRI spine image, and the matrix is ​​the Mask image; The DUnet model includes 4 layers of Encoder modules, 1 layer of dilated convolution module and 2 layers of Decoder modules. The Encoder module includes 2 layers of 3x3 convolution. The Encoder module is activated by Leaky Relu function and normalized by BatchNormalization. The dilated convolution module includes 4 3x3 dilated convolution layers with dilated convolution rates of 1, 2, 3 and 5 respectively. The dilated convolution module also includes a 1x1 convolution layer.

3. The method for intervertebral disc scanning positioning and vertebral identification based on deep learning in MRI spine images according to claim 1, characterized in that: In step s2, all foreground pixels in the Mask image are fitted to obtain a plurality of mutually independent polygons composed of these foreground pixels.

4. The method for intervertebral disc scanning positioning and vertebra identification based on deep learning in MRI spine images according to claim 1, characterized in that: In step s4, the polygons filtered in step s3 are sorted from small to large according to the y-axis coordinate values ​​of the image reference coordinate system specified by the industry standard DICOM. After the sorting is completed, polygon labels can be directly added at the center points of the polygons in combination with the scanning position and manual assistance, and the single polygon is regarded as a single vertebra to complete the vertebral identification.

5. The method for intervertebral disc scanning positioning and vertebra identification based on deep learning in MRI spine images according to claim 1, characterized in that: In step s5, each sorted polygon is traversed, the perimeter C of the polygon is calculated, and the polygon curve is approximated with an accuracy P so that the polygon is close to a quadrilateral, where η represents an empirical value, P=η×C.

6. The method for intervertebral disc scanning positioning and vertebral identification based on deep learning in MRI spine images according to claim 1, characterized in that: In step s6, the following steps are included: Step s601: Determine the coordinates of the center point (x0, y0) and the vertex (x i ,y i ), and divide the vertices into two sets with x0 as the boundary: those less than x0 are divided into one set, in which the upper left vertex and the lower left vertex are found; those greater than x0 are divided into another set, in which the upper right vertex and the lower right vertex are found; Step s602: Calculate the distance d from the polygon vertex set to the center point using the following formula:

Images