A semantic segmentation method for spinal region ultrasound images based on deep learning

Abstract

The present invention is applicable to the field of medical image processing technology, and provides a semantic segmentation method for spinal region ultrasound images based on deep learning, including the following steps: converting the collected spinal region ultrasound video data of scoliosis patients into a set of JPG format pictures, and preprocessing the images with clear spinous process and transverse process regions; annotating the spinous process and transverse process regions to obtain labels for semantic segmentation, generating auxiliary class labels using the labels of the spinous process class and the transverse process class, and constructing a spinal region ultrasound image dataset; dividing the dataset into a training set, a validation set, and a test set; generating a probability heat map reflecting prior knowledge; designing a specific neural network model structure; training and testing the model on the dataset to achieve accurate segmentation of the spinous process class and the transverse process class in the images. The present invention not only realizes accurate segmentation of spinal region ultrasound images, but also provides strong support for subsequent spinal model reconstruction and Cobb angle measurement work.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and particularly relates to a semantic segmentation method for spinal region ultrasound images based on deep learning. Background Art

[0002] Scoliosis refers to a three-dimensional deformity in which the spine shows a lateral curvature in the coronal plane, usually accompanied by axial rotation and abnormalities in the sagittal plane. Scoliosis not only affects the patient's body shape, but also seriously affects the patient's physiological functions in severe cases. For patients with adolescent idiopathic scoliosis, wearing a brace can significantly reduce the probability of high-risk curves progressing to the surgical threshold. Therefore, it is very important to detect and intervene in scoliosis as early as possible. In clinical diagnosis, the Cobb angle is the standard for evaluating scoliosis. However, the current method of diagnosing scoliosis through X-Ray films may seriously endanger the patient's physical health due to long-term exposure to radiation. In addition to the X-Ray method, the MRI method can also be used to achieve the imaging task of the spine and measure the Cobb angle of scoliosis. However, due to the long scanning time and high cost of MRI, it is difficult to be popularized in clinical scoliosis diagnosis. In contrast, the ultrasound method is safer, more convenient and more economical, and is very suitable for clinical scenarios where scoliosis patients need to be examined multiple times.

[0003] When using the ultrasound method to diagnose scoliosis, it is first necessary to identify the spinal region in the ultrasound image, and then reconstruct the spinal model based on the recognition result to measure the Cobb angle. However, due to the imaging principle of ultrasound images, there is a lot of interference information in the ultrasound images, and accurately segmenting the spinal region has become a major challenge. For this reason, the present invention proposes a semantic segmentation method for spinal region ultrasound images based on deep learning to accurately segment the spinal region in the ultrasound image. Summary of the Invention

[0004] The purpose of the present invention is to provide a semantic segmentation method for spinal region ultrasound images based on deep learning, aiming to solve the problems proposed in the above background art.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] A semantic segmentation method for spinal region ultrasound images based on deep learning, comprising the following steps:

[0007] Step A, constructing a data set: converting the collected spinal region ultrasound video data into a set of JPG format pictures, selecting images with clear spinous process and transverse process regions for preprocessing, and annotating the spinous process and transverse process regions in the images to obtain corresponding label maps, namely spinous process class labels and transverse process class labels, generating auxiliary class labels using the spinous process class labels and transverse process class labels, constructing a spinal region ultrasound image data set, and the image size in the data set is 448pix×448pix;

[0008] Step B, Divide the dataset: Divide the obtained dataset into three parts, namely the training set, the validation set, and the test set;

[0009] Step C, Generate probability heatmaps: Generate probability heatmaps that can reflect prior knowledge using the class labels in the training set obtained in Step B;

[0010] Step D, Build a neural network model: Design a neural network model SpineHNet that can combine the probability heatmaps obtained in Step C to improve the segmentation performance;

[0011] Step E, Model training and validation: Use the training set and the validation set divided in Step B to train and validate the neural network model SpineHNet designed in Step E;

[0012] Step F, Model testing: Use the test set divided in Step B to test the performance of the trained neural network model SpineHNet.

[0013] Furthermore, in the above Step A, the specific operation of generating the auxiliary class label is as follows: For an image that simultaneously contains spinous processes and transverse processes, calculate the centroid P of the spinous process class through the corresponding label map of the image 1 ; Using the abscissa of the centroid P 1 as the dividing line, divide the transverse process class region into two left and right regions, and calculate the centroids P 2 , P 3 of the two left and right regions through their respective label maps; Construct a triangular region with P 1 , P 2 and P 3 as vertices, map the triangular region to the original image, and calculate the median m of the pixel values in the triangular region in the original image; Set the points with pixel values less than m and belonging to the background class in the triangular region as the auxiliary class.

[0014] Furthermore, in the above Step B, the picture data of a single case only exists in a single set.

[0015] Furthermore, in the above Step C, the specific operation of generating the probability heatmaps is as follows: Set a zero matrix Z with the same size as the images in the dataset and the initial center point coordinates C O ; For each label map with auxiliary classes in the training set, obtain the coordinate set M 1 of the non-background class points in the map, calculate the centroid P A of the auxiliary class region in the label map, and calculate the abscissa difference d A between P O and C x , the ordinate difference d y , and for M 1The horizontal and vertical coordinates of each item in are respectively subtracted from d x and d y to obtain a new coordinate set M 2 . The coordinates of each item in M 2 are mapped to the matrix Z. For the coordinate positions within the range of the matrix Z, their values are incremented by 1; the values of each item in the matrix Z are divided by the total number of images with auxiliary class labels in the training set to obtain the probability heatmap H.

[0016] Furthermore, in the step D, the neural network model SpineHNet includes two processing stages, and both processing stages use an Encoder-Decoder type network as the backbone network;

[0017] The processing flow of the neural network model SpineHNet is as follows: The input image is processed by the network in the first stage to output a prediction map of the auxiliary class; the centroid P A' of the auxiliary class is determined through the prediction map, and the center point C O of the probability heatmap H is calculated. The horizontal coordinate difference d A' and the vertical coordinate difference d x' between C y' and P 3 are obtained. A matrix H' with the same shape as the probability heatmap H is set. It is defined that M 3 is the coordinate set of all non-zero items in the probability heatmap H. If the coordinates in M x' added with d y' are still within the range of H, then the value at the corresponding coordinate position in H' is set to the value at the position obtained by adding the horizontal and vertical coordinates of this coordinate in H with d x' and d y' ; otherwise, the value at the corresponding coordinate position in H' is 0. After traversing M 3 , the repositioned heatmap H' is obtained; the H' processed by the Sigmoid function is concatenated with the input image in the channel dimension and then input into the network in the second stage for processing to obtain the prediction map of the target class.

[0018] Furthermore, in the step E, the model parameters of the neural network model SpineHNet are trained using the training set, and the performance of the neural network model SpineHNet is verified using the validation set.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] The semantic segmentation method of spinal region ultrasound images based on deep learning generates heatmaps that can reflect the prior knowledge between target classes in spinal region ultrasound images, and constructs a neural network model SpineHNet combined with the heatmaps, significantly improving the segmentation accuracy of the spinous process and transverse process regions. This method not only achieves accurate segmentation of spinal region ultrasound images, but also provides strong support for subsequent spinal model reconstruction and Cobb angle measurement work. Brief Description of the Drawings

[0021] Figure 1 This is the flowchart of the method.

[0022] Figure 2 This is the processing flowchart of the neural network model SpineHNet. Detailed Embodiments

[0023] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] The following describes the specific implementation of the present invention in detail with reference to specific embodiments.

[0025] Example 1: As Figure 1 shown, a method for semantic segmentation of spinal region ultrasound images based on deep learning provided by an embodiment of the present invention includes the following steps:

[0026] A. Construct a dataset: Use an ultrasound probe to move linearly from top to bottom along the back of a scoliosis patient to collect spinal region ultrasound videos. Convert the collected video data into a set of JPG format images. Select the images with relatively clear spinous process and transverse process regions, perform preprocessing such as cropping and resizing on them, and label the spinous process and transverse process regions in the images to obtain corresponding label maps, that is, target class labels, which are spinous process class labels and transverse process class labels respectively. For an image that has both a spinous process and a transverse process, calculate the centroid P 1 of the spinous process class in the image through its label map; divide the transverse process class region into left and right two regions according to the abscissa of P 1 , and calculate the centroids P 2 , P 3 of the two regions respectively through their respective label maps; with P 1 , P 2 and P 3Construct a triangular region for the vertices; map this triangular region to the original image, calculate the median m of the pixel values in this region of the original image; set the points with pixel values less than m and belonging to the background class in this region as the auxiliary class. Collect the ultrasound data of 30 patients, and finally obtain a dataset containing 3,136 ultrasound images. The image size in the dataset is 448 pix × 448 pix.

[0027] B. Dataset division: Divide the dataset into three parts: the training set, the validation set, and the test set, ensuring that the image data of a single case only exists in a single set. The number of images in the divided training set is 1,308, the number of images in the validation set is 259, and the number of images in the test set is 1,569.

[0028] C. Heat map generation: Set a zero matrix Z with the same size as the images in the dataset and the initial center point coordinates C O ; for a label map with auxiliary classes in a training set, the coordinate set M of non-background class points can be obtained 1 , calculate the centroid P of the auxiliary class region in this label map A , and calculate the horizontal coordinate difference d A between P O and C x , and the vertical coordinate difference d y , subtract the horizontal and vertical coordinates of each item in M 1 from d x and d y respectively to obtain a new set M 2 , map the coordinates of each item in M 2 to Z, and add 1 to the values at the coordinate positions that do not exceed the range of Z; for the matrix Z after the above operation is performed on all label maps with auxiliary classes in the training set, divide each value in the matrix Z by the total number of images with auxiliary class labels in the training set to obtain the probability heat map H.

[0029] D. Neural network model construction: Design a neural network model SpineHNet that can combine the probability heat map obtained in step C to improve the segmentation performance; the neural network model SpineHNet includes two processing stages, and both processing stages use a network of the Encoder-Decoder type as the backbone network. In this embodiment, the U-Net structure is used as the backbone network for both stages. Refer to Figure 2 , the processing flow of the neural network model SpineHNet is as follows: The input image is processed by the network in the first stage to obtain a prediction map of the auxiliary class; determine the centroid P of the auxiliary class through this prediction map A' , calculate the center point C of the probability heat map H O and the horizontal coordinate difference d A' between P x' and the vertical coordinate difference dy' ; Set up a matrix H' with the same shape as the probability heatmap H. For the values of each item in H', they can be determined through the set M of non-zero item coordinates in H 3 as follows: If the coordinates in M 3 and d x' 、d y' after addition are still within the range of H, then the value at this coordinate position in H' is the value at the position obtained by adding the horizontal and vertical coordinates of this coordinate in H and d x' 、d y' ; otherwise, the value at this coordinate position in H' is 0. After traversing M 3 , the relocated heatmap H' is obtained; Concatenate the H' processed by the Sigmoid function with the input image in the channel dimension, and then input it into the second-stage network for processing to obtain the prediction map of the target class.

[0030] E. Model training and verification: Use the training set to train the model parameters of the neural network model SpineHNet. The initial learning rate is set to 1e-5, the batch size is set to 4, the training epoch is set to 50 rounds. Use a loss function that mixes the cross-entropy loss function and the Dice loss function to calculate the loss. Use the RMSprop optimizer with a momentum of 0.9 and a weight decay of 1e-8 to optimize the model parameters. When training, use data augmentation methods such as randomly adjusting the brightness and randomly adjusting the contrast for the read images. Use the validation set to verify the performance of the neural network model SpineHNet.

[0031] F. Model testing: Use common performance indicators for segmentation tasks such as the Dice similarity coefficient and the Hausdorff distance to test the performance of the neural network model SpineHNet on the test set.

[0032] The above is only the preferred implementation manner of the present invention. It should be noted that for those skilled in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicability of the patent.

Claims

1. A semantic segmentation method of spinal region ultrasound images based on deep learning, characterized in that: The following steps are involved: Step A, constructing a data set: converting the collected ultrasound video data of the spinal region into a set of images in JPG format, selecting clear images of the spinous process and transverse process regions for preprocessing, and annotating the spinous process and transverse process regions in the images to obtain corresponding label maps, i.e., spinous process labels and transverse process labels, and using the spinous process labels and transverse process labels to generate auxiliary labels, constructing a spinal region ultrasound image data set, in which the image size is 448pix×448pix; Step B, divide the data set: divide the obtained data set into three parts, namely, training set, validation set and test set; Step C, generating a probability heat map: using various labels in the training set obtained in step B to generate a probability heat map that can reflect prior knowledge; Step D, building a neural network model: designing a neural network model SpineHNet that can be combined with the probability heat map obtained in step C to improve segmentation performance; Step E, model training and verification: Use the training set and verification set divided in step B to train and verify the neural network model SpineHNet designed in step D; Step F, model testing: Use the test set divided in step B to test the performance of the trained neural network model SpineHNet; In step C, the specific operation of generating the probability heat map is: setting an all-0 matrix Z with the same size as the image in the data set and the initial center point coordinate C O ; For each label image with auxiliary classes in the training set, obtain the coordinate set M1 of the non-background points in the image, and calculate the centroid P of the auxiliary class area in the label image A , and calculate P A With C O The horizontal axis difference d x , vertical coordinate difference d y , and the horizontal and vertical coordinates of each item in M1 are respectively x ,d y Subtract them to get a new coordinate set M2, map each coordinate in M2 to the matrix Z, and add 1 to the coordinate position that does not exceed the range of the matrix Z; divide each value in the matrix Z by the total number of images with auxiliary class labels in the training set to get the probability heat map H; In the step D, the neural network model SpineHNet includes two processing stages, and both processing stages use an Encoder-Decoder type network as the backbone network; The processing flow of the neural network model SpineHNet is as follows: the input image is processed by the first stage of network processing, and the prediction map of the auxiliary class is output; the centroid P of the auxiliary class is determined by the prediction map A' , calculate the initial center point coordinates C of the probability heat map H O With P A' The horizontal axis difference d x' , vertical coordinate difference d y' ; Set a matrix H' with the same shape as the probability heat map H, define M3 as the coordinate set of all non-zero items in the probability heat map H, if the coordinates in M3 are the same as d x' ,d y' If the added position is still within the range of H, the value of the corresponding coordinate position in H' is set to the sum of the horizontal and vertical coordinates of the coordinate in H and d x' ,d y' The value of the position after addition, otherwise the value of the corresponding coordinate position in H' is 0. After traversing M3, the relocated heat map H' is obtained; H' processed by the Sigmoid function is concatenated with the input image in the channel dimension and input into the second stage network for processing to obtain the prediction map of the target class.

2. The method for semantic segmentation of spinal region ultrasound images based on deep learning according to claim 1, characterized in that: In step A, the specific operation of generating auxiliary class labels is as follows: for an image containing both spinous processes and transverse processes, the centroid P1 of the spinous process class is calculated through the label map corresponding to the image; the transverse process class area is divided into two left and right areas with the horizontal coordinate of the centroid P1 as the dividing line, and the centroids P2 and P3 of the left and right areas are calculated through their respective label maps; a triangular area is constructed with P1, P2 and P3 as vertices, the triangular area is mapped to the original image, and the median m of the pixel value of the triangular area in the original image is calculated; and points in the triangular area whose pixel values ​​are less than m and are background class are set as auxiliary classes.

3. The method for semantic segmentation of spinal region ultrasound images based on deep learning according to claim 1, characterized in that: In step B, the image data of a single case only exists in a single set.

4. The method for semantic segmentation of spinal region ultrasound images based on deep learning according to claim 1, characterized in that: In step E, the model parameters of the neural network model SpineHNet are trained using the training set, and the performance of the neural network model SpineHNet is verified using the validation set.

Images