Deep learning-based spinal region ultrasound image semantic segmentation method

Abstract

The invention is suitable for the technical field of medical image processing, and provides a deep learning-based spinal region ultrasonic image semantic segmentation method, which comprises the following steps of: converting acquired spinal region ultrasonic video data of a scoliosis patient into a picture set in a JPG format, and preprocessing clear images in spinous process and transverse process regions; marking spinous process and transverse process regions to obtain labels used for semantic segmentation, generating auxiliary class labels by using the labels of spinous process classes and transverse process classes, and constructing a spine region ultrasonic image data set; dividing the data set into a training set, a verification set and a test set; generating a probability heat map reflecting prior knowledge; designing a specific neural network model structure; and training and testing the model on the data set so as to realize accurate segmentation of spinous process classes and transverse process classes in the image. According to the method, accurate segmentation of the ultrasonic image of the spine area is achieved, and powerful support is provided for subsequent spine model reconstruction and Cobb angle measurement work.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a semantic segmentation method of spinal region ultrasound images based on deep learning. Background Art

[0002] Scoliosis refers to a three-dimensional deformity of the spine with lateral curvature in the coronal plane, usually accompanied by axial rotation and sagittal plane abnormalities. Scoliosis not only affects the patient's body shape and posture, but also affects the patient's physiological function in severe cases. For patients with adolescent idiopathic scoliosis, wearing a brace can significantly reduce the probability of high-risk curves progressing to the surgical threshold. Therefore, it is very important to detect and intervene in scoliosis as early as possible. In clinical diagnosis, the Cobb angle is the standard for evaluating scoliosis. However, the current method of diagnosing scoliosis through X-Ray films may seriously endanger the patient's health due to long-term exposure to radiation. In addition to the X-Ray method, the MRI method can also achieve the imaging task of the spine and measure the Cobb angle of scoliosis, but because MRI scanning is time-consuming and expensive, it is difficult to popularize in clinical scoliosis diagnosis. In contrast, the ultrasound method is safer, more convenient and economical, and is very suitable for clinical scenarios where scoliosis patients need to undergo multiple examinations.

[0003] When diagnosing scoliosis using ultrasound, the spinal region in the ultrasound image must first be identified, and then the spinal model must be reconstructed based on the identification results to measure the Cobb angle. However, due to the limitations of the ultrasound imaging principle, there is a lot of interference information in the ultrasound image, and accurate segmentation of the spinal region becomes a major challenge. To this end, the present invention proposes a semantic segmentation method for spinal region ultrasound images based on deep learning, which is used to accurately segment the spinal region in ultrasound images. Summary of the invention

[0004] The purpose of the present invention is to provide a semantic segmentation method of spinal region ultrasound images based on deep learning, aiming to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A semantic segmentation method of spinal region ultrasound images based on deep learning, comprising the following steps: Step A, constructing a data set: converting the collected ultrasound video data of the spinal region into a set of images in JPG format, selecting clear images of the spinous process and transverse process regions for preprocessing, and annotating the spinous process and transverse process regions in the images to obtain corresponding label maps, i.e., spinous process labels and transverse process labels, and using the spinous process labels and transverse process labels to generate auxiliary labels, constructing a spinal region ultrasound image data set, in which the image size is 448pix×448pix; Step B, divide the data set: divide the obtained data set into three parts, namely, training set, validation set and test set; Step C, generating a probability heat map: using various labels in the training set obtained in step B to generate a probability heat map that can reflect prior knowledge; Step D, building a neural network model: designing a neural network model SpineHNet that can be combined with the probability heat map obtained in step C to improve segmentation performance; Step E, model training and verification: Use the training set and verification set divided in step B to train and verify the neural network model SpineHNet designed in step E; Step F, model testing: Use the test set divided in step B to test the performance of the trained neural network model SpineHNet.

[0006] Furthermore, in step A, the specific operation of generating auxiliary class labels is as follows: for an image containing both spinous processes and transverse processes, the centroid P of the spinous process class is calculated through the label map corresponding to the image. 1 ; Take the center of mass P 1 The horizontal axis is the dividing line, dividing the transverse process area into two left and right areas, and the centroid P of the left and right areas is calculated through their respective label maps. 2 , P 3 ; 1 , P 2 and P 3 Construct a triangular area for the vertex, map the triangular area to the original image, and calculate the median m of the pixel value of the triangular area in the original image; set the points in the triangular area with pixel values ​​less than m and belonging to the background class as auxiliary classes.

[0007] Furthermore, in step B, the image data of a single case only exists in a single set.

[0008] Furthermore, in step C, the specific operation of generating the probability heat map is: setting an all-0 matrix Z with the same size as the image in the data set and the initial center point coordinate C O ; For each label image with auxiliary classes in the training set, obtain the coordinate set M of the non-background points in the image 1 , calculate the centroid P of the auxiliary class area in the label map A , and calculate P A With C O The horizontal axis difference d x , vertical coordinate difference d y , M 1 The horizontal and vertical coordinates of each item are respectively x ,d y Subtract and get the new coordinate set M 2 , M2 The coordinates of each item in the matrix Z correspond to each item in the matrix Z. For the coordinate positions that do not exceed the range of the matrix Z, their values ​​are increased by 1. The values ​​of each item in the matrix Z are divided by the total number of images with auxiliary class labels in the training set to obtain the probability heat map H.

[0009] Furthermore, in step D, the neural network model SpineHNet includes two processing stages, and both processing stages use an Encoder-Decoder type network as a backbone network; The processing flow of the neural network model SpineHNet is as follows: the input image is processed by the first stage of network processing, and the prediction map of the auxiliary class is output; the centroid P of the auxiliary class is determined by the prediction map A' , calculate the center point C of the probability heat map H O With P A' The horizontal axis difference d x' , vertical coordinate difference d y' ; Set a matrix H' with the same shape as the probability heat map H, and define M 3 is the coordinate set of all non-zero items in the probability heat map H. If M 3 The coordinates in d x' ,d y' If the added position is still within the range of H, the value of the corresponding coordinate position in H' is set to the sum of the horizontal and vertical coordinates of the coordinate in H and d x' ,d y' The value of the position after addition, otherwise the value of the corresponding coordinate position in H' is 0. 3 After that, the relocated heat map H' is obtained; H' processed by the Sigmoid function is concatenated with the input image in the channel dimension and input into the second-stage network for processing to obtain the prediction map of the target class.

[0010] Furthermore, in step E, the training set is used to train the model parameters of the neural network model SpineHNet, and the validation set is used to validate the performance of the neural network model SpineHNet.

[0011] Compared with the prior art, the present invention has the following beneficial effects: This deep learning-based semantic segmentation method for spinal ultrasound images significantly improves the segmentation accuracy of the spinous process and transverse process regions by generating a heat map that reflects the prior knowledge between target classes in spinal ultrasound images and constructing a neural network model SpineHNet that combines the heat map. This method not only achieves accurate segmentation of spinal ultrasound images, but also provides strong support for subsequent spinal model reconstruction and Cobb angle measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flowchart of this method.

[0013] Figure 2 This is the processing flow chart of the neural network model SpineHNet. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0015] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0016] Embodiment 1, as Figure 1 As shown, an embodiment of the present invention provides a method for semantic segmentation of a spinal region ultrasound image based on deep learning, comprising the following steps: A. Constructing a data set: Use an ultrasound probe to move in a straight line from top to bottom along the back of a scoliosis patient to collect ultrasound videos of the spinal region. Convert the collected video data into a set of images in JPG format, select images with clearer spinous and transverse process regions, crop them, resize them, and annotate the spinous and transverse process regions in the image to obtain the corresponding label maps, i.e., target class labels, which are spinous process labels and transverse process labels, respectively. For an image with both spinous processes and transverse processes, calculate the centroid P of the spinous process class in the image through its label map. 1 ; 1 The horizontal axis divides the transverse process area into two areas, the left and right areas, and the centroids P of the two areas are calculated through their respective label maps. 2 , P 3 ; 1 , P 2 and P 3 Construct a triangular region for the vertex; correspond the triangular region to the original image, calculate the median m of the pixel value of the region in the original image; set the points in the region with pixel values ​​less than m and belonging to the background class as auxiliary classes. Collect ultrasound data of 30 patients, and finally obtain a data set containing 3136 ultrasound images, with an image size of 448pix×448pix.

[0017] B. Dataset division: The data set is divided into three parts: training set, validation set and test set. It is necessary to ensure that the image data of a single case exists only in a single set. The number of images in the training set is 1308, the number of images in the validation set is 259, and the number of images in the test set is 1569.

[0018] C. Heat map generation: Set a full zero matrix Z and initial center point coordinates C with the same size as the image in the dataset O; For a label image with auxiliary classes in the training set, the coordinate set M of the non-background class points can be obtained 1 , calculate the centroid P of the auxiliary class area in the label map A , and calculate P A With C O The horizontal axis difference d x , vertical coordinate difference d y , M 1 The horizontal and vertical coordinates of each item are respectively x ,d y Subtract and get a new set M 2 , M 2 The coordinates of each item in correspond to Z, and the values ​​at the coordinate positions that do not exceed the range of Z are added by 1; for all label images with auxiliary classes in the training set, the matrix Z after the previous operation is performed, and the values ​​of each item in the matrix Z are divided by the total number of images with auxiliary class labels in the training set to obtain the probability heat map H.

[0019] D. Neural network model construction: Design a neural network model SpineHNet that can be combined with the probability heat map obtained in step C to improve segmentation performance; the neural network model SpineHNet includes two processing stages, both of which use the Encoder-Decoder type network as the backbone network. This embodiment uses the U-Net structure as the backbone network of the two stages. Figure 2 The processing flow of the neural network model SpineHNet is as follows: the input image is processed by the first stage of network processing to obtain the prediction map of the auxiliary class; the auxiliary class centroid P is determined by the prediction map A' , calculate the center point C of the probability heat map H O With P A' The horizontal axis difference d x' , vertical coordinate difference d y' ; Set a matrix H' with the same shape as the probability heat map H. For the values ​​of each item in H', you can use the coordinate set M of the non-zero items in H 3 To determine, specifically: if M 3 The coordinates in d x' ,d y' If the added position is still within the range of H, then the value of the coordinate position in H' is the sum of the horizontal and vertical coordinates of the coordinate in H and d x' ,d y' The value of the position after addition, otherwise the value of the coordinate position in H' is 0. 3 After that, the relocated heat map H' is obtained; H' processed by the Sigmoid function is concatenated with the input image in the channel dimension, and then input into the two-stage network for processing to obtain the prediction map of the target class.

[0020] E. Model training and validation: The training set is used to train the model parameters of the neural network model SpineHNet. The initial learning rate is set to 1e-5, the batch size is set to 4, the training epoch is set to 50 rounds, and the loss function mixed with the cross entropy loss function and the Dice loss function is used to calculate the loss. The RMSprop optimizer with a momentum of 0.9 and a weight decay of 1e-8 is used to optimize the model parameters. During training, data enhancement methods such as random adjustment of brightness and random adjustment of contrast are used for the read images. The performance of the neural network model SpineHNet is verified using the validation set.

[0021] F. Model testing: Use the Dice similarity coefficient, Hausdorff distance and other common performance indicators of segmentation tasks on the test set to test the performance of the neural network model SpineHNet.

[0022] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A semantic segmentation method of spinal region ultrasound images based on deep learning, characterized in that: The following steps are involved: Step A, constructing a data set: converting the collected ultrasound video data of the spinal region into a set of images in JPG format, selecting clear images of the spinous process and transverse process regions for preprocessing, and annotating the spinous process and transverse process regions in the images to obtain corresponding label maps, i.e., spinous process labels and transverse process labels, and using the spinous process labels and transverse process labels to generate auxiliary labels, constructing a spinal region ultrasound image data set, in which the image size is 448pix×448pix; Step B, divide the data set: divide the obtained data set into three parts, namely, training set, validation set and test set; Step C, generating a probability heat map: using various labels in the training set obtained in step B to generate a probability heat map that can reflect prior knowledge; Step D, building a neural network model: designing a neural network model SpineHNet that can be combined with the probability heat map obtained in step C to improve segmentation performance; Step E, model training and verification: Use the training set and verification set divided in step B to train and verify the neural network model SpineHNet designed in step E; Step F, model testing: Use the test set divided in step B to test the performance of the trained neural network model SpineHNet; In step C, the specific operation of generating the probability heat map is: setting an all-0 matrix Z with the same size as the image in the data set and the initial center point coordinate C O ; For each label image with auxiliary classes in the training set, obtain the coordinate set M1 of the non-background points in the image, and calculate the centroid P of the auxiliary class area in the label image A , and calculate P A With C O The horizontal axis difference d x , vertical coordinate difference d y , and the horizontal and vertical coordinates of each item in M1 are respectively x d y Subtract them to get a new coordinate set M2, map each coordinate in M2 to the matrix Z, and add 1 to the coordinate position that does not exceed the range of the matrix Z; divide each value in the matrix Z by the total number of images with auxiliary class labels in the training set to get the probability heat map H; In the step D, the neural network model SpineHNet includes two processing stages, and both processing stages use an Encoder-Decoder type network as the backbone network; The processing flow of the neural network model SpineHNet is as follows: the input image is processed by the first stage of network processing, and the prediction map of the auxiliary class is output; the centroid P of the auxiliary class is determined by the prediction map A' , calculate the center point C of the probability heat map H O With P A' The horizontal axis difference d x' , vertical coordinate difference d y' ; Set a matrix H' with the same shape as the probability heat map H, define M3 as the coordinate set of all non-zero items in the probability heat map H, if the coordinates in M3 are the same as d x' d y' If the added position is still within the range of H, the value of the corresponding coordinate position in H' is set to the sum of the horizontal and vertical coordinates of the coordinate in H and d x' d y' The value of the position after addition, otherwise the value of the corresponding coordinate position in H' is 0. After traversing M3, the relocated heat map H' is obtained; H' processed by the Sigmoid function is concatenated with the input image in the channel dimension and input into the second stage network for processing to obtain the prediction map of the target class.

2. The method for semantic segmentation of spinal region ultrasound images based on deep learning according to claim 1, characterized in that: In step A, the specific operation of generating auxiliary class labels is as follows: for an image containing both spinous processes and transverse processes, the centroid P1 of the spinous process class is calculated through the label map corresponding to the image; the transverse process class area is divided into two left and right areas with the horizontal coordinate of the centroid P1 as the dividing line, and the centroids P2 and P3 of the left and right areas are calculated through their respective label maps; a triangular area is constructed with P1, P2 and P3 as vertices, the triangular area is mapped to the original image, and the median m of the pixel value of the triangular area in the original image is calculated; and points in the triangular area whose pixel values ​​are less than m and are background class points are set as auxiliary classes.

3. The method for semantic segmentation of spinal region ultrasound images based on deep learning according to claim 1, characterized in that: In step B, the image data of a single case only exists in a single set.

4. The method for semantic segmentation of spinal region ultrasound images based on deep learning according to claim 1, characterized in that: In step E, the model parameters of the neural network model SpineHNet are trained using the training set, and the performance of the neural network model SpineHNet is verified using the validation set.

Images