A cobb angle detection method based on residual network

Abstract

A Cobb angle detection method based on a residual network, first establishes a model to position the center of the vertebral body, and then tracks the four corner markers of the vertebral body through the learned angle offset on this basis, which can ensure the relative order of the feature points of the vertebral body. Compared with the method based on regression and segmentation, the results show that the method has certain advantages in Cobb angle measurement and feature point detection in low contrast and blurred X-ray images. The MICCAI and SKI10 data sets are used for result verification, and the experimental results show that the average precision of the algorithm is improved by 9%, and the detection speed can reach 30fps.

Description

TECHNICAL FIELD

[0001] The application relates to the field of target detection, and particularly relates to a Cobb angle detection method based on a residual network. BACKGROUND

[0002] With the continuous breakthrough and innovation of computer vision related technologies, the Cobb angle detection algorithm has made great progress in the field of medical image processing. Cobb angle is an angle index for measuring the degree of scoliosis, and is of great significance for the diagnosis and treatment of scoliosis. The traditional Cobb angle detection method needs to be measured manually by doctors, and has the problems of subjectivity and large error.

[0003] In recent years, the Cobb angle detection algorithm based on deep learning has been widely applied. In 2019, Padmini et al. proposed a Cobb angle measurement method based on deep learning and image analysis technology, which can effectively improve the accuracy and stability of Cobb angle measurement. In 2020, Li et al. proposed a Cobb angle measurement method based on deep learning, which combines image segmentation and regression technology, and can effectively improve the accuracy and stability of Cobb angle measurement. At present, the mainstream research adopts the target detection algorithm based on convolutional neural network, such as R-CNN, Fast R-CNN, Faster R-CNN, SSD, YOLO, YOLOv3, etc., among which the One-stage detection algorithm has become a research hotspot due to its high efficiency and precision advantage. In addition, there are also some methods based on segmentation and regression, such as U-Net and FCN. These algorithms can autonomously complete the learning of Cobb angle features and extract key information, and thus have strong robustness and accuracy. With the continuous innovation and development of technology, the Cobb angle detection algorithm will play a more important role in clinical application.

[0004] The existing technologies for solving the related problems are as follows:

[0005] Application No. CN202111053719.8, application name: a Cobb angle automatic measurement method based on spine hierarchical reconstruction, the application discloses a Cobb angle automatic measurement method based on spine hierarchical reconstruction, which comprises the following steps: (1) inputting the cross-sectional CT sequence image, pre-processing the DICOM original data, hierarchically reconstructing the superimposed sagittal graph, selecting the sagittal range of the spine by template matching method, and then selecting the middle one-third position interval to reconstruct the bone coronal graph; (2) based on the deep learning network, training the vertebral segmentation model based on the bone coronal graph, detecting and segmenting each vertebral body of the spine by using the model, and threshold processing and optimizing the intervertebral space; (3) extracting the center point of each vertebral body, and using a six-order polynomial to fit the center point set to obtain a spine curve; (4) calculating the curvature between each feature point in the spine curve, and iteratively calculating the Cobb angle.

[0006] It adopts hierarchical method, reconstructs the superimposed sagittal graph, selects the sagittal range of the spine by template matching method, and then selects the middle one-third position interval to reconstruct the bone coronal graph.

[0007] Based on the deep learning network, the vertebral segmentation model based on the bone coronal graph is trained, and the model is used to detect and segment each vertebral body of the spine and threshold processing and optimization of the intervertebral space. The present application is trained by residual network, which first locates the center of the vertebral body based on the model, and then tracks the four corner marks of the vertebral body through the learned angle offset. The detection adopts the feature point focusing detection method, and the comparison results show that the method has certain advantages in Cobb angle measurement and feature point detection in low contrast and fuzzy X-ray images.

[0008] Application No. CN202011568345.9, application name: a method and system for automatically measuring Cobb angle of scoliosis, which comprises: S1: collecting X-ray image; S2: extracting a plurality of vertebral boundary points of the X-ray image through a neural network unit; S3: generating vertebral boundary information corresponding to each vertebral body based on the plurality of vertebral boundary points; S4: extracting vertebral endplates corresponding to the vertebrae based on all the vertebral boundary information; S5: generating Cobb angle of scoliosis based on all the vertebral endplates. The present application automatically detects a plurality of vertebral boundary points of the X-ray image by using a neural network unit, extracts the vertebral endplates corresponding to the vertebrae through the vertebral boundary points, and generates Cobb angle of scoliosis through all the vertebral endplates, realizes automatic detection of Cobb angle of scoliosis, avoids the manual calibration and calculation steps in traditional Cobb angle, reduces the influence of human factors on the detection results, and effectively improves the detection accuracy of Cobb angle of scoliosis;

[0009] The neural network unit is used to extract a plurality of vertebral boundary points of the X-ray image, and based on the plurality of vertebral boundary points, vertebral boundary information corresponding to each vertebra is generated, and based on all the vertebral boundary information, a vertebral endplate corresponding to the vertebra is extracted. The center point heat map construction method directly locates the center of the vertebra, uses a convolutional neural network to learn to distinguish different vertebrae. After locating the vertebra, the convolutional layer is used to regress four corner markers of each vertebra, and the relative order of the feature points is more accurately maintained.

[0010] The application number CN202211005578.7, the application name is a kind of spinal Cobb angle measurement method, device, equipment and storage medium, it provides a kind of spinal Cobb angle measurement method, device, equipment and storage medium, by the image to be measured input spinal positioning model, to obtain the position coordinates of a plurality of target vertebrae;Wherein, the training data of the spinal positioning model is a plurality of training images and the rotation parameter of each training vertebra corresponding to the plurality of training images;Determine the spinal Cobb angle in the image to be measured by the position coordinates of the plurality of target vertebrae, improve the accuracy and efficiency of spinal vertebra detection, so as to improve the detection efficiency of spinal Cobb angle,

[0011] It adopts the image to be measured input spinal positioning model, to obtain the position coordinates of a plurality of target vertebrae;Wherein, the training data of the spinal positioning model is a plurality of training images and the rotation parameter of each training vertebra corresponding to the plurality of training images, the required parameters are many. The present application uses a method based on supervised learning to solve this problem, which learns more stable spinal features through convolutional layer, and combines deep features with shallow features using skip connection in convolutional layer, so as to utilize high-level semantic information and low-level detailed information, thereby eliminating the need for training a large number of parameters.

[0012] The application number CN202011070304.7, the application name is a kind of based on artificial intelligence-image recognition's spinal full length X-ray film measures and calculates spinal scoliosis Cobb angle's system and method, it includes spinal X-ray image pretreatment module, image deep learning network module, spinal X-ray vertebra image segmentation module, spline function fitting reconstruction module, Cobb angle measurement module and data labeling module, automatic measurement of Cobb angle software system interface module. The present application also discloses a kind of based on artificial intelligence-image recognition's spinal full length X-ray film measures and calculates spinal scoliosis Cobb angle's method. The above-mentioned system and method can truly realize the function of batch automatic measurement of spinal Cobb angle, the whole process from picture input to automatic measurement of Cobb angle does not need staff to carry out additional operation again, measurement is accurate and time-consuming is short, avoids the error introduced by manual or semi-manual method for spinal Cobb angle measurement, has better reliability.

[0013] It adopts a spine X-ray image preprocessing module, an image deep learning network module, a spine X-ray vertebra image segmentation module, a spline function fitting reconstruction module, a Cobb angle measurement module and data labeling module, an automatic Cobb angle measurement software system interface module, the required image input resolution is high, and the segmentation method positioning is not accurate enough. A new idea is proposed in this paper based on residual network, that is, each newly added layer should more easily contain the original function as one of its elements, thereby avoiding the above situation. That is, in the residual block, the input can be transmitted faster forward through the cross-layer data line, so that the new deep neural network can better learn new features, thereby improving the performance. SUMMARY

[0014] To solve the above problems, the application provides a Cobb angle detection method based on a residual network, which positions the center of the vertebra by establishing a model, and tracks the four corner marks of the vertebra based on the learned angle offset, so as to ensure the relative order of the feature points of the vertebra. Compared with the method based on regression and segmentation, the method has certain advantages in Cobb angle measurement and feature point detection in low-contrast and blurred X-ray images.

[0015] To achieve the above object, the technical scheme adopted by the application is:

[0016] A Cobb angle detection method based on a residual network, the specific steps are as follows:

[0017] Step 1, the user uploads the spine CT picture to the server;

[0018] Step 2, pre-processing the input image;

[0019] Step 3, the residual network training process is specifically represented as:

[0020] Let be a general constant, the pre-processed gray picture Real classification label The input layer of the residual network is composed of an unbiased convolution:

[0021]

[0022] Where the matrix The elements to be determined, the output feature Y0 is the input data of the following module:

[0023] Y p+1 =Y p +F(Y p , ω p ), p = 0, …, P-1

[0024] Where Y p Let F represent the input features of the p-th residual block, where F is the feature composition function, and ω is the feature composite function. p These are adjustable parameters that can be adjusted during training. Indicates the total number of network blocks;

[0025] Through matrix multiplication and error calculation, the deep features Y of the spine image are obtained. p In the output layer, it will be transformed into an m×1 dimensional vector, which is then normalized by softmax to become the classification prediction probability of the input image x in the residual network.

[0026]

[0027] in It is a weight matrix. Here, σ is the bias vector, and σ is the softmax function. The overall feature training process is optimized as follows:

[0028] Under conditions

[0029] Y0 = s(x), Y p+1 =Y p +F(Y p ω p p = 0, ..., p-1

[0030] Minimum loss function

[0031]

[0032] Where ||·|| is the distance measure, used to compare the true classification h(x) and the predicted classification M(Y) of image x in the training set Ω. p The differences between the vertebral feature point models were evaluated using a test dataset after multiple training iterations and parameter adjustments.

[0033] Step 4: Fix the size of the grayscale image processed in Step 2 to 1024×512, use the ResNet34 Conv1-5 trained in Step 3 to extract the high-level semantic features of the input image, and then use skip connections to combine the deep features with the shallow features.

[0034] Step 5: Use convolutional layers to construct a center point heatmap:

[0035] Object detection and pose joint localization are performed using a center-point heatmap. For each point k, its ground truth is an unnormalized two-dimensional Gaussian disk. This means that the size of the cone determines the size of the radius σ, and the construction parameters are optimized according to a variation of Focalloss;

[0036]

[0037] Step 6, using convolutional layers to build the center offset map:

[0038] The output feature map of the network is reduced compared to the input image, and a position (x, y) on the input image is mapped to a position on the reduced feature map where n is the down-sampling factor, after extracting the center points from the reduced feature map, these points are mapped to the original input image using the center offset, which is defined as:

[0039]

[0040] Step 7, using the symmetric mean absolute percentage error (SMAPE) to evaluate the accuracy of the measured Cobb angle:

[0041]

[0042] Step 8, by comparing the predicted vertebral angle marker position and the actual measured angle marker position to evaluate the accuracy of the algorithm, the average detection error is:

[0043]

[0044] As a further improvement of the present application, the step 2 preprocessing process adopts the weighted average method for image gray processing, assuming that the weights of the three channels are wr, wg, wb, for each pixel point (i, j), its values in R, G, B channels are R(i, j), G(i, j), B(i, j), and the weighted value of other indicators is k, then the gray value calculation formula of the pixel point is expressed as:

[0045] Gray(i, j) = wrR(i, j) + wgG(i, j) + wbB(i, j) + k.

[0046] Beneficial effects:

[0047] This invention discloses a Cobb angle detection method based on residual networks. The Cobb angle is an important indicator for doctors to assess the spinal rehabilitation of patients. However, in clinical practice, manually assessing the degree of scoliosis is very time-consuming and inaccurate. Automatic detection based on image processing is a future trend. However, current regression- or segmentation-based methods suffer from low recognition accuracy and low detection efficiency. Therefore, improving the accuracy and speed of Cobb angle detection is of great significance for clinical spinal rehabilitation assessment. This invention proposes a novel Cobb angle focusing detection method. First, a model is established to locate the vertebral body center. Then, based on this, the four corner markers of the vertebral body are tracked using learned angle offsets. This method ensures the relative order of vertebral feature points. Comparison with regression- and segmentation-based methods shows that this method has certain advantages in Cobb angle measurement and feature point detection in low-contrast and blurred X-ray images. The results are validated using the MICCAI and SKI10 datasets. Experimental results show that the average accuracy of the proposed algorithm is improved by 9%, and the detection speed can reach 30fps. Attached Figure Description

[0048] Figure 1 Residual network design diagram of this invention;

[0049] Figure 2 Comparison chart of regression, segmentation, and errors of this method;

[0050] Figure 3 Comparison of center point heat map, center offset map, and vertebral angle map;

[0051] Figure 3 (a) is the center point heatmap. Figure 3 (b) is the center offset diagram. Figure 3 (c) is the final vertebral trigonometric diagram;

[0052] Figure 4 Comparison chart of regression, segmentation, and detection methods using this approach. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments:

[0054] As a specific embodiment of the present invention, the present invention provides a Cobb angle detection method based on residual networks, wherein the residual network design diagram is as follows: Figure 1 As shown, the specific steps are as follows:

[0055] Step 1, the user uploads the spine CT picture to the server, collects multiple public medical data sets (MICCAI 2019, CSI 2019, SKI10, OAI-ZIB) a total of 2580 pictures as original image data for training test evaluation. All images are anteroposterior X-ray images of the spine, and each picture contains 17 thoracic and lumbar vertebrae.

[0056] Step 2, pre-process the input image, use the weighted average method to perform grayscale processing on the image, assume that the weights of the three channels are wr, wg, wb, for each pixel point (i, j), its values in R, G, B channels are R(i, j), G(i, j), B(i, j), and the weighted value of other indicators is k, then the gray value calculation formula of the pixel point can be expressed as:

[0057] Gray(i, j) = wrR(i, j) + wgG(i, j) + wbB(i, j) + k

[0058] Step 3, the residual network training process is specifically represented as:

[0059] Let be a general constant, the pre-processed grayscale image the true classification label The input layer of the residual network is composed of an unbiased convolution:

[0060]

[0061] Where the matrix The elements to be determined, the output feature Y0 is the input data of the following module:

[0062] Y p+1 = Y p +F(Y p , ω p ), p = 0, …, p-1

[0063] Where Y p represents the input feature of the pth residual block, F is a feature composite function, ω p is an adjustable parameter, which is adjusted during training, represents the total number of network blocks.

[0064] Through matrix multiplication and error operation, the deep features Y p of the spine image are converted into an m x 1 dimensional vector at the output layer, and the residual network input image x classification prediction probability is normalized by softmax

[0065]

[0066] Where is the weight matrix, is the bias vector, and σ is the softmax function. The overall feature training process can be optimized as: in the condition

[0067] Y0= s(x), Y p+1 = Y p +F(Y p , ω p ), p = 0,..., P-1

[0068] the minimum loss function

[0069]

[0070] where ||·|| is the distance measure, used to compare the difference between the true classification h(x) of the picture x in the training set Ω and the predicted classification M(Y p ) through multiple training iterations and parameter adjustment, the test data set is used to evaluate the vertebral feature point model.

[0071] Hardware uses NVIDIA 3060 GPU, based on VSCODE software, for code writing, model training, testing and evaluation. Pre-train the residual network ResNet34 on ImageNet. Use standard Gaussian distribution to initialize the weights of the network. In the algorithm, the input resolution of each picture is corrected to 1024x512, so that the output resolution is fixed to 256x128. Standard data enhancement algorithm is adopted, including cropping, random expansion, brightness distortion and contrast method, which reduces the overfitting of the model. Use Adam optimizer to optimize the network, set the initial learning rate to 2.5x10 -4 . Train the residual for 1000 cycles, and stop training when the validation loss does not decrease significantly.

[0072] Step 4, fix the size of the gray image processed in step 2 to 1024x512. Use ResNet34 Conv1-5 trained in step 3 to extract high-level semantic features of the input image. Then, use skip connection to combine deep features with shallow features, so that high-level semantic information and low-level fine details can be used.

[0073] Step 5, use convolutional layers to build center point heat map, as shown in Figure 3 (a):

[0074] Use the center point heat map for object detection and joint pose positioning. For each point k, its ground truth is an unnormalized two-dimensional Gaussian disc, as shown in the figure x, which can be represented as The size of the vertebral body determines the size of the radius σ. According to the variant of Focal loss to optimize the construction parameters.

[0075]

[0076] where parameter i indexes each location of the feature map. N is the total number of locations on the feature map, p i and y i represent the predicted and ground truth values, respectively. In the preliminary training, parameters a = 2 and b = 4 were set.

[0077] Step 6, a center offset map was constructed using the convolutional layer, where the center offset map is shown in Figure 3 (b).

[0078] The output feature map of the network is down-sampled compared to the input image. This not only saves the computational cost, but also alleviates the imbalance problem between positive and negative points due to the reduction of output resolution. Therefore, a location (x, y) on the input image is mapped to a location on the down-sampled feature map, where n is the down-sampling factor. After extracting the center points from the down-sampled feature map, these points are mapped to the original input image using the center offset. The center offset is defined as:

[0079]

[0080] Step 7, the accuracy of the measured Cobb angle was evaluated using the Symmetric Mean Absolute Percentage Error (SMAPE):

[0081]

[0082] where i represents the three Cobb angles of the proximal thoracic (PT), main thoracic (MT), and thoracolumbar (TL), j represents the jth image, N represents the total number of detected images. a and b represent the Cobb angles predicted by the algorithm and the actual measured Cobb angles, respectively. The SMAPE of the PT, MT, and TL regions were also calculated, represented as SMAPE PT , SMAPE MT , and SMAPE TL , respectively.

[0083] Step 8, the accuracy of the algorithm was evaluated by comparing the predicted vertebrae landmark positions and the actually measured landmark positions, where the final obtained vertebrae landmark map is shown in Figure 3 (c). The average detection error is:

[0084]

[0085] where d i = (d x，i , d y，i ) and g i = (g x，i , gy，i ) are the predicted and actual landmark positions respectively, and M is the total number of landmarks in the whole test image.

[0086] The method of the present application is compared with regression-based methods and segmentation-based methods. Qualitative and quantitative results are shown in Figure 2 and Figure 4 The image input resolution of regression-based methods is smaller due to the large number of parameters in the fully connected layers and the limited GPU memory. The same data augmentation and training approach is used for the segmentation-based methods, which obtain the accurate landmark positions by decoding on the corner points of the minimum bounding rectangle of the vertebral body segmentation mask.

[0087] Table 1 Comparison of detection results of each method

[0088]

[0089] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application in any other form, and any modification or equivalent change made according to the technical essence of the present application still falls within the scope of the present application.

Claims

1. A Cobb angle detection method based on a residual network, characterized in that, The specific steps are as follows: Step 1, the user uploads the spine CT picture to the server; Step 2, pre-processing the input image; Step 3, the residual network training process is specifically represented as: Let is a constant, pre-processed grayscale image true classification label unbiased convolutional composition of the residual network input layer: where the matrix The elements to be determined in the matrix are the inputs of the following module: Y p+1 = Y p + F(Y p , ω p ), p = 0,..., p - 1 wherein Y p represents the input feature of the pth residual block, F is a feature composite function, ω p is an adjustable parameter, which is adjusted during training, represents the total number of network blocks; Through matrix multiplication and error operation, the deep features Y of the spine image p In the output layer, it will be converted into an m x 1-dimensional vector, and normalized by softmax to predict the probability of classifying the input image x of the residual network. wherein is a weight matrix, is a bias vector, σ is a softmax function, and the overall feature training process is optimized as: Under the condition Y0= S(x), Y p+1 = Y p + F(Y p , ω p ), p = 0,..., P - 1 The minimum loss function where ||·|| is a distance measure used to compare the difference between the true classification h(x) of the picture x in the training set Ω and the predicted classification M(Y p ) through multiple training iterations and parameter adjustments, the vertebral feature point model is evaluated using the test data set; Step 4, fix the size of the gray image processed in step 2 to 1024x512, use ResNet34Conv1-5 trained in step 3 to extract the high-level semantic features of the input image, then use skip connection to combine deep features with shallow features; Step 5, use convolution layer to build center point heat map: Object detection and pose joint localization using center point heat map, for each point k, its ground truth is an unnormalized two-dimensional Gaussian disc, denoted as The size of the disc determines the size of the radius σ, and the construction parameters are optimized according to the variant of Focal loss; Step 6, use convolution layer to build center offset map: The output feature map of the network is down-sampled compared to the input image, a location (x, y) on the input image is mapped to a location on the down-sampled feature map where n is the down-sampling factor, after extracting the center points from the down-sampled feature map, these points are mapped to the original input image using a center offset defined as: Step 7, use symmetric mean absolute percentage error (SMAPE) to evaluate the accuracy of the measured Cobb angle: Step 8, by comparing the predicted vertebra angle mark position and the actually measured angle mark position, the accuracy of the algorithm is evaluated, and the average detection error is:

2. The Cobb angle detection method based on residual network according to claim 1, wherein, The pre-processing process of step 2 uses weighted average method to perform gray processing on the image, assuming that the weights of the three channels are wr, wg, wb, for each pixel point (i, j), its values in R, G, B channels are R(i, j), G(i, j), B(i, j), and the weighted value of other indicators is k, then the gray value calculation formula of the pixel point is represented as: Gray(i, j) = wrR(i, j) + wgG(i, j) + wbB(i, j) + k.

Images