Spine Cobb angle prediction method based on centrum segmentation

Abstract

The invention provides a spine Cobb angle prediction method based on centrum segmentation. The spine Cobb angle prediction method comprises the following steps: acquiring an X-ray image of a spine to be measured; inputting the X-ray image into an encoder to obtain a feature map of the X-ray image; inputting the feature map into a spine decoder to obtain a spine segmentation mask; inputting the feature map into a vertebra decoder to obtain a vertebra segmentation mask; performing center line prediction according to the spine segmentation mask to obtain center line representation of the spine; performing feature extraction according to the vertebra segmentation mask to obtain a target feature vector; the target feature vector is used for representing the form and structure information of each vertebra; and according to the center line representation and the target feature vector, predicting to obtain a target Cobb angle corresponding to the X-ray image of the spine to be measured. According to the method, the overhigh dependence on data quality can be reduced, the prediction capability under various image qualities is improved, the time consumption of manual operation is reduced, the accuracy is improved, and the robustness and interpretability of Cobb angle prediction are enhanced.

Description

Technical Field

[0001] The present application relates to the computer field, and more specifically, to a spinal Cobb angle prediction method based on vertebral segmentation in the computer field. Background Art

[0002] Scoliosis refers to the deviation of the spine of the back from the central axis of the trunk to one side or both sides, forming a "C" or "S" shaped curve. Severe scoliosis not only affects the patient's physical appearance, but may also cause cardiopulmonary dysfunction, bringing serious psychological and economic burdens. Therefore, it is very important to choose an appropriate method for the diagnosis and evaluation of scoliosis. The Cobb angle is usually used for clinical evaluation. The vertebrae (end vertebrae) with the largest inclination at the top and bottom of the spinal curve are identified in the anteroposterior (AP View, AP) X-ray film, and then the Cobb angle is measured by the angle of the line drawn between the upper endplate of the upper end vertebra and the lower endplate of the lower end vertebra. However, manual measurement of the Cobb angle is time-consuming, cumbersome, and relies on the judgment of the operator, resulting in large differences between and within observers. Therefore, it is necessary to automatically estimate the Cobb angle to improve interpretability.

[0003] In recent years, the research on computer-aided imaging diagnosis has become more and more abundant, and some traditional methods have been produced that extract features through simple image processing technology and then automatically estimate the Cobb angle. For example, a DNN (Deep Neural Network) structure based on Hough transform automatically calculates the slope of the vertebra to obtain the Cobb angle. There are also some other traditional methods that use active contour models, customized filters, and charged particle-based curve fitting models to segment the spine mask to calculate the Cobb angle. Although these methods have alleviated the internal variability caused by manual measurement to a certain extent and improved the efficiency of prediction, they are sensitive to noise, cannot obtain accurate results when the spine boundary is not clear, and the amount of calculation is also very large.

[0004] With the development of deep learning technology, some methods based on some intermediate representations of the spine have been developed to predict the Cobb angle. These methods include mask-based methods, vertebra-based methods, and centerline-based methods. The mask-based method directly regresses the Cobb angle from the segmented mask and cannot explain the location of scoliosis. The vertebra-based method predicts the Cobb angle by predicting the position of each vertebral corner point and reconstructing the shape of each vertebra. Although this method can improve the interpretability to a certain extent, it often fails to obtain accurate results when faced with low-quality or blurred images where the human eye cannot accurately identify the position of each vertebra, and requires expensive real value annotations for training. The centerline-based method obtains the center point by predicting the contour of the spine boundary, and then fits the center point to the center line, and estimates the Cobb angle by calculating the slope of the center line. Although this method improves both robustness and interpretability, choosing a suitable fitting method is crucial for the prediction of the final result. Once the wrong fitting method is selected, it may lead to problems such as inaccurate prediction, sensitivity to noise, and inconvenience for training.

[0005] Although the above methods can obtain relatively accurate estimates, their robustness is still limited and they perform poorly for low-quality images. Summary of the invention

[0006] The present application provides a spinal Cobb angle prediction method based on vertebral segmentation, which can reduce the excessive dependence on data quality, improve the prediction ability under various image qualities, reduce the time consumption of manual operation and improve the accuracy, and enhance the robustness and interpretability of Cobb angle prediction.

[0007] In a first aspect, a method for predicting the Cobb angle of the spine based on vertebral segmentation is provided, the method comprising: obtaining an X-ray image of the spine to be tested; inputting the X-ray image into an encoder to obtain a feature map of the X-ray image; inputting the feature map into a spine decoder to obtain a spine segmentation mask; wherein the spine segmentation mask reflects the overall contour of the spine; inputting the feature map into a vertebral decoder to obtain a vertebral segmentation mask; wherein the vertebral segmentation mask reflects the shape information of each vertebra; performing centerline prediction based on the spine segmentation mask to obtain a centerline representation of the spine; performing feature extraction based on the vertebral segmentation mask to obtain a target feature vector; wherein the target feature vector is used to characterize the morphology and structural information of each vertebra; and predicting a target Cobb angle corresponding to the X-ray image of the spine to be tested based on the centerline representation and the target feature vector.

[0008] The above technical solution proposes an innovative dual-branch structure that combines vertebrae and spine centerline-based methods for Cobb angle prediction. This dual-branch structure allows two independent branches to correct errors with each other, and can still generate accurate and highly interpretable results when processing low-quality or blurred images, solving the problem that the existing technology is susceptible to interference when the image quality is poor. In addition, the same encoder and different decoder structures are used to simultaneously generate two different types of masks: spine segmentation mask and vertebrae segmentation mask. The spine segmentation mask is used for centerline prediction, while the vertebrae segmentation mask is used for angle regression. This design greatly improves the efficiency of multi-task segmentation processing, while ensuring the segmentation accuracy under different tasks, optimizing resource utilization and computational efficiency. The robustness of the prediction is enhanced by fusing the vertebrae regression features (i.e., target feature vectors) with the centerline regression features (i.e., centerline representations). The feature extraction result of the vertebrae segmentation mask regression, i.e., the target feature vector, is used to strengthen the features obtained by the centerline regression. By combining them together, the global geometric information and local structural information of the spinal curvature can be more effectively captured, thereby improving the stability and accuracy of the final Cobb angle prediction.

[0009] In combination with the first aspect, in some possible implementations, feature extraction is performed based on the vertebral segmentation mask to obtain a target feature vector, including: inputting the vertebral segmentation mask into a pre-trained DenseNet to obtain intermediate features; wherein the intermediate features can capture multi-level information of vertebral morphology and structure; inputting the intermediate features into a first multi-layer perceptron, extracting a first high-dimensional feature vector in the penultimate layer of the first multi-layer perceptron, and using the first high-dimensional feature vector as the target feature vector.

[0010] In combination with the first aspect, in some possible implementations, a target Cobb angle corresponding to the X-ray image of the spine to be tested is predicted based on the centerline representation and the target feature vector, including: performing a curve slope analysis based on the centerline representation to obtain a first Cobb angle corresponding to the X-ray image of the spine to be tested; resampling based on the centerline representation to obtain a resampling center point; inputting the resampling center point into a second multi-layer perceptron, and extracting a second high-dimensional feature vector in the penultimate layer of the second multi-layer perceptron; concatenating the first high-dimensional feature vector and the second high-dimensional feature vector to obtain a joint feature vector; wherein the joint feature vector integrates the global geometric information of the spine centerline and the local structure and morphology information of the vertebrae; inputting the joint feature vector into a third multi-layer perceptron to output a regression predicted second Cobb angle corresponding to the X-ray image of the spine to be tested; and performing weighted processing based on the first Cobb angle and the second Cobb angle to obtain the target Cobb angle corresponding to the X-ray image of the spine to be tested.

[0011] In combination with the first aspect, in some possible implementations, the loss function Loss used by the vertebral decoder during training is vertebra_mask for:

[0012] Loss vertebra_mask =Loss vertebra_BCE +Loss vertebra_SSIM +Loss vertebra_IOU +Loss vertebra

[0013] Among them, Loss vertebra_BCE is the binary cross entropy loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask. vertebra_SSIM is the structural similarity index loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask, Loss vertebra_IOU is the intersection-over-union loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask, Loss vertebra is the vertebra prediction loss function used to measure the difference between the predicted value of the vertebra and the true value of the vertebra.

[0014] In combination with the first aspect, in some possible implementations, Loss vertebra The expression is as follows:

[0015]

[0016] Where N represents the number of pixels in the image; C represents the total number of vertebrae; c represents the index of the vertebrae, traversing from the first vertebra to the last vertebra; i represents the index of the pixel, traversing from the first pixel to the last pixel; and are the predicted value and true value of a pixel in the vertebral segmentation mask, respectively.

[0017] In combination with the first aspect, in some possible implementations, the loss function Loss used by the spine decoder during training spine_mask for:

[0018] Loss spine_mask =Loss spine_BCE +Loss spine_SSIM +Loss spine_IOU

[0019] Among them, Loss spine_BCE is the binary cross entropy loss function used to measure the difference between the predicted spine mask and the true spine mask. spine_SSIM is the structural similarity index loss function used to measure the difference between the predicted spine mask and the true spine mask. spine_IOUis the intersection-over-union loss function used to measure the difference between the predicted spine mask and the true spine mask.

[0020] In combination with the first aspect, in some possible implementations, after the feature map is input into the spine decoder to obtain the spine segmentation mask, the method also includes: inputting the spine segmentation mask into a fine segmentation model to remove noise in the spine segmentation mask through the fine segmentation model to obtain a fine mask corresponding to the spine segmentation mask; performing centerline prediction based on the spine segmentation mask to obtain a centerline representation of the spine, including: performing centerline prediction based on the fine mask corresponding to the spine segmentation mask to obtain a centerline representation of the spine.

[0021] In combination with the first aspect, in some possible implementations, centerline prediction is performed based on a fine mask corresponding to a spinal segmentation mask to obtain a centerline representation of the spine, including: predicting an initial center point from the fine mask; predicting Hermite spline parameters based on the initial center point predicted from the fine mask; constructing a Hermite curve based on the Hermite spline parameters, and using the Hermite curve as the centerline representation of the spine.

[0022] In a second aspect, an electronic device is provided, comprising: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, so that the hybrid electronic device executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0023] According to a third aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method according to the first aspect or any possible implementation of the first aspect.

[0024] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic flow chart of a spinal Cobb angle prediction method based on vertebral segmentation provided in an embodiment of the present application;

[0026] Figure 2 It is a schematic diagram of an implementation framework of a spinal Cobb angle prediction method based on vertebral segmentation provided in an embodiment of the present application;

[0027] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0030] The present application studies the Cobb angle prediction method in the prior art and finds that the prior art mainly has the following problems:

[0031] (1) It takes a lot of time and effort for clinicians to manually select vertebrae to estimate the Cobb angle, and there are differences within and between observers. It also requires rich experience to carry out these annotations at the same time. For beginners, it is not easy to successfully annotate the correct angle. Therefore, the introduction of artificial intelligence technology for auxiliary annotation and prediction of Cobb angle can improve efficiency and interpretability.

[0032] (2) Existing technologies are mainly divided into traditional methods and data-driven methods. Although traditional methods can automatically identify the vertebra with the largest slope to estimate the Cobb angle, the end vertebra still needs to be manually selected. Therefore, the results obtained by this method still have internal variability, and traditional methods are usually based on simple image processing techniques and are easily affected by noise. Data-driven methods mainly estimate the Cobb angle based on some intermediate representations of the spine. This method often only focuses on a certain intermediate representation without considering the overall structural information. For example, the method of fitting the center line alone may lead to incorrect prediction of the center point due to unclear segmentation of the spine contour, which further leads to problems in curve fitting; the method of predicting corner points alone cannot accurately identify the position of the corner points when faced with low-quality images that cannot be accurately recognized by the human eye, which will produce very poor results.

[0033] (3) The accuracy of existing technologies depends on the quality of the dataset. When faced with high-quality datasets, these models can obtain good results. However, when faced with low-quality or blurred images, their performance is often poor and not universal.

[0034] (4) Although existing techniques can obtain relatively accurate estimates, their robustness and interpretability are still limited because small errors in vertebrae prediction or centerline fitting may lead to serious errors in Cobb angle prediction.

[0035] In order to at least solve the above technical problems, the embodiment of the present application provides a spinal Cobb angle prediction method based on vertebral segmentation, which uses artificial intelligence technology to solve the problems of existing methods such as high data quality requirements, cumbersome manual labeling, weak robustness and interpretability, and inaccurate results in special situations. It improves the efficiency of prediction and reduces errors in clinical practice.

[0036] Considering that existing methods often fail to obtain accurate results when processing low-quality images, their clinical applicability is limited. The embodiments of the present application combine multiple intermediate representations to reduce excessive dependence on data quality and improve the prediction ability under various image qualities.

[0037] Considering that the traditional manual operation method is not only time-consuming and labor-intensive, but also has limited accuracy, the embodiment of the present application uses artificial intelligence technology to assist Cobb angle prediction, reduce the workload of doctors, and provide more interpretable results, thereby reducing the time consumption of manual operation and improving accuracy.

[0038] Considering that although the existing methods can provide relatively accurate estimates, they perform poorly in certain special cases and lack sufficient robustness and interpretability, which limits their clinical reference value. The embodiment of the present application aims to improve the robustness and interpretability of spinal curvature estimation through an improved Cobb angle prediction method, thereby providing more reliable results in the face of special cases.

[0039] Figure 1 : is a schematic flow chart of a spinal Cobb angle prediction method based on vertebral segmentation provided in an embodiment of the present application:

[0040] For example, Figure 1 As shown, the method includes:

[0041] Step 101: Acquire an X-ray image of the spine to be tested.

[0042] Step 102: Input the X-ray image into the encoder to obtain a feature map of the X-ray image.

[0043] Step 103: Input the feature map into the spine decoder to obtain a spine segmentation mask; wherein the spine segmentation mask reflects the overall contour of the spine.

[0044] Step 104: Input the feature map into the vertebra decoder to obtain a vertebra segmentation mask; wherein the vertebra segmentation mask reflects the shape information of each vertebra.

[0045] Step 105: Perform centerline prediction based on the spine segmentation mask to obtain a centerline representation of the spine.

[0046] Step 106: Perform feature extraction based on the vertebra segmentation mask to obtain a target feature vector; wherein the target feature vector is used to characterize the morphology and structural information of each vertebra.

[0047] Step 107: predicting a target Cobb angle corresponding to the X-ray image of the spine to be measured based on the centerline representation and the target feature vector.

[0048] exist Figure 1 In the illustrated embodiment, an innovative dual-branch structure is proposed, which combines methods based on vertebrae and spine centerlines for Cobb angle prediction. This dual-branch structure allows two independent branches to correct errors with each other, and can still generate accurate and highly interpretable results when processing low-quality or blurred images, solving the problem that the prior art is susceptible to interference when the image quality is poor. In addition, the same encoder and different decoder structures are used to simultaneously generate two different types of masks: spine segmentation mask and vertebrae segmentation mask. The spine segmentation mask is used for centerline prediction, while the vertebrae segmentation mask is used for angle regression. This design greatly improves the efficiency of multi-task segmentation processing, while ensuring the segmentation accuracy under different tasks, optimizing resource utilization and computational efficiency. The robustness of the prediction is enhanced by fusing the vertebrae regression features (i.e., the target feature vector) with the centerline regression features (i.e., the centerline representation). The feature extraction result of the vertebrae segmentation mask regression, i.e., the target feature vector, is used to strengthen the features obtained by the centerline regression. By combining them together, the global geometric information and local structural information of the spinal curvature can be more effectively captured, thereby improving the stability and accuracy of the final Cobb angle prediction.

[0049] Below Figure 1 The specific implementation method of each step in the embodiment shown is described as follows:

[0050] In step 101, the X-ray image of the spine to be measured is the X-ray image of the spine for which Cobb angle prediction is to be performed. Exemplarily, the X-ray image of the patient's spine can be derived from the hospital's radiology department information system. The X-ray image is usually digitized and can be directly used for subsequent processing.

[0051] In step 102, the X-ray image is input into an encoder to obtain a feature map of the X-ray image. For example, a convolutional neural network can be used as an encoder, such as the classic VGG (Visual Geometry Group), ResNet (Residual Network) or a network designed specifically for medical images. The encoder converts the input X-ray image into a set of high-level feature maps that capture important patterns and structures in the image. The feature maps are then input into two different decoders, namely, a spine decoder in step 103 and a vertebral decoder in step 104.

[0052] In step 103, the feature map is input into the spine decoder to obtain a spine segmentation mask; wherein the spine segmentation mask reflects the overall contour of the spine and is used for subsequent spine centerline prediction.

[0053] For example, the loss function Loss used by the spine decoder during training spine_mask for:

[0054] Loss spine_mask =Loss spine_BCE +Loss spine_SSIM +Loss spine_IOU

[0055] Among them, Loss spine_BCE is the Binary Cross-Entropy (BCE) loss function used to measure the difference between the predicted spine mask and the true spine mask. spine_SSIM is the structural similarity index loss function used to measure the difference between the predicted spine mask and the true spine mask. spine_IOU is the intersection-over-union loss function used to measure the difference between the predicted spine mask and the true spine mask.

[0056] Specifically, measuring the difference between the predicted spine mask and the true spine mask is to ensure that the spine decoder can accurately capture the overall spine contour, thereby providing precise input for subsequent spine centerline prediction and reducing errors.

[0057] For example, the above Loss spine_BCE The expression is as follows:

[0058]

[0059] Where N represents the number of pixels in the sample image. If the sample image format is 512x256, the value of N is 131072. i represents the index of each pixel, from 0 to 131071. y1(i) corresponds to the true value of pixel i in the spine segmentation mask. For this pixel i, if the pixel i belongs to the spine region, y1(i) is 1; otherwise, it is 0. is the predicted probability of whether each pixel belongs to the spine, The closer the value is to 1, the greater the probability that the pixel i belongs to the spine area, and the closer the value is to 0, the greater the probability that the pixel i belongs to the background.

[0060] For example, the above Loss spine_SSIM The expression is as follows:

[0061]

[0062] Loss spine_SSIM =1-SSIM(mask1,GT1)

[0063] in, and are the average values ​​of all pixels of the predicted spine mask mask1 and the true spine mask GT1, and are the variances of all pixels of the predicted spine mask mask1 and the true spine mask GT1, σ mask1,GT1 is the covariance between the predicted spine mask mask1 and the true spine mask GT1, c1 and c2 are constants used to prevent the denominator from being zero, and are usually chosen to be a very small constant value.

[0064] During training, the goal of the SSIM loss is to maximize the SSIM value. To calculate the loss, (1-SSIM) is usually used as the loss function to minimize. In this way, the network will be optimized in the direction of making the predicted spine mask and the true spine mask structure similar.

[0065] For example, the above Loss spine_IOU The expression is as follows:

[0066]

[0067] Loss spine_IOU =1-IOU(mask1, GT1)

[0068] Among them, mask1 and GT1 are the predicted spine mask and the true spine mask respectively, mask1 ∩ GT1 represents the intersection of mask1 and GT1, and the intersection is the overlapping part of the two regions, namely mask1 and GT1. Mask1 ∪ GT1 represents the union of maskl and GTl, and the union is the merged part of the two regions, namely mask1 and GT1. Because the larger the value of IOU(mask1, GT1), the more overlap there is between the predicted spine mask mask1 and the true spine mask GT1, and the better the segmentation effect, therefore, (1-IOU) is used as the loss function for minimization.

[0069] In summary, during the training of the spine decoder, BCE, SSIM, and IOU losses are used to measure the difference between the predicted spine mask and the true spine mask.

[0070] In step 104, the feature map is input into the vertebra decoder to obtain a vertebra segmentation mask, wherein the vertebra segmentation mask reflects the shape information of each vertebra. For example, the vertebra segmentation mask shows the shape information of 17 vertebrae, which is used for subsequent regression and angle prediction.

[0071] Exemplarily, the loss function Loss used by the vertebral decoder during training vertebra_mask For: Loss vertebra_mask =Loss vertebra_BCE +Loss vertebra_SSIM +Loss vertebra_IOU +Loss vertebra

[0072] Among them, Loss vertebra_BCE is the binary cross entropy loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask. vertebra_SSIM is the structural similarity index loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask, Loss vertebra_IOU is the intersection-over-union loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask, Loss vertebra is the vertebra prediction loss function used to measure the difference between the predicted value of the vertebra and the true value of the vertebra.

[0073] During the training of the vertebra decoder, in order to further optimize the effect of the vertebra segmentation mask, in addition to the binary cross entropy (BCE), structural similarity (SSIM) and intersection over union (IOU) losses, a vertebra prediction loss function Loss is introduced. vertebra. This loss function is used to measure the difference in coordinates between the vertebral area predicted by the model and the actual marked area, thereby improving the accuracy and reliability of the segmentation results. In the vertebrae prediction loss function, the predicted value of the vertebra represents the vertebral area coordinate information generated by the vertebrae decoder, while the true value comes from the vertebral area coordinates pre-marked in the data set. In order to more comprehensively evaluate the segmentation quality, this embodiment is no longer limited to the prediction of vertebral vertebrae (ie, "corner points"), but uses the area range of the entire vertebra as the main basis for judgment. This method avoids reliance on a single feature point and enables the model to learn the overall characteristics of the vertebrae area more comprehensively.

[0074] For example, Loss vertebra The expression is as follows:

[0075]

[0076] Where N represents the number of pixels in the sample image; C represents the total number of vertebrae; c represents the index of the vertebrae, traversing from the first vertebra to the last vertebra; i represents the index of the pixel, traversing from the first pixel to the last pixel; and are the predicted value and true value of a pixel in the vertebral segmentation mask, respectively. i represents a pixel, and c represents whether the pixel corresponds to the vertebral area or the background area.

[0077] For example, Loss vertebra_BCE The expression is as follows:

[0078]

[0079] Where N represents the number of pixels in the sample image. If the format of the sample image is 512×256, the value of N is 131072. i represents the index of each pixel, from 0 to 131071. y2(i) corresponds to the true value of pixel i in the vertebral segmentation mask. For this pixel i, if the pixel i belongs to the vertebral region, y2(i) is 1; otherwise, it is 0. is the predicted probability of each pixel belonging to a vertebra, The closer the value is to 1, the greater the probability that the pixel i belongs to the vertebral area, and the closer the value is to 0, the greater the probability that the pixel i belongs to the background.

[0080] For example, the above Loss vertebra_SSIM The expression is as follows:

[0081]

[0082] Loss vertebra_SSIM =1-SSIM(mask2,GT2)

[0083] in, and are the average values ​​of all pixels of the predicted vertebrae mask mask2 and the true vertebrae mask GT2, respectively. and are the variances of all pixels of the predicted vertebrae mask mask2 and the true vertebrae mask GT2, σ mask2,GT2 is the covariance between the predicted vertebra mask mask2 and the true vertebra mask GT2, c1 and c2 are constants used to prevent the denominator from being zero, and are usually chosen to be a very small constant value.

[0084] In the specific implementation, the segmentation mask of each vertebra can be distinguished, that is, for each individual vertebra, the SSIM value of its predicted mask and the true mask is calculated separately, and the SSIM loss of these segmented areas is weighted or averaged as part of the overall loss. This block processing can perform fine-grained optimization for each vertebra, ensuring that the model can more accurately capture the detailed features of different vertebrae.

[0085] During the training process, the goal of the SSIM loss is to maximize the SSIM value. To calculate the loss, (1-SSIM) is usually used as the loss function to minimize. In this way, the network will be optimized in the direction of making the predicted vertebral mask and the true vertebral mask structure similar.

[0086] For example, the above Loss vertebra_IOU The expression is as follows:

[0087]

[0088] Loss vertebra_IOU =1-IOU(mask2, GT2)

[0089] Among them, mask2 and GT2 are the predicted vertebra mask and the true vertebra mask respectively, mask2 ∩ GT2 represents the intersection of mask2 and GT2, and the intersection is the overlapping part of the two regions, namely mask2 and GT2. Mask2 ∪ GT2 represents the union of mask2 and GT2, and the union is the merged part of the two regions, namely mask2 and GT2. Because the larger the value of IOU(mask2, GT2), the more overlap there is between the predicted vertebra mask and the true vertebra mask, and the better the segmentation effect, therefore, (1-IOU) is used as the loss function for minimization.

[0090] In step 105, the centerline is predicted based on the spine segmentation mask to obtain the centerline representation of the spine. e Curve prediction model estimation Hermit eThe spline parameters are fitted to obtain the centerline representation of the spine. e The curve prediction model can be pre-trained, Hermit e The curve prediction model is used to predict and estimate the Hermite spline parameters and fit the Hermite curve based on the input spine segmentation mask, and the fitted Hermite e The curve is represented as the centerline of the spine.

[0091] Exemplarily, after the feature map is input into the spine decoder to obtain the spine segmentation mask, it also includes: inputting the spine segmentation mask into a fine segmentation model to remove noise in the spine segmentation mask through the fine segmentation model to obtain a fine mask corresponding to the spine segmentation mask; correspondingly, the implementation method of the above step 105 may include: performing centerline prediction based on the fine mask corresponding to the spine segmentation mask to obtain a centerline representation of the spine.

[0092] The spine segmentation mask obtained from the spine decoder may contain noise, but unclear spine boundaries may cause large errors in the subsequent centerline prediction. To solve this problem, a fine segmentation model is used to generate a fine mask from the noisy spine segmentation mask. The fine segmentation model uses an unpaired image-to-image mapping technique, which is a CycleGAN (Cycle-Consistent Generative Adversarial Networks) trained with a set of spine segmentation masks generated by a segmentation model and a set of real masks. Compared with the initial spine segmentation mask, the result of the fine segmentation model can greatly improve the quality of the spine segmentation mask, making its boundaries clearer and more continuous, thereby reducing errors in subsequent centerline predictions and improving the robustness and reliability of the model.

[0093] Exemplarily, centerline prediction is performed based on the fine mask corresponding to the spine segmentation mask to obtain a centerline representation of the spine, including: predicting an initial center point from the fine mask; predicting Hermite spline parameters based on the initial center point predicted from the fine mask; constructing a Hermite curve based on the Hermite spline parameters, and using the Hermite curve as the centerline representation of the spine.

[0094] Specifically, the Resnet model (ResNet101) can be used to predict the initial center points (estimated center points) based on the refined mask. Among them, the last layer of ResNet101 is modified so that it can predict the coordinates of 34 center points under the supervision of the real spine center points. Then, in order to predict the Hermite spline parameters from the estimated center points of the X-ray image of the spine to be tested, the initial center points predicted from the refined mask are used as input to predict the Hermite spline parameters.

[0095] The Hermite spline parameters are the position of the center point and the derivative value at the center point. After obtaining the Hermite spline parameters, the Hermite basis function interpolation method is used to construct the Hermite curve. The Hermite curve is an interpolation curve that defines a smooth curve by giving the starting point, end point, and tangent direction at the starting point and end point. The characteristic of the Hermite curve is that the slope of the curve at the starting point and end point can be accurately controlled, thereby achieving more flexible and accurate curve fitting.

[0096] In step 106, feature extraction is performed based on the vertebra segmentation mask to obtain a target feature vector, wherein the target feature vector is used to characterize the morphology and structural information of each vertebra. The target feature vector captures the morphology and structural information of each vertebra, providing support for the calculation of the Cobb angle.

[0097] Exemplarily, feature extraction is performed based on the vertebral segmentation mask to obtain a target feature vector, including:

[0098] The vertebral segmentation mask is input into the pre-trained DenseNet to obtain the intermediate features, wherein the intermediate features can capture the multi-level information of the vertebral morphology and structure; the intermediate features are input into the first multi-layer perceptron, the first high-dimensional feature vector is extracted in the penultimate layer of the first multi-layer perceptron, and the first high-dimensional feature vector is used as the target feature vector.

[0099] Specifically, in this embodiment, a pre-trained DenseNet (Densely Connected Convolutional Networks) is fine-tuned, and the vertebral segmentation mask is used as input. The powerful feature extraction capability of DenseNet is used to extract the intermediate deep feature representation (i.e., intermediate features), which can capture the multi-level information of vertebral morphology and structure. Next, these intermediate features are input into a 4-layer multi-layer perceptron (i.e., the first multi-layer perceptron) for further processing, and a 128-dimensional feature vector (i.e., the first high-dimensional feature vector) is extracted from the penultimate layer of the first multi-layer perceptron as the target feature vector. This high-dimensional feature vector will be involved in the subsequent target Cobb angle prediction, thereby achieving accurate prediction of the Cobb angle. In this way, the detailed features and global morphological information in the vertebral segmentation mask can be effectively utilized to improve the accuracy and stability of the prediction. The above-mentioned first high-dimensional feature vector can be understood as a high-dimensional feature vector of the vertebra.

[0100] In a neural network, features closer to the output layer tend to be more specific (for example, features directly related to angle prediction), while features closer to the input layer tend to be low-level morphological features. The 128-dimensional feature vector of the penultimate layer contains multi-level information about the morphology and structure of the vertebrae after several feature extractions, and is an intermediate feature that is more useful for the Cobb angle prediction task. In addition, directly using the penultimate layer will result in insufficient information, affecting the accuracy of angle prediction, while the penultimate layer may contain too many irrelevant features, increasing the computational burden. Therefore, in this embodiment, extracting a 128-dimensional feature vector in the penultimate layer can retain sufficient contextual information while avoiding excessive information loss.

[0101] In step 107, the target Cobb angle corresponding to the X-ray image of the spine to be measured is predicted by a linear combination of slope analysis and regression analysis based on the centerline representation and the target feature vector.

[0102] Exemplarily, the implementation of step 107 includes the following steps 1071 to 1076:

[0103] Step 1071: Perform a curve slope analysis based on the centerline representation to obtain a first Cobb angle corresponding to the X-ray image of the spine to be tested.

[0104] Specifically, curve slope analysis refers to: given a set of Hermite spline parameters {H i}, 17 points are sampled uniformly from the predicted Hermite curve, i.e., the centerline representation, according to the parameters. These 17 points correspond to 12 thoracic vertebrae and 5 lumbar vertebrae, and each point corresponds to a coordinate (S x , Sy ), then, another 17 points are sampled close enough to these points. These 17 resampled points also correspond to 12 thoracic vertebrae and 5 lumbar vertebrae, and each point also corresponds to a coordinate (S′ x , S′ y ), the resampled point is the result of a slight shift based on the original sampling point, that is, S′ x =S x +ε,S′ y =S y +ε. In actual execution, the value of ε is set to 5e-2. After obtaining two sets of coordinates through the above method, the slope between each two corresponding points can be calculated. For example, the slope k between two corresponding points is calculated by the following formula:

[0105]

[0106] In order to prevent the situation where the slope does not exist at some key points, before calculating the slope, the predicted Hermite curve can be rotated 90 degrees counterclockwise, and then the slope is calculated based on the 17 sampled points. Since scoliosis usually occurs in the thoracic spine area, the maximum slope and minimum slope of the entire curve can be used to calculate MT (Main ThoracicCurv e The Cobb angle of the main thoracic curve is calculated by using the maximum and minimum slopes of the upper thoracic region to calculate the PT (ProximalThoracic Curv e The Cobb angle of the upper thoracic curve is calculated by using the maximum and minimum slopes of the lumbar region to calculate the TL (Thoracolumbar Curv e , lumbar bend) Cobb angle, the calculation of Cobb angle follows the following formula:

[0107]

[0108] Among them, k MT_max and k MT_min Represent the maximum slope and minimum slope of the entire curve, k PT_max and k PT_min Represent the maximum slope and minimum slope of the upper part of the curve, k TL_max and k TL_min Represents the maximum slope and minimum slope of the lower part of the curve. MTs Represents the Cobb angle of MT. PTs Cobb angle representing PT, Angle TLs The final curve slope analysis is used to obtain the first Cobb angle Angle corresponding to the X-ray image of the spine to be tested. slope It can be expressed as follows:

[0109] Angle slope = {Angle MTs ,Angle PTs ,Angle TLs}.

[0110] Step 1072: Resample according to the center line representation to obtain the resampled center point.

[0111] Specifically, since the Hermite spline parameters are predicted based on the center point, and the center point is obtained based on the fine mask, the accuracy of the Hermite curve fitting may be affected by the quality of the fine mask. In order to improve the accuracy and robustness of the algorithm, a set of Hermite spline parameters {H i Generate a set of resampled center points under the supervision of the true value of the center point. For example, the 34 center points obtained above are resampled to obtain 34 resampled center points. In the specific implementation, the Hermite curve can be resampled to obtain a set of new discrete center points as the resampled center points.

[0112] Step 1073: Input the resampled center point into the second multi-layer perceptron, and extract the second high-dimensional feature vector in the penultimate layer of the second multi-layer perceptron.

[0113] Specifically, the resampled center point is used as the input of a 4-layer multilayer perceptron (i.e., the second multilayer perceptron), and a 128-dimensional feature vector is extracted from the penultimate layer of the second multilayer perceptron as the second high-dimensional feature vector. The second high-dimensional feature vector can be understood as a centerline high-dimensional feature vector.

[0114] Step 1074: concatenate the first high-dimensional feature vector and the second high-dimensional feature vector to obtain a joint feature vector; wherein the joint feature vector integrates the global geometric information of the spine centerline and the local structure and morphology information of the vertebrae.

[0115] Specifically, in combination with the above, the 128-dimensional feature vector (i.e., the second high-dimensional feature vector) extracted from the penultimate layer of the second multi-layer perceptron is concatenated with the 128-dimensional feature vector (i.e., the first high-dimensional feature vector) extracted from the penultimate layer of the first multi-layer perceptron to form a 256-dimensional joint feature vector. This joint feature not only combines the global geometric information of the spine centerline, but also integrates the local structure and morphological information of the vertebrae. This feature fusion strategy utilizes the synergistic effect of the two complementary information and provides more comprehensive feature support for the prediction of the Cobb angle.

[0116] Step 1075: Input the joint feature vector into a third multi-layer perceptron, and output the second Cobb angle corresponding to the X-ray image of the spine to be tested that is predicted by regression.

[0117] Specifically, the above 256-dimensional joint feature vector is input into another 4-layer multilayer perceptron (i.e., the third multilayer perceptron), and further processed, and finally the regression prediction result, i.e., the second Cobb angle Angle, is output. regression = {Angle MTr ,Angle PTr ,Angle TLr}.

[0118] Step 1076: Perform weighted processing according to the first Cobb angle and the second Cobb angle to obtain a target Cobb angle corresponding to the X-ray image of the spine to be measured.

[0119] Specifically, considering Angle slope (i.e. the first Cobb angle) and Angle regression (i.e., the second Cobb angle) has a different error distribution. Since the two methods are complementary in feature extraction and noise processing, when one method predicts an overestimation, the other method will usually predict an underestimation. Therefore, the final target Cobb angle Angle can be obtained by linear combination:

[0120] Angle = αAngle slope +(1-α)Angle regression

[0121] Among them, α is a parameter used to control the weight of the results of slope analysis and regression analysis in the final result. In the AASCE2019 dataset, α for the thoracic spine, upper thoracic spine, and lumbar spine was set to 0.6, 0.7, and 0.6, respectively. This linear combination method uses the complementary characteristics of slope and regression analysis to reduce errors, improve accuracy, and enhance adaptability and stability in different clinical situations.

[0122] Figure 2 It is a schematic diagram of an implementation framework of a spinal Cobb angle prediction method based on vertebral segmentation provided in an embodiment of the present application, including: a segmentation module 201, a centerline prediction module 202, a vertebral regression module 203 and an angle calculation module 204.

[0123] The segmentation module 201 is used to analyze the input spinal X-ray image to obtain a spinal segmentation mask describing the overall outline of the spine and a vertebral segmentation mask describing the outline of each vertebra. Then, the spinal segmentation mask is used to estimate the Hermit eThe spline parameters are calculated and the curve is fitted to obtain the centerline representation of the spine. The vertebral segmentation mask is passed through the vertebral regression module 203 to obtain the target feature vector, which captures the morphological and structural information of each vertebra and provides support for the calculation of the Cobb angle. Finally, in the angle calculation module 204, the Cobb angle and the position of the end plate are estimated through a linear combination of slope analysis and regression analysis. The following is a detailed introduction to each module:

[0124] The segmentation module 201 is introduced below:

[0125] The input of the segmentation module 201 is a spinal X-ray image, ie, an X-ray image of the spinal column to be tested.

[0126] The segmentation module 201 segments the input image to obtain a mask. Specifically, the segmentation module 201 can be understood as a lightweight segmentation model, which is an SSFormer with an additional structural similarity index (SSIM) loss. The input image first passes through an encoder to obtain a feature map, and then the feature map is input into two different decoders, namely the spine decoder and the vertebrae decoder. After the feature map is decoded by the spine decoder, a spine segmentation mask is output, and after the feature map is decoded by the vertebrae decoder, a vertebrae segmentation mask is output.

[0127] The spine decoder and vertebra decoder are trained under the supervision of the true spine mask and the true vertebra mask, respectively. During the training process, BCE, SSIM and IOU losses are used to measure the difference between the predicted spine mask and the true spine mask, and the predicted vertebra mask and the true vertebra mask. At the same time, in order to better optimize the result of the vertebra segmentation mask, in the training process of the vertebra decoder, the corner prediction loss is additionally introduced, that is, the vertebra prediction loss function Loss mentioned above, which is used to measure the difference between the predicted value of the vertebra and the true value of the vertebra vertebra .

[0128] The spine segmentation mask (i.e., predicted spine mask) output by the spine decoder reflects the overall contour of the spine and is used for subsequent centerline prediction. The vertebra segmentation mask (i.e., predicted vertebra mask) output by the vertebra decoder shows the shape information of the 17 vertebrae for subsequent regression and angle prediction. Optionally, when training the segmentation model, the input X-ray image size is resized to 512×256.

[0129] Considering that the spinal segmentation mask obtained from the segmentation module may contain noise, unclear spinal boundaries may cause large errors in the subsequent centerline prediction. In order to solve this problem, in this embodiment, a fine segmentation model is used to generate a fine mask from the spinal segmentation mask with noise. The fine segmentation model uses an unpaired image-to-image mapping technology, which is a CycleGAN trained with a set of spinal masks generated by a segmentation model and a set of real masks. Compared with the initial spinal segmentation mask, the result of the fine segmentation model can greatly improve the quality of the spinal segmentation mask, making its boundaries clearer and more continuous, thereby reducing errors in subsequent centerline predictions and improving the robustness and reliability of the model.

[0130] The centerline prediction module 202 is introduced below:

[0131] The centerline prediction module 202 uses the Hermite curve prediction model to predict the Hermite spline parameters from the fine mask. The Hermite spline parameters are the position of the center point and the derivative value at the center point. Since predicting the Hermite spline parameters directly from the segmentation mask may bring unstable results, the center points are first predicted from the fine mask, and then these center points are used as input to predict the Hermite spline parameters. In order to predict the center point, this embodiment modifies the last layer of the ResNet101 model to predict the position of 34 center points under the supervision of the real center point. The loss of predicting the center point is obtained by the following L2 norm:

[0132]

[0133] Among them, point_pre i Represents the center point of the i-th prediction, centerline i represents the i-th true center point.

[0134] After obtaining the center point, a 4-layer multilayer perceptron is used to continue predicting the derivative value of each center point and use it as the starting point or end point derivative of the piecewise Hermite curve. Therefore, the derivative prediction loss can be defined as:

[0135]

[0136] Among them, derivatives_pre i 、derivatives_GT i Represent the predicted value and true value of the derivative at the i-th center point respectively.

[0137] After obtaining the Hermite spline parameters, the Hermite basis function interpolation method is used to construct the Hermite curve. Since the Hermite spline parameters are obtained based on the predicted center point, and the center point is obtained based on the fine mask, the accuracy of the Hermite curve fitting may be affected by the quality of the segmentation mask. In order to improve the accuracy and robustness of the algorithm, this embodiment proposes a resampling loss and uses a set of Hermite spline parameters {H i A set of resampled center points is generated under the supervision of the true value of the center point. This set of resampled center points will be used as input. Under the supervision of the true value of the Cobb angle, a 4-layer multi-layer perceptron is used to predict the angle. The resampling loss Loss resample Defined as:

[0138]

[0139] Among them, sampled_points i Represents the predicted value of the i-th resampling point, centerline i Represents the true value of the i-th center point.

[0140] In summary, the overall loss of the centerline prediction module 202 is curve for:

[0141] Loss curve =λ1Loss points +λ2Loss derivatives +λ3Loss resample

[0142] In the specific implementation, ResNet and multi-layer perceptron will be trained jointly, and the true value of the center point and the true value of the Hermite spline parameters will be used as supervised prediction of the center point and Hermite spline parameters, and λ1, λ2, and λ3 are set to 1, 0.1, and 0.1, respectively.

[0143] The vertebral regression module 203 is described in detail below:

[0144] The vertebral regression module 203 predicts the Cobb angle by regression analysis of the vertebral segmentation mask. Specifically, this embodiment fine-tunes a pre-trained DenseNet, takes the vertebral segmentation mask as input, and uses the powerful feature extraction capability of DenseNet to extract the intermediate deep feature representation. The feature map can capture the multi-level information of the vertebral morphology and structure. Next, these intermediate features are input into a 4-layer multi-layer perceptron (i.e., the first multi-layer perceptron, schematically shown as 4-MLP in the figure) for further processing, and a 128-dimensional high-dimensional feature vector (i.e., the first high-dimensional feature vector, schematically shown as 128-dimensional features in the figure) is extracted from the penultimate layer of the network. The high-dimensional feature vector is input into the angle calculation module 204, and the vertebral regression module 203 further processes and learns the feature vector by regression analysis, thereby realizing the accurate prediction of the Cobb angle. In this way, the detailed features and global morphological information in the vertebral segmentation mask can be effectively utilized to improve the accuracy and stability of the prediction.

[0145] The angle calculation module 204 is specifically introduced below:

[0146] The angle calculation module 204 calculates the angle of the Hermit according to the predicted e The curve slope analysis is performed, the slope is calculated to estimate the Cobb angle (i.e., the first Cobb angle), the Cobb angle (i.e., the second Cobb angle) is regressed from the resampled center point and the vertebral segmentation mask, and finally the final target Cobb angle is obtained by combining the slope and regression.

[0147] In the centerline prediction module 202, the 34 center points obtained are resampled to obtain 34 resampled center points, and the resampled center points are used as the input of a 4-layer multilayer perceptron (i.e., the second multilayer perceptron, schematically shown as 4-MLP in the figure), and the angle is predicted under the supervision of the true value of the angle. In order to further improve the accuracy and robustness of the Cobb angle prediction, in this embodiment, the 128-dimensional feature vector (i.e., the second high-dimensional feature vector, schematically shown as 128-dimensional feature) extracted from the penultimate layer of the second multilayer perceptron is spliced ​​with the 128-dimensional feature (i.e., the first high-dimensional feature vector) extracted by the vertebra regression module 203 to form a 256-dimensional joint feature vector. The joint feature vector not only combines the global geometric information of the spinal centerline, but also integrates the local structure and morphological information of the vertebrae. This feature fusion strategy utilizes the synergistic effect of two complementary information to provide more comprehensive feature support for the prediction of the Cobb angle. Finally, this 256-dimensional joint feature vector is input into another 4-layer multilayer perceptron (i.e., the third multilayer perceptron, schematically shown as MLP in the figure) for further processing, and finally outputs the regression prediction result, i.e., the second Cobb angle.

[0148] Finally, the angle calculation module 204 performs weighted processing on the first Cobb angle (PT1, MT1, TL1) and the second Cobb angle (PT2, MT2, TL2), and outputs the final target Cobb angle (PT, MT, TL). PT1, MT1, TL1 respectively represent: the Cobb angle of the upper thoracic curve, the Cobb angle of the main thoracic curve, and the Cobb angle of the lumbar curve obtained by the curve slope analysis. PT2, MT2, TL2 respectively represent: the Cobb angle of the upper thoracic curve, the Cobb angle of the main thoracic curve, and the Cobb angle of the lumbar curve obtained by the resampling center point and the vertebral segmentation mask regression. PT, MT, TL respectively represent: the final Cobb angle of the upper thoracic curve, the Cobb angle of the main thoracic curve, and the Cobb angle of the lumbar curve obtained by the combination of slope and regression.

[0149] In summary, the final output is the Cobb angles of the main thoracic curve, upper thoracic curve, and lumbar curve and their corresponding positions.

[0150] In actual implementation, this example uses the AASCE2019 dataset to evaluate the performance of the proposed method. The dataset contains 609 full-length anteroposterior images of the spine. Each image consists of 12 thoracic vertebrae and 5 lumbar vertebrae, and the position coordinates of the vertebral vertex of each vertebra are marked. Figure 2 To apply this dataset in the framework shown, it is necessary to obtain the true values ​​of the spine segmentation mask, vertebra segmentation mask, center point and Hermite spline parameters. In order to obtain this information, first connect the 34 points on the left and the 34 points on the right to obtain the contours of the left and right boundaries of the spine to construct the spine segmentation mask, and use the top of the first vertebra and the bottom of the last vertebra as the top and bottom of the mask to obtain the spine segmentation mask; then connect the four vertebrae of each vertebra to form a rectangle, which describes the shape and position information of a vertebra. Each image includes information of 17 vertebrae to construct the true vertebrae mask. Next, according to the coordinates of the vertebral vertebrae, extract the center of the top and bottom boundaries of each vertebra to obtain the initial center point. Then, by calculating the derivative of each initial center point, fit an initial Hermite curve. Uniformly sample 34 points on this initial Hermite curve as the true value of the center point. Further calculate the derivatives of the true values ​​of these center points, and use them together with the true values ​​of the center points to form the true values ​​of the Hermite spline parameters.

[0151] In the specific implementation, the code of the method in this embodiment is implemented by PyTorch and trained on a server equipped with a 3.6GHz CPU and an NVIDIA RTX 4060 GPU. A variety of data enhancement strategies are used during training, including random flipping, rotation, and geometric scaling to improve the robustness of the model. After data enhancement, the input image is combined with the initial center point and the Hermite spline parameters for training. Figure 2 The segmentation module, centerline prediction module, vertebra regression module, and angle calculation module are shown. During training, the initial learning rate of the segmentation module is set to 1e-3, while the initial learning rate of the centerline prediction module, vertebra regression module, and angle calculation module is 1e-4. All modules use the Adam optimizer and adopt a learning rate scheduler with a decay mechanism. The learning rate is updated every 1000 iterations with a decay rate of 0.9. In addition, due to the inconsistent image resolution in the dataset, all images are resized to a uniform size of 512×256 before training.

[0152] In the training phase, the segmentation module is first trained to generate the spine segmentation mask and the vertebra segmentation mask. Before the segmentation module starts training, the training rounds are pre-set. When the last training is completed, the model parameters are saved, and then the next phase is entered. In this phase, the other three modules (centerline prediction module, vertebra regression module, and angle calculation module) are jointly trained. Before the joint training starts, the training rounds are also pre-set. When the last training is completed, the model parameters are saved. Whether it is a training process is controlled by the "stage" parameter: when "stage" is set to "train", the above training process is executed.

[0153] The above joint model includes: 1. Centerline prediction module: used to predict the center point and predict the Hermite spline parameters based on the center point. 2. Vertebra regression module: used to process the vertebra segmentation mask and extract vertebrae-related features. 3. Angle calculation module: used to predict the Cobb angle using slope analysis through the fitted Hermite curve; predict the Cobb angle using regression analysis based on the high-dimensional feature vector obtained from the centerline regression analysis and the vertebrae regression analysis, and then linearly combine the results obtained from the slope analysis and regression analysis to predict the final Cobb angle.

[0154] Figure 2The framework shown combines multiple networks to work together. The input image is first cropped to a fixed size of 512×256, and a predicted segmentation mask is generated under the supervision of the true values ​​of the spine segmentation mask and the vertebrae segmentation mask. The predicted results of the spine segmentation mask and the true values ​​of the spine segmentation mask are used as input to train a fine segmentation model to obtain a more accurate spine segmentation mask result. Then, by jointly training ResNet and a multi-layer perceptron MLP, the Hermite spline parameters are predicted under the supervision of the true values ​​of the center points and the true values ​​of the Hermite spline parameters. The Hermite spline parameters include the location of the center point and the derivative value at the center point. The spinal curvature is then calculated based on these parameters to predict the Cobb angle. At the same time, 34 points are uniformly sampled on the fitted Hermite curve, and these resampled points are input into the centerline regression model to generate 128-dimensional features about the centerline. In addition, for the vertebra segmentation mask, we generate 128-dimensional features about the vertebra by training Densenet and MLP, and concatenate them with the features obtained from the centerline regression model to further enhance the centerline prediction results. Finally, these high-dimensional features pass through 4 layers of MLP, and under the supervision of the true angle value, the final predicted Cobb angle is obtained.

[0155] In the angle calculation module, the slope analysis and regression analysis methods are combined. By linearly combining the results of these two methods, a more accurate angle value is obtained. In the slope analysis, based on the predicted Hermite curve, 17 points are uniformly sampled, and another 17 points are sampled near these points, and the slope is calculated through these two sets of points. Among them, the hyperparameter ∈ is set to 5e-2. In order to prevent special circumstances during slope calculation (such as the non-existence of the slope at certain key points), the Hermite curve is rotated 90 degrees counterclockwise as a whole before calculation. Subsequently, the Cobb angle of the spine is calculated based on the slope: the maximum and minimum slopes of the entire curve are used to calculate MT, the maximum and minimum slopes of the upper part of the curve are used to calculate PT, and the maximum and minimum slopes of the lower part of the curve are used to calculate TL.

[0156] When "stage" is set to "test", the test process is executed. First, the spine segmentation mask and vertebrae segmentation mask of the images in the test set are generated using the model parameters saved by the segmentation module. Then, each image in the test set is analyzed using the model parameters saved by the other three modules during joint training, and the corresponding Cobb angle value is finally generated. For each Cobb angle, the mean absolute error loss (MAE) is used to measure the accuracy of the estimate. The mean absolute error loss can be defined as:

[0157] Loss MAE =Angle pre -Angle

[0158] Among them, Anglepre and Angle represent the predicted value and true value of the angle respectively.

[0159] For all Cobb angles, the symmetric mean absolute percentage error (SMAPE) is used to measure the accuracy of the estimation. The symmetric mean absolute percentage error can be defined as:

[0160]

[0161] Among them, Θ={MT, PT, TL}, Angle pre and Angle represent the predicted and true values ​​of the angle respectively, and N represents the total number of images. In order to prevent the denominator from overflowing due to 0, a sufficiently small value ∈ needs to be added to the denominator. In actual execution, ∈ is set to 1e-7.

[0162] In summary, the main key points of the prediction method provided in the embodiment of the present application are as follows:

[0163] (1) A dual-branch structure is proposed, which integrates the vertebral angle-based and spine centerline-based methods for Cobb angle prediction. The accuracy and robustness of the prediction are significantly improved by joint training and mutual supervision of the two methods. This dual-branch structure allows two independent branches to correct errors with each other, and can still generate accurate and highly interpretable results when processing low-quality or blurred images, solving the problem that the existing technology is easily disturbed when the image quality is poor.

[0164] (2) A lightweight segmentation network with the same encoder and different decoders is designed to simultaneously generate spine segmentation masks and vertebrae segmentation masks. The spine segmentation mask is used for centerline prediction, while the vertebrae segmentation mask is used for angle regression. This design greatly improves the efficiency of multi-task segmentation processing, while ensuring the segmentation accuracy under different tasks, optimizing resource utilization and computational efficiency.

[0165] (3) In the regression analysis, the vertebral regression features are used to enhance the centerline regression features, which improves the robustness of the regression results. The feature extraction results of the vertebral segmentation mask regression are used to enhance the features obtained by the centerline regression. By combining them together, the global geometric information and local structural information of the spinal curvature can be more effectively captured, which improves the stability and accuracy of the final Cobb angle prediction.

[0166] (4) Optimization and application of linear combination strategy: In order to solve the complementarity problem of different methods in feature extraction and noise processing, this embodiment of the present application introduces a linear combination strategy to combine the slope analysis results with the regression analysis results. By adjusting the weight parameters, the accuracy and robustness of the Cobb angle prediction are improved. This strategy ensures that reliable prediction results can be obtained even when facing low-quality or blurred images.

[0167] The main beneficial effects of the prediction method provided in the embodiment of the present application are as follows:

[0168] (1) It solves the problem that traditional operations are time-consuming and labor-intensive. Manual measurement of the Cobb angle is not only time-consuming and labor-intensive, but also relies on the doctor's subjective judgment. The embodiment of the present application can automatically, quickly and accurately estimate the Cobb angle, which reduces the burden on doctors and improves the efficiency of spinal curvature estimation.

[0169] (2) Improving the robustness and interpretability of prediction results. The solution proposed in the embodiment of the present application jointly considers the vertebral representation and the centerline representation. Compared with the method of only considering a certain intermediate representation, it reduces the estimation error and makes the result more accurate. In addition, because two different methods are jointly trained and supervised by each other, more interpretable results can be produced when facing special images.

[0170] (3) Reduced dependence on data quality. Existing methods for automatically estimating spinal curvature usually cannot obtain good results when faced with low-quality or blurred images. The solution proposed in the embodiment of the present application is suitable for spinal curvature estimation of low-quality images, and can obtain accurate and interpretable results, thereby enhancing the universality of the algorithm.

[0171] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0172] For example, Figure 3 As shown, the electronic device 300 includes: a memory 301 and a processor 302, wherein the memory 301 stores an executable program code 3011, and the processor 3602 is used to call and execute the executable program code 3011 to perform a spinal Cobb angle prediction method based on vertebral segmentation.

[0173] In addition, an embodiment of the present application also protects a device, which may include a memory and a processor, wherein the memory stores an executable program code, and the processor is used to call and execute the executable program code to execute a spinal Cobb angle prediction method based on vertebral segmentation provided in an embodiment of the present application.

[0174] In this embodiment, the functional modules of the device can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0175] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to an electronic device, the processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device in executing related program codes, etc.

[0176] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits shown in conjunction with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0177] In addition, the device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a spinal Cobb angle prediction method based on vertebral segmentation provided in the above embodiment.

[0178] This embodiment also provides a computer-readable storage medium, which stores a computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement a spinal Cobb angle prediction method based on vertebral segmentation provided in the above embodiment.

[0179] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement a spinal Cobb angle prediction method based on vertebral segmentation provided in the above embodiment.

[0180] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0181] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0182] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0183] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for predicting the Cobb angle of the spine based on vertebral segmentation, characterized in that: The method comprises: Obtaining an X-ray image of the spine to be tested; Inputting the X-ray image into an encoder to obtain a feature map of the X-ray image; Inputting the feature map into a spine decoder to obtain a spine segmentation mask; wherein the spine segmentation mask reflects the overall contour of the spine; Inputting the feature map into a vertebra decoder to obtain a vertebra segmentation mask; wherein the vertebra segmentation mask reflects the shape information of each vertebra; Performing centerline prediction based on the spine segmentation mask to obtain a centerline representation of the spine; Performing feature extraction based on the vertebra segmentation mask to obtain a target feature vector; wherein the target feature vector is used to characterize the morphology and structural information of each vertebra; The target Cobb angle corresponding to the X-ray image of the spine to be measured is predicted based on the centerline representation and the target feature vector.

2. The method according to claim 1, characterized in that The step of extracting features based on the vertebral segmentation mask to obtain a target feature vector includes: Inputting the vertebral segmentation mask into a pre-trained DenseNet to obtain intermediate features; wherein the intermediate features can capture multi-level information of vertebral morphology and structure; The intermediate features are input into a first multi-layer perceptron, a first high-dimensional feature vector is extracted from the penultimate layer of the first multi-layer perceptron, and the first high-dimensional feature vector is used as the target feature vector.

3. The method according to claim 1, characterized in that The step of predicting a target Cobb angle corresponding to the X-ray image of the spine to be measured according to the centerline representation and the target feature vector comprises: Performing a curve slope analysis according to the center line representation to obtain a first Cobb angle corresponding to the X-ray image of the spine to be tested; Resampling is performed according to the center line representation to obtain a resampling center point; Inputting the resampled center point into a second multi-layer perceptron, and extracting a second high-dimensional feature vector in the penultimate layer of the second multi-layer perceptron; The first high-dimensional feature vector and the second high-dimensional feature vector are concatenated to obtain a joint feature vector; wherein the joint feature vector integrates the global geometric information of the spine centerline and the local structure and morphology information of the vertebrae; Inputting the joint feature vector into a third multi-layer perceptron, and outputting a second Cobb angle corresponding to the X-ray image of the spine to be tested that is predicted by regression; A weighted processing is performed according to the first Cobb angle and the second Cobb angle to obtain a target Cobb angle corresponding to the X-ray image of the spine to be measured.

4. The method according to claim 1, characterized in that: The loss function Loss used by the vertebral decoder during training vertebra_mask for: Loss vertebra_mask =Loss vertebra_BCE +Loss vertebra_SSIM +Loss vertebra_IOU +Loss vertebra Among them, Loss vertebra_BCE is the binary cross entropy loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask. vertebra_SSIM is the structural similarity index loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask, Loss vertebra_IOU is the intersection-over-union loss function used to measure the difference between the predicted vertebrae mask and the true vertebrae mask, Loss vertebra is the vertebra prediction loss function used to measure the difference between the predicted value of the vertebra and the true value of the vertebra.

5. The method according to claim 4, characterized in that The Loss vertebra The expression is as follows: Where N represents the number of pixels in the image; C represents the total number of vertebrae; c represents the index of the vertebrae, traversing from the first vertebra to the last vertebra; i represents the index of the pixel, traversing from the first pixel to the last pixel; and are the predicted value and true value of a pixel in the vertebral segmentation mask, respectively.

6. The method according to claim 1, characterized in that The loss function Loss used by the spine decoder during training s p ine_mask for: Loss spine_mask =Loss spine_BCE +Loss spine_SSIM +Loss spine_IoU Among them, Loss spine_BCE is the binary cross entropy loss function used to measure the difference between the predicted spine mask and the true spine mask. spine_SSIM is the structural similarity index loss function used to measure the difference between the predicted spine mask and the true spine mask. spine_IOU is the intersection-over-union loss function used to measure the difference between the predicted spine mask and the true spine mask.

7. The method according to claim 1, characterized in that After inputting the feature map into the spine decoder to obtain the spine segmentation mask, the method further includes: Inputting the spine segmentation mask into a fine segmentation model to remove noise in the spine segmentation mask through the fine segmentation model to obtain a fine mask corresponding to the spine segmentation mask; The performing centerline prediction according to the spine segmentation mask to obtain the centerline representation of the spine includes: The centerline is predicted based on the refined mask corresponding to the spine segmentation mask to obtain a centerline representation of the spine.

8. The method according to claim 7, characterized in that The step of performing centerline prediction according to the fine mask corresponding to the spine segmentation mask to obtain the centerline representation of the spine includes: predicting an initial center point from the refined mask; Predicting Hermite spline parameters based on an initial center point predicted from the refined mask; A Hermite curve is constructed according to the Hermite spline parameters, and the Hermite curve is used as the center line of the spine.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the hybrid electronic device executes the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 8 is implemented.

Images