Scoliosis classification and angle joint estimation method, system, device and storage medium

Abstract

The application discloses a scoliosis grading and angle joint estimation method, system, device and storage medium. First, patient back images are selected and cropped from original data sets to form a scoliosis data set, then a visual attention module I is constructed to extract scoliosis image features, then a ordinal regression module II is constructed to perform scoliosis grading or angle estimation, and finally a scoliosis grading estimation branch and a scoliosis angle estimation branch are constructed to form a complete network model for scoliosis grading and angle joint estimation. The application adopts an end-to-end deep learning framework to learn scoliosis grading and angle estimation, uses a visual attention network based on large kernel attention to learn scoliosis image features, and solves a classification problem with order according to ordinal regression, so that a scoliosis grading and angle joint estimation system is constructed.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, device, and storage medium for joint estimation of scoliosis grading and angle. Background Technology

[0002] Currently, computer-aided diagnostics (CAD) is widely used in medical image processing to identify diseases. Using CAD, doctors use the output of computer-processed medical images as a "second opinion" to make the final decision.

[0003] CAD (Computer-Aided Design) also provides a new direction for scoliosis screening. One method of scoliosis screening is to first manually observe the direction of the spine on the patient's back to determine if scoliosis is present, and then roughly judge the severity of scoliosis based on the degree of curvature. Finally, X-rays are used for diagnosis. Using artificial intelligence technology to replace the initial screening of scoliosis by human eyes in the screening process can achieve large-scale, low-cost screening and outreach for scoliosis.

[0004] Currently, there are two main approaches to scoliosis screening using artificial intelligence: judging from RGB images of the patient's back and X-ray images of the patient's back. Using X-ray images is the more common and generally more accurate approach, but it hasn't effectively promoted large-scale, low-cost scoliosis screening and outreach. Research on scoliosis screening using RGB images of the patient's back is relatively limited, but this approach offers a lower cost, greater convenience, and higher efficiency for scoliosis screening and severity assessment. Previously, Lino Ramirez proposed training a Support Vector Machine (SVM) model that combines patient back images with clinical data to classify and predict the degree of scoliosis, achieving an accuracy rate of 69-85%. He also found that the SVM classifier outperformed other machine learning classifiers such as decision trees. In 2013, PhanP et al. used a Self-Organizing Map (SOM) network model to classify adolescent idiopathic scoliosis cases from a large database, achieving an accuracy rate of 82%.

[0005] Although progress has been made in screening for scoliosis using RGB images of patients' backs, the accuracy of scoliosis classification has not reached the expected level, and there is room for further improvement. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a method, system, device and storage medium for joint estimation of scoliosis grading and angle.

[0007] The method for jointly estimating the scoliosis grading and angle proposed in this invention comprises the following steps:

[0008] S01: Select and crop images of patients' backs from the original dataset to form a scoliosis dataset;

[0009] S02: Construct visual attention module I to extract features from scoliosis images;

[0010] S03: Using the scoliosis image features extracted in step S02, construct ordinal regression module II to perform scoliosis grading or angle estimation.

[0011] S04: Construct a scoliosis grading estimation branch and a scoliosis angle estimation branch. Both branches are composed of visual attention module I and ordinal regression module II in sequence. The two branches form the overall scoliosis grading and angle joint estimation network model.

[0012] S05: Train the overall scoliosis grading and angle joint estimation network model, which consists of a scoliosis grading estimation branch and a scoliosis angle estimation branch, using the training dataset, and update the parameters of the overall scoliosis grading and angle joint estimation network model using a gradient-based optimization method.

[0013] S06: Input a given image of a patient's back with scoliosis into a trained network model that jointly estimates the overall scoliosis grade and angle to predict the scoliosis grade and angle.

[0014] Preferably, the visual attention module I in S02 includes several stages with different output spatial resolutions. For each stage, its input is downsampled, and the downsampling rate is controlled by the step size. After downsampling, all other layers in a stage maintain the same output spatial resolution and number of channels. Each stage includes L sets of sequences, with the output of the previous sequence serving as the input of the next sequence.

[0015] Preferably, the visual attention module I comprises four stages, with L values ​​of 3, 5, 27, and 3 in each stage. Each sequence sequentially includes a batch normalization layer II, attention layers I-II, batch normalization layers I-III, and feedforward network layers I-IV. The input of the sequence is used as the input of batch normalization layer II. Within batch normalization layer II, the input is batch normalized once, and the result is used as the output of batch normalization layer II. The output of batch normalization layer II is used as the input of attention layer I-II. After processing by attention layer I-II, the result is obtained and used as the input of attention layer I. -II output; sum the sequence input with the output of attention layer I-II, and use the result as the input of batch normalization layer I-III. Within batch normalization layer I-III, batch normalization is performed on the input once, and the result is used as the output of batch normalization layer I-III; use the output of batch normalization layer I-III as the input of feedforward network layer I-IV, process it through feedforward network layer I-IV to obtain the result, and use the result as the output of feedforward network layer I-IV; sum the input of batch normalization layer I-III with the output of feedforward network layer I-IV, and use the result as the sequence output.

[0016] Preferably, the attention layers I-II sequentially include a 1×1 convolution, a GELU activation, a large kernel attention module, and a 1×1 convolution; wherein the large kernel attention module can be represented as follows:

[0017] Attention = Conv 1×1 (DW-D-Conv((DW-Conv(F))))

[0018]

[0019] In the formula: F represents the input feature; DW-Conv represents depthwise convolution; DW-D-Conv represents depthwise dilated convolution; Conv 1×1 represents a 1×1 convolution; Attention represents the attention map; Output represents the output of the large kernel attention module.

[0020] Preferably, the ordinal regression module II in S03 includes the same number of binary classifiers as the scoliosis grade or angle. Each binary classifier corresponds to a different scoliosis grade or a different scoliosis angle. Each binary classifier determines whether the scoliosis grade or angle of the current input image is greater than or equal to the scoliosis grade or angle corresponding to the binary classifier. If it is greater than or equal to, the binary classifier outputs 1; otherwise, it outputs 0, thereby obtaining the predicted value q of the scoliosis grade or angle.

[0021] Preferably, the scoliosis grade or angle prediction value q is calculated as follows:

[0022]

[0023] In the formula: x represents the input image; f k (x)∈{0,1} represents the result of the k-th binary classifier for the input image x; K represents the number of binary classifiers.

[0024] Preferably, the loss function used by ordinal regression module II is:

[0025]

[0026] In the formula: x represents the output of the t-th binary classifier for the i-th input image; i This represents the i-th input image; W represents the weight of the k-th binary classifier for the i-th input image; t λ represents the network parameters of the t-th binary classifier. t represents the importance coefficient of the t-th task; N represents the number of input images; T represents the number of binary classifiers.

[0027] The scoliosis grading and angle joint estimation system proposed in this invention includes:

[0028] The image acquisition unit is used to obtain images of the patient's back from the original dataset to form a scoliosis dataset.

[0029] The scoliosis image feature extraction unit includes a visual attention module I, which uses a large kernel attention module to extract scoliosis image features;

[0030] The scoliosis grading or angle estimation unit includes ordinal regression module II, which uses multiple binary classifiers to achieve scoliosis grading or angle estimation.

[0031] The parameter optimization unit is used to calculate the parameters and loss function values ​​of the bi-branch global scoliosis grading and angle joint estimation network model, which consists of visual attention module I and ordinal regression module II, and to update the parameters based on the gradient optimization method.

[0032] The present invention proposes a computer device comprising a memory and a processor, wherein:

[0033] The memory is used to store the above-described system;

[0034] The processor is coupled to the memory to execute the system described above.

[0035] The present invention provides a computer-readable storage medium storing the above-described system.

[0036] Beneficial technical effects of the present invention:

[0037] (1) In the visual attention module, the present invention uses large kernel attention to generate attention maps. Large kernel attention combines the advantages of convolution and self-attention, and takes into account local contextual information, large receptive field, linear complexity and dynamic process.

[0038] (2) In the ordinal regression module, the present invention uses multiple binary classifiers to process the order relationship between different categories.

[0039] (3) The present invention can directly perform scoliosis classification and angle estimation through the RGB image of the patient's back, which facilitates large-scale scoliosis screening so that patients can detect scoliosis early and intervene in a timely manner. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the method for joint estimation of scoliosis grading and angle proposed in this invention.

[0041] Figure 2 This is a schematic diagram of the structure of one stage in the visual attention module I proposed in this invention;

[0042] Figure 3 This is a schematic diagram of the structure of the ordinal regression module II proposed in this invention;

[0043] Figure 4 This is a schematic diagram of the structure of the large-core attention module proposed in this invention;

[0044] Figure 5 This is a schematic diagram of the entire neural network model for joint estimation of scoliosis grading and angle based on ordinal regression and attention networks proposed in this invention. Detailed Implementation

[0045] The present invention will be further explained below with reference to specific embodiments.

[0046] Example 1

[0047] Reference Figure 1 The present invention proposes a method for joint estimation of scoliosis grading and angle based on ordinal regression and attention network. The method steps are as follows:

[0048] S01: Select and crop RGB images of patients' backs from the original dataset to form a scoliosis dataset.

[0049] In the scoliosis grading estimation branch, the dataset is divided into four levels: 0, 1, 2, and 3, corresponding to no scoliosis (0°-10°), mild scoliosis (11°-20°), moderate scoliosis (21°-45°), and severe scoliosis (above 45°). In the scoliosis angle estimation branch, the dataset is divided according to the angle.

[0050] S02: Constructing a visual attention module I to extract scoliosis image features

[0051] Visual attention module I is used as the backbone network to learn scoliosis image features. Visual attention module I includes four stages with different output spatial resolutions. For each stage, its input is downsampled, and the downsampling rate is controlled by the step size. After downsampling, all other layers in a stage maintain the same output spatial resolution and number of channels.

[0052] like Figure 2 As shown, each stage includes L sets of sequences, with the output of the previous sequence serving as the input to the next sequence. The values ​​of L in the four stages are 3, 5, 27, and 3, respectively. Each sequence sequentially includes batch normalization layer II, attention layers I-II, batch normalization layers I-III, and feedforward network layers I-IV. The input of the sequence is used as the input to batch normalization layer II. Within batch normalization layer II, batch normalization is performed on the input, and the result is used as the output of batch normalization layer II. The output of batch normalization layer II is used as the input to attention layer I-II. After processing by attention layer I-II, the result is obtained, and the result is... The input of the sequence is the output of attention layer I-II; the input of the sequence is summed with the output of attention layer I-II, and the result is used as the input of batch normalization layer I-III. In batch normalization layer I-III, the input is batch normalized once, and the result is used as the output of batch normalization layer I-III; the output of batch normalization layer I-III is used as the input of feedforward network layer I-IV, and the result is used as the output of feedforward network layer I-IV; the input of batch normalization layer I-III is summed with the output of feedforward network layer I-IV, and the result is used as the output of the sequence.

[0053] like Figure 2 As shown, attention layers I-II sequentially include 1×1 convolution, GELU activation, a large kernel attention module, and another 1×1 convolution. The key step in attention layers I-II is generating the attention map. This can be achieved using either self-attention or large kernel convolution. To overcome the shortcomings of both methods and combine their advantages, a large kernel attention module is employed. This module enables self-attention to be adaptive and possess long-range relevance while avoiding its drawbacks. It includes depthwise convolution, depthwise dilated convolution, and 1×1 convolution. Depthwise convolution extracts local contextual information, depthwise dilated convolution captures long-range dependencies, and 1×1 convolution captures relationships along the channel dimension. Combined with the attention mechanism, channel-dimensional adaptability is introduced. The steps for generating the attention map by decomposing a large kernel convolution operation are as follows:

[0054] (1) Decompose a K×K convolution into a The dilation rate is d for a depthwise dilated convolution, a (2d-1)×(2d-1) depthwise convolution, and a 1×1 convolution.

[0055] (2) By decomposing (1), long-distance relationships can be captured with less cost and parameters, and then the importance of different parts can be estimated and attention maps can be generated through long-distance relationships.

[0056] like Figure 4 As shown, the large-kernel attention module can be represented as follows:

[0057] Attention = Conv 1×1 (DW-D-Conv((DW-Conv(F))))

[0058]

[0059] Where: F represents the input feature, DW-Conv represents depthwise convolution, DW-D-Conv represents depthwise dilated convolution, and Conv... 1×1 represents a 1×1 convolution, Attention represents the attention map, and Output represents the output of the large kernel attention module.

[0060] S03: Using the scoliosis image features extracted in step S02, construct ordinal regression module II to perform scoliosis grading or angle estimation.

[0061] Since scoliosis grading and angle estimation are both classification problems with order, in addition to classification loss, the order relationship between different categories also needs to be considered. To handle the order relationship between different categories, ordinal regression module II is used for scoliosis grading or angle estimation, such as... Figure 3 As shown, the ordinal regression module II includes the same number of binary classifiers as the scoliosis grade or angle. Each binary classifier corresponds to a different scoliosis grade or a different scoliosis angle. Each binary classifier determines whether the scoliosis grade or angle of the current input image is greater than or equal to the scoliosis grade or angle corresponding to its own binary classifier. If it is greater than or equal to, the binary classifier outputs 1; otherwise, it outputs 0, thus obtaining the predicted scoliosis grade or angle value q. The definition of q is:

[0062]

[0063] Where: x represents the input image, f k (x)∈{0,1} represents the result of the k-th binary classifier of the input image x, where K represents the number of binary classifiers and q represents the predicted grade or angle of scoliosis.

[0064] The loss function used in Ordinal Regression Module II is:

[0065]

[0066] in, Let x represent the output of the t-th binary classifier for the i-th input image. i This represents the i-th input image. W represents the weight of the k-th binary classifier for the i-th input image. t Let λ represent the network parameters of the t-th binary classifier. t Let represent the importance coefficient of the t-th task, N represent the number of input images, and T represent the number of binary classifiers.

[0067] S04: Constructing scoliosis grading estimation branches and scoliosis angle estimation branches

[0068] Both branches are composed of visual attention module I and ordinal regression module II in sequence, and the two branches form a network model for joint estimation of scoliosis grading and angle.

[0069] S05: Train the overall scoliosis grading and angle joint estimation network model, consisting of a scoliosis grading estimation branch and a scoliosis angle estimation branch, using the training dataset. Update the parameters of the overall scoliosis grading and angle joint estimation network model using a gradient-based optimization method.

[0070] The entire neural network model for joint estimation of scoliosis grading and angle is trained using an end-to-end method based on ordinal regression and attention networks (e.g.) Figure 5 In step S01, the original dataset is processed into two scoliosis datasets according to the two branches of scoliosis grading and angle. Based on the two datasets, the scoliosis grading estimation branch is first trained to obtain the grade prediction, and then the scoliosis angle estimation branch is trained to obtain the angle prediction. Finally, the grade and angle predictions are combined to obtain the final prediction result.

[0071] S06: Input a given back image of a scoliosis patient into a trained network model that jointly estimates the overall scoliosis grade and angle, and predict the scoliosis grade and angle.

[0072] The overall scoliosis grading and angle joint estimation network model can obtain the grade prediction results and the angle prediction results. If the predicted angle is consistent with the grade, the angle prediction can be considered very accurate. Otherwise, the grade prediction is taken as the standard, and the median of the grade interval is used as the predicted angle.

[0073] Based on the scoliosis dataset constructed in this embodiment, the final scoliosis grading prediction accuracy was 94.90%, and the scoliosis angle prediction accuracy was 79.62%. The scoliosis angle prediction was a fine-grained Cobb angle range prediction, with a 5° range limit for severe scoliosis (below 45°) and a range greater than 45° for severe scoliosis (above 45°) considered a correct prediction.

[0074] Example 2

[0075] The scoliosis grading and angle joint estimation system proposed in this invention includes:

[0076] The image acquisition unit is used to obtain images of the patient's back from the original dataset to form a scoliosis dataset.

[0077] The scoliosis image feature extraction unit includes a visual attention module I, which uses a large kernel attention module to extract scoliosis image features;

[0078] The scoliosis grading or angle estimation unit includes ordinal regression module II, which uses multiple binary classifiers to achieve scoliosis grading or angle estimation.

[0079] The parameter optimization unit is used to calculate the parameters and loss function values ​​of the bi-branch global scoliosis grading and angle joint estimation network model, which consists of visual attention module I and ordinal regression module II, and to update the parameters based on the gradient optimization method.

[0080] The specific estimation method of the system in this embodiment is the same as that in Embodiment 1.

[0081] Example 3

[0082] The present invention proposes a computer device comprising a memory and a processor, wherein:

[0083] The memory is used to store the system in Embodiment 2;

[0084] The processor is coupled to the memory to execute the system in Embodiment 2.

[0085] Example 4

[0086] The present invention provides a computer-readable storage medium storing the system of Embodiment 2.

Claims

1. A method for jointly estimating the grading and angle of scoliosis, characterized in that, The steps are as follows: S01: Select and crop images of patients' backs from the original dataset to form a scoliosis dataset; S02: Construct visual attention module I to extract features from scoliosis images; S03: Using the scoliosis image features extracted in step S02, construct ordinal regression module II to perform scoliosis grading or angle estimation. S04: Construct a scoliosis grading estimation branch and a scoliosis angle estimation branch. Both branches are composed of visual attention module I and ordinal regression module II in sequence. The two branches form the overall scoliosis grading and angle joint estimation network model. S05: Train the overall scoliosis grading and angle joint estimation network model, which consists of a scoliosis grading estimation branch and a scoliosis angle estimation branch, using the training dataset, and update the parameters of the overall scoliosis grading and angle joint estimation network model using a gradient-based optimization method. S06: Input a given image of a patient’s back with scoliosis into a trained network model for joint estimation of scoliosis grading and angle to predict the scoliosis grade and angle. The visual attention module I includes four stages, with L values ​​of 3, 5, 27, and 3 in each stage. Each sequence includes batch normalization layer II, attention layer I-II, batch normalization layer I-III, and feedforward network layer I-IV in sequence. Each stage consists of L sets of sequences, with the output of the previous sequence serving as the input to the next sequence; The attention layers I-II sequentially include a 1×1 convolution, GELU activation, a large kernel attention module, and another 1×1 convolution; the large kernel attention module can be represented as follows: ； In the formula: F represents the input feature; Represents depthwise convolution; This represents depthwise dilated convolution; Represents a 1×1 convolution; Represents an attention map; This represents the output of the large kernel attention module; The ordinal regression module II includes the same number of binary classifiers as the scoliosis grade or angle.

2. The method for joint estimation of scoliosis grading and angle according to claim 1, characterized in that, In S02, the visual attention module I includes several stages with different output spatial resolutions. For each stage, its input is downsampled, and the downsampling rate is controlled by the step size. After downsampling, all other layers in a stage maintain the same output spatial resolution and number of channels.

3. The method for jointly estimating the grading and angle of scoliosis according to claim 2, characterized in that, The sequence input is used as the input of batch normalization layer II. Within batch normalization layer II, the input is batch normalized once, and the result is used as the output of batch normalization layer II. The output of batch normalization layer II is used as the input of attention layer I-II. After processing by attention layer I-II, the result is obtained and used as the output of attention layer I-II. The sequence input is summed with the output of attention layer I-II, and the result is used as the input of batch normalization layer I-III. Within batch normalization layer I-III, the input is batch normalized once, and the result is used as the output of batch normalization layer I-III. The output of batch normalization layer I-III is used as the input of feedforward network layer I-IV. After processing by feedforward network layer I-IV, the result is used as the output of feedforward network layer I-IV. The input of batch normalization layer I-III and the output of feedforward network layer I-IV are summed, and the result is used as the sequence output.

4. The method for joint estimation of scoliosis grading and angle according to claim 1, characterized in that, Each binary classifier corresponds to a different scoliosis level or a different scoliosis angle. Each binary classifier determines whether the scoliosis level or angle of the current input image is greater than or equal to the scoliosis level or angle corresponding to the binary classifier. If it is greater than or equal to, the binary classifier outputs 1; otherwise, it outputs 0, thereby obtaining the predicted value q of the scoliosis level or angle.

5. The method for jointly estimating the grading and angle of scoliosis according to claim 4, characterized in that, The calculation method for the predicted grade or angle of scoliosis, q, is as follows: ； In the formula: x represents the input image; This represents the result of the k-th binary classifier for the input image x; K represents the number of binary classifiers.

6. A scoliosis grading and angle joint estimation system, characterized in that, include: The image acquisition unit is used to obtain images of the patient's back from the original dataset to form a scoliosis dataset. The scoliosis image feature extraction unit includes a visual attention module I, which uses a large kernel attention module to extract scoliosis image features; The scoliosis grading or angle estimation unit includes ordinal regression module II, which uses multiple binary classifiers to achieve scoliosis grading or angle estimation. The parameter optimization unit is used to calculate the parameters and loss function values ​​of the bi-branch global scoliosis grading and angle joint estimation network model, which is composed of visual attention module I and ordinal regression module II in sequence, and to update the parameters based on the gradient optimization method. The visual attention module I includes four stages, with L values ​​of 3, 5, 27, and 3 in each stage. Each sequence includes batch normalization layer II, attention layer I-II, batch normalization layer I-III, and feedforward network layer I-IV in sequence. Each stage consists of L sets of sequences, with the output of the previous sequence serving as the input to the next sequence; The attention layers I-II sequentially include a 1×1 convolution, GELU activation, a large kernel attention module, and another 1×1 convolution; the large kernel attention module can be represented as follows: ； In the formula: Represents input features; DW-Conv represents depthwise convolution; This represents depthwise dilated convolution; Represents a 1×1 convolution; Represents an attention map; This represents the output of the large kernel attention module; The ordinal regression module II includes the same number of binary classifiers as the scoliosis grade or angle.

7. A computer device, characterized in that, Includes memory and processor, wherein: The memory is used to store the system according to claim 6; The processor is coupled to the memory for executing the system of claim 6.

8. A computer-readable storage medium, characterized in that, The system of claim 6 is stored thereon.

Images