Scoliosis grading and angle joint estimation method, system and equipment and storage medium

The scoliosis grading and angle estimation network model constructed by the visual attention module and the ordinal regression module solves the problem of low accuracy in scoliosis screening in the existing technology, and achieves high-accuracy scoliosis grading and Angle estimates have improved the effectiveness of large-scale low-cost screening.

CN118982702AActive Publication Date: 2024-11-19ANHUI PROVINCIAL HOSPITAL
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202410998595.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-19
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

In the prior art, when using RGB images of the patient's back for scoliosis screening, the classification accuracy rate did not reach a high expected value, and there is room for improvement.

Method used

The visual attention module and ordinal regression module are used to construct a scoliosis grading and angle estimation network model. The scoliosis image features are extracted through the visual attention module, and the ordinal regression module is used to perform scoliosis grading and angle estimation, combining multiple The binary classifier processes the sequential relationship and updates the model parameters using the gradient optimization method.

Benefits of technology

High accuracy grading and angle estimation of scoliosis in the patient's back RGB image was achieved, which improved the accuracy of scoliosis screening, achieving a 94.90% grading prediction and a 79.62% angular prediction accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982702A_ABST
    Figure CN118982702A_ABST
Patent Text Reader

Abstract

The invention discloses a scoliosis grading and angle joint estimation method, system and device and a storage medium, and the method comprises the steps: selecting and cutting out a patient back image from an original data set to form a scoliosis data set, and constructing a visual attention module I to extract scoliosis image features; constructing an ordinal number regression module II to perform scoliosis grading or angle estimation, and finally constructing a scoliosis grading estimation branch and a scoliosis angle estimation branch to form a complete network model to perform scoliosis grading and angle joint estimation. According to the method, scoliosis grading and angle estimation are learned by adopting an end-to-end deep learning framework, scoliosis image features are learned by utilizing a visual attention network based on big kernel attention, and a sequential classification problem is solved according to ordinal regression, so that construction of a scoliosis grading and angle joint estimation system is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system, device and storage medium for jointly estimating scoliosis grading and angle. Background Art

[0002] Currently, computer-aided diagnosis (CAD) has been widely used in medical image processing to identify diseases. Using CAD, doctors use the output of computer-processed medical images as a "second opinion" and make a final decision.

[0003] CAD also provides a new direction for scoliosis screening. One of the methods of scoliosis screening is to first use manual observation of the direction of the patient's back spine surface to determine whether there is scoliosis, and then roughly judge the severity of scoliosis based on the degree of curvature of the direction, and finally use X-rays to confirm the diagnosis. Using artificial intelligence technology instead of the initial screening of scoliosis by human eyes in the screening method can achieve large-scale, low-cost screening and visits for scoliosis.

[0004] At present, there are two main ways to use artificial intelligence technology to achieve scoliosis screening, namely, judging on the patient's back RGB image and the patient's back X-ray. Among them, using patient X-rays for scoliosis screening is a more common way, and the results are usually more accurate, but this approach has not effectively promoted large-scale, low-cost scoliosis screening and visits. There are relatively few studies on scoliosis screening using RGB images of the patient's back, but this approach can relatively achieve low-cost, convenient and efficient scoliosis screening and grade judgment. Previously, Lino Ramirez proposed training a support vector machine model (SVM) to combine the patient's back image with clinical data to classify and predict the degree of scoliosis in patients. The test accuracy can reach 69-85%. At the same time, it was found that the performance of the SVM classifier is better than other machine learning classifiers such as decision trees. In 2013, PhanP et al. used the self-organizing map (SOM) network model to classify adolescent idiopathic scoliosis cases from a large database with an accuracy of 82%.

[0005] Although there has been progress in scoliosis screening on RGB images of patients' backs, the accuracy of scoliosis classification has not reached high expectations and there is room for further improvement. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a method, system, device and storage medium for jointly estimating scoliosis grading and angle.

[0007] The present invention proposes a method for scoliosis classification and angle joint estimation, and the method steps are as follows:

[0008] S01: Select and crop the patient's back images from the original dataset to form the scoliosis dataset;

[0009] S02: Construct visual attention module I to extract scoliosis image features;

[0010] S03: using the scoliosis image features extracted in step S02, constructing an ordinal regression module II to perform scoliosis grading or angle estimation;

[0011] S04: Construct a scoliosis grade estimation branch and a scoliosis angle estimation branch. Both branches are sequentially composed of a visual attention module I and an ordinal regression module II. The two branches form an overall scoliosis grade and angle joint estimation network model.

[0012] S05: Using the training data set to train the overall scoliosis grade and angle joint estimation network model consisting of the scoliosis grade estimation branch and the scoliosis angle estimation branch, and updating the parameters of the overall scoliosis grade and angle joint estimation network model using a gradient-based optimization method;

[0013] S06: Input the back image of a given scoliosis patient into the trained overall scoliosis grade and angle joint estimation network model to predict the scoliosis grade and angle.

[0014] Preferably, the visual attention module I in S02 includes several stages with different output spatial resolutions. For each stage, its input is downsampled, and the downsampling rate is controlled using the step number. After the downsampling, all other layers in a stage maintain the same output spatial resolution and number of channels; each stage includes L groups of sequences, and the output of the previous sequence serves as the input of the next sequence.

[0015] Preferably, the visual attention module I includes four stages, the L values ​​in each stage are 3, 5, 27, and 3 respectively, and each sequence includes a batch normalization layer II, an attention layer I-II, a batch normalization layer I-III, and a feedforward network layer I-IV in sequence; the input of the sequence is used as the input of the batch normalization layer II, and the input is batch normalized once in the batch normalization layer II, and the result is used as the output of the batch normalization layer II; the output of the batch normalization layer II is used as the input of the attention layer I-II, and the result is obtained after being processed by the attention layer I-II, and the result is used as the output of the attention layer I -II; add the input of the sequence with the output of the attention layer I-II, and use the result as the input of the batch normalization layer I-III. In the batch normalization layer I-III, the input is batch normalized once, and the result is used as the output of the batch normalization layer I-III; use the output of the batch normalization layer I-III as the input of the feedforward network layer I-IV, obtain the result after being processed by the feedforward network layer I-IV, and use the result as the output of the feedforward network layer I-IV; add the input of the batch normalization layer I-III and the output of the feedforward network layer I-IV, and use the result as the output of the sequence.

[0016] Preferably, the attention layers I-II sequentially include 1×1 convolution, GELU activation, large core attention module, and 1×1 convolution; wherein the large core attention module can be expressed as follows:

[0017] Attention=Conv 1×1 (DW-D-Conv((DW-Conv(F))))

[0018]

[0019] Where: F represents input features; DW-Conv represents deep convolution; DW-D-Conv represents deep dilated convolution; Conv 1×1 represents 1×1 convolution; Attention represents the attention map; Output represents the output of the large-core attention module.

[0020] Preferably, the ordinal regression module II in S03 includes the same number of binary classifiers as the scoliosis grade or angle, each binary classifier corresponds to a different scoliosis grade or a different scoliosis angle, and each binary classifier determines whether the scoliosis grade or angle of the current input image is greater than or equal to the scoliosis grade or angle corresponding to the binary classifier. If so, the binary classifier outputs 1, otherwise, it outputs 0, thereby obtaining the scoliosis grade or angle prediction value q.

[0021] Preferably, the scoliosis grade or angle prediction value q is calculated as follows:

[0022]

[0023] Where: x represents the input image; f k (x)∈{0,1} represents the result of the kth binary classifier for the input image x; K represents the number of binary classifiers.

[0024] Preferably, the loss function used by the ordinal regression module II is:

[0025]

[0026] Where: represents the output of the t-th binary classifier for the ith input image; x i represents the i-th input image; represents the weight of the kth binary classifier of the i-th input image; W t represents the network parameters of the t-th binary classifier; λ t represents the important coefficient of the t-th task; N represents the number of input images; T represents the number of binary classifiers.

[0027] The scoliosis classification and angle joint estimation system proposed by the present invention comprises:

[0028] An image acquisition unit, used for acquiring a patient's back image from an original data set to form a scoliosis data set;

[0029] A scoliosis image feature extraction unit, including a visual attention module I, which extracts scoliosis image features using a large core attention module;

[0030] A scoliosis grading or angle estimation unit, comprising an ordinal regression module II, which uses multiple binary classifiers to achieve scoliosis grading or angle estimation;

[0031] The parameter optimization unit is used to calculate the parameters and loss function values ​​of the dual-branch holistic scoliosis grading and angle joint estimation network model composed of the visual attention module I and the ordinal regression module II, and update the parameters based on the gradient optimization method.

[0032] The present invention provides a computer device, comprising a memory and a processor, wherein:

[0033] The memory is used to store the above system;

[0034] The processor is coupled to the memory and is configured to execute the above system.

[0035] The present invention provides a computer-readable storage medium storing the above system.

[0036] Beneficial technical effects of the present invention:

[0037] (1) In the visual attention module, the present invention uses large kernel attention to generate an attention map. The large kernel attention combines the advantages of convolution and self-attention, and takes into account local context information, large receptive field, linear complexity and dynamic process.

[0038] (2) In the ordinal regression module, the present invention uses multiple binary classifiers to process the order relationship between different categories.

[0039] (3) The present invention can perform scoliosis classification and angle joint estimation directly through the RGB image of the patient's back, which is convenient for large-scale scoliosis screening, so that patients can detect scoliosis at an early stage and intervene in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the method for scoliosis classification and angle joint estimation proposed by the present invention;

[0041] Figure 2 A schematic diagram of the structure of a stage in the visual attention module I proposed in the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of the ordinal regression module II proposed in the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of the large core attention module proposed in the present invention;

[0044] Figure 5 This is a schematic diagram of the structure of the entire neural network model for joint estimation of scoliosis grading and angle based on ordinal regression and attention network proposed in the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further explained below in conjunction with specific embodiments.

[0046] Example 1

[0047] Reference Figure 1 The present invention proposes a method for joint estimation of scoliosis grading and angle based on ordinal regression and attention network, and the method steps are as follows:

[0048] S01: Select and crop the patient's back RGB images from the original dataset to form the scoliosis dataset

[0049] In the scoliosis grade estimation branch, the dataset is divided into 4 levels, namely 0, 1, 2, and 3, corresponding to no scoliosis (0°-10°), mild scoliosis (11°-20°), moderate scoliosis (21°-45°), and severe scoliosis (above 45°). In the scoliosis angle estimation branch, the dataset is divided according to the angle.

[0050] S02: Constructing Visual Attention Module I to Extract Scoliosis Image Features

[0051] Visual attention module I is used as the backbone network to learn scoliosis image features. Visual attention module I includes four stages with different output spatial resolutions. For each stage, its input is downsampled and the downsampling rate is controlled by the number of steps. After downsampling, all other layers in a stage maintain the same output spatial resolution and number of channels.

[0052] like Figure 2 As shown in the figure, each stage includes L groups of sequences, the output of the previous sequence is used as the input of the next sequence, the L values ​​in the four stages are 3, 5, 27, and 3 respectively, and each sequence includes batch normalization layer II, attention layer I-II, batch normalization layer I-III, and feedforward network layer I-IV in turn; the input of the sequence is used as the input of batch normalization layer II, and the input is batch normalized once in batch normalization layer II, and the result is used as the output of batch normalization layer II; the output of batch normalization layer II is used as the input of attention layer I-II, and the result is obtained after being processed by attention layer I-II, and the result is used as is the output of attention layer I-II; the input of the sequence is added to the output of attention layer I-II, and the result is used as the input of batch normalization layer I-III. In batch normalization layer I-III, the input is batch normalized once, and the result is used as the output of batch normalization layer I-III; the output of batch normalization layer I-III is used as the input of feedforward network layer I-IV, and the result is obtained after being processed by feedforward network layer I-IV, and the result is used as the output of feedforward network layer I-IV; the input of batch normalization layer I-III is added to the output of feedforward network layer I-IV, and the result is used as the output of the sequence.

[0053] like Figure 2 As shown in the figure, the attention layers I-II include 1×1 convolution, GELU activation, large-core attention module, and 1×1 convolution in sequence; the key step of the attention layers I-II is to generate the attention map, which can be generated by using the self-attention mechanism and large-core convolution. The large-core attention module is used to overcome the shortcomings of the two methods and combine their advantages; the large-core attention module makes the self-attention adaptive and remotely relevant, while avoiding the shortcomings of self-attention, including deep convolution, deep dilated convolution, and 1×1 convolution. The deep convolution extracts local context information, the deep dilated convolution captures remote dependencies, and the 1×1 convolution captures the relationship in the channel dimension. Combined with the attention mechanism, the adaptability of the channel dimension is introduced; the steps to generate the attention map by decomposing a large-core convolution operation are as follows:

[0054] (1) Decompose a K×K convolution into a A depth-wise dilated convolution with a dilation rate of d, a (2d-1)×(2d-1) depth-wise convolution, and a 1×1 convolution.

[0055] (2) Through the decomposition in (1), long-range relations can be captured with smaller costs and parameters. The importance of different parts is then estimated through long-range relations and an attention map is generated.

[0056] like Figure 4 As shown, the large core attention module can be expressed as follows:

[0057] Attention=Conv 1×1 (DW-D-Conv((DW-Conv(F))))

[0058]

[0059] Among them: F represents input features, DW-Conv represents deep convolution, DW-D-Conv represents deep dilated convolution, Conv 1×1 represents 1×1 convolution, Attention represents the attention map, and Output represents the output of the large-core attention module.

[0060] S03: Using the scoliosis image features extracted in step S02, construct an ordinal regression module II to perform scoliosis classification or angle estimation

[0061] Since scoliosis grading and angle estimation are both sequential classification problems, in addition to the classification loss, the sequential relationship between different categories must also be considered; in order to deal with the sequential relationship between different categories, the ordinal regression module II is used for scoliosis grading or angle estimation, such as Figure 3 As shown, the ordinal regression module II includes the same number of binary classifiers as the scoliosis grade or angle, each binary classifier corresponds to a different scoliosis grade or a different scoliosis angle, and each binary classifier determines whether the scoliosis grade or angle of the current input image is greater than or equal to the scoliosis grade or angle corresponding to the binary classifier. If so, the binary classifier outputs 1, otherwise, it outputs 0, thereby obtaining the scoliosis grade or angle prediction value q, and q is defined as:

[0062]

[0063] Where: x represents the input image, f k (x)∈{0,1} represents the result of the kth binary classifier of the input image x, K represents the number of binary classifiers, and q represents the predicted value of scoliosis grade or angle.

[0064] The loss function used in the ordinal regression module II is:

[0065]

[0066] in, represents the output of the t-th binary classifier for the ith input image, x i represents the i-th input image, represents the weight of the k-th binary classifier for the i-th input image, W t represents the network parameters of the t-th binary classifier, λ t represents the important coefficient of the t-th task, N represents the number of input images, and T represents the number of binary classifiers.

[0067] S04: Constructing the scoliosis grade estimation branch and the scoliosis angle estimation branch

[0068] Both branches are composed of visual attention module I and ordinal regression module II in sequence, and the two branches constitute the overall scoliosis grading and angle joint estimation network model.

[0069] S05: Use the training dataset to train the overall scoliosis grade and angle joint estimation network model consisting of the scoliosis grade estimation branch and the scoliosis angle estimation branch, and update the parameters of the overall scoliosis grade and angle joint estimation network model using a gradient-based optimization method

[0070] The entire neural network model for joint estimation of scoliosis grade and angle based on ordinal regression and attention network is trained in an end-to-end manner (e.g. Figure 5 ). In step S01, the original data set is processed into two scoliosis data sets according to the two branches of scoliosis grade and angle. According to the two data sets, the scoliosis grade estimation branch is first trained to obtain grade prediction, and then the scoliosis angle estimation branch is trained to obtain angle prediction. Finally, the grade and angle predictions are combined to obtain the final prediction result.

[0071] S06: Input the back image of a given scoliosis patient into the trained overall scoliosis grade and angle joint estimation network model to predict the scoliosis grade and angle

[0072] The grade prediction results and angle prediction results can be obtained through the overall scoliosis grading and angle joint estimation network model. If the predicted angle is consistent with the grade, the angle prediction can be considered very accurate. Otherwise, the grade prediction is taken as the standard, and the median of the grade interval is used as the predicted angle.

[0073] Based on the scoliosis dataset constructed in this embodiment, the final scoliosis grade prediction accuracy is 94.90%, and the scoliosis angle prediction accuracy is 79.62%. Among them, the scoliosis angle prediction is a fine-grained Cobb angle range prediction, with 5° as a range limit for severe scoliosis (below 45°), and a range greater than 45° for severe scoliosis (above 45°) is considered correct.

[0074] Example 2

[0075] The scoliosis classification and angle joint estimation system proposed by the present invention comprises:

[0076] An image acquisition unit, used for acquiring a patient's back image from an original data set to form a scoliosis data set;

[0077] A scoliosis image feature extraction unit, including a visual attention module I, which extracts scoliosis image features using a large core attention module;

[0078] A scoliosis grading or angle estimation unit, comprising an ordinal regression module II, which uses multiple binary classifiers to achieve scoliosis grading or angle estimation;

[0079] The parameter optimization unit is used to calculate the parameters and loss function values ​​of the dual-branch holistic scoliosis grading and angle joint estimation network model composed of the visual attention module I and the ordinal regression module II, and update the parameters based on the gradient optimization method.

[0080] The specific estimation method of the system in this embodiment refers to Example 1.

[0081] Example 3

[0082] The present invention provides a computer device, comprising a memory and a processor, wherein:

[0083] The memory is used to store the system in embodiment 2;

[0084] The processor is coupled to the memory and is used to execute the system in Example 2.

[0085] Example 4

[0086] The present invention provides a computer-readable storage medium, which stores the system in Example 2.

Claims

1. A method for jointly estimating scoliosis grading and angle, characterized in that: The steps are as follows: S01: Select and crop the patient's back images from the original dataset to form the scoliosis dataset; S02: Construct visual attention module I to extract scoliosis image features; S03: using the scoliosis image features extracted in step S02, constructing an ordinal regression module II to perform scoliosis grading or angle estimation; S04: Construct a scoliosis grade estimation branch and a scoliosis angle estimation branch. Both branches are sequentially composed of a visual attention module I and an ordinal regression module II. The two branches form an overall scoliosis grade and angle joint estimation network model. S05: Using the training data set to train the overall scoliosis grade and angle joint estimation network model consisting of the scoliosis grade estimation branch and the scoliosis angle estimation branch, and updating the parameters of the overall scoliosis grade and angle joint estimation network model using a gradient-based optimization method; S06: Input the back image of a given scoliosis patient into the trained overall scoliosis grade and angle joint estimation network model to predict the scoliosis grade and angle.

2. A method for jointly estimating scoliosis classification and angle according to claim 1, characterized in that: The visual attention module I in S02 includes several stages with different output spatial resolutions. For each stage, its input is downsampled, and the step number is used to control the downsampling rate. After downsampling, all other layers in a stage maintain the same output spatial resolution and number of channels; each stage includes L groups of sequences, and the output of the previous sequence is used as the input of the next sequence.

3. A method for jointly estimating scoliosis grading and angle according to claim 2, characterized in that: The visual attention module I includes four stages, the L value in each stage is 3, 5, 27, and 3 respectively, and each sequence includes batch normalization layer II, attention layer I-II, batch normalization layer I-III, and feedforward network layer I-IV in sequence; the input of the sequence is used as the input of batch normalization layer II, and the input is batch normalized once in batch normalization layer II, and the result is used as the output of batch normalization layer II; The output of batch normalization layer II is used as the input of attention layer I-II, and the result is obtained after being processed by attention layer I-II, and the result is used as the output of attention layer I-II; The input of the sequence is added to the output of the attention layer I-II, and the result is used as the input of the batch normalization layer I-III. In the batch normalization layer I-III, the input is batch normalized once, and the result is used as the output of the batch normalization layer I-III; The output of batch normalization layer I-III is used as the input of feedforward network layer I-IV, and the result is obtained after processing by feedforward network layer I-IV, and the result is used as the output of feedforward network layer I-IV; the input of batch normalization layer I-III and the output of feedforward network layer I-IV are added, and the result is used as the output of the sequence.

4. A method for jointly estimating scoliosis classification and angle according to claim 3, characterized in that: The attention layers I-II include 1×1 convolution, GELU activation, large core attention module, and 1×1 convolution in sequence; the large core attention module can be expressed as follows: Attention=Conv 1×1 (DW-D-Conv((DW-Conv(F)))) Where: F represents input features; DW-Conv represents deep convolution; DW-D-Conv represents deep dilated convolution; Conv 1×1 represents 1×1 convolution; Attention represents the attention map; Output represents the output of the large-core attention module.

5. The method for jointly estimating scoliosis classification and angle according to claim 1, characterized in that: The ordinal regression module II in S03 includes the same number of binary classifiers as the scoliosis grades or angles, each of which corresponds to a different scoliosis grade or a different scoliosis angle. Each binary classifier determines whether the scoliosis grade or angle of the current input image is greater than or equal to the scoliosis grade or angle corresponding to the binary classifier. If so, the binary classifier outputs 1, otherwise, it outputs 0, thereby obtaining the scoliosis grade or angle prediction value q.

6. A method for jointly estimating scoliosis classification and angle according to claim 5, characterized in that: The calculation method of the predicted value q of scoliosis grade or angle is: Where: x represents the input image; f k (x)∈{0,1} represents the result of the kth binary classifier for the input image x; K represents the number of binary classifiers.

7. A method for jointly estimating scoliosis classification and angle according to claim 5, characterized in that: The loss function used in the ordinal regression module II is: Where: represents the output of the t-th binary classifier for the ith input image; x i represents the i-th input image; represents the weight of the kth binary classifier of the i-th input image; W t represents the network parameters of the t-th binary classifier; λ t represents the important coefficient of the t-th task; N represents the number of input images; T represents the number of binary classifiers.

8. A scoliosis classification and angle joint estimation system, characterized in that: include: An image acquisition unit, used for acquiring a patient's back image from an original data set to form a scoliosis data set; A scoliosis image feature extraction unit, including a visual attention module I, which extracts scoliosis image features using a large core attention module; A scoliosis grading or angle estimation unit, comprising an ordinal regression module II, which uses multiple binary classifiers to achieve scoliosis grading or angle estimation; The parameter optimization unit is used to calculate the parameters and loss function values ​​of the dual-branch holistic scoliosis grading and angle joint estimation network model composed of the visual attention module I and the ordinal regression module II, and update the parameters based on the gradient optimization method.

9. A computer device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store the system according to claim 8; The processor is coupled to the memory and is configured to execute the system of claim 8.

10. A computer-readable storage medium, characterized in that: The system according to claim 8 is stored.

Citation Information

Patent Citations

  • Fine-grained image classification method and system, computer equipment and storage medium

    CN112163465A

  • Scoliosis image detection model training method and device, scoliosis image detection model determining method and device and equipment

    CN113052842A

  • Scoliosis detection method and medical platform

    CN113284090A

  • Face image age estimation method based on global-to-local ordinal regression network

    CN116524568A

  • Chest multi-disease diagnosis model construction method based on visual self-attention

    CN117522877A