A scoliosis measurement method based on RGB images

Abstract

This application relates to a scoliosis measurement method based on RGB images. The method involves acquiring an RGB image, extracting the target area, identifying and marking key points within the target area, correcting the key point positions using a key point correction algorithm, performing preliminary screening by establishing a coarse-grained spinal curve, and calculating the Cobb angle through fine-grained spinal fitting. This application addresses the issue of scoliosis detection's inability to accurately screen patients across a wide range of conditions, at the level of classification accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a scoliosis measurement method based on RGB images. Background Art

[0002] Scoliosis is a complex three-dimensional spinal deformity that mainly occurs in adolescent children. Early identification of scoliosis, grading of disease severity and effective intervention are particularly important.

[0003] Existing scoliosis detection methods generally fail to balance accuracy, detection efficiency, and non-invasiveness. For example, the traditional Adam's test and trunk rotator examination suffer from low classification accuracy, complex operation, and long duration in clinical diagnosis. The Moiré fringe method has the advantages of being fast and non-invasive, but its accuracy is easily affected by optical layout, lighting conditions, and residual grating. EOS three-dimensional imaging technology can accurately measure key parameters such as the Cobb angle and AVR angle, but its equipment is expensive and lacks portability, which limits its application in widespread screening. Similarly, neural network-based X-ray image analysis has both accuracy and detection efficiency, but cannot address the issues of radiation damage and high costs.

[0004] Deep learning-based image classification technology has been applied to categorize the severity of bare back images. While its performance approaches that of human experts, its accuracy is only 55.5%. Customized RIM cutout models and YOLOv5 models can quickly detect scoliosis, but they do not classify its severity.

[0005] In summary, existing technologies have been able to initially achieve a balance between accuracy, detection efficiency, and non-invasiveness, but they still cannot achieve large-scale screening in clinical practice due to the lack of classification accuracy. Summary of the Invention

[0006] This application provides a scoliosis measurement method based on RGB images, which solves the problem that scoliosis detection cannot achieve large-scale accurate screening from the level of classification accuracy.

[0007] The technical solution of this application is as follows:

[0008] A scoliosis measurement method based on RGB images, comprising the following steps:

[0009] S1) obtaining an RGB / BGR image of the subject's back;

[0010] S2) extracting the shoulder and back region of the RGB / BGR image as the target region, and identifying and marking the positions of key points in the target region;

[0011] The key points include fixed points and dynamic points. The fixed points are located at the edge of the target area, namely the left / right inner shoulder, left / right outer shoulder, left / right armpit, the innermost side of the left / right waist, and the top of the intergluteal groove. The dynamic points are distributed at equal heights at the edge of the area enclosed by two sets of adjacent fixed points. The key points at corresponding positions on the left and right sides of the spine constitute a pair of key points.

[0012] S3) correcting the key point positions in S2) using a loss function in a key point correction algorithm, the correction contents including local symmetry correction for normal areas, curvature correction for curved areas, and error correction; the normal area is defined as an area where the angle between a line connecting a set of key point pairs and a horizontal line is less than or equal to a set threshold; the curved area is defined as an area where the angle between a line connecting a set of key point pairs and a horizontal line is greater than a set threshold;

[0013] The loss function is as follows:

[0014]

[0015] is the regression loss function, is the error function between the key point coordinates and the true annotation point coordinates, is the local symmetry constraint loss function, is the loss function of the curved area, λ1, λ2, and λ3 are weighting coefficients, which are adjusted by five-fold cross validation;

[0016] S4) connecting each pair of key points of the spine to generate a key point line, calculating the coordinates of the midpoint of each key point line and connecting the two points with the largest left and right offsets to form a coarse-grained spinal line, calculating the angle between the coarse-grained spinal line and the vertical direction and setting a binary classification threshold. When the angle is less than or equal to the binary classification threshold, it is determined that there is no scoliosis. When the angle is greater than the binary classification threshold, it is determined that there is scoliosis.

[0017] S5) In the thoracic spine, every two pairs of keypoints connected sequentially from top to bottom are considered a vertebra; in the lumbar spine, every three pairs of keypoints are considered a vertebra; the Cobb angle is calculated based on the vertebral bodies and the Cobb angle definition.

[0018] Furthermore, the key point correction algorithm architecture includes a data pre-processing module, a backbone network module, a head network module and a post-processing module;

[0019] The data preprocessing module takes the original RGB image or BGR image as input, normalizes it using the mean and standard deviation of each channel, and converts the BGR image into an RGB image output;

[0020] The backbone network module takes the normalized RGB image as input, extracts the high-level semantic features of the RGB image and outputs a feature map;

[0021] The head network takes the feature map as input and outputs the coordinates of the key points through the convolution layer, ReLU activation function, KL divergence loss function and fully connected layer;

[0022] The post-processing module takes the coordinates of the key points as input and uses The loss function corrects the coordinates of key points.

[0023] Furthermore, in step S3), the method for local symmetry correction of the normal area is: dividing the target area into several local areas containing at least one set of key point pairs; using the quadratic Newton interpolation method to fit the center line of the local area, and constraining the key point coordinates in each local area according to the following formula:

[0024]

[0025] Where, L local-symmertry is the local symmetry constraint loss function, x i L and x i R They are the coordinates of the key points on the left and right sides of the local symmetric region, x center is the horizontal coordinate of the center line of the back area, N local Indicates the number of key point pairs contained in the local area.

[0026] Furthermore, in step S3), the formula for the bending correction of the curved area is:

[0027]

[0028] Where, w i is the weight of the curved region, is the total number of keypoint pairs in the curved area, x i L and x i R These are the coordinates of the key points on the left and right sides of the curved area, respectively.

[0029] Furthermore, the loss of error correction in step S3) is calculated as follows:

[0030]

[0031] Where, N is the number of key points, is the position of the real key point, To predict the location of key points.

[0032] Furthermore, the total number of key points is 41.

[0033] Furthermore, in step S5), the fitted vertebral body is drawn using OpenCV, the width of the vertebral body is 1 / 6 of the distance between the left and right axillae, and the height of the vertebral body is determined according to the ratio of the vertebral body width to the height in the RGB image / X-ray image.

[0034] Due to the adoption of the above technical solution, the beneficial effects of this application are as follows:

[0035] 1. This application has designed a two-stage detection strategy. First, in S4), coarse-grained spinal lines are generated based on the maximum left and right offsets for coarse-grained screening. This approach can take into account both detection accuracy and efficiency, helps to achieve large-scale screening, and paves the way for fine-grained screening, which helps to improve the accuracy and efficiency of fine-grained screening.

[0036] 2. This application enables highly accurate and efficient scoliosis screening using only a single RGB image. Model training expands the database through brightening and darkening, improving the model's robustness and generalization capabilities, and contributing to improved prediction accuracy. Using only a single RGB image can significantly increase the reach and convenience of scoliosis screening.

[0037] 3. This application realizes non-invasive detection. The technical solution of this application does not require expensive professional equipment and instruments. RGB images of the subject's back can be collected through common imaging devices such as mobile phones and cameras, avoiding the radiation risks of traditional X-ray examinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0039] Figure 1 A flowchart of a scoliosis measurement method based on RGB images provided in this application;

[0040] Figure 2 A schematic diagram of the locations of key points in this application;

[0041] Figure 3 This is an architectural diagram of the key point correction algorithm described in this application;

[0042] Figure 4 Schematic diagram of vertebral body sampling in step S5);

[0043] Figure 5 Schematic diagram of the calculation of the Cobb angle in step S5). DETAILED DESCRIPTION

[0044] Based on the background technology, see the attached Figure 1 , the present application provides a scoliosis measurement method based on RGB images, comprising the following steps:

[0045] S1) obtaining an RGB / BGR image of the subject's back;

[0046] During training, the acquired RGB / BGR images can be augmented with data to form a training set through brightening and darkening processing. The specific operation process of brightening and darkening processing is to multiply each pixel value in the image by a number greater than 1 and less than 1, so as to make the image variable or darker. As an example and not a limitation, in the brightening process, the multiplied constant is defined as the brightness factor. When the brightness factor is 1, the image maintains the original brightness. When the brightness factor increases linearly to 1.4, the image brightness also increases accordingly. The image after the increase is 40% brighter than the original image. The same applies to the darkening process.

[0047] S2) extracting the shoulder and back region of the RGB / BGR image as the target region, and identifying and marking the positions of key points in the target region.

[0048] The key points include fixed points and dynamic points. The fixed points are located at the left / right inner shoulder, left / right outer shoulder, left / right armpit, innermost side of left / right waist, and top of intergluteal groove at the edge of the target area; the dynamic points are distributed at equal heights at the edge of the area surrounded by two groups of adjacent fixed points; the key points at corresponding positions on the left and right sides of the spine form a group of key point pairs.

[0049] Step S2) can be achieved using a target detection algorithm. In this application, the target area is the shoulder and back. The subject stands upright with their arms hanging naturally at their sides. The target detection algorithm removes unnecessary areas based on the settings, retaining the shoulder and back area.

[0050] See attached Figure 3 In this embodiment, the Faster-RCNN target detection algorithm is used, which is an improved version of the R-CNN series of methods. By introducing the Region Proposal Network (RPN), the detection speed and accuracy are significantly improved.

[0051] As a preferred embodiment of this application, as shown in the attached Figure 2As shown in Table 1, there are 41 key points. In addition to the nine fixed points, 16 key point pairs are marked in the target area. It should be noted that for subjects of different body shapes and heights, the number of key points in the area enclosed by the two fixed points is the same, but the spacing between the key points varies. The key point markings are shown in Table 1.

[0052] Table 1 Names and locations of key points

[0053]

[0054] S3) using the loss function in the key point correction algorithm to correct the key point positions in S2), the correction content includes local symmetry correction for normal areas, curvature correction for curved areas, and error correction;

[0055] The loss function is as follows:

[0056]

[0057] is the regression loss function, is the error function between the key point coordinates and the true annotation point coordinates, is the local symmetry constraint loss function, is the loss function for the curved region, and λ1, λ2, and λ3 are weighting coefficients, adjusted through five-fold cross-validation. It should be noted that the RGB / BRG images input to the model contain images with real-world annotations. Through supervised learning, keypoints are corrected based on the real-world annotations.

[0058] See attached Figure 3 The key point correction algorithm architecture includes a data preprocessing module, a backbone network module, a head network module and a post-processing module; the data preprocessing module takes the original RGB image or BGR image as input, uses the mean and standard deviation of each channel to normalize, and converts the BGR image into an RGB image output; the backbone network module takes the normalized RGB image as input, extracts the high-level semantic features of the RGB image and outputs a feature map; the head network takes the feature map as input, and outputs the coordinates of the key points through the convolution layer, ReLU activation function, KL divergence loss function and full connection layer; the post-processing module takes the coordinates of the key points as input, uses The loss function corrects the coordinates of key points.

[0059] As an example, not a limitation, the data preprocessing module uses the mean and standard deviation of each image channel for normalization to reduce pixel value deviations and improve model training stability and convergence speed. The open-source Swin Transformer is selected as the backbone network module to extract high-level semantic features of the image. The advantage of the Swin Transformer is that its window-based self-attention mechanism effectively captures local and global information, making it suitable for extracting structured features from images. In this embodiment, the backbone network module outputs a feature map of size (32 × 32) (with 1024 channels). The convolution kernel of the head network module is 7 × 7, and the convolution stride and padding are adjusted to a size suitable for downstream tasks. The ReLU activation function is used to enhance nonlinear expression capabilities. In the regression task, the KL divergence is used as the loss function to help the model optimize the relative positional relationships of the predicted keypoints. Finally, after passing through the fully connected layer, the output has 41 channels, indicating the coordinates of 41 keypoints. In the post-processing module, since the keypoint coordinates output by the head network may contain noise or outliers, a loss function is used to correct unreasonable keypoint coordinates.

[0060] In the model, the normal area is defined as the area where the angle between the line connecting a set of key point pairs and the horizontal line is less than or equal to the set threshold. The method for local symmetry correction of the normal area is as follows: the target area is divided into several local areas containing at least one set of key point pairs; the center line of the local area is fitted using the quadratic Newton interpolation method, and the coordinates of the key points in each local area are constrained according to the following formula:

[0061]

[0062] Where, L local-symmertry is the local symmetry constraint loss function, x i L and x i R They are the coordinates of the key points on the left and right sides of the local symmetric region, x center is the horizontal coordinate of the center line of the back area, N local Indicates the number of key point pairs contained in the local area.

[0063] The formula for quadratic Newton interpolation is:

[0064]

[0065]

[0066] Where, xi For the label i The horizontal coordinate of the midpoint of the line connecting the key points of N 3 (x) is the Newton fitting function, m is the number of key points, k The order to compute the difference quotient.

[0067] Local symmetry correction is for the normal area, that is, the key points in the normal area are set to be symmetrical, which is used to correct the key point positions in the normal area. The key points in the curved area are asymmetric and need to be determined based on weights.

[0068] The curved area is defined as the area where the angle between the line connecting a set of key point pairs and the horizontal line is greater than the set threshold. The formula for the curvature correction of the curved area is:

[0069]

[0070] Where, w i is the weight of the curved area, which is continuously calculated and updated through iterative training. is the total number of keypoint pairs in the curved area, x i L and x i R These are the coordinates of the key points on the left and right sides of the curved area, respectively.

[0071] The loss of error correction in step S3) is calculated as follows:

[0072]

[0073] Where, N is the number of key points, is the position of the real key point, To predict the location of key points.

[0074] During the simulation phase, key points were labeled by professionals. As an example and not a limitation, the dataset was divided into training, validation, and test sets in a 6:2:2 ratio.

[0075] S4) Connect each pair of key point pairs of the spine to generate key point lines, calculate the midpoint coordinates of each key point line, and connect the two points with the largest left and right offsets to form a coarse-grained spinal line. Calculate the angle between the coarse-grained spinal line and the vertical direction to set a binary classification threshold. When the angle is less than or equal to the binary classification threshold, it is determined that there is no scoliosis. When the angle is greater than the binary classification threshold, it is determined that there is scoliosis.

[0076] The coarse-grained spine line is determined by simply connecting the midpoints of the two largest offsets. This method is simple and fast, and after key point correction, the accuracy is reliable.

[0077] S5) In the thoracic spine, every two pairs of keypoints connected sequentially from top to bottom are considered a vertebra; in the lumbar spine, every three pairs of keypoints are considered a vertebra; the Cobb angle is calculated based on the vertebral bodies and the Cobb angle definition.

[0078] See attached Figure 4 and attached Figure 5 According to the 17 vertebrae of the human body, the thoracic vertebrae are marked as T1~T12, and the lumbar vertebrae are marked as L1~L5. Every two groups of key point pairs of the thoracic vertebrae are connected but not repeated to form a vertebra. The first and third groups of key point pairs of the three groups of lumbar vertebrae are selected to form the vertebrae, and the second group of vertebrae are not connected. The width of the vertebra does not affect the calculation of the Cobb angle. For the convenience of calculation, in this embodiment, the width is 1 / 6 of the distance between the left / right axillary fossa, and the height can be determined based on the vertebral proportions of the RGB image or X-ray image on the premise that the width is determined. Finally, the fitted vertebra is drawn by Opencv, and then the Cobb angle is calculated according to the definition of the Cobb angle. The Cobb angle is defined as the angle between the extension line of the upper end plate of the most inclined vertebra and the extension line of the lower end plate of the most inclined vertebra in the curved segment, as shown in Figure 5 As shown, m1 is the slope of the most inclined upper end plate of the vertebra, m2 is the slope of the most inclined lower end plate of the vertebra, and angle is the Cobb angle.

[0079] As follows:

[0080]

[0081] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0082] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A scoliosis measurement method based on RGB images, characterized in that: The following steps are involved: S1) obtaining an RGB / BGR image of the subject's back; S2) extracting the shoulder and back region of the RGB / BGR image as the target region, and identifying and marking the positions of key points in the target region; The key points include fixed points and dynamic points. The fixed points are located at the edge of the target area, namely the left / right inner shoulder, left / right outer shoulder, left / right armpit, the innermost side of the left / right waist, and the top of the intergluteal groove. The dynamic points are distributed at equal heights at the edge of the area enclosed by two sets of adjacent fixed points. The key points at corresponding positions on the left and right sides of the spine constitute a pair of key points. S3) correcting the key point positions in S2) using a loss function in a key point correction algorithm, the correction contents including local symmetry correction for normal areas, curvature correction for curved areas, and error correction; the normal area is defined as an area where the angle between a line connecting a set of key point pairs and a horizontal line is less than or equal to a set threshold; the curved area is defined as an area where the angle between a line connecting a set of key point pairs and a horizontal line is greater than a set threshold; The loss function is as follows: is the regression loss function, is the error function between the key point coordinates and the true annotation point coordinates, is the local symmetry constraint loss function, is the loss function of the curved area, λ1, λ2, and λ3 are weighting coefficients, which are adjusted by five-fold cross validation; S4) connecting each pair of key points of the spine to generate a key point line, calculating the coordinates of the midpoint of each key point line and connecting the two points with the largest left and right offsets to form a coarse-grained spinal line, calculating the angle between the coarse-grained spinal line and the vertical direction and setting a binary classification threshold. When the angle is less than or equal to the binary classification threshold, it is determined that there is no scoliosis. When the angle is greater than the binary classification threshold, it is determined that there is scoliosis. S5) In the thoracic spine, every two pairs of keypoints connected sequentially from top to bottom are considered a vertebra; in the lumbar spine, every three pairs of keypoints are considered a vertebra; the Cobb angle is calculated based on the vertebral bodies and the Cobb angle definition.

2. The scoliosis measurement method based on RGB images according to claim 1, characterized in that: The key point correction algorithm architecture includes data preprocessing module, backbone network module, head network module and post-processing module; The data preprocessing module takes the original RGB image or BGR image as input, normalizes it using the mean and standard deviation of each channel, and converts the BGR image into an RGB image output; The backbone network module takes the normalized RGB image as input, extracts the high-level semantic features of the RGB image and outputs a feature map; The head network takes the feature map as input and outputs the coordinates of the key points through the convolution layer, ReLU activation function, KL divergence loss function and fully connected layer; The post-processing module takes the coordinates of the key points as input and uses The loss function corrects the coordinates of key points.

3. The scoliosis measurement method based on RGB images according to claim 1, characterized in that: In step S3), the method for local symmetry correction of the normal area is as follows: the target area is divided into several local areas containing at least one set of key point pairs; the center lines of the local areas are fitted using the quadratic Newton interpolation method, and the coordinates of the key points in each local area are constrained according to the following formula: Where, L local-symmertry is the local symmetry constraint loss function, x i L and x i R They are the coordinates of the key points on the left and right sides of the local symmetric region, x center is the horizontal coordinate of the center line of the back area, N local Indicates the number of key point pairs contained in the local area.

4. The method for measuring scoliosis based on RGB images according to claim 3, wherein: In step S3), the formula for the bending correction of the curved area is: Where, w i is the weight of the curved region, is the total number of keypoint pairs in the curved area, x i L and x i R These are the coordinates of the key points on the left and right sides of the curved area, respectively.

5. The method for measuring scoliosis based on RGB images according to claim 4, characterized in that: The loss of error correction in step S3) is calculated as follows: Where, N is the number of key points, is the position of the real key point, To predict the location of key points.

6. The method for measuring scoliosis based on RGB images according to claim 1, characterized in that: The total number of key points is 41.

7. The method for measuring scoliosis based on RGB images according to claim 1, characterized in that: In step S5), the fitted vertebral body is drawn using OpenCV. The width of the vertebral body is 1 / 6 of the distance between the left and right axillae, and the height of the vertebral body is determined according to the ratio of the vertebral body width to height in the RGB image / X-ray image.

Images