Scoliosis measuring method based on RGB image

Abstract

The invention relates to a scoliosis measurement method based on an RGB image. The scoliosis measurement method comprises the following steps: acquiring an RGB image, extracting a target area, identifying and marking key points in the target area, correcting the positions of the key points by using a key point correction algorithm, performing preliminary screening by establishing a coarse-grained spine curve, and calculating a Cobb angle through fine-grained spine fitting. The problem that large-range accurate screening cannot be realized in scoliosis detection is solved from the classification precision level.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a method for measuring scoliosis based on RGB images. Background Art

[0002] Scoliosis is a complex three-dimensional spinal deformity that mainly occurs in adolescent children. Early identification of scoliosis, grading of disease severity and effective intervention are particularly important.

[0003] Existing scoliosis detection methods generally fail to balance accuracy, detection efficiency, and non-invasiveness. For example, the traditional Adam test and trunk rotator examination have the problems of low classification accuracy, complex operation, and long time consumption in clinical diagnosis; the moiré fringe method has the advantages of being fast and non-invasive, but its accuracy is easily affected by optical layout, lighting conditions, and residual gratings; EOS three-dimensional imaging technology can accurately measure key parameters such as cobb angle and AVR angle, but its equipment is expensive and not portable, which limits its application in wide screening; similarly, X-ray image analysis based on neural networks has accuracy and detection efficiency, but cannot solve the problems of radiation damage and high cost.

[0004] Image classification technology based on deep learning has been applied to the severity classification of naked back images, and its classification performance is close to that of human experts, but the accuracy rate is only 55.5%. The custom RIM cutout model and YOLOv5 model can quickly detect scoliosis, but they do not achieve severity classification.

[0005] In summary, existing technologies have been able to initially achieve a balance between accuracy, detection efficiency, and non-invasiveness, but in terms of classification accuracy, they are still unable to achieve large-scale screening in clinical practice. Summary of the invention

[0006] The present application provides a scoliosis measurement method based on RGB images, which solves the problem that scoliosis detection cannot achieve large-scale accurate screening from the level of classification accuracy.

[0007] The technical solution of this application is as follows:

[0008] A method for measuring scoliosis based on RGB images, comprising the following steps:

[0009] S1) obtaining an RGB / BGR image of the back of the subject;

[0010] S2) extracting the shoulder and back region of the RGB / BGR image as the target region, and identifying and marking the positions of key points in the target region;

[0011] The key points include fixed points and dynamic points. The fixed points are located at the left / right inner shoulder, left / right outer shoulder, left / right armpit, innermost side of left / right waist, and top of intergluteal groove at the edge of the target area. The dynamic points are distributed at equal height differences at the edge of the area surrounded by two groups of adjacent fixed points. The key points at the corresponding positions on the left and right sides of the spine are a group of key point pairs.

[0012] S3) using the loss function in the key point correction algorithm to correct the key point position in S2), wherein the correction content includes local symmetry correction for the normal area, bending correction for the curved area, and error correction;

[0013] The loss function is as follows:

[0014] L total =λ1·L MSE +λ2·L local-symmetry +λ3·L curvature

[0015] L total is the regression loss function, L MSE is the error function between the key point coordinates and the true annotation point coordinates, L local-symmetry is the local symmetry constraint loss function, L curvature is the loss function of the curved area, λ1, λ2, and λ3 are weighting coefficients, which are adjusted by five-fold cross validation;

[0016] S4) connecting each group of key point pairs of the spine to generate key point lines, calculating the coordinates of the midpoints of each key point line and connecting the two points with the largest leftward and rightward offsets to form a coarse-grained spine line, calculating the angle between the coarse-grained spine line and the vertical direction to set a binary classification threshold, when the angle is less than or equal to the binary classification threshold, it is determined that there is no scoliosis, and when the angle is greater than the binary classification threshold, it is determined that there is scoliosis;

[0017] S5) In the thoracic vertebrae, every two pairs of key points connected in sequence from top to bottom are taken as a vertebrae; in the lumbar vertebrae, every three pairs of key points connected in sequence from top to bottom are taken as a vertebrae; the Cobb angle is calculated based on the vertebrae and the Cobb angle definition.

[0018] Furthermore, the key point correction algorithm architecture includes a data preprocessing module, a backbone network module, a head network module, and a post-processing module;

[0019] The data preprocessing module takes the original RGB image or BGR image as input, normalizes it using the mean and standard deviation of each channel, and converts the BGR image into an RGB image output;

[0020] The backbone network module takes the normalized RGB image as input, extracts the high-level semantic features of the RGB image and outputs a feature map;

[0021] The head network takes the feature map as input, and outputs the coordinates of the key points through the convolution layer, ReLU activation function, KL divergence loss function and fully connected layer;

[0022] The post-processing module takes the coordinates of the key points as input and uses L total The loss function corrects the coordinates of key points.

[0023] Further, in step S3), the normal area is defined as an area where the angle between a line connecting a set of key point pairs and a horizontal line is less than or equal to a set threshold, and the method for local symmetry correction of the normal area is: dividing the target area into a number of local areas containing at least one set of key point pairs; fitting the center line of the local area using the quadratic Newton interpolation method, and constraining the key point coordinates in each local area according to the following formula:

[0024]

[0025] Where, L local-symmertry is the local symmetry constraint loss function, x i L and x i R are the coordinates of the key points on the left and right sides of the local symmetric region, respectively, center is the horizontal coordinate of the center line of the back area, N local Indicates the number of key point pairs contained in the local area.

[0026] Further, in step S3), the curved area is defined as an area where the angle between a set of key point pairs and the horizontal line is greater than a set threshold, and the formula for the curvature correction of the curved area is:

[0027]

[0028] In the formula, w i is the weight of the bending area, N eurnature is the total number of keypoint pairs in the curved area, x i L and x i R These are the coordinates of the key points on the left and right sides of the curved area, respectively.

[0029] Furthermore, the loss of error correction in step S3) is calculated as follows:

[0030]

[0031] Where N is the number of key points, y i is the position of the real key point, To predict the location of key points.

[0032] Furthermore, the total number of key points is 41.

[0033] Further, in step S5), the fitted vertebral body is drawn using Opencv, the width of the vertebral body is 1 / 6 of the distance between the left / right axilla, and the height of the vertebral body is determined according to the ratio of the vertebral body width to the height in the RGB image / X-ray image.

[0034] Due to the adoption of the above technical solution, the beneficial effects of this application are as follows:

[0035] 1. This application has designed a two-stage detection strategy in terms of method. First, in S4), a coarse-grained spinal line is generated according to the left and right maximum offsets for coarse-grained screening. This method can take into account both detection accuracy and efficiency, help to achieve large-scale screening, and pave the way for fine-grained screening, which helps to improve the accuracy and efficiency of fine-grained screening.

[0036] 2. This application can achieve high-precision and high-efficiency scoliosis screening with only one RGB image. Model training expands the database through brightening and darkening processing, improves the robustness and generalization ability of the model, and helps to improve the prediction accuracy of the model. Only by using one RGB image, the popularity of scoliosis screening can be greatly improved, and the convenience of detection can be greatly improved.

[0037] 3. The present application realizes non-invasive detection. The technical solution of the present application does not require expensive professional equipment and instruments. The RGB image of the subject's back can be collected through common imaging devices such as mobile phones and cameras, avoiding the radiation risk of traditional X-ray examinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0039] Figure 1 A flow chart of a scoliosis measurement method based on RGB images provided in this application;

[0040] Figure 2 A schematic diagram of the locations of key points in this application;

[0041] Figure 3 This is a diagram of the architecture of the key point correction algorithm described in this application;

[0042] Figure 4 Schematic diagram of vertebral sampling in step S5);

[0043] Figure 5 Schematic diagram of calculation of Cobb angle in step S5). DETAILED DESCRIPTION

[0044] Based on the background technology, see the attached Figure 1 The present application provides a method for measuring scoliosis based on RGB images, comprising the following steps:

[0045] S1) obtaining an RGB / BGR image of the back of the subject;

[0046] During training, the acquired RGB / BGR images can be expanded by brightening and darkening to form a training set. The specific operation process of brightening and darkening is to multiply each pixel value in the image by a number greater than 1 and less than 1, so as to make the image variable or darker. As an example and not a limitation, in the brightening process, the multiplied constant is defined as the brightness factor. When the brightness factor is 1, the image maintains the original brightness. The brightness factor increases linearly to 1.4, and the image brightness also increases accordingly. The image after the increase is 40% brighter than the original image. The same applies to the darkening process.

[0047] S2) extracting the shoulder and back region of the RGB / BGR image as the target region, and identifying and marking the positions of key points in the target region.

[0048] The key points include fixed points and dynamic points. The fixed points are located at the left / right inner shoulder, left / right outer shoulder, left / right armpit, innermost side of left / right waist, and top of intergluteal groove at the edge of the target area; the dynamic points are distributed at equal heights at the edge of the area surrounded by two groups of adjacent fixed points; the key points at corresponding positions on the left and right sides of the spine are a group of key point pairs.

[0049] Step S2) can be achieved using a target detection algorithm. The target area in this application is the shoulder and back area. The subject stands upright with his arms hanging naturally on both sides of the body. The target detection algorithm removes unnecessary positions according to the settings and retains the shoulder and back area.

[0050] See attached Figure 3 In this embodiment, the Faster-RCNN target detection algorithm is used, which is an improved version of the R-CNN series of methods. By introducing the Region Proposal Network (RPN), the detection speed and accuracy are significantly improved.

[0051] As a preferred embodiment of the present application, Figure 2 As shown in Table 1, the number of key points is 41. In addition to the nine fixed points, 16 key point pairs are marked in the target area. It should be noted that for subjects of different body shapes and heights, the number of key points in the area surrounded by the two sets of fixed points is the same, but the spacing between the key points is different. The marking of the key points is shown in Table 1.

[0052] Table 1 Names and locations of key points

[0053] name Location 1 / 2 Left / right inner shoulder 9 / 10 Left / right outer shoulder 15 / 16 Left / right armpit 29 / 30 Left / right inner side of waist 41 Top of intergluteal groove Dynamic Points The remaining points are equally spaced

[0054] S3) using the loss function in the key point correction algorithm to correct the key point position in S2), wherein the correction content includes local symmetry correction for the normal area, bending correction for the curved area, and error correction;

[0055] The loss function is as follows:

[0056] L total =λ1·L MSE +λ2·L local-symmetry +λ3·L curvature

[0057] L total is the regression loss function, L MSE is the error function between the key point coordinates and the true annotation point coordinates, L local-symmetry is the local symmetry constraint loss function, L curvature is the loss function of the curved area, λ1, λ2, and λ3 are weighted coefficients, which are adjusted through five-fold cross validation. It should be noted that the RGB / BRG images input to the model contain images with real annotation points. Through supervised learning, the key points are corrected according to the real annotation points.

[0058] See attached Figure 3 The key point correction algorithm architecture includes a data preprocessing module, a backbone network module, a head network module and a post-processing module; the data preprocessing module takes the original RGB image or BGR image as input, uses the mean and standard deviation of each channel for normalization, and converts the BGR image into an RGB image for output; the backbone network module takes the normalized RGB image as input, extracts the high-level semantic features of the RGB image and outputs a feature map; the head network takes the feature map as input, and outputs the coordinates of the key points through a convolutional layer, a ReLU activation function, a KL divergence loss function and a fully connected layer; the post-processing module takes the coordinates of the key points as input, uses L total The loss function corrects the coordinates of key points.

[0059] As an example but not a limitation, the mean and standard deviation of each channel of the image are used for normalization in the data preprocessing module to reduce the deviation of the image pixel value and improve the stability and convergence speed of the model training. The backbone network module is selected as the open source Swin Transformer to extract high-level semantic features of the image. The advantage of Swin Transformer is that its window-based self-attention mechanism can effectively capture local and global information, which is suitable for structured feature extraction in images. In this embodiment, the backbone network module outputs a feature map of size (32×32) (the number of channels is 1024). The convolution kernel of the head network module is 7×7, and the convolution step size and padding method are adjusted to a size suitable for downstream tasks. The ReLU activation function is used to enhance the nonlinear expression ability. KL divergence is used as a loss function in the regression task to help the model optimize the relative position relationship of the key point prediction. Finally, after the fully connected layer, the output channel number is 41, that is, the coordinates of 41 key points. In the post-processing module, since the key point coordinates output by the head network may contain noise or outliers, it is necessary to use a loss function to correct unreasonable key point coordinates.

[0060] In the model, the normal area is defined as the area where the angle between the line connecting a set of key point pairs and the horizontal line is less than or equal to the set threshold. The method for local symmetry correction of the normal area is: divide the target area into several local areas containing at least one set of key point pairs; use the quadratic Newton interpolation method to fit the center line of the local area, and constrain the key point coordinates in each local area according to the following formula:

[0061]

[0062] Where, L local-symmertry is the local symmetry constraint loss function, x i L and x i R are the coordinates of the key points on the left and right sides of the local symmetric region, respectively, center is the horizontal coordinate of the center line of the back area, N local Indicates the number of key point pairs contained in the local area.

[0063] The formula for quadratic Newton interpolation is:

[0064] N3(x)=f(x0)+f[x0,x1](x-x0)+

[0065] f[x0,x1,x2](x-x0)(x-x1)

[0066]

[0067] In the formula, x iis the horizontal coordinate of the midpoint of the line connecting the key points labeled i, N3(x) is the Newton fitting function, m is the number of key points, and k is the order of difference quotient calculation.

[0068] Local symmetry correction is for the normal area, that is, the key point pairs in the normal area are set to be symmetrical, which is used to correct the key point positions in the normal area. The key points in the curved area are asymmetric and need to be determined based on weights.

[0069] The curved area is defined as the area where the angle between a set of key point pairs and the horizontal line is greater than the set threshold. The formula for the curvature correction of the curved area is:

[0070]

[0071] In the formula, w i is the weight of the curved area, which is continuously calculated and updated through iterative training. eurnature is the total number of keypoint pairs in the curved area, x i L and x i R These are the coordinates of the key points on the left and right sides of the curved area, respectively.

[0072] The loss of error correction in step S3) is calculated as follows:

[0073]

[0074] Where N is the number of key points, y i is the position of the real key point, To predict the location of key points.

[0075] In the simulation stage, key points are labeled by professionals. As an example but not a limitation, the dataset is divided into training set, validation set, and test set in a ratio of 6:2:2.

[0076] S4) Connect each group of key point pairs of the spine to generate key point lines, calculate the midpoint coordinates of each key point line and connect the two points with the largest left and right offsets to form a coarse-grained spinal line, calculate the angle between the coarse-grained spinal line and the vertical direction to set a binary classification threshold, when the angle is less than or equal to the binary classification threshold, it is determined that there is no scoliosis, and when the angle is greater than the binary classification threshold, it is determined that there is scoliosis.

[0077] The coarse-grained spine line is determined by simply connecting the midpoints of the two largest offsets. This method is simple and fast, and after key point correction, the accuracy is reliable.

[0078] S5) In the thoracic vertebrae, every two pairs of key points connected in sequence from top to bottom are taken as a vertebrae; in the lumbar vertebrae, every three pairs of key points connected in sequence from top to bottom are taken as a vertebrae; the Cobb angle is calculated based on the vertebrae and the Cobb angle definition.

[0079] See attached Figure 4 and attached Figure 5 According to the 17 vertebrae of the human body, the thoracic vertebrae are marked as T1 to T12, and the lumbar vertebrae are marked as L1 to L5. Every two groups of key point pairs in the thoracic vertebrae are connected but not repeated to form a vertebra. The first and third groups of key point pairs of the three groups of lumbar vertebrae are selected to form a vertebra, and the second group of vertebrae are not connected. The width of the vertebra does not affect the calculation of the Cobb angle. For the convenience of calculation, in this embodiment, the width is 1 / 6 of the distance between the left / right axillary, and the height can be determined based on the vertebral proportion of the RGB image or the X-ray image on the premise that the width is determined. Finally, the fitted vertebra is drawn through Opencv, and then the Cobb angle is calculated according to the definition of the Cobb angle. The Cobb angle is defined as the angle between the extension line of the upper end plate of the most inclined vertebra and the extension line of the lower end plate of the most inclined vertebra in the curved segment, such as Figure 5 As shown, m1 is the slope of the most inclined upper end plate of the vertebra, m2 is the slope of the most inclined lower end plate of the vertebra, and angle is the Cobb angle.

[0080] As follows:

[0081]

[0082] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0083] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for measuring scoliosis based on RGB images, characterized in that: The following steps are involved: S1) obtaining an RGB / BGR image of the back of the subject; S2) extracting the shoulder and back region of the RGB / BGR image as the target region, identifying and marking the positions of key points in the target region; The key points include fixed points and dynamic points. The fixed points are located at the left / right inner shoulder, left / right outer shoulder, left / right armpit, innermost side of left / right waist, and top of intergluteal groove at the edge of the target area. The dynamic points are distributed at equal height differences at the edge of the area surrounded by two groups of adjacent fixed points. The key points at the corresponding positions on the left and right sides of the spine are a group of key point pairs. S3) using the loss function in the key point correction algorithm to correct the key point position in S2), wherein the correction content includes local symmetry correction for the normal area, bending correction for the curved area, and error correction; The loss function is as follows: L total =λ1·L MSE +λ2·L local-symmetry +λ3·L curvature L total is the regression loss function, L MSE is the error function between the key point coordinates and the true annotation point coordinates, L local-symmetry is the local symmetry constraint loss function, L curvature is the loss function of the curved area, λ1, λ2, and λ3 are weighting coefficients, which are adjusted by five-fold cross validation; S4) connecting each group of key point pairs of the spine to generate key point lines, calculating the coordinates of the midpoints of each key point line and connecting the two points with the largest leftward and rightward offsets to form a coarse-grained spine line, calculating the angle between the coarse-grained spine line and the vertical direction to set a binary classification threshold, when the angle is less than or equal to the binary classification threshold, it is determined that there is no scoliosis, and when the angle is greater than the binary classification threshold, it is determined that there is scoliosis; S5) In the thoracic vertebrae, every two pairs of key points connected in sequence from top to bottom are taken as a vertebrae; in the lumbar vertebrae, every three pairs of key points connected in sequence from top to bottom are taken as a vertebrae; the Cobb angle is calculated based on the vertebrae and the Cobb angle definition.

2. The method for measuring scoliosis based on RGB images according to claim 1, characterized in that: The key point correction algorithm architecture includes data preprocessing module, backbone network module, head network module and post-processing module; The data preprocessing module takes the original RGB image or BGR image as input, normalizes it using the mean and standard deviation of each channel, and converts the BGR image into an RGB image output; The backbone network module takes the normalized RGB image as input, extracts the high-level semantic features of the RGB image and outputs a feature map; The head network takes the feature map as input, and outputs the coordinates of the key points through the convolution layer, ReLU activation function, KL divergence loss function and fully connected layer; The post-processing module takes the coordinates of the key points as input and uses L total The loss function corrects the coordinates of key points.

3. The method for measuring scoliosis based on RGB images according to claim 1, characterized in that: In step S3), the normal area is defined as an area where the angle between a line connecting a set of key point pairs and a horizontal line is less than or equal to a set threshold. The method for local symmetry correction of the normal area is: dividing the target area into a number of local areas containing at least one set of key point pairs; fitting the center line of the local area using the quadratic Newton interpolation method, and constraining the key point coordinates in each local area according to the following formula: Where, L local-symmertry is the local symmetry constraint loss function, x i L and x i R are the coordinates of the key points on the left and right sides of the local symmetric region, respectively, center is the horizontal coordinate of the center line of the back area, N local Indicates the number of key point pairs contained in the local area.

4. The method for measuring scoliosis based on RGB images according to claim 3, characterized in that: In step S3), the curved area is defined as an area where the angle between a set of key point pairs and the horizontal line is greater than a set threshold, and the formula for the curvature correction of the curved area is: In the formula, w i is the weight of the bending area, N eurnature is the total number of keypoint pairs in the curved area, x i L and x i R These are the coordinates of the key points on the left and right sides of the curved area, respectively.

5. The method for measuring scoliosis based on RGB images according to claim 4, characterized in that: The loss of error correction in step S3) is calculated as follows: Where N is the number of key points, y i is the position of the real key point, To predict the location of key points.

6. The method for measuring scoliosis based on RGB images according to claim 1, characterized in that: The total number of key points is 41.

7. The method for measuring scoliosis based on RGB images according to claim 1, characterized in that: In step S5), the fitted vertebral body is drawn using Opencv, the width of the vertebral body is 1 / 6 of the distance between the left / right axilla, and the height of the vertebral body is determined according to the ratio of the vertebral body width to the height in the RGB image / X-ray image.

Images