Method for calculating a scoliosis index in a back profile image

Abstract

The present application belongs to the field of image data processing, and proposes a method for calculating scoliosis index in back contour image, comprising the following steps: first, obtaining back contour image, and obtaining shoulder feature curve and thoracolumbar segment feature curve from the back contour image; second, registering the shoulder feature curve and the thoracolumbar segment feature curve; then, calculating the difference value of the registered shoulder feature curve, the difference value of the upper end point of the thoracolumbar segment feature curve and the difference value of the thoracolumbar segment feature curve respectively, and obtaining the scoliosis index through the calculated difference value. The present application can detect back contour curve according to human back contour image, and judge whether the scoliosis index is normal through the shoulder curve and the thoracolumbar segment curve in the back contour curve, and further judge posture abnormality or suspected scoliosis.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method for calculating the scoliosis index in a back contour image. Background Technology

[0002] Scoliosis is a three-dimensional structural deformity of the spine with a lateral curvature of ≥10° in the coronal plane (Cobb method). It is often accompanied by vertebral rotation and increased or decreased kyphosis or lordosis in the sagittal plane, rotational and tilting deformities of the ribs and pelvis, and abnormalities of the paravertebral ligaments and muscles. The global incidence is 2%–3%, while in China it is approximately 0.11%–3.51%. This disease is most common during peak developmental periods, particularly childhood and adolescence. Idiopathic scoliosis (IS) of unknown cause accounts for 85% of cases. If IS is not detected effectively in its early stages in children and adolescents, it can progress rapidly, leading to proprioceptive disturbances, motor imbalances, and even lower back pain, cardiopulmonary injury, and increased risk of death. Currently, there are no highly efficient and accurate methods for large-scale screening of scoliosis. Currently, hospitals primarily rely on visual inspection, using tools such as protractors, markers, torsion bottles, and scoliosis measuring instruments to measure Cobb angles and make diagnoses based on these measurements. Researchers are also continuously exploring alternative methods, such as 3D ultrasound, thermal infrared imaging, and Moiré imaging. However, all of these methods suffer from low accuracy, inefficiency, and a lack of self-learning capabilities, failing to meet the public health service needs for scoliosis. Summary of the Invention

[0003] The purpose of this invention is to provide a method for calculating the scoliosis index in a back contour image. By processing the back contour image, the method can obtain the shoulder feature curve and the thoracolumbar feature curve, accurately calculate the difference between the shoulder feature curve, the difference between the upper endpoints of the thoracolumbar feature curve, and the difference between the thoracolumbar feature curve, and accurately reflect the scoliosis index through the three differences.

[0004] The technical solution adopted by this invention to solve its technical problem is as follows:

[0005] A method for calculating the scoliosis index in a back contour image includes the following steps:

[0006] Obtain the back contour image, and extract the shoulder feature curve and thoracolumbar feature curve from the back contour image;

[0007] Register the shoulder feature curve and the thoracolumbar feature curve;

[0008] The differences between the shoulder characteristic curves, the upper endpoints of the thoracolumbar characteristic curves, and the thoracolumbar characteristic curves after registration are calculated separately, and the scoliosis index is obtained through the calculated differences.

[0009] As a further optimization, after obtaining the back contour image, the image annotation tool labelme is used for annotation, and the annotated areas are the contour areas of the shoulders and the lumbar region.

[0010] As a further optimization, the labeled back contour images are divided into training and testing sets in a ratio of 7:3.

[0011] As a further optimization, a Mask R-CNN neural network was used to obtain shoulder feature curves and thoracolumbar feature curves from the back contour image.

[0012] As a further optimization, the step of using a Mask R-CNN neural network to obtain shoulder feature curves and thoracolumbar feature curves from the back contour image includes:

[0013] The back contour image is used as the input to the Mask R-CNN neural network. The size of the input image is H×W, and then it is fed into the backbone network.

[0014] Features were extracted using a ResNet50 network as a feature extractor, resulting in a 5-layer feature map with the following sizes:

[0015]

[0016] Feature fusion is performed using the Feature Pyramid Network (FPN) structure to obtain enhanced features;

[0017] In the Region Candidate Network (RPN) processing stage, each feature map of size h×w generates h×w×9 bounding boxes. For all boxes, a classification and regression task is performed. The classification task is used to determine whether there is a feature curve in the box and calculate the probability of having a feature curve. The regression task is used to generate the coordinates of each box.

[0018] The special layer ROI Align for object detection unifies bounding boxes of different sizes into a single dimension;

[0019] The category of the feature curve and its coordinates in the image are obtained through fully connected (FC) layers.

[0020] As a further optimization, the calculation of the difference in the shoulder feature curves after registration, and the scoliosis index obtained from the calculated difference, includes:

[0021] Determine the right shoulder contour line s1 and the left shoulder contour line s2 in the shoulder feature curve, and select a cutoff line at the neck position. The point where the left shoulder contour line s2 intersects the cutoff line is the cutoff point a.

[0022] After horizontally flipping s1, we get s1'. Then, we translate s1' so that its upper endpoint coincides with point a.

[0023] For each pixel on the horizontal axis, calculate the difference between pixels on the vertical axis, take the absolute value and find the maximum value as ais. If ais is greater than 5 and less than 10, it indicates incorrect posture and uneven shoulders. The sign of ais is used to determine the height of the left and right shoulders. If the absolute value of ais is greater than 10, it indicates suspected scoliosis. If the absolute value of ais is less than 5, it is within the normal range.

[0024] As a further optimization, the calculation of the difference between the upper endpoints of the characteristic curves of the thoracolumbar region, and the scoliosis index obtained from the calculated difference, includes:

[0025] Determine the right thoracolumbar characteristic curve L1 and the left thoracolumbar characteristic curve L2 in the thoracolumbar characteristic curve, and obtain the upper endpoint at1 of L1 and the upper endpoint at2 of L2.

[0026] The absolute value of the height difference between the endpoints at1 and at2 along the vertical axis is Ais_at;

[0027] If Ais_at is less than 5, it indicates normal posture; if Ais_at is greater than 5 but less than 10, it indicates incorrect posture; if Ais_at is greater than 10, it indicates suspected scoliosis.

[0028] As a further optimization, the calculation of the difference in the characteristic curves of the thoracolumbar region, and the scoliosis index obtained from the calculated difference, includes:

[0029] Flip L1 horizontally to get L1` and at1`;

[0030] Translate L1' so that at1' and at2 coincide;

[0031] Given that the x-axis point set of L1' is {x1,x2,x3,...,xn} and the x-axis point set of L2 is {x1`,x2`,x3`,...,xn`}, calculate the difference between the two point sets. Take the absolute value of each difference and find the maximum value as ais_L. If ais_L is less than 5, it indicates normal posture; if ais_L is greater than 5 and less than 10, it indicates abnormal posture; if ais_L is greater than 10, it indicates suspected scoliosis.

[0032] The beneficial effects of the present invention are: by using the above-mentioned method for calculating the scoliosis index in a back contour image, the back contour curve can be detected based on the human back contour image, and the shoulder curve and thoracolumbar curve in the back contour curve can be used to determine whether the scoliosis index is normal, thereby determining abnormal posture or suspected scoliosis. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a method for calculating the scoliosis index in a back contour image according to an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the Mask R-CNN network structure in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the shoulder contour line in an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the thoracolumbar contour curve in an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of horizontal flipping using L1 in an embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram of the RCNN+ResNet50+FPN network structure in an embodiment of the present invention.

[0039] Figure 7 This is a schematic diagram of ICP registration of the contour curve in an embodiment of the present invention.

[0040] Figure 8 As described in embodiment I of the present invention ais A diagram illustrating the value calculation process. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0042] Example

[0043] This embodiment provides a method for calculating the scoliosis index in a back contour image, the flowchart of which is shown below. Figure 1 The method includes the following steps:

[0044] S1. Obtain the back contour image, and extract the shoulder feature curve and the thoracolumbar feature curve from the back contour image;

[0045] S2. Register the shoulder feature curve and the thoracolumbar feature curve;

[0046] S3. Calculate the difference between the shoulder characteristic curve, the difference between the upper endpoints of the thoracolumbar characteristic curve, and the difference between the thoracolumbar characteristic curve after registration, and obtain the scoliosis index through the calculated difference.

[0047] In practical applications, scoliosis can cause changes in a patient's body shape, manifesting as an uneven back, chest deformity, pelvic tilt, leg length discrepancy, and poor posture. In this embodiment, the patient's body curve can be acquired, and after extracting body features, deep learning processing can be performed to directly measure whether the patient has scoliosis. Therefore, by acquiring back images of the subject in a standing posture using a camera, a Mask R-CNN neural network model based on the back contour curve is trained. Using the Mask R-CNN network, an accurate target mask can be generated to detect the back contour curve, establishing a back contour fitting model. The algorithm evaluates the degree of difference in the contour curve model, achieving rapid, harmless, and non-contact detection of postural abnormalities and scoliosis.

[0048] In this embodiment, a back contour feature curve detection model can be established first, and the shoulder feature curve and thoracolumbar segment feature curve can be obtained from the back contour image. Then, a back contour feature curve model can be established to accurately calculate the scoliosis index.

[0049] When building a back contour feature curve detection model, after acquiring the back contour image, to enable the model to learn the features of specific areas, the image annotation tool LabelMe can be used for annotation before network training. The annotated areas are the contour regions of the shoulders and thoracolumbar region. All annotated data is divided into training and test sets in a 7:3 ratio. When the total amount of data is large, a more common split is a 6:2:2 ratio for the training, validation, and test sets.

[0050] To obtain the contour curve, a Mask R-CNN neural network can be used. In this embodiment, the Mask R-CNN used follows the idea of ​​Faster R-CNN, employing a ResNet (Residual Network)-FPN (Feature Pyramid Network) architecture for feature extraction, and adding a Mask prediction branch to predict the binary mask. It can not only detect objects in the image but also provide a high-quality segmentation result for each object. Furthermore, it can be extended to other tasks such as keypoint detection. Figure 2 As shown.

[0051] In the Mask R-CNN network structure, the input image (size H×W) is used. In the backbone stage, a ResNet50 network is used as the feature extractor to extract features, resulting in 5 layers of feature maps with the following sizes: In image processing, low-level features often contain a lot of detailed information (color, contour, texture). Therefore, the FPN (Feature Fusion Network) structure is used for feature fusion to obtain enhanced features. In the RPN (Region Proposal Network) processing stage, each feature map (h x w) generates h × w × 9 bounding boxes. For all boxes, a classification and regression task is performed. Classification determines whether an object is inside the box and calculates the probability of its presence, while regression generates the coordinates (top, bottom right) of each box. ROI Align unifies the boxes of different sizes to a single dimension. Finally, fully connected layers (FC) obtain the object category and its coordinates in the image.

[0052] In another embodiment, the model used to obtain the contour curve can also be Mask RCNN+ResNet50+FPN, and the specific network structure of this model can be found in [link to documentation]. Figure 6 This study employs a Mask R-CNN+ResNet50+FPN model to obtain the back mask image, and then extracts the shoulder feature curves and the thoracolumbar segment feature curves. Mask R-CNN follows the idea of ​​Faster R-CNN, using the ResNet-FPN architecture for feature extraction, and adds a mask prediction branch to predict the binary mask. This not only detects objects in the image but also provides a high-quality segmentation result for each object, and can also be extended to other tasks such as keypoint detection.

[0053] When the back contour feature curve model is established, it is mainly based on the shoulder feature curve and thoracolumbar feature curve obtained by the back contour feature curve detection model.

[0054] First, based on the two types of characteristic curves (chest-waist and shoulder) obtained in the previous steps, there are a total of four curves. The shoulder curves can be used to determine if there is uneven shoulder posture, such as... Figure 3As shown. s1 and s2 are the outlines of the right and left shoulders, respectively. A cutoff line is selected at the neck position. Therefore, the point where the left shoulder outline s2 intersects the cutoff line is cutoff point a. s1 is horizontally flipped to obtain s1'. s1' is translated so that its upper endpoint coincides with point a. Finally, for each pixel on the horizontal axis, the difference between pixels on the vertical axis is calculated. The absolute value is taken to find the maximum value, ais. If ais is greater than 5 and less than 10, it indicates incorrect posture and uneven shoulders. The sign of ais is used to determine the height of the left and right shoulders. If the absolute value of ais is greater than 10, it indicates suspected scoliosis; if the absolute value of ais is less than 5, it is within the normal range.

[0055] Secondly, for the thoracolumbar curve, such as Figure 4 As shown in the figure, L1 and L2 are the left and right thoracolumbar curves, respectively, and at1 and at2 are the upper endpoints of the left and right thoracolumbar curves, respectively. Therefore, the absolute value of the height difference between the vertical axes of the endpoints at1 and at2 is Ais_at. If Ais_at is less than 5, it indicates normal posture; if Ais_at is greater than 5 but less than 10, it indicates incorrect posture; and if Ais_at is greater than 10, it indicates suspected scoliosis.

[0056] Finally, flip L1 horizontally to obtain L1' and at1'. Then, translate L1' so that at1' and at2' coincide. For example... Figure 5 As shown. Given that the x-axis point set of L1' is {x1,x2,x3,...,xn} and the x-axis point set of L2 is {x1`,x2`,x3`,...,xn`}, calculate the difference between the two point sets. Take the absolute value of each difference and find the maximum value as ais_L. If ais_L is less than 5, it indicates normal posture; if ais_L is greater than 5 and less than 10, it indicates abnormal posture; if ais_L is greater than 10, it indicates suspected scoliosis.

[0057] In another embodiment, the scoliosis index is calculated in a different manner; see [link to relevant documentation]. Figure 7 and Figure 8 First, after obtaining two types of feature curves (thoracolumbar and shoulder), curve registration is calculated for each type of curve. The optimal registration between the two curves is found, and the error is gradually reduced through iterative solutions. Finally, a point-to-ICP algorithm is executed to obtain the optimal position and orientation.<R,t> Let L1 and L2 be the left contour point set and L2 be the right contour point set, respectively, and m and n be the sizes of the two point sets. Let the rotation matrix be R and the translation matrix be t. Let f(R, t) represent the error between the L1 point set and the target point set L2 under the transformation matrix (R, t).

[0058] L1={p i |p i ∈R3, i=1,2,3……n};

[0059] L2={q i |q i ∈R3, i=1,2,3……m};

[0060]

[0061] After obtaining the optimal registration result, the Euclidean distance between the points on the curves is calculated. For each point X, two points y are determined for line segments L1 and L2, denoted as Δy, where d1 = Δy. Δy is calculated for all x points and summed. Finally, the average value is calculated to obtain the result. I ais The higher the value, the greater the likelihood of scoliosis.

[0062] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for calculating the scoliosis index in a back contour image, characterized in that, Includes the following steps: Obtain the back contour image, and extract the shoulder feature curve and thoracolumbar feature curve from the back contour image; Register the shoulder feature curve and the thoracolumbar feature curve; The differences between the shoulder characteristic curves, the upper endpoints of the thoracolumbar characteristic curves, and the thoracolumbar characteristic curves after registration are calculated respectively, and the scoliosis index is obtained through the calculated differences. The calculation of the difference in the shoulder characteristic curves after registration, and the scoliosis index obtained from the calculated difference, includes: Determine the right shoulder contour line s1 and the left shoulder contour line s2 in the shoulder feature curve, and select a cutoff line at the neck position. The point where the left shoulder contour line s2 intersects the cutoff line is the cutoff point a. After horizontally flipping s1, we get s1'. Then, we translate s1' so that its upper endpoint coincides with point a. For each pixel on the horizontal axis, calculate the difference between pixels on the vertical axis, take the absolute value and find the maximum value as ais. If ais is greater than 5 and less than 10, it indicates that the posture is incorrect and there is uneven shoulders. The height of the left and right shoulders is determined based on the sign of ais. If the absolute value of ais is greater than 10, it indicates suspected scoliosis. If the absolute value of ais is less than 5, it indicates that it is within the normal range. Alternatively, the calculation of the difference between the upper endpoints of the characteristic curves of the thoracolumbar region, and the scoliosis index obtained from the calculated difference, includes: Determine the right thoracolumbar characteristic curve L1 and the left thoracolumbar characteristic curve L2 in the thoracolumbar characteristic curve, and obtain the upper endpoint at1 of L1 and the upper endpoint at2 of L2. The absolute value of the height difference between the endpoints at1 and at2 along the vertical axis is Ais_at; If Ais_at is less than 5, it indicates normal posture; if Ais_at is greater than 5 but less than 10, it indicates incorrect posture; if Ais_at is greater than 10, it indicates suspected scoliosis. Alternatively, the calculation of the difference in the characteristic curves of the thoracolumbar region, and the scoliosis index obtained from the calculated difference, includes: Flip L1 horizontally to get L1` and at1`; Translate L1' so that at1' and at2 coincide; Given that the x-axis point set of L1' is {x1,x2,x3,...,xn} and the x-axis point set of L2 is {x1`,x2`,x3`,...,xn`}, calculate the difference between the two point sets. Take the absolute value of each difference and find the maximum value as ais_L. If ais_L is less than 5, it indicates normal posture; if ais_L is greater than 5 and less than 10, it indicates abnormal posture; if ais_L is greater than 10, it indicates suspected scoliosis.

2. The method for calculating the scoliosis index in a back contour image according to claim 1, characterized in that, After obtaining the back contour image, the image is labeled using the image annotation tool labelme, and the labeled areas are the contour areas of the shoulders and the chest and waist.

3. The method for calculating the scoliosis index in a back contour image according to claim 2, characterized in that, The labeled back contour images were divided into training and testing sets in a ratio of 7:

3.

4. The method for calculating the scoliosis index in a back contour image according to claim 1, characterized in that, The Mask R-CNN neural network was used to extract shoulder feature curves and thoracolumbar feature curves from the back contour image.

5. The method for calculating the scoliosis index in a back contour image according to claim 1, characterized in that, The Mask R-CNN neural network was used to extract shoulder and thoracolumbar feature curves from the back contour image, including: The back contour image is used as the input to the Mask R-CNN neural network. The size of the input image is H×W, and then it is fed into the backbone network. Features were extracted using a ResNet50 network as a feature extractor, resulting in a 5-layer feature map with dimensions of ( ). ),( ),( ),( ),( ); Feature fusion is performed using the Feature Pyramid Network (FPN) structure to obtain enhanced features; In the Region Candidate Network (RPN) processing stage, each feature map of size h×w generates h×w×9 bounding boxes. For all boxes, a classification and regression task is performed. The classification task is used to determine whether there is a feature curve in the box and calculate the probability of having a feature curve. The regression task is used to generate the coordinates of each box. The special layer ROI Align for object detection unifies bounding boxes of different sizes into a single dimension; The category of the feature curve and its coordinates in the image are obtained through fully connected (FC) layers.

Images