A method for automatic assessment of Cobb angle in scoliosis based on deep learning

Abstract

The present invention relates to a method for automatic assessment of the Cobb angle for scoliosis based on deep learning. Through a network model consisting of an HRNet-18 encoding unit and a decoding unit with a resolution equivalent fusion module, the coordinates of the vertebral center points and the vertebral inclination vectors in X-ray images are predicted. This eliminates the indirect errors caused by point-based Cobb angle prediction in existing work and enables high-precision automatic assessment of the Cobb angle. At the same time, the vertebral center point coordinates and vertebral inclination vectors are combined and visualized using OpenCV to achieve visual interpretability.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for automatic evaluation of the Cobb angle of scoliosis based on deep learning. Background Art

[0002] Scoliosis refers to the lateral curvature of one or more vertebrae in the human spine accompanied by vertebral rotational deformity. It is a common type of spinal disease. Scoliosis can affect the growth and development of infants, children and adolescents, causing body deformation, and in severe cases, can affect cardiopulmonary function. Scoliosis is usually diagnosed using standing X-ray images. Early detection of related symptoms has a positive effect because doctors can treat earlier to inhibit the deterioration of the disease. In the diagnosis and treatment decision-making of scoliosis, the measurement of the Cobb angle is crucial, which is usually used to measure the degree of scoliosis. Figure 1 As shown in (a), clinically, the Cobb angle is the angle between the upper edge of the upper vertebra and the lower edge of the lower vertebra; the upper and lower vertebrae refer to the vertebrae with the largest inclination toward the concave side of the scoliosis. The upper end plate of the upper vertebra and the lower end plate of the lower vertebra are extended respectively. The angle between the two extension lines or their perpendicular lines is the required Cobb angle; for scoliosis assessment, three Cobb angles are generally required to be determined, and the largest angle is considered the primary Cobb angle, such as Figure 1 (b)

[0003] Manual measurement of the Cobb angle in clinical practice is cumbersome and time-consuming, and is easily influenced by subjective vertebral selection, resulting in certain deviations. This can lead to differences of 5-10 degrees in Cobb angle measurements between different observers. Therefore, automated Cobb angle assessment methods will help eliminate the subjective influence caused by manual intervention, thereby obtaining clinically reliable and repeatable assessment results, forming an integrated solution from imaging to diagnostic report generation, which has great practical value and research significance.

[0004] At present, many methods for automatically evaluating Cobb angles have been proposed. Among them, the method based on deep learning does not require complex feature engineering and has high robustness compared to traditional methods, showing great advantages. Deep learning methods are mainly divided into two categories. One is the method of directly regressing the Cobb angle. This type of method establishes the relationship between the shape of the spine and the Cobb angle by learning the overall shape features of the spine; for the method of directly regressing the Cobb angle, since it can only obtain the angles of three Cobb angles, it cannot specifically know the vertebrae that make up these Cobb angles, which means that it cannot give a clear visual explanation result, thus hindering its promotion in real clinical diagnosis scenarios; the other type is based on vertebral key points. These methods use an indirect key point-driven approach to calculate the spinal Cobb angle. Although the idea is simple and easy to construct and due to the extensiveness of the key point detection task, this type of method can borrow advanced network frameworks and However, the need to predict a large number of key points in an X-ray image potentially increases the learning difficulty of the network and reduces its generalization ability, which cannot be ignored. This is because the essence of the task of evaluating the Cobb angle is to evaluate the degree of inclination of the vertebrae. The automatic evaluation method based on vertebral key points calculates the vertebral inclination by calculating the slope of the curve formed by four key points, thereby indirectly estimating the Cobb angle. The quality of its Cobb angle evaluation depends heavily on the accuracy of the key point positioning, which means that a small error in the prediction of the key point coordinates may lead to a huge deviation in the final result. Due to the problem of overlapping occlusion of organs and tissues in APxray images, the vertebral key points may be obscured, and errors in the prediction of key point coordinates are usually inevitable.

[0005] In addition, in past studies, the Cobb angle calculated based on key points did not take clinical rationality into consideration, resulting in the calculated Cobb angle not conforming to clinical rules and potentially leading to misdiagnosis; because the calculation of the Cobb angle in clinical diagnosis not only requires the two vertebrae with the largest inclination, but more importantly, the Cobb angle can only include one scoliosis. Existing methods for calculating the Cobb angle include: a calculation method based on vertebral key points and a calculation method based on vertebral regions. The method based on vertebral key points regards the upper and lower boundaries of the vertebra as parallel, and thus uses the slope of the transverse midline of the vertebra as the degree of vertebral inclination, where the transverse midline of the vertebra is determined by its four corner points, and then the angle formed by the transverse midlines of any two vertebrae is calculated, where the maximum angle is the Cobb angle. The MBR method is a vertebral region-based method. Unlike the vertebral key point-based method, this method only marks the vertebral contour and uses a minimum rectangle to contain the contour. The boundaries of the rectangle are used as the upper and lower boundaries of the vertebra, and the Cobb angle is calculated using a strategy similar to the vertebral key point-based method. The existing Cobb angle calculation method only considers the former and ignores the latter. In order to be better applied in the clinical diagnosis of scoliosis, correct Cobb angle calculation is essential. Summary of the Invention

[0006] The problem solved by the present invention is how to automatically evaluate the Cobb angle of scoliosis that complies with clinical rules with visual interpretability and without indirect errors.

[0007] To solve the above problems, the present invention provides a method for automatic assessment of the Cobb angle of scoliosis based on deep learning, comprising:

[0008] Step 1: Collect several X-ray images and calculate the vertebral center points and vertebral inclination vectors in the X-ray images as labels, which serve as a training sample set;

[0009] Step 2: Based on the training sample set, a network model is constructed for obtaining an estimated vertebral center point and an estimated vertebral tilt vector. The network model includes an encoding unit and a decoding unit. The encoding unit includes HRNet-18, which is used to output a four-layer feature map. The decoding unit includes a multi-resolution equivalent fusion module, which is used to fuse the feature map with strong semantic information at low resolution and the feature map with rich spatial information at high resolution to obtain a feature image.

[0010] Step 3: Obtain an estimated vertebral center point and an estimated vertebral tilt vector based on feature image decoding;

[0011] Step 4: construct an overall loss function based on the vertebral center point and vertebral tilt vector calculated in step 3 and the estimated vertebral center point and estimated vertebral tilt vector output in step 2 to optimize the network model;

[0012] Step 5: Construct a reasonable angle matrix for calculating the Cobb angle;

[0013] Step 6: Calculate the Cobb angle for spinal assessment based on the network model and the reasonable angle matrix and visualize the Cobb angle using OpenCV.

[0014] The beneficial effects of the present invention are as follows: through a network model composed of a HRNet-18 encoding unit and a decoding unit of a resolution equivalent fusion module, the coordinates of the vertebral center point and the vertebral inclination vector in the X-ray image are predicted, thereby eliminating the errors indirectly generated by the point-based prediction of the Cobb angle in the existing work, and realizing high-precision automatic evaluation of the Cobb angle; at the same time, the coordinates of the vertebral center point and the vertebral inclination vector are combined and visualized through OpenCV to achieve visual interpretability.

[0015] Preferably, the step 1 specifically includes:

[0016] Step 101: The collected X-ray image includes n vertebrae, each vertebrae includes four corner point coordinates: upper left, upper right, lower left, and lower right. The coordinates of the center point of each vertebrae are obtained by calculating the average of the coordinates of the four corner points of each vertebrae:

[0017]

[0018] Where, lt n ,rt n ,lb n ,rb n Represent the coordinates of the upper left corner, upper right corner, lower left corner, and lower right corner of the nth vertebra respectively;

[0019] Step 102: Generate a Gaussian heat map of the vertebral center points according to the coordinates of the vertebral center points:

[0020]

[0021] Where (X, Y) is the pixel coordinate in the Gaussian heat map, (x n ,y n ) is the coordinate of the center point of the nth vertebra; σ n is the radius of the disk formed by the Gaussian heat map;

[0022] Step 103: merge the Gaussian heat maps of the n vertebrae center points in the X-ray image into a Gaussian heat map H of the vertebrae center points:

[0023] H=H1+H2+……+H n ;

[0024] Step 104: Obtain the center points of the left and right sides of the vertebra using the coordinates of the four corner points of the vertebra:

[0025]

[0026]

[0027] Then the vertebral inclination vector of the nth vertebra is:

[0028]

[0029] Where a is the tilt component of the vertebra in the X direction, and b is the tilt component in the Y direction;

[0030] Preferably, the feature image includes a predicted center point Gaussian heat map H pred and vertebral tilt vector prediction feature map V pred , the step 2 specifically includes:

[0031] Step 201: Input the X-ray image into the network model, first pass it through the HRNet-18 encoding unit to output a 4-layer feature map {C1, C2, C3, C4}, and the resolution of the feature map is {C'1, C'2, C'3, C'4};

[0032] Step 202: The feature maps {C1, C2, C3, C4} are fused through a multi-resolution equivalent fusion module to generate the feature maps {P1, P2, P3} of the decoding stage. Specifically:

[0033] The feature map C4 is upsampled by a multiple of Then, the upsampled feature map C4 is processed through a 3×3 convolution layer, and the number of channels of the feature map C4 is adjusted to the number of channels of the feature map C3. The feature maps C4 and C3 with the same number of channels are fused by channel splicing to generate the feature map P1 in the decoding stage.

[0034] P1 is upsampled by a multiple of The upsampled feature map P1 is processed through a 3×3 convolution layer, and the number of channels of P1 is adjusted to the number of channels of the feature map C2. The P1 and feature map C2 of the same channel are fused by channel splicing to generate the feature map P2 in the decoding stage.

[0035] Similarly, P2 is upsampled, and the upsampling multiple is The upsampled P2 is processed through a 3×3 convolution layer, and the number of channels of P2 is adjusted to the number of channels of the feature map C1. P2 and the feature map C1 with the same number of channels are fused through channel splicing to generate the feature map P3 in the decoding stage.

[0036] Step 203: The decoding stage feature map P3 is passed through a 3×3 convolution layer and a 1×1 convolution layer to generate the predicted center point Gaussian heat map H pred and vertebral tilt vector prediction feature map V pred ;

[0037] Compared with the existing CNN technology that causes spatial information loss through downsampling operations or bilinear upsampling, the multi-resolution equivalent fusion module is used to fuse feature maps with strong semantic information at low resolution and feature maps with rich spatial information at high resolution through channel splicing. With less increase in computational complexity, the strong semantic information and rich spatial information are combined, which helps to locate the vertebral center point and estimate the vertebral tilt vector, avoid spatial information loss, and improve prediction accuracy.

[0038] Preferably, the step 3 comprises:

[0039] Step 301: predict the Gaussian heat map H of the center point pred After performing pooling in a 3×3 pooling layer, the n points with the largest response values are selected as the estimated vertebral center points of the X-ray image:

[0040] Step 302: Obtain an estimated vertebral tilt vector based on the estimated vertebral center point. The estimated vertebral tilt vector of the nth vertebra is:

[0041]

[0042] 8. The method for automatic assessment of Cobb angle of scoliosis based on deep learning according to claim 4, wherein step 4 specifically comprises:

[0043] Step 401: Construct an overall loss function:

[0044] Loss = L hm +L vec

[0045] Where, L hm Used to optimize the prediction center point Gaussian heat map H pred ;L vec For optimizing the vertebral tilt vector prediction feature map V pred ;in:

[0046] The weighted MSE of vertebra positioning is used to optimize the prediction of the center point Gaussian heat map H pred :

[0047]

[0048]

[0049] Where δ and α are hyperparameters. i When it is close to 0, the weight W is close to 1. i When it is close to 1 near the center point, the weight W is close to β+α;

[0050] Optimize the vertebral tilt vector prediction feature map V pred :

[0051]

[0052] Vec gt is the value of the vertebral tilt vector calculated in step 1, Vec pred is the value of the estimated vertebral tilt vector output from step 3, M is the number of samples, and N is the number of vertebrae;

[0053] According to the coordinate position and area characteristics of the spine in the X-ray image, the weighted MSE is used to locate the vertebrae, strengthen the regional spatial constraints, focus on the differences in the area around the spine, and ignore the loss of the distant background.

[0054] Preferably, the step 5 specifically includes:

[0055] Step 501: Construct an angle matrix Angle based on n vertebrae to store all vertebrae combinations that can form the Cobb angle:

[0056]

[0057] Vec i ,Vec j refer to the inclination of the i-th vertebra and the j-th vertebra, respectively;

[0058] Step 502: Calculate the vertebral inclination angle tmpvec based on the vertebral inclination vector:

[0059]

[0060] Where, i is the i-th vertebra;

[0061] Step 503: Determine the inclination direction Orientation[v] of each vertebra in turn based on the inclination angle tmpvec of the vertebra:

[0062] A1. Determine the inclination angle tmpvec of the vth vertebra, where v = 1;

[0063] A2. Check whether the inclination angle tmpvec of the vth vertebra is less than -3°. If so, set Orientation[v]←-1 and proceed to A3. Otherwise, check whether the inclination angle tmpvec of the vth vertebra satisfies -3°≤tmpvec≤3°. If so, set Orientation[v]←0 and proceed to A3. Otherwise, if the inclination angle tmpvec of the vth vertebra is greater than 3°, set Orientation[v]←1 and proceed to A3.

[0064] A3.v=v+1;

[0065] A4. Determine whether v≤n is satisfied. If so, return to A1; otherwise, proceed to step 504;

[0066] Step 504: Based on the clinically prescribed positive and negative change rules of the vertebral slope of the Cobb angle, obtain the reasonable matrix ReaMat:

[0067] B1.CurVec = Orientation[v], used to record the inclination direction of the upper vertebra in the Cobb angle; and define flag = True to record whether the inclination direction of the vertebra has changed. The default state is True, indicating that the inclination direction of the vertebra has not changed; when the inclination direction of the vertebra has not changed, flag = False;

[0068] B2. Determine whether curvec == 0. If so, proceed to B3; if not, proceed to B6.

[0069] B3. Find the next vertebra k whose inclination direction is not 0, k = v + 1;

[0070] B4. Determine whether Orientation[k] is equal to 0. If so, proceed to B5. If not, the inclination direction of curvec is consistent with that of Orientation[k], and proceed to B6.

[0071] B5.k=k+1 and returns to B4;

[0072] B6. Locate the lower vertebra j; j = v;

[0073] B7. Determine whether j≤n. If so, proceed to B8; if not, proceed to B11.

[0074] B8. Determine whether the inclination direction of lower vertebra j satisfies Orientation[j] == 0. If so, vertebra v and vertebra j can form a Cobb angle, and proceed to B10. If not, proceed to B9.

[0075] B9. Determine whether the inclination direction of the upper vertebra v and the lower vertebra j satisfies Orientation[j]! = CurVec. If so, output flag = False and proceed to B10. If not, determine whether flag == False. If so, the vth vertebra and the jth vertebra cannot form a Cobb angle and proceed to B11. If not, proceed to B10.

[0076] B10. Vertebra v and vertebra j can form a Cobb angle, ReaMat[v,j] = ReaMat[j,v] = 1, j = j + 1, and return to B7.

[0077] B11.v=v+1;ReaMat[v,j:]=ReaMat[j:,v]=0;and determine whether v≤n. If so, return to B2; if not, output the reasonable matrix ReaMat and end.

[0078] Step 505: obtain a reasonable angle matrix RationalCobb based on the dot product of the reasonable matrix ReaMat and the angle matrix, RationalCobb=ReaMat*Angle;

[0079] By comprehensively considering the angle formed by any two vertebrae and combining the clinical rule that one Cobb angle only contains one scoliosis, angle rationality constraints are added through a reasonable angle matrix, making the Cobb angle calculation applicable to any scoliosis case.

[0080] Preferably, the step 6 specifically includes:

[0081] Step 601: Calculate the first three Cobb angles θ1, θ2, and θ3 for evaluating scoliosis:

[0082] C1. Filter out the largest Cobb angle θ1 from RationalCobb:

[0083]

[0084] Vec top and Vec bottom They are the upper and lower vertebrae that form the Cobb angle;

[0085] C2. Obtain the reasonable angle of the vertebrae that does not form the Cobb angle RationalCobb'∈R n

[0086] RationalCobb'[:top]=RationalCobb[top,:top]

[0087] RationalCobb'[bottom+1]=RationalCobb[bottom,bottom+1]

[0088] Next, the largest Cobb angle θ2 is selected from the remaining RationalCobb':

[0089]

[0090] C3. Obtain the reasonable angle of the vertebrae that does not form the Cobb angle RationalCobb"∈R n

[0091] RationalCobb"[:top']=RationalCobb'[top',:top']

[0092] RationalCobb"[bottom'+1]=RationalCobb'[bottom',bottom'+1]

[0093] Next, the largest Cobb angle θ3 is selected from the remaining RationalCobb:

[0094]

[0095] Step 602: Based on the upper vertebra Vec top 、Vec' top' 、Vec” top” and lower vertebrae Vec bottom 、Vec' bottom' 、Vec” bottom” The corresponding vertebral center point K = (x, y) and the vertebral inclination vector Vec = (a, b), the slope of the transverse midline of the vertebra where the vertebral center point is located is obtained through the vertebral center point Combined with the coordinates of the vertebral center point, the offset of the straight line equation of the vertebral transverse midline is obtained: S = y-Cx;

[0096] Then the equation of the line through the transverse midline of the vertebra is:

[0097] Step 603: Based on the linear equation of the transverse midline of the vertebra, OpenCV is used to draw the transverse centers of the Cobb angle in the X-ray image to visualize the Cobb angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 (1a) is a schematic diagram of the clinical Cobb angle, which is the angle between the upper edge of the upper vertebra and the lower edge of the lower vertebra. (1b) is a schematic diagram of the main Cobb angle for scoliosis assessment.

[0099] Figure 2 It is a structural diagram of the network model of the present invention;

[0100] Figure 3 This is the output effect diagram of the X-ray image after passing through the network model of the present invention;

[0101] Figure 4 Schematic diagram of the Cobb angle for left convexity and right convexity in this clinical diagnosis;

[0102] Figure 5 This is the output effect diagram of the Cobb angle visualization of the present invention. DETAILED DESCRIPTION

[0103] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0104] A method for automatic assessment of Cobb angle of scoliosis based on deep learning, comprising:

[0105] Step 1: Collect several X-ray images and calculate the vertebral center points and vertebral tilt vectors in the X-ray images as labels, which serve as the training sample set. This includes:

[0106] Step 101: The collected X-ray image includes n vertebrae. In this specific embodiment, n=17. Each vertebrae includes four corner coordinates: upper left, upper right, lower left, and lower right. The coordinates of the center point of each vertebrae are obtained by calculating the average of the coordinates of the four corner points of each vertebrae:

[0107]

[0108] Where, lt n ,rt n ,lb n ,rb n Represent the coordinates of the upper left corner, upper right corner, lower left corner, and lower right corner of the nth vertebra respectively;

[0109] Step 102: Generate a Gaussian heat map of the vertebral center points according to the coordinates of the vertebral center points:

[0110]

[0111] Where (X, Y) is the pixel coordinate in the Gaussian heat map, (x n ,y n ) is the coordinate of the center point of the nth vertebra; σ n is the radius of the disk formed by the Gaussian heat map;

[0112] Step 103: Merge the Gaussian heat maps of the 17 vertebrae center points in the X-ray image into a Gaussian heat map H of the vertebrae center points:

[0113] H=H1+H2+……+H n ;

[0114] Step 104: Obtain the center points of the left and right sides of the vertebra using the coordinates of the four corner points of the vertebra:

[0115]

[0116]

[0117] Then the vertebral inclination vector of the nth vertebra is:

[0118]

[0119] Where a is the tilt component of the vertebra in the X direction, and b is the tilt component in the Y direction;

[0120] The vertebral center point and vertebral tilt vector calculated in step 1 are used as labels in the corresponding X-ray image;

[0121] Step 2: Based on the training sample set, a network model is constructed for obtaining the estimated vertebral center point and the estimated vertebral tilt vector. The network model includes an encoding unit and a decoding unit. The encoding unit includes HRNet-18, which is used to output a 4-layer feature map. The decoding unit includes a multi-resolution equivalent fusion module, which is used to fuse the feature map with strong semantic information at low resolution and the feature map with rich spatial information at high resolution to obtain a feature image. The feature image includes the Gaussian heat map H of the predicted center point. pred and vertebral tilt vector prediction feature map V pred ,like Figure 2 As shown, specifically including:

[0122] Step 201: Input the X-ray image into the network model, and first pass it through the HRNet-18 encoding unit to output 4 layers of feature maps {C1, C2, C3, C4}. The resolutions of the feature maps are {C'1, C'2, C'3, C'4}. In this specific embodiment, the resolutions of the feature maps are {256×128×18, 128×64×36, 64×32×72, 32×16×144} respectively. In this specific embodiment, the size of the feature map is 1 / 4 of the original X-ray image size.

[0123] Step 202: The feature maps {C1, C2, C3, C4} are fused through a multi-resolution equivalent fusion module to generate the feature maps {P1, P2, P3} of the decoding stage. Specifically:

[0124] The feature map C4 is upsampled by a factor of 2. The upsampled feature map C4 is then processed through a 3×3 convolutional layer, and the number of channels of the feature map C4 is adjusted to the number of channels of the feature map C3. The feature maps C4 and C3 with the same number of channels are spliced together to achieve feature fusion, generating the feature map P1 of the decoding stage.

[0125] P1 is upsampled by a factor of 2. The upsampled feature map P1 is processed through a 3×3 convolution layer, and the number of channels of P1 is adjusted to the number of channels of the feature map C2. P1 and the feature map C2 of the same channel are fused by channel concatenation to generate the feature map P2 in the decoding stage.

[0126] Similarly, P2 is upsampled by a factor of 2. The upsampled P2 is processed through a 3×3 convolution layer, and the number of channels of P2 is adjusted to the number of channels of the feature map C1. P2 and the feature map C1 with the same number of channels are fused by channel concatenation to generate the feature map P3 in the decoding stage.

[0127] Step 203: The decoding stage feature map P3 is passed through a 3×3 convolution layer and a 1×1 convolution layer to generate a 256×128×1 prediction center point Gaussian heat map H. pred and the 256×128×2 vertebral tilt vector prediction feature map V pred ;

[0128] Compared to existing CNNs that lose spatial information through downsampling or bilinear upsampling, this approach uses a multi-resolution equivalent fusion module to fuse feature maps with strong semantic information at low resolution and rich spatial information at high resolution through channel splicing. This combination of strong semantic information and rich spatial information, while minimizing computational effort, facilitates vertebral center location and vertebral tilt vector estimation, avoiding spatial information loss and improving prediction accuracy.

[0129] Step 3: Obtaining an estimated vertebral center point and an estimated vertebral tilt vector based on feature image decoding; specifically including:

[0130] Step 301: predict the Gaussian heat map H of the center point pred After performing pooling in a 3×3 pooling layer, the 17 points with the largest response values are selected as the estimated vertebral center points of the X-ray image:

[0131] Step 302: Obtain an estimated vertebral tilt vector based on the estimated vertebral center point. The estimated vertebral tilt vector of the nth vertebra is:

[0132]

[0133] Step 4: Construct an overall loss function based on the vertebral center point and vertebral tilt vector calculated in step 3 and the estimated vertebral center point and estimated vertebral tilt vector output in step 2 to optimize the network model; specifically, the following steps are included:

[0134] Step 401: Construct an overall loss function:

[0135] Loss = L hm +L vec

[0136] Where, L hm Used to optimize the prediction center point Gaussian heat map H pred ;L vec For optimizing the vertebral tilt vector prediction feature map V pred ;in:

[0137] The weighted MSE of vertebra positioning is used to optimize the prediction of the center point Gaussian heat map H pred :

[0138]

[0139]

[0140] Where δ and α are hyperparameters. i When it is close to 0, the weight W is close to 1. i When it is close to 1 near the center point, the weight W is close to β+α;

[0141] Optimize the vertebral tilt vector prediction feature map V pred :

[0142]

[0143] Vec gt is the value of the vertebral tilt vector calculated in step 1, Vec pred is the value of the estimated vertebral tilt vector output in step 3, M is the number of samples, N is the number of vertebrae, N = 17;

[0144] According to the coordinate position and area characteristics of the spine in the X-ray image, the weighted MSE is used to locate the vertebrae, strengthen the regional spatial constraints, focus on the differences in the area around the spine, and ignore the loss of the distant background;

[0145] Step 5: Construct a reasonable angle matrix for calculating the Cobb angle; specifically,

[0146] Step 501: Construct an angle matrix Angle based on n vertebrae to store all vertebrae combinations that can form the Cobb angle:

[0147]

[0148] Vec i ,Vec j refer to the inclination of the i-th vertebra and the j-th vertebra, respectively;

[0149] Step 502: Calculate the vertebral inclination angle tmpvec based on the vertebral inclination vector:

[0150]

[0151] Where, i is the i-th vertebra;

[0152] Step 503: Determine the inclination direction Orientation[v] of each vertebra in turn based on the inclination angle tmpvec of the vertebra:

[0153] A1. Determine the inclination angle tmpvec of the vth vertebra, where v = 1;

[0154] A2. Check whether the inclination angle tmpvec of the vth vertebra is less than -3°. If so, set Orientation[v]←-1 and proceed to A3. Otherwise, check whether the inclination angle tmpvec of the vth vertebra satisfies -3°≤tmpvec≤3°. If so, set Orientation[v]←0 and proceed to A3. Otherwise, if the inclination angle tmpvec of the vth vertebra is greater than 3°, set Orientation[v]←1 and proceed to A3.

[0155] A3.v=v+1;

[0156] A4. Determine whether v≤17 is satisfied. If so, return to A1; otherwise, proceed to step 504;

[0157] The local shape of the vertebrae based on the Cobb angle in clinical diagnosis is divided into left convex and right convex, such as Figure 3 As shown, CobbA is left convex and CobbB is right convex. According to clinical regulations, Cobb cannot contain two scoliosis, so the curvature signs of the corresponding curve ends of a Cobb angle must be consistent. Based on this principle, a Cobb angle must contain only one scoliosis. The positive and negative changes in the slope of the transverse midline of the vertebrae that constitute the Cobb angle that meets clinical regulations are shown in Table 1:

[0158]

[0159] Table 1

[0160] Step 504: Based on the clinically prescribed positive and negative change rules of the vertebral slope of the Cobb angle, obtain the reasonable matrix ReaMat:

[0161] B1.CurVec = Orientation[v], used to record the inclination direction of the upper vertebra in the Cobb angle; and define flag = True to record whether the inclination direction of the vertebra has changed. The default state is True, indicating that the inclination direction of the vertebra has not changed; when the inclination direction of the vertebra has not changed, flag = False;

[0162] B2. Determine whether curvec == 0. If so, proceed to B3; if not, proceed to B6.

[0163] B3. Find the next vertebra k whose inclination direction is not 0, k = v + 1;

[0164] B4. Determine whether Orientation[k] is equal to 0. If so, proceed to B5. If not, the inclination direction of curvec is consistent with that of Orientation[k], and proceed to B6.

[0165] B5.k=k+1 and returns to B4;

[0166] B6. Locate the lower vertebra j; j = v;

[0167] B7. Determine whether j≤n. If so, proceed to B8; if not, proceed to B11.

[0168] B8. Determine whether the inclination direction of lower vertebra j satisfies Orientation[j] == 0. If so, vertebra v and vertebra j can form a Cobb angle, and proceed to B10. If not, proceed to B9.

[0169] B9. Determine whether the inclination direction of the upper vertebra v and the lower vertebra j satisfies Orientation[j]! = CurVec. If so, output flag = False, indicating that the inclination direction of the upper vertebra v and the lower vertebra j has changed, and proceed to B10. If not, determine whether flag == False. If so, the vth vertebra and the jth vertebra cannot form a Cobb angle, and proceed to B11. If not, proceed to B10.

[0170] B10. Vertebra v and vertebra j can form a Cobb angle, ReaMat[v,j] = ReaMat[j,v] = 1, j = j + 1, and return to B7.

[0171] B11.v=v+1;ReaMat[v,j:]=ReaMat[j:,v]=0;and determine whether v≤17. If so, return to B2; if not, output the reasonable matrix ReaMat and end.

[0172] Step 505: obtain a reasonable angle matrix RationalCobb based on the dot product of the reasonable matrix ReaMat and the angle matrix, RationalCobb=ReaMat*Angle;

[0173] By comprehensively considering the angle formed by any two vertebrae and combining the clinical rule that one Cobb angle only contains one scoliosis, and adding angle rationality constraints through a reasonable angle matrix, the Cobb angle calculation can be applied to any scoliosis case.

[0174] Step 6: Calculate the Cobb angle for spinal assessment based on the network model and the reasonable angle matrix and visualize the Cobb angle using OpenCV, including:

[0175] Step 601: Calculate the first three Cobb angles θ1, θ2, and θ3 for evaluating scoliosis:

[0176] C1. Filter out the largest Cobb angle θ1 from RationalCobb:

[0177]

[0178] Vec top and Vec bottom They are the upper and lower vertebrae that form the Cobb angle;

[0179] C2. Obtain the reasonable angle of the vertebrae that does not form the Cobb angle RationalCobb'∈R 17

[0180] RationalCobb'[:top]=RationalCobb[top,:top]

[0181] RationalCobb'[bottom+1]=RationalCobb[bottom,bottom+1]

[0182] Next, the largest Cobb angle θ2 is selected from the remaining RationalCobb':

[0183]

[0184] C3. Obtain the reasonable angle of the vertebrae that does not form the Cobb angle RationalCobb"∈R 17

[0185] RationalCobb"[:top']=RationalCobb'[top',:top']

[0186] RationalCobb"[bottom'+1]=RationalCobb'[bottom',bottom'+1]

[0187] Next, the largest Cobb angle θ3 is selected from the remaining RationalCobb:

[0188]

[0189] Step 602: Based on the upper vertebra Vec top 、Vec' top' 、Vec” top” and lower vertebrae Vec bottom 、Vec' bottom' 、Vec” bottom” The corresponding vertebral center point K = (x, y) and the vertebral inclination vector Vec = (a, b), the slope of the transverse midline of the vertebra where the vertebral center point is located is obtained through the vertebral center point Combined with the coordinates of the vertebral center point, the offset of the straight line equation of the vertebral transverse midline is obtained: S = y-Cx;

[0190] Then the equation of the line through the transverse midline of the vertebra is:

[0191] Step 603: Based on the linear equation of the transverse midline of the vertebra, OpenCV is used to draw the transverse center of the Cobb angle in the X-ray image to visualize the Cobb angle, as shown in FIG. Figure 5 shown.

[0192] Although the present disclosure is disclosed as above, the protection scope of the present disclosure is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for automatic assessment of Cobb angle of scoliosis based on deep learning, characterized in that: include: Step 1: Collect several X-ray images and calculate the vertebral center points and vertebral inclination vectors in the X-ray images as labels, which serve as a training sample set; Step 2: Based on the training sample set, a network model is constructed for obtaining an estimated vertebral center point and an estimated vertebral tilt vector. The network model includes an encoding unit and a decoding unit. The encoding unit includes HRNet-18, which is used to output a four-layer feature map. The decoding unit includes a multi-resolution equivalent fusion module, which is used to fuse the feature map with strong semantic information at low resolution and the feature map with rich spatial information at high resolution to obtain a feature image. Step 3: Obtain an estimated vertebral center point and an estimated vertebral tilt vector based on feature image decoding; Step 4: Construct an overall loss function based on the vertebral center point and vertebral tilt vector calculated in step 3 and the estimated vertebral center point and estimated vertebral tilt vector output in step 2 to optimize the network model; specifically, the following steps are included: Step 401: Construct an overall loss function: ； Where, Used to optimize the prediction center point Gaussian heat map ; Feature map for optimizing vertebral tilt vector prediction ;in: Using vertebrae localization weighted MSE to optimize the prediction of the center point Gaussian heat map : ； ； Where, and is a hyperparameter, when When it is close to 0, the weight W is close to ,when When it is close to 1 near the center point, the weight W is close to ; Optimizing the vertebral tilt vector prediction feature map : ； is the value of the vertebral tilt vector calculated in step 1, is the value of the estimated vertebral tilt vector output from step 3, M is the number of samples, and N is the number of vertebrae; Step 5: Construct a reasonable angle matrix for calculating the Cobb angle; specifically, Step 501: Based on The vertebrae construct an angle matrix that stores all possible combinations of vertebrae that can form the Cobb angle. : ； Refers to the vertebrae and the the inclination of the vertebrae; Step 502: Calculate the vertebral inclination angle based on the vertebral inclination vector : ； Where, For the vertebrae; Step 503: Based on the inclination angle of the vertebra Determine the inclination direction of each vertebra in turn : A1. Determine the The inclination angle of the vertebra , ; A2. The inclination angle of the vertebra Is it less than -3°? If so, , and go to A3; otherwise, judge The inclination angle of the vertebra Whether -3°≤ ≤3°, if yes, then , and go to A3; otherwise, The inclination angle of the vertebra More than 3°, , enter A3; A3. ； A4. Determine whether If yes, then return to A1; otherwise, go to step 504; Step 504: Based on the clinically prescribed positive and negative change rules of the vertebral slope of the Cobb angle, obtain a reasonable matrix : B1. , used to record the tilt direction of the upper vertebra in the Cobb angle; and define Used to record whether the vertebral tilt direction changes. By default, , and the vertebral tilt direction does not change; when the vertebral tilt direction does not change, ; B2. Determine whether ==0, if yes, go to B3, if no, go to B6; B3. Find the next vertebra whose tilt direction is not 0 , ; B4. Judgment Is it equal to 0? If so, go to B5. If not, and The inclination direction is consistent and enters B6; B5. , and returns to B4; B6. Locate the lower vertebra ; ; B7. Determine whether ; If yes, go to B8, if no, go to B11; B8. Identify the lower vertebra Does the tilt direction satisfy If so, it means vertebrae and If the vertebrae can form the Cobb angle, they will enter B10; if not, they will enter B9; B9. Determine the upper vertebra and lower vertebrae Does the tilt direction satisfy If so, then output , and enter B10, if not, then determine whether it satisfies If so, then vertebrae and If the vertebrae do not form a Cobb angle, proceed to B11; if not, proceed to B10; B10. No. vertebrae and The vertebrae can form the Cobb angle, , , and return to B7; B11. ; ; and determine whether If yes, then return B2; if no, then output the reasonable matrix ,Finish; Step 505: Based on the reasonable matrix Multiply the matrix with the angle matrix to get a reasonable angle matrix , ; Step 6: Calculate the Cobb angle for spinal assessment based on the network model and the reasonable angle matrix and visualize the Cobb angle using OpenCV, including: Step 601: Calculate the first three Cobb angles for scoliosis assessment 、 and : C1. From Filter out the largest Cobb angle : ； and They are the upper and lower vertebrae that form the Cobb angle; C2. Obtain reasonable vertebral angles without forming Cobb angles ; ； ； Then, from the remaining Filter out the largest Cobb angle : ； C3. Obtain reasonable vertebral angles without forming Cobb angles ; ； ； Then, from the remaining Filter out the largest Cobb angle : ； Step 602: Based on the upper vertebra 、 and lower vertebrae 、 、 Corresponding vertebral center point and the vertebral tilt vector , get the slope of the transverse midline of the vertebra where the vertebral center point is located through the vertebral center point ; Combine the coordinates of the vertebral center point to obtain the offset of the straight line equation of the vertebral transverse midline: ; Then the equation of the line through the transverse midline of the vertebra is: ; Step 603: Based on the linear equation of the transverse midline of the vertebra, OpenCV is used to draw the transverse centers of the Cobb angle in the X-ray image to visualize the Cobb angle.

2. The method for automatic assessment of Cobb angle of scoliosis based on deep learning according to claim 1, characterized in that: The step 1 specifically includes: Step 101: The collected X-ray image includes n vertebrae, each vertebrae includes four corner point coordinates: upper left, upper right, lower left, and lower right. The coordinates of the center point of each vertebrae are obtained by calculating the average of the coordinates of the four corner points of each vertebrae: ； Where, Represent the coordinates of the upper left corner, upper right corner, lower left corner, and lower right corner of the nth vertebra respectively; Step 102: Generate a Gaussian heat map of the vertebral center points according to the coordinates of the vertebral center points: ； Where, are the pixel coordinates in the Gaussian heat map, is the coordinate of the center point of the nth vertebra; is the radius of the disk formed by the Gaussian heat map; Step 103: Merge the Gaussian heat maps of the n vertebrae center points in the X-ray image into a Gaussian heat map of the vertebrae center point. : ； Step 104: Obtain the center points of the left and right sides of the vertebra using the coordinates of the four corner points of the vertebra: ； ； Then the vertebral inclination vector of the nth vertebra is: ； Where a is the tilt component of the vertebra in the X direction, and b is the tilt component in the Y direction.

3. The method for automatic assessment of Cobb angle of scoliosis based on deep learning according to claim 1, characterized in that: The feature image includes a predicted center point Gaussian heat map and vertebral tilt vector prediction feature map , the step 2 specifically includes: Step 201: Input the X-ray image into the network model and output a 4-layer feature map through the HRNet-18 encoding unit. , the resolution of the feature map ; Step 202: The feature map Generate feature maps in the decoding stage through a multi-resolution equivalent fusion module ; Specifically: The feature map Upsampling is performed, and the upsampling multiple is ; Then the upsampled feature map After the 3×3 convolution layer, the feature map is processed and the feature map is transformed into The number of channels is adjusted to the feature map The number of channels; the feature maps with the same number of channels and feature maps Feature fusion is achieved through channel splicing to generate feature maps in the decoding stage ; Will Upsampling is performed, and the upsampling multiple is ; Feature map after upsampling After the 3×3 convolution layer, the feature is processed and the The number of channels is adjusted to the feature map The number of channels of the same channel and feature maps Feature fusion is achieved through channel splicing to generate feature maps in the decoding stage ; Similarly, Upsampling is performed, and the upsampling multiple is ; After upsampling After the 3×3 convolution layer, the feature is processed and the The number of channels is adjusted to the feature map The number of channels; the same number of channels and feature maps Feature fusion is achieved through channel splicing to generate feature maps in the decoding stage ; Step 203: Decoding the feature map After a 3×3 convolution layer and a 1×1 convolution layer, the predicted center point Gaussian heat map is generated respectively. and vertebral tilt vector prediction feature map .

4. The method for automatic assessment of Cobb angle of scoliosis based on deep learning according to claim 3, characterized in that: The step 3 comprises: Step 301: Gaussian heat map of the predicted center point After performing pooling in a 3×3 pooling layer, the n points with the largest response values are selected as the estimated vertebral center points of the X-ray image: ; Step 302: Obtain an estimated vertebral tilt vector based on the estimated vertebral center point. The estimated vertebral tilt vector of the nth vertebra is: 。

Images