Method and system for assessing scoliosis angle based on regression vertebral bending direction

Abstract

The application discloses a kind of based on regression spine curvature direction evaluation scoliosis angle method and system, belong to scoliosis evaluation field.Method includes: extracting the spine mask in X-ray image, after the spine mask is spliced with the X-ray image, input the spine detection network and obtain the k bounding box of the k spine;Three scales of spine feature blocks corresponding to the k bounding box are cut out on the output feature map of three scales of the spine detection network;The k spine block corresponding to the k bounding box is cut out on the X-ray image;Spine block is input into spine regression network, and the features of spine detection network and the features of spine direction regression are fused, the direction of each spine block on end plate and lower end plate is output, after splicing the direction of upper end plate and lower end plate of each spine block, the curvature direction of whole spine is obtained, and Cobb angle is calculated.The application can reduce the difficulty of spine direction regression, and improve the accuracy of Cobb angle calculation.

Description

Technical Field

[0001] This invention belongs to the field of scoliosis assessment, and more specifically, relates to a method and system for assessing the angle of scoliosis based on the regression of the direction of spinal curvature. Background Technology

[0002] Scoliosis is a disease characterized by three-dimensional structural deformation of the spine, often occurring during adolescence. It can cause physical pain such as back pain, cardiopulmonary compression, and difficulty breathing, as well as psychological distress due to the physical deformity. Early diagnosis and treatment of scoliosis can prevent further deterioration and gradually correct the deformed posture.

[0003] The Cobb angle is a commonly used clinical criterion for diagnosing scoliosis, primarily measured manually by a physician on coronal X-ray images. On a coronal X-ray, the physician first identifies the two most tilted vertebrae in the scoliosis segment and manually marks the two superior endplate angles of the upper vertebra and the two inferior endplate angles of the lower vertebra. Then, lines are drawn connecting these two endplate angles on the same side of each vertebra, and the angle between these two lines is the Cobb angle. Figure 1 As shown in (a) above. Medically, a spine is typically defined as having three Cobb angles, which, from top to bottom, are the superior thoracic curve (PT), the thoracic curve (MT), and the lumbar curve (TL). Due to the overlapping of anisotropic soft tissues, X-ray images of some patients are unclear, requiring doctors to expend considerable effort when measuring the Cobb angles, such as... Figure 1 As shown in (b) and (c) in the figure. This traditional method of doctors manually measuring the Cobb angle is time-consuming and laborious, and inevitably involves inter-observer and intra-observer errors.

[0004] In existing techniques, some researchers extract the centerline or edge line of the spine from X-ray images and then calculate the normal at each position of the curve to calculate the Cobb angle. However, the measurement results have a larger error compared to manual measurements by doctors. Other researchers, intending to implement this in clinical practice, first locate the angle points of the spine and then calculate the Cobb angle from these points. However, this method suffers from significant interference from noise in the X-ray images, and the subsequent manual calibration of the angle points is cumbersome, resulting in measurement results that lack stability and flexibility. Still other researchers use segmentation masks of the spine to directly regress the three Cobb angles. This method, by omitting a large amount of intermediate reasoning, lacks necessary clinical interpretability and ignores the importance of human-computer interaction in clinical practice.

[0005] Therefore, proposing a method that can accurately measure the Cobb angle has great guiding significance and application value in clinical practice. Summary of the Invention

[0006] In view of the shortcomings of existing technologies and the need for improvement, this invention provides a method and system for assessing the angle of scoliosis based on the regression of the direction of spinal curvature, with the aim of improving the accuracy of Cobb angle calculation.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for assessing scoliosis angle based on regression of spinal curvature direction is provided, comprising:

[0008] Extract the spine mask from the X-ray image, where the X-ray image contains k vertebrae, k≥1;

[0009] A spine detection network is constructed. The spine mask is stitched together with the X-ray image and then input into the spine detection network to obtain k bounding boxes for the k vertebrae. The spine feature blocks F corresponding to the k bounding boxes at three scales are then cropped from the output feature maps at three scales of the spine detection network. p ={f p1 ,f p2 ,f p3};

[0010] The k vertebral blocks I corresponding to the k bounding boxes are cropped from the X-ray image. p ;

[0011] Construct a spinal regression network and extract the k spinal blocks I p The features are used to obtain the first feature map f. reg1 , the first feature map f reg1 With the characteristic block of the spine f p1 After adding elements one by one, features are extracted to obtain the second feature map f. reg2 The second feature map f reg2 With the characteristic block of the spine f p2 After adding elements one by one, features are extracted to obtain the third feature map f. reg3 The third feature map f reg3 With the characteristic block of the spine f p3 After adding elements one by one, features are extracted to obtain the fused feature map f. p_reg The fused feature map f p_reg Input a fully connected layer and output the directions of the upper and lower endplates of k vertebral blocks;

[0012] By splicing together the directions of the upper and lower endplates of k vertebral blocks, the curvature direction of the entire spine is obtained, and the Cobb angle is calculated.

[0013] Furthermore, the method includes the step of selecting bounding boxes containing the complete spine from the k bounding boxes, specifically including the following sub-steps:

[0014] Step S11: Calculate the Intersection over Union (IOU) ratio between the i-th bounding box and the k labeled bounding boxes to obtain the set of the k IOU ratios of the i-th bounding box, where 1≤i≤k;

[0015] Step S12: Find the largest Intersection over Union (IOU) ratio in the set IOUs. max With the set threshold I t Compare sizes, if IOU max ≥I t Then the i-th bounding box is a bounding box that contains the complete spine, denoted as . Otherwise, discard the i-th bounding box;

[0016] Step S13: Repeat steps S11 and S12 to obtain m selected bounding boxes containing complete spines. in, Let m be the coordinates of two opposite vertices of the bounding box, where 1 ≤ m ≤ k.

[0017] Furthermore, the labeled bounding box is obtained through the following steps:

[0018] Choose the smallest bounding box from the k bounding boxes that can completely enclose the four corner points of the corresponding spine;

[0019] The minimum bounding box is expanded by a set multiple to obtain the labeled bounding box.

[0020] Furthermore, the ROIAlign cropping module, which has a noise perturbation mechanism, crops the spine feature blocks F corresponding to the k bounding boxes at the three scales on the output feature maps of the spine detection network. p And cropping out the k vertebral blocks I corresponding to the k bounding boxes from the X-ray image. p ;

[0021] After adding Gaussian noise to the m selected bounding boxes containing complete spines, the resulting bounding boxes are: Where, n i =[x ni1 ,y ni1 ,x ni2 ,y ni2 ];

[0022] With the bounding box as BN * The X-ray image and the output feature map F of the spine detection network at three scales. loc ={f loc1 ,f loc2 ,f loc3} is the input to ROIAlign. Through ROIAlign pruning, the three-scale spine feature blocks F corresponding to the m bounding boxes are cropped from the output feature maps of the spine detection network at the three scales. p ={f p1 ,f p2 ,f p3}, cut out m vertebral blocks I corresponding to the m bounding boxes from the X-ray image. p .

[0023] Furthermore, the process of extracting the spinal mask from the X-ray image includes:

[0024] A spine segmentation network was constructed, which used ResNet50 and EfficientNet-b2 as the backbone networks for feature extraction.

[0025] The X-ray images are input into two feature extraction backbone networks for feature extraction, resulting in two feature maps of the same scale. The two feature maps of the same scale are then concatenated by channel and convolved to obtain a fused feature map.

[0026] The fused feature map is then subjected to spatial pyramid pooling, followed by upsampling to restore the features and output the spine mask.

[0027] Furthermore, the orientations of the k vertebrae are spliced ​​together, including:

[0028] The directions of the superior and inferior endplates of each vertebral segment are summed to obtain the final direction vector of each vertebral segment.

[0029] Based on the center point of the spine corresponding to the k bounding boxes, the entire spine is divided into three segments: the upper thoracic curve, the thoracic curve, and the lumbar curve.

[0030] For each segment, the angle between the final direction vectors of the vertebral blocks within that segment is compared pairwise, and the largest angle within that segment is determined as the Cobb angle of that segment.

[0031] Furthermore, the spine detection network employs a Yolov5s convolutional neural network;

[0032] The spine regression network uses an Efficientnet-b1 convolutional neural network.

[0033] Furthermore, prior to the process of extracting the spine mask from the X-ray image, the process includes training the spine segmentation network, the training process including:

[0034] The dataset is input into the spine segmentation network to obtain the spine mask in the X-ray images. The dataset consists of multiple X-ray images containing k vertebrae, labeled as the four corner points of the vertebrae.

[0035] Furthermore, it also includes training the spinal regression network, the training process including:

[0036] The loss between the labeled spinal orientation and the upper and lower endplate orientations of the spinal block output through the fully connected layer is calculated and the parameters of the spinal regression network are adjusted inversely. Training continues until the loss converges to the expected range, and then training ends.

[0037] The marked spinal direction is obtained through the following steps:

[0038] Calculate the intersection-union ratio (CUI) of the labeled bounding boxes and the corresponding bounding boxes containing the complete spine selected from the X-ray images;

[0039] Choose the bounding box containing the complete spine with the largest intersection-union ratio as the largest bounding box;

[0040] Calculate the superior endplate direction and inferior endplate direction of the vertebrae corresponding to the maximum bounding box;

[0041] The normalized directions of the upper and lower endplates are used as the labeled vertebral directions.

[0042] According to another aspect of the present invention, a system for assessing scoliosis angle based on regression of spinal curvature direction is provided, comprising:

[0043] A spine segmentation network is used to extract spine masks from X-ray images containing k vertebrae, where k ≥ 1.

[0044] The stitching module is used to stitch the spinal mask with the X-ray image to obtain a stitched image.

[0045] A spine detection network is used to take the stitched image as input and output k bounding boxes for the k vertebrae; and to crop the spine feature blocks F at three scales corresponding to the k bounding boxes from the output feature maps at three scales. p ={f p1 ,f p2 ,f p3};

[0046] The vertebral block generation module is used to crop out k vertebral blocks I corresponding to the k bounding boxes from the X-ray image. p ;

[0047] A spinal regression network is used to extract the k spinal blocks I. p The features are used to obtain the first feature map f. reg1 , the first feature map f reg1 With the characteristic block of the spine f p1 After adding elements one by one, features are extracted to obtain the second feature map f.reg2 The second feature map f reg2 With the characteristic block of the spine f p2 After adding elements one by one, features are extracted to obtain the third feature map f. reg3 The third feature map f reg3 With the characteristic block of the spine f p3 After adding elements one by one, features are extracted to obtain the fused feature map f. p_reg The fused feature map f p_reg Input a fully connected layer and output the directions of the upper and lower endplates of k vertebral blocks;

[0048] The Cobb angle calculation module is used to splice the directions of the upper and lower endplates of k vertebrae to obtain the curvature direction of the entire spine and calculate the Cobb angle.

[0049] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0050] (1) The method and system for evaluating scoliosis angle based on the regression of spinal curvature direction of the present invention involves cropping the three-scale spinal feature blocks corresponding to each bounding box from the output feature maps of the three scales of the spinal detection network, and simultaneously cropping the spinal blocks corresponding to each bounding box from the X-ray image. After the spinal blocks are input into the spinal regression network for feature extraction, they are added element by element to the spinal feature blocks at the same scale to obtain the feature map at the next scale. In a similar manner, the fused feature map is finally obtained. The fused feature map contains the features of the spinal detection network and the features of spinal direction regression, which is conducive to promoting the joint learning of the two networks and greatly reducing the difficulty of spinal direction regression. In contrast, the existing technology simply crops the spinal blocks corresponding to each bounding box from the X-ray image and then inputs the spinal blocks into the spinal regression network. This results in the spinal detection network and the spinal regression network being trained separately, with no information exchange between the two networks. This cannot alleviate the influence of spinal detection error on spinal direction regression, nor can it promote the joint learning of the two networks.

[0051] Meanwhile, compared with existing technologies that assess scoliosis by extracting the center line or edge line of the spine from X-ray images or by directly regressing the Cobb angle, this invention directly regresses the direction of the superior and inferior endplates of each vertebra. Therefore, the model's decision-making process has stronger clinical interpretability, and the spinal direction can be easily corrected manually, resulting in more flexible clinical interactivity.

[0052] (2) Preferably, the present invention employs a pruning module ROIAlign with differentiable properties, which enables the loss gradient to propagate backward from the spine regression network to the spine detection network, further promoting mutual learning between the two networks, deepening the information exchange between the two networks, and enabling the two networks to be trained together in an end-to-end manner.

[0053] Furthermore, compared to the existing technology of directly adding noise to the network model, this invention adds noise perturbation to the detected bounding boxes before cropping the spine blocks or spine feature blocks. This method makes the cropping region change each time, which enhances the diversity of data samples to a certain extent. At the same time, the bounding boxes with added noise perturbation are cropped directly on the feature map, which enables the spine detection network to capture richer sample features, expands the receptive field of the model, makes the input feature patterns of the spine regression network richer, and further enhances the generalization ability of the entire model.

[0054] (3) Compared with directly inputting X-ray images into the spine detection network to detect each vertebral block, which may result in false positives due to deviation from the entire spine and false negatives due to missing data, this invention first segments the spine using a spine segmentation network before using the spine detection network. Based on the fact that the entire spine has a large area with high contrast and continuous distribution on the X-ray image, the entire spine is segmented. The segmented entire spine is used to constrain the detection range of the spine detection network. This is simpler and more accurate than detecting a single small vertebra, and can avoid false positives and false negatives due to missing data, thereby improving the accuracy and robustness of the evaluation results. Attached Figure Description

[0055] Figure 1 The figures show the measurement method of the Cobb angle and examples of unclear patient X-ray images. In the figure, (a) shows the measurement method of the Cobb angle, and (b) and (c) show examples of unclear patient X-ray images.

[0056] Figure 2 Figure (a) shows the quadrilateral formed by the four vertices of the vertebrae and the directions of the superior and inferior endplates of each vertebra. Figure (b) shows the directions of the superior and inferior endplates of each vertebra.

[0057] Figure 3 This is a flowchart of the algorithm for evaluating Cobb angle based on regression of the spinal direction in an embodiment of the present invention.

[0058] Figure 4 This is the spine segmentation network framework in this embodiment of the invention.

[0059] Figure 5 This is a schematic diagram of a trimming module with a noise disturbance mechanism in an embodiment of the present invention.

[0060] Figure 6 The figure shown is a simulation result diagram of an embodiment of the present invention, wherein, Figure 6 Figures (a)-(c) show the simulation results when the test set consists of 36, 98, and 128 images, respectively. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0062] In this invention, the terms "first," "second," etc., used in the invention and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] like Figure 2 As shown in (a) and (b) in the present invention, each vertebra is regarded as a quadrilateral, and the four corner points of each vertebra are the four vertices of the quadrilateral. The coordinates of the four corner points of each vertebra are denoted as p1, p2, p3, p4, and p2-p1 represents the direction vector of the upper endplate of the vertebra, and p4-p3 represents the direction vector of the lower endplate of the vertebra. By regressing the directions of the upper and lower endplates corresponding to each vertebra, the obtained directions of the upper and lower endplates of the vertebra are spliced ​​together to obtain the curvature direction of the entire spine. The Cobb angle is calculated based on the curvature direction of the entire spine.

[0064] like Figure 3 As shown, the method for assessing scoliosis angle based on regression of spinal curvature direction according to the present invention includes:

[0065] Extract the spine mask from the X-ray image, where the X-ray image contains k vertebrae, k≥1;

[0066] A spinal detection network, LocNet, is constructed. The spinal mask is stitched together with the X-ray image and then input into the LocNet network to obtain k bounding boxes B = [b1, b2, ..., b...]. k ], b i =[x bi1 ,y bi1 ,x bi2 ,y bi2 [i] represents the coordinates of the two opposite vertices of the bounding box, 1 ≤ i ≤ k; k spine feature blocks F corresponding to the bounding boxes at the three scales are cropped from the output feature maps of the three scales of the spine detection network LocNet. p ={f p1 ,f p2 ,fp3};

[0067] Cut out k vertebral blocks corresponding to k bounding boxes from the X-ray image. p ;

[0068] Construct a RegNet spinal regression network and extract k spinal blocks I p The features are used to obtain the first feature map f. reg1 , the first feature map f reg1 With the characteristic block of the spine f p1 After adding elements one by one, features are extracted to obtain the second feature map f. reg2 The second feature map f reg2 With the characteristic block of the spine f p2 After adding elements one by one, features are extracted to obtain the third feature map f. reg3 The third feature map f reg3 With the characteristic block of the spine f p3 After adding elements one by one, features are extracted to obtain the fused feature map f. p_reg The fused feature map f p_reg Input a fully connected layer to obtain the orientation V of the superior and inferior endplates for each vertebral segment. p =[v p1 ,v p2 ], where v p1 v represents the endplate direction vector on each vertebral block. p2 This represents the direction vector of the endplate under each vertebral block, and f at the corresponding scale. pi and f regi They have the same size, and 1 ≤ i ≤ 3.

[0069] By splicing together the superior and inferior endplates of each vertebral segment, the curvature direction of the entire spine is obtained, and the Cobb angle is calculated.

[0070] Specifically, such as Figure 4 As shown, the spine mask in X-ray images is extracted using the SegNet spine segmentation network. In this embodiment, the SegNet spine segmentation network adopts an encoding path and a decoding path structure. The process of extracting the spine mask in X-ray images includes:

[0071] In the encoding path, ResNet50 and EfficientNet-b2 are used as the backbone networks for feature extraction;

[0072] X-ray images are input into ResNet50 and EfficientNet-b2 feature extraction backbone networks respectively for feature extraction, resulting in two feature maps of the same scale. The two feature maps of the same scale are then concatenated by channel and a convolution operation is performed to fuse the features of the two feature maps, resulting in a fused feature map.

[0073] The fused feature map is then expanded by spatial pyramid pooling to increase the receptive field of the convolutional kernel and add global image information. Finally, it is decoded by upsampling to restore the features and output a binary mask of the spine.

[0074] In this embodiment, the decoding path uses a decoder with three sets of deconvolutional modules and four sets of convolutional modules cascaded alternately to decode the features. The deconvolutional module is a network layer composed of deconvolution, batch normalization, LeakyReLU activation function, convolution (3x3 kernel size, stride 1), batch normalization, and LeakyReLU activation function stacked sequentially. The convolutional module is a network layer composed of convolution, batch normalization, LeakyReLU activation function stacked sequentially. In the decoder's output layer, a sigmoid activation function is used to output a binary mask S of the spine.

[0075] Specifically, in the spine detection network LocNet, the spine mask output by the spine segmentation network SegNet is combined with X-ray images to assist in spine detection. In this embodiment of the invention, a Yolov5s convolutional neural network is selected as the spine detection network LocNet. Due to the assistance of the segmentation results from the spine segmentation network SegNet, the spine detection network LocNet can effectively reduce false positives and significantly reduce the regression error of the downstream spine orientation regression task.

[0076] Preferably, to avoid a large error between the bounding boxes output by the LocNet spine detection network and the labeled bounding boxes, which would affect the accuracy of the final detection results, the method further includes the step of: selecting bounding boxes containing the complete spine from the k bounding boxes output by the LocNet spine detection network, and discarding bounding boxes with large errors compared to the labeled bounding boxes. This specifically includes the following sub-steps:

[0077] Step S11: Calculate the i-th bounding box b i =[x bi1 ,y bi1 ,x bi2 ,y bi2 The intersection-union ratios (IOUs) of the i-th bounding box G with the k labeled bounding boxes are used to obtain the set of k IOUs for the i-th bounding box, where 1 ≤ i ≤ k;

[0078] Step S12: Find the largest Intersection over Union (IOU) ratio in the set IOUs. max With the set threshold I t Compare sizes, if IOU max≥I t Then the i-th bounding box is a bounding box that contains the complete spine, denoted as . Otherwise, discard the i-th bounding box;

[0079] Step S13: Repeat steps S11 and S12 to obtain m selected bounding boxes containing complete spines. in, Let be the coordinates of the two opposite vertices of the bounding box, and m represent the number of bounding boxes containing the complete spine selected from k bounding boxes in an X-ray image, where 1 ≤ m ≤ k.

[0080] The bounding box is obtained through the following steps:

[0081] Choose the smallest bounding box from the k bounding boxes that can completely enclose the four corner points of the corresponding spine;

[0082] The minimum bounding box is enlarged by a set factor to obtain the labeled bounding box; in this embodiment, it is enlarged by a factor of 1.1.

[0083] In this embodiment, k = 17 is chosen, that is, G = [g1, g2, ..., g 17 ], where g i =[x gi1 ,y gi1 ,x gi2 ,y gi2 ], x gi1 ,y gi1 ,x gi2 ,y gi2 These represent the coordinates of two opposite vertices of the bounding box, respectively, and the set IOUs = [iou1, iou2, ..., iou...]. 17 ], where iou i This represents the IOU between the bounding box and the i-th labeled bounding box, where 1 ≤ i ≤ 17. The set threshold I... t It is 0.5.

[0084] As a preferred option, such as Figure 5 As shown, a cropping module with a noise perturbation mechanism is used to crop k bounding boxes corresponding to the three scales of the spine detection network LocNet, resulting in spine feature blocks F at three scales. p And cropping k vertebral blocks corresponding to k bounding boxes from the X-ray image. p This involves adding Gaussian noise to the bounding boxes output by the LocNet spine detection network, and then using the noise-perturbed bounding boxes to crop the corresponding spine blocks and spine feature blocks based on the differentiable ROIAlign cropping module. This enhances the tolerance of the spine orientation regression network to the localization errors of the LocNet spine detection network, while also improving the generalization ability of the spine orientation regression network. Specifically, the steps include:

[0085] Add a set Gaussian noise N = [n1, n2, ..., n] to the m selected bounding boxes containing complete spines. m ]∈R 4×m Then, the bounding box with added noise perturbation is obtained as follows: Where, n i =[x ni1 ,y ni1 ,x ni2 ,y ni2 ], 1≤i≤m;

[0086] Using the bounding box as BN * Output feature maps F at three scales of the LocNet network for X-ray imaging and spinal cord detection loc ={f loc1 ,f loc2 ,f loc3} is the input to ROIAlign. Through ROIAlign cropping, m bounding boxes corresponding to the three scales of the spine feature blocks F are cropped from the output feature maps of the three scales of the spine detection network LocNet. p ={f p1 ,f p2 ,f p3}, cut out m vertebral blocks I corresponding to m bounding boxes from the X-ray image. p .

[0087] In this embodiment, the output feature maps F at three scales loc The outputs of the three C3 modules of the YOLOv5s convolutional neural network produce feature sizes at three different scales, ranging from 64×384×192, 128×192×96, and 256×96×48, respectively. The multipliers for each size group are the number of feature channels, the feature height, and the feature width, respectively, corresponding to the feature block F. p ={f p1 ,f p2 ,f p3 The sizes, from largest to smallest, are m×64×48×64, m×128×24×32, and m×256×12×16. The multipliers for each size group are the batch size, the number of feature channels, the height of the feature, and the width of the feature, respectively.

[0088] Since the pruning process of the ROIAlign module is differentiable, the two stages can be trained in an end-to-end manner, which can deeply integrate the learning of the two stages of the spinal detection network and the spinal regression network.

[0089] Specifically, in this embodiment, the spine detection network RegNet uses an Efficientnet-b1 convolutional neural network to regress the orientation of each spine block.

[0090] Specifically, the orientation of each spine block output by the RegNet spine detection network is concatenated, including:

[0091] The directions of the superior and inferior endplates of each vertebral segment are summed to obtain the final direction vector of each vertebral segment.

[0092] Based on the center point of the spine corresponding to each bounding box output by the LocNet spinal detection network, the entire spine is divided into three segments: upper thoracic curve (PT), thoracic curve (MT), and lumbar curve (TL).

[0093] For each segment, the angle between the final direction vectors of each vertebral block within that segment is compared pairwise. The angle with the largest angle within that segment is determined as the Cobb angle of that segment, thus obtaining the Cobb angles of the three segments.

[0094] Specifically, before the process of extracting the spine mask from X-ray images, the process also includes training the spine segmentation network SegNet. The training process includes:

[0095] The dataset is input into the SegNet spine segmentation network to obtain the spine mask in the X-ray images. The dataset consists of multiple X-ray images containing k vertebrae, labeled as the four corner points of the vertebrae, where k≥1.

[0096] Using the shape similarity loss function Loss Dice and binary cross-entropy loss function Loss Bce Weighted loss function Loss Seg The parameters of the SegNet spinal segmentation network are adjusted in reverse, and training continues until the loss converges to the expected range, at which point training ends.

[0097] Among them, the loss function Loss Seg for:

[0098] Loss Seg =Loss Dice +λLoss BCE (1)

[0099] In the formula, λ is the weighting coefficient, which is taken as 1.0 in this embodiment of the invention.

[0100] Loss Dice and Loss BCE As shown in equations (2) and (3) respectively:

[0101]

[0102]

[0103] In the formula, Si It is the result of the segmentation network prediction, Y i is the spine annotation mask, and n is the amount of data used in each iteration of training.

[0104] Before using the dataset, it is normalized. Simultaneously, due to the small data volume, to prevent overfitting, this invention also employs data augmentation methods to increase the data volume. These methods include, but are not limited to, edge sharpening, affine transformation, histogram equalization, and random contrast enhancement. To standardize the image size input to the network, the data in this embodiment is deformed and padded to a resolution of 1536×768 pixels, where 1536 is the image height and 768 is the image width.

[0105] Before using the spinal detection network LocNet, the process also includes training the spinal detection network LocNet. The output of the spinal segmentation network SegNet is stitched together with the corresponding X-ray image and used as the input to train the spinal detection network LocNet.

[0106] Before using the spine detection network RegNet, training the spine detection network RegNet is also included. The training process includes:

[0107] The loss between the labeled vertebral orientation and the upper and lower endplate orientations of each vertebral block output through the fully connected layer is calculated and the parameters of the RegNet vertebral detection network are adjusted in reverse. Training continues until the loss converges to the expected range, and then training ends.

[0108] The indicated spinal direction is obtained through the following steps:

[0109] Calculate the intersection-union ratio (CUI) of the labeled bounding boxes and the corresponding bounding boxes containing the complete spine selected from the X-ray images;

[0110] The bounding box containing the complete spine (i.e., the bounding box predicted by the spine detection network RegNe) with the largest intersection and union is selected as the largest bounding box.

[0111] Calculate the superior endplate direction and inferior endplate direction of the vertebrae corresponding to the maximum bounding box;

[0112] Normalized superior and inferior endplate directions are used as labeled vertebral directions V. g .

[0113] The loss function of the spine detection network RegNet is Loss. Reg It is the vector distance loss. Dist Loss of vector angle Ang The weighted average, i.e.:

[0114] Loss Reg =Loss Dist +αLoss Ang (4)

[0115] In the formula, α is a weighting coefficient, which is taken as 0.25 in this embodiment of the invention.

[0116] Loss Dist and Loss Ang As shown in equations (5) and (6) respectively:

[0117]

[0118]

[0119] In the formula, ΔV=V g -V p =[v g1 -v p1 ,v g2 -v p2 ], ω determines the nonlinear range, ε limits the curvature of the nonlinear region, and C is a constant and

[0120] V p =[v p1 ,v p2 [V] represents the orientation of the superior and inferior endplates of each vertebra as output by the fully connected layers of the RegNet spinal detection network; g =[v g1 ,v g2 [] indicates the direction of the spine.

[0121] The present invention also provides a system for assessing scoliosis angle based on regression of spinal curvature direction, the system comprising:

[0122] SegNet, a spine segmentation network, is used to extract spine masks from X-ray images containing k vertebrae, where k ≥ 1.

[0123] The stitching module is used to stitch the spinal mask with the X-ray image to obtain the stitched image.

[0124] The LocNet spine detection network takes a stitched image as input and outputs k bounding boxes for k vertebrae. It then crops three scales of spine feature blocks F corresponding to these k bounding boxes from the output feature maps of LocNet at three scales. p ={f p1 ,f p2 ,f p3};

[0125] The spine block generation module is used to crop k spine blocks I corresponding to k bounding boxes from an X-ray image. p ;

[0126] The RegNet spinal detection network is used to extract k vertebral blocks I. p The features are used to obtain the first feature map f. reg1 , the first feature map f reg1 With the characteristic block of the spine f p1 After adding elements one by one, features are extracted to obtain the second feature map f. reg2 The second feature map f reg2 With the characteristic block of the spine f p2 After adding elements one by one, features are extracted to obtain the third feature map f. reg3 The third feature map f reg3 With the characteristic block of the spine f p3 After adding elements one by one, features are extracted to obtain the fused feature map f. p_reg The fused feature map f p_reg Input a fully connected layer and output the directions of the upper and lower endplates of k vertebral blocks;

[0127] The Cobb angle calculation module is used to splice the directions of the upper and lower endplates of k vertebrae to obtain the curvature direction of the entire spine and calculate the Cobb angle.

[0128] Preferably, to avoid large errors between the bounding boxes output by the LocNet spine detection network and the labeled bounding boxes, which would affect the accuracy of the final detection results, a bounding box filtering module is also included. This module is used to filter out bounding boxes containing the complete spine from the k bounding boxes output by the LocNet spine detection network, and discard bounding boxes with large errors compared to the labeled bounding boxes.

[0129] The bounding box filtering module specifically includes:

[0130] The intersection-union ratio (IoU) calculation module is used to calculate the i-th bounding box b. i =[x bi1 ,y bi1 ,x bi2 ,y bi2 The intersection-union ratios (IOUs) of the i-th bounding box G with the k labeled bounding boxes are used to obtain the set of k IOUs for the i-th bounding box, where 1 ≤ i ≤ k;

[0131] The judgment module is used to find the largest intersection-union ratio (IOU) in the set IOUs. max With the set threshold I t Compare sizes, if IOU max ≥I t Then the i-th bounding box is a bounding box that contains the complete spine, denoted as . Otherwise, discard the i-th bounding box;

[0132] The filtering module is used to repeatedly execute the intersection-union calculation module and the judgment module to obtain m selected bounding boxes containing the complete spine. in, Let be the coordinates of the two opposite vertices of the bounding box, and m represent the number of bounding boxes containing the complete spine selected from k bounding boxes in an X-ray image, where 1 ≤ m ≤ k.

[0133] In this embodiment of the invention, the Cobb angle is evaluated using Mean Absolute Error (MAE) and Symmetric Mean Absolute Percentage Error (SMAPE), with the following expressions:

[0134]

[0135]

[0136] In the formula, T is the data size of the test set, and a ij Let b represent the j-th Cobb angle predicted for the i-th data point. ij Let represent the j-th Cobb angle of the i-th data point, where i = 1, 2, ..., T, j = 1, 2, 3.

[0137] To fully verify the effectiveness of the present invention, the performance of the embodiments of the present invention was tested on three datasets. The first dataset is the official validation set of the MICCAI2019 scoliosis assessment competition, containing 128 X-ray images; the second dataset is the test set of the aforementioned competition, containing 98 X-ray images; the third dataset is a test set collected from a local hospital, containing 36 X-ray images. Each test set includes annotations of 68 vertebral angles and 3 Cobb angles calculated from the angles, as labeled by professional doctors.

[0138] Table 1 presents the experimental results of quantitatively evaluating the Cobb angle using embodiments of the present invention. The results show that the present invention achieved good experimental performance on all three datasets, with the absolute error consistently below 4 degrees and the relative error remaining below 11%. Furthermore, as... Figure 6 As shown in Figures (a)-(c), this embodiment of the invention also demonstrates the regressed spinal direction vector. Solid arrows in the figures represent the predicted direction vector, while dashed arrows represent the labeled direction vector. The test results from the three datasets show that the regressed direction vector and the labeled direction vector largely coincide, indicating that the prediction effect of this invention is good. Therefore, this invention has significant application value in the clinical diagnosis of scoliosis.

[0139] Table 1: Experimental results of the embodiments of the present invention on three test sets

[0140] Test data (sheets) MAE(degree) SMAPE (%) MICCAI2019 Validation Set (128) 2.92 6.87 MICCAI2019 Test Set (98) 3.47 11.00 Private test set (36) 3.61 5.90

[0141] The present invention relates to a method and system for assessing scoliosis angle based on the regression of spinal curvature direction. This method involves cropping spinal feature blocks corresponding to each bounding box at three scales from the output feature maps of the three-scale spinal detection network LocNet, and simultaneously cropping spinal blocks corresponding to each bounding box from X-ray images. These spinal blocks are then input into the spinal detection network RegNet for feature extraction, and each feature block is element-wise added to the corresponding feature block at each scale to obtain the feature map at the next scale. This process is repeated to obtain a fused feature map. The fused feature map contains features from both the LocNet and spinal direction regression networks, which facilitates joint learning between the two networks and significantly reduces the difficulty of spinal direction regression. In contrast, existing technologies simply crop spinal blocks corresponding to each bounding box from X-ray images and input these blocks into the RegNet network. This results in LocNet and RegNet being trained independently, with no information exchange between them. This approach fails to mitigate the impact of spinal detection errors on spinal direction regression and also fails to promote joint learning between the two networks.

[0142] Furthermore, this invention employs a pruning module ROIAlign with differentiable properties, enabling the loss gradient to propagate backward from the spine detection network RegNet to the spine detection network LocNet, further promoting mutual learning between the two networks, deepening the information exchange between the two networks, and enabling the two networks to be trained together in an end-to-end manner.

[0143] Furthermore, compared to existing methods that assess scoliosis by extracting the center line or edge line of the spine from X-ray images or by directly regressing the Cobb angle, this invention directly regresses the direction of the superior and inferior endplates of each vertebra. Therefore, the model's decision-making process has stronger clinical interpretability, and the spinal direction can be easily corrected manually, resulting in more flexible clinical interactivity.

[0144] Furthermore, compared to directly inputting X-ray images into the LocNet spinal detection network to detect each vertebral block, which can result in false positives due to deviations from the entire spine and false negatives due to missing segments, this invention first segments the spine using the SegNet spinal segmentation network before using the LocNet. Based on the fact that the entire spine has a large, continuously distributed region with high contrast on the X-ray image, the entire spine is segmented. The segmented entire spine is used to constrain the detection range of the LocNet spinal detection network. This is simpler and more accurate than detecting individual small vertebrae, and can avoid false positives and false negatives due to missing segments, thereby improving the accuracy and robustness of the evaluation results.

[0145] Furthermore, this invention adds random Gaussian noise to the detected bounding boxes, enabling the RegNet spine detection network to adapt to more visual features and increasing the model's generalization ability to some extent.

[0146] Those skilled in the art will readily understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating a scoliosis angle based on a regression spine curvature direction, characterized by, The method comprises the following steps: extracting a spine mask in an X-ray image, wherein the X-ray image contains k vertebrae, and k≥1; The spine detection network is built, the spine mask is spliced with the X-ray image, and the k bounding boxes of the k vertebrae are obtained by inputting the spine detection network. ; cropping out the k vertebral blocks corresponding to the k bounding boxes on the X-ray image ; Build a spine regression network to extract the k spine blocks , and obtain a first feature map , and obtain a second feature map , and obtain a third feature map , and obtain a fused feature map , and obtain a fourth feature map , and obtain a fifth feature map , and obtain a sixth feature map , and obtain a seventh feature map , and obtain an eighth feature map , and obtain a ninth feature map , and obtain a tenth feature map , and output the directions of the upper and lower endplates of the k spine blocks. splicing the upper endplate and lower endplate directions of the k vertebrae blocks to obtain the bending direction of the whole spine, and calculating the Cobb angle; filtering the bounding boxes containing complete vertebrae from the k bounding boxes, specifically comprising the following sub-steps: Step S11, calculating the intersection-over-union (IOU) of the i-th bounding box and the k labeled bounding boxes to obtain the set of k intersection-over-unions (IOUs) of the i-th bounding box, wherein 1≤i≤k; Step S12, take the maximum intersection over union in the set IOUs with the set threshold Compare the size, if ≥ , the i-th bounding box is the bounding box containing the complete spine, denoted as , otherwise, discard the i-th bounding box; Step S13: Repeat steps S11 and S12 to obtain m selected bounding boxes containing complete spines. =[ , ,…, ],in, , where are the coordinates of two opposite vertices of the bounding box, 1≤m≤k; cropping the k bounding boxes corresponding to the k bounding boxes on the X-ray image through the ROIAlign cropping module with the noise disturbance mechanism and cropping the k bounding boxes corresponding to the k bounding boxes on the X-ray image through the ROIAlign cropping module with the noise disturbance mechanism ; After adding Gaussian noise to the m selected bounding boxes containing complete vertebrae, the bounding boxes are [ , , , , , ], wherein =[ , , , ] ; The bounding box is taken as , the output feature map of the three scales of the X-ray image and the spine detection network is the input of ROIAlign, and the three-scale spine feature blocks corresponding to the m bounding boxes are cropped on the output feature map of the three scales of the spine detection network through ROIAlign cropping , the m spine blocks corresponding to the m bounding boxes are cropped on the X-ray image ; The labeled bounding box is obtained by the following steps: selecting the smallest bounding box that can completely enclose the four corner points of the corresponding vertebrae from the k bounding boxes; enlarging the smallest bounding box by a certain multiple to obtain the labeled bounding box.

2. The method of claim 1, wherein, The process of extracting the spine mask in the X-ray image comprises: building a spine segmentation network, wherein the spine segmentation network takes Resnet50 and Efficientnet-b2 as the feature extraction backbone network; inputting the X-ray image into the two feature extraction backbone networks respectively to perform feature extraction, obtaining two feature maps of the same scale, splicing the two feature maps of the same scale by channel, and performing convolution to obtain a fused feature map; performing spatial pyramid pooling on the fused feature map, and then performing feature restoration through upsampling operation to output the spine mask.

3. The method of claim 1, wherein, Splicing the directions of the k vertebrae blocks comprises: summing the upper endplate and lower endplate directions of each vertebrae block to obtain the final direction vector of each vertebrae block; dividing the whole spine into three segments, i.e., upper thoracic bend, thoracic bend and lumbar bend, according to the center points of the vertebrae corresponding to the k bounding boxes; for each segment, comparing the angles between the final direction vectors of the vertebrae blocks in the segment to determine the maximum angle in the segment as the Cobb angle of the segment.

4. The method of claim 2, wherein, The spine detection network adopts a Yolov5s convolutional neural network; The spine regression network adopts an Efficientnet-b1 convolutional neural network.

5. The method of claim 3, wherein, Before the process of extracting the spine mask in the X-ray image, the spine segmentation network is trained, and the training process comprises: inputting a data set into the spine segmentation network to obtain the spine mask in the X-ray image, wherein the data set is a plurality of X-ray images containing k vertebrae, and the labels are the four corner points of the vertebrae.

6. The method of claim 5, wherein, The spine regression network is also trained, and the training process comprises: calculating the loss between the labeled vertebrae direction and the upper endplate and lower endplate directions of the vertebrae block output by the fully connected layer, and adjusting the parameters of the spine regression network in reverse, continuously training until the loss converges to the expected range, and ending the training; wherein the labeled vertebrae direction is obtained by the following steps: calculating the intersection-over-union (IOU) of the labeled bounding box and the bounding box containing complete vertebrae filtered from the corresponding X-ray image; selecting the bounding box containing complete vertebrae with the largest intersection-over-union (IOU) as the maximum bounding box; calculating the upper endplate direction and lower endplate direction of the vertebrae corresponding to the maximum bounding box; normalizing the upper endplate direction and lower endplate direction as the labeled vertebrae direction.

7. A system for assessing the angle of scoliosis based on the direction of the regression of the curvature of the spine, characterized in that, ​ A spine segmentation network is used to extract a spine mask in an X-ray image, wherein the X-ray image contains k vertebrae, and k≥1; A splicing module is used to splice the spine mask and the X-ray image to obtain a spliced image; The spine detection network is configured to input the spliced image and output k bounding boxes of the k spines; and cut out three-scale spine feature blocks corresponding to the k bounding boxes on three-scale output feature maps ; wherein, the k bounding boxes are filtered to obtain a bounding box containing a complete spine, specifically including the following sub-steps: In step S11, the intersection over union (IOU) of the i th bounding box and the k labeled bounding boxes is calculated to obtain a set of k intersection over unions (IOUs) of the i th bounding box, wherein 1≤i≤k; Step S12, take the maximum intersection over union in the set IOUs with the set threshold Compare the size, if ≥ , the i-th bounding box is the bounding box containing the complete spine, denoted as , otherwise, discard the i-th bounding box; Step S13: Repeat steps S11 and S12 to obtain m selected bounding boxes containing complete spines. =[ , ,…, ],in, , where are the coordinates of two opposite vertices of the bounding box, 1≤m≤k; a vertebra block generation module configured to crop out k vertebra blocks corresponding to the k bounding boxes on the X-ray image ; a spine regression network for extracting the k spine blocks to obtain a first feature map extracting features from the first feature map and the spine feature blocks after element-wise addition to obtain a second feature map extracting features from the second feature map and the spine feature blocks after element-wise addition to obtain a third feature map extracting features from the third feature map and the spine feature blocks after element-wise addition to obtain a fused feature map inputting the fused feature map into a fully connected layer to output directions of upper and lower endplates on the k spine blocks A Cobb angle calculation module is used to splice the upper end plate and the lower end plate of the k vertebrae blocks to obtain the bending direction of the whole spine, and calculate the Cobb angle; The ROIAlign clipping module with a noise disturbance mechanism clips three scales of spine feature blocks corresponding to the k bounding boxes from the output feature maps of the three scales of the spine detection network and clips k spine blocks corresponding to the k bounding boxes from the X-ray image ; After adding Gaussian noise to the m selected bounding boxes containing complete vertebrae, the bounding boxes are [ , , , , , ], wherein =[ , , , ] ; The bounding box is taken as , the output feature map of the three scales of the X-ray image and the spine detection network is the input of ROIAlign, and the three-scale spine feature blocks corresponding to the m bounding boxes are cropped on the output feature map of the three scales of the spine detection network through ROIAlign cropping , the m spine blocks corresponding to the m bounding boxes are cropped on the X-ray image ; The labeled bounding box is obtained by the following steps: In the k bounding boxes, the smallest bounding box capable of completely enclosing the four corner points of the corresponding vertebrae is selected; The smallest bounding box is enlarged by a set multiple to obtain the labeled bounding box.

Images