Thoracolumbar spine key point automatic identification and Cobb angle calculation method based on long and short range characteristics

Abstract

The invention relates to a thoracolumbar spine key point automatic identification and Cobb angle calculation method based on long and short range features. The method comprises the following steps: forming a sample set; a network model is constructed, the constructed network model comprises an encoder with d encoding layers, a decoder with d decoding layers, d-1 centrum position capturing modules and a processing unit, and each centrum position capturing module comprises a short-range feature connection module and a long-range feature connection module, the processing unit comprises a center positioning module, a center comparison learning module, a center offset module and an angle positioning module; all sample images in the training set are input into the constructed network model in batches for training, and after multiple times of training and verification, an optimal network model is screened out; inputting a to-be-tested image in the test set into the optimal network model to obtain a key point positioning result of the to-be-tested image; the method reduces the influence of the shielding of foreign objects on the detection of the thoracolumbar key points, and enables the recognition of the thoracolumbar key points to be more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for automatically identifying thoracic and lumbar key points and calculating Cobb angles based on long- and short-range features. Background Art

[0002] Scoliosis is mainly diagnosed by measuring the Cobb angle in the thoracolumbar segment of the spine in the anteroposterior X-ray image to quantify and diagnose spinal deformity. The Cobb angle is the angle between the upper end plate of the uppermost vertebra where the scoliosis begins and the lower end plate of the lowermost vertebra where the scoliosis ends in the anteroposterior X-ray image of the spine. Usually, a complete spine has three Cobb angles.

[0003] In recent years, many convolutional neural network-based Cobb angle automatic estimation methods have shown great potential compared to traditional machine learning algorithms. Existing methods are mainly divided into two categories: 1) regression-based methods, and 2) vertebral key point detection-based methods. Regression-based methods map images to Cobb angles by omitting the complex intermediate steps. Although regression-based methods can directly obtain three Cobb angles, they lack the demonstration of the basis for the results, so they cannot be promoted in clinical environments. Compared with regression-based methods, vertebral key point detection-based methods calculate Cobb angles by finding four key points of the vertebra in the image or segmenting the vertebra, thus providing an intuitive basis for clinical decision-making.

[0004] However, the accuracy of the Cobb angle estimated by the key point detection method depends on the accuracy of the vertebral key point detection. The vertebral key point detection is usually affected by the occlusion of tissues, organs and other foreign objects in the image. Therefore, the accurate identification of the vertebra is still a major problem in the automatic estimation of the Cobb angle. For this reason, further improvements to the existing technology are needed. Summary of the invention

[0005] The first technical problem to be solved by the present invention is to provide a method for automatically identifying key points of the thoracic and lumbar vertebrae based on long- and short-range features, which can improve detection accuracy, in view of the above-mentioned prior art.

[0006] The second technical problem to be solved by the present invention is to provide a Cobb angle calculation method using the above-mentioned automatic identification method of thoracic and lumbar key points based on long- and short-range features.

[0007] The technical solution adopted by the present invention to solve the above-mentioned first technical problem is: a method for automatically identifying thoracic and lumbar key points based on long- and short-range features, which is characterized by comprising the following steps:

[0008] Step 1, obtain a certain number of anterior-posterior X-ray images of the spine, and mark the upper left, upper right, lower left and lower right positions of each thoracic and lumbar vertebra in each X-ray image to form a sample set;

[0009] Step 2: Divide the sample set into training set, validation set and test set;

[0010] Step 3, constructing a network model, wherein the constructed network model includes an encoder with d encoding layers, a decoder with d decoding layers, d-1 vertebral position capture modules and a processing unit, where d is a positive integer greater than 1;

[0011] The d encoding layers are connected in sequence, the d decoding layers are also connected in sequence, and the output end of the d encoding layer is connected to the input end of the 1st decoding layer; the 1st vertebral position capture module is arranged between the 2nd encoding layer and the 3rd encoding layer, the 2nd vertebral position capture module is arranged between the 3rd encoding layer and the 4th encoding layer, ..., the d-2nd vertebral position capture module is arranged between the d-1th encoding layer and the dth encoding layer, the d-1th vertebral position capture module is arranged between the dth encoding layer and the 1st decoding layer, the 1st vertebral position capture module is jump-connected to the dth decoding layer, the 2nd vertebral position capture module is jump-connected to the d-1th decoding layer, ..., the d-1th vertebral position capture module is jump-connected to the 2nd decoding layer;

[0012] The vertebral position capture module includes a short-range feature connection module and a long-range feature connection module. The specific processing process of the vertebral position capture module is: the original feature map input to the vertebral position capture module is divided into blocks twice, one block division is to evenly divide the original feature map into q first feature blocks, and the other block division is to evenly divide the original feature map into q-1 second feature blocks; the image size of the second feature block is the same as that of the first feature block, and q is a positive integer greater than 1;

[0013] Then, the short-range feature connection module is used to perform self-attention processing on each first feature block and each second feature block respectively, to obtain a first feature map corresponding to each first feature block and a second feature map corresponding to each second feature block, and all the first feature maps are spliced ​​in the order of the first feature blocks to obtain a first output feature map; in addition, all the second feature maps are spliced ​​in the order of the second feature blocks to obtain a second output feature map, and the first output feature map and the second output feature map have the same image size as the original feature map. Finally, the first output feature map and the second output feature map are fused to obtain a feature map output by the short-range feature connection module;

[0014] In addition, the long-range feature connection module is used to perform global average pooling processing on each first feature block and each second feature block, respectively, to obtain a third feature map corresponding to each first feature block and a fourth feature map corresponding to each second feature block; then, self-attention processing is performed on all third feature maps to obtain a fifth feature map, and the fifth feature map is added to the original feature map to obtain a third output feature map; similarly, self-attention processing is performed on all fourth feature maps to obtain a sixth feature map, and the obtained sixth feature map is added to the original feature map block to obtain a fourth output feature map; finally, the third output feature map and the fourth output feature map are fused to obtain a feature map output by the long-range feature connection module;

[0015] The feature map output by the short-range feature connection module, the feature map output by the long-range feature connection module and the original feature map are added together to obtain the output of the vertebral position capture module;

[0016] The processing unit includes a center positioning module, a center contrast learning module, a center offset module and an angle positioning module, and the center positioning module, the center contrast learning module, the center offset module and the angle positioning module are connected to the output end of the dth decoding layer;

[0017] Step 4: Input all sample images in the training set into the network model constructed in step 3 in batches for training, and use all sample images in the validation set to verify the performance of the trained network model; after multiple training and validation, select the optimal network model;

[0018] Step 5: Input the image to be tested in the test set into the optimal network model to obtain the key point positioning result of the image to be tested; the key point positioning result of the image to be tested includes the coordinates of the center point of each vertebra in the image to be tested predicted by the center offset module and the coordinates of the endpoints of each vertebra in the image to be tested predicted by the angle positioning module.

[0019] Preferably, before dividing the second feature map blocks, a distance of h / 2q is first left above and below the original feature map, and then the image except the left positions in the original feature map is evenly divided, so as to obtain q-1 second feature map blocks, where h is the height of the original feature map.

[0020] Specifically, the jth feature block in the feature map output by the short-range feature connection module The calculation formula is:

[0021]

[0022] in, is the jth first feature map in the first output feature map, is the jth first feature map in the second output feature map, and Mean(.) is the mean processing.

[0023] Specifically, the jth feature block in the feature map output by the long-range feature connection module The calculation formula is:

[0024]

[0025] in, is the jth feature map in the third output feature map, is the j-th feature map in the fourth output feature map.

[0026] Specifically, each time the network model is trained in step 4, the total loss function L is calculated all , and according to the total loss function L all Reversely update the network parameters in the network model to obtain a network model after the training is completed;

[0027] L all The calculation formula is:

[0028] L all =L hm +L hc +L cenoff +L coroff

[0029] Among them, L hm is the first loss function calculated based on the output of the center positioning module, L hc is the second loss function calculated based on the output of the center contrast learning module, L cenoff is the third loss function calculated based on the output of the center offset module, L coroff is the fourth loss function calculated according to the output result of the angular positioning module.

[0030] Preferably, the specific calculation process of the center positioning module is: by calculating the average value of the coordinates of the four endpoints of the upper left, upper right, lower left and lower right of each vertebra in the original image, and taking it as the center point coordinate of each vertebra, then converting the center point of each vertebra into a Gaussian heat map through Gaussian transformation, and finally using the Gaussian heat map to predict the center point coordinate of each vertebra; the above original image is a training sample in the training set;

[0031] The first loss function L hm The calculation formula is:

[0032]

[0033] Among them, N is the total number of pixels on the Gaussian heat map, i is the pixel position index on the Gaussian heat map, and y i is the pixel value at the i-th position on the original image, pi is the pixel value at the i-th position on the Gaussian heat map, and α and β are both hyperparameters.

[0034] Preferably, the specific calculation process of the center contrast learning module is:

[0035] The output end of the d-th decoding layer is passed through the feature extraction layer to obtain an output feature map, which has the same size as the Gaussian heat map. The pixel positions in the Gaussian heat map with pixel values ​​greater than T are screened out and recorded as the first position set. Each feature in the output feature map that is in the first position set is used as a positive sample, and the features in the output feature map other than the positive sample are used as negative samples.

[0036] The second loss function L hc The calculation formula is:

[0037]

[0038] Among them, P i is the position set of positive samples in the output feature map, A is the total number of positive samples in the output feature map, i is the position of all samples in the output feature map, E i is the output feature map, i + is the position of the positive sample in the output feature map, is the feature map corresponding to the positive sample in the output feature map, τ is a hyperparameter; N i is the location set of negative samples in the output feature map, i - is the position of negative samples in the output feature map, is the feature map corresponding to the negative sample in the output feature map, and sim(.) is the cosine similarity.

[0039] Preferably, the center offset module predicts the coordinates of the center point of each vertebra in the original image based on the coordinates of the center point of each vertebra predicted by the center positioning module;

[0040] The third loss function L cenoff The calculation formula is:

[0041]

[0042] Where M is the total number of vertebral center points, x′ k and y′ k To predict the horizontal and vertical coordinates of the center point of the kth vertebra in the original image, x k and k is the theoretical horizontal and vertical coordinates of the center point of the kth vertebra in the original image, n is the ratio of the original image size to the feature map restored by the decoder, is the floor function.

[0043] Preferably, the angle positioning module predicts the coordinates of each vertebral endpoint in the original image based on the center point coordinates of each vertebra predicted by the center offset module;

[0044] The fourth loss function L coroff The calculation formula is:

[0045]

[0046] Where P is the set of four endpoints of each vertebra, x i ′ and y i ′ is the predicted horizontal and vertical coordinates of the center point of the i-th vertebra in the original image, x i and i is the theoretical horizontal and vertical coordinates of the center point of the i-th vertebra in the original image, x i,p and i,p To predict the horizontal and vertical coordinates of the endpoint p of the i-th vertebra in the original image.

[0047] The technical solution adopted by the present invention to solve the above second technical problem is: a Cobb angle calculation method, characterized by: applying the above-mentioned automatic identification method of thoracic and lumbar key points based on long- and short-range features.

[0048] Compared with the prior art, the advantages of the present invention are: by setting a short-range feature connection module and a long-range feature connection module in the vertebral position capture module, the short-range feature connection module and the long-range feature connection module are utilized to strengthen the short- and long-range feature dependence of the thoracic and lumbar key points, thereby strengthening the connection between the corresponding position feature points and improving the network recognition accuracy, reducing the influence of foreign object occlusion on the detection of thoracic and lumbar key points, and making the recognition of thoracic and lumbar key points more accurate; in addition, through the center positioning module, the center comparison learning module, the center offset module and the angle positioning module, the target vertebra and the non-target vertebra are further distinguished, thereby removing the interference with the detection of the thoracic key points and improving the detection accuracy of the thoracic and lumbar key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the network model in an embodiment of the present invention;

[0050] Figure 2 for Figure 1 Schematic diagram of the architecture of the mid-vertebral position capture module. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below with reference to the accompanying drawings.

[0052] The method for automatically identifying thoracic and lumbar key points based on long-range and short-range features in this embodiment includes the following steps:

[0053] Step 1, obtain a certain number of anterior-posterior X-ray images of the spine, and mark the upper left, upper right, lower left and lower right positions of each thoracic and lumbar vertebra in each X-ray image to form a sample set;

[0054] The X-ray images acquired in this embodiment were provided by two radiologists with at least 5 years of clinical experience, focusing on 17 vertebrae consisting of thoracic and lumbar vertebrae, and using the Labelme tool to annotate the upper left, upper right, lower left and lower right positions of each vertebra. Each X-ray image had a total of 68 position labels and was not cropped to form a sample set;

[0055] Step 2: Divide the sample set into training set, validation set and test set;

[0056] Step 3: construct a network model, wherein the constructed network model includes an encoder with d encoding layers, a decoder with d decoding layers, d-1 vertebral position capture modules and a processing unit, where d is a positive integer greater than 1; Figure 1 As shown, d=4 in this embodiment, that is, the encoder includes 4 encoding layers, the decoder includes 4 decoding layers, and the number of vertebral position capture modules is 3; the encoding layer includes a first feature extraction module and an upsampling module, and the decoding layer includes a second feature extraction module and a downsampling module; the encoding layer and the decoding layer are both prior art and will not be described in detail here;

[0057] The d encoding layers are connected in sequence, the d decoding layers are also connected in sequence, and the output end of the d encoding layer is connected to the input end of the 1st decoding layer; the 1st vertebral position capture module is arranged between the 2nd encoding layer and the 3rd encoding layer, the 2nd vertebral position capture module is arranged between the 3rd encoding layer and the 4th encoding layer, ..., the d-2nd vertebral position capture module is arranged between the d-1th encoding layer and the dth encoding layer, the d-1th vertebral position capture module is arranged between the dth encoding layer and the 1st decoding layer, the 1st vertebral position capture module is jump-connected to the dth decoding layer, the 2nd vertebral position capture module is jump-connected to the d-1th decoding layer, ..., the d-1th vertebral position capture module is jump-connected to the 2nd decoding layer;

[0058] The vertebral position capture module includes a short-range feature connection module and a long-range feature connection module, such as Figure 2As shown, the specific processing process of the vertebral position capture module is: the original feature map input to the vertebral position capture module is divided into blocks twice, one block division is to evenly divide the original feature map into q first feature blocks, and the other block division is to evenly divide the original feature map into q-1 second feature blocks; the image size of the second feature block is the same as that of the first feature block, and q is a positive integer greater than 1; in this embodiment, before the second feature block is divided, the upper and lower parts of the original feature map are first vacated with a distance of h / 2q, and then the image in the original feature map except the vacated position is evenly divided, that is, q-1 second feature blocks are obtained, and h is the height of the original feature map;

[0059] Then, the short-range feature connection module is used to perform self-attention processing on each first feature block and each second feature block respectively, to obtain a first feature map corresponding to each first feature block and a second feature map corresponding to each second feature block, and all the first feature maps are spliced ​​in the order of the first feature blocks to obtain a first output feature map; in addition, all the second feature maps are spliced ​​in the order of the second feature blocks to obtain a second output feature map, and the first output feature map and the second output feature map have the same image size as the original feature map. Finally, the first output feature map and the second output feature map are fused to obtain a feature map output by the short-range feature connection module;

[0060] In addition, the long-range feature connection module is used to perform global average pooling processing on each first feature block and each second feature block, respectively, to obtain a third feature map corresponding to each first feature block and a fourth feature map corresponding to each second feature block; then, self-attention processing is performed on all third feature maps to obtain a fifth feature map, and the fifth feature map is added to the original feature map to obtain a third output feature map; similarly, self-attention processing is performed on all fourth feature maps to obtain a sixth feature map, and the obtained sixth feature map is added to the original feature map block to obtain a fourth output feature map; finally, the third output feature map and the fourth output feature map are fused to obtain a feature map output by the long-range feature connection module;

[0061] The feature map output by the short-range feature connection module, the feature map output by the long-range feature connection module and the original feature map are added together to obtain the output of the vertebral position capture module;

[0062] In this embodiment, q=8, such as Figure 2 As shown, the eight first feature blocks are S 1 , S 2 , S 3 , S 4 , S 5 , S 6 , S 7 and S 8; The 7 second feature blocks are M 1 、M 2 、M 3 、M 4 、M 5 、M 6 and M 7 ; Let k = 1, 2, 3...8 in sequence, and follow Calculated For S 1 The corresponding first feature map, For S 2 The corresponding first feature map, For S 8 The first feature map corresponding to it; Selfatt(.) is the self-attention processing, which is a prior art and will not be elaborated here; similarly, let k = 1, 2, 3...7 in turn, and follow Calculated M 1 The corresponding second feature map, M 2 The corresponding second feature map, M 7 The corresponding second feature map;

[0063] The first output feature map Cat(.) means concatenating all feature blocks head to tail, and the second output feature map

[0064] The jth feature block in the feature map output by the short-range feature connection module The calculation formula is:

[0065]

[0066] in, is the jth first feature map in the first output feature map, is the jth first feature map in the second output feature map, and Mean(.) is the mean processing;

[0067] like Figure 2 As shown, the fifth characteristic graph S ia =Selfatt(S′ 1 ,S′ 2 ,,…,S′ 8 ), where S′ 1 , S′ 2 ,,…,S′ 8 For S 1 , S 2 …S 8A third feature map obtained by performing global average pooling processing; are 8 feature images in the fifth feature map respectively; let k = 1, 2, 3...8 in sequence, and follow

[0068] Calculate each feature image in the third output feature map

[0069] Similarly, the sixth characteristic graph M ia =Selfatt(M′ 1 ,M′ 2 ,…,M′ 7 ), M′ 1 ,M′ 2 ,…,M′ 7 For M 1 、M 2 、M 3 、M 4 、M 5 、M 6 and M 7 The fourth feature map obtained by global average pooling is are the 7 feature images in the sixth feature map, respectively, and k=1, 2, 3...7 in sequence, and Calculate each feature image in the fourth output feature map

[0070] The jth feature block in the feature map output by the long-range feature connection module The calculation formula is:

[0071]

[0072] in, is the jth feature map in the third output feature map, is the j-th feature map in the fourth output feature map;

[0073] The processing unit includes a center positioning module, a center contrast learning module, a center offset module and an angle positioning module, and the center positioning module, the center contrast learning module, the center offset module and the angle positioning module are connected to the output end of the dth decoding layer;

[0074] The specific calculation process of the center positioning module in this embodiment is: by calculating the average value of the coordinates of the four endpoints of the upper left, upper right, lower left and lower right of each vertebra in the original image, and taking it as the coordinate of the center point of each vertebra, then converting the center point of each vertebra into a Gaussian heat map through Gaussian transformation, and finally using the Gaussian heat map to predict the coordinate of the center point of each vertebra; the above original image is a training sample in the training set;

[0075] In this embodiment, the calculation formula of the Gaussian heat map generated according to the center of the Nth vertebral body is:

[0076]

[0077] Where (X, Y) is the pixel coordinate on the Gaussian heat map generated according to the center of the Nth vertebra, (X N ,Y N ) is the coordinate of the center point of the Nth vertebra, a is the radius of the disk formed on the Gaussian heat map;

[0078] The specific process of using the Gaussian heat map to predict the coordinates of the center point of each vertebra is as follows: use the maximum pooling of size b*b to perform non-maximum suppression on the Gaussian heat map, and select the point with the largest pixel value as the center prediction point of the current vertebra; b is a positive integer, and in this embodiment b=3;

[0079] The specific calculation process of the center contrast learning module is:

[0080] The output end of the d-th decoding layer is passed through the feature extraction layer to obtain an output feature map, which has the same size as the Gaussian heat map. The pixel positions in the Gaussian heat map with pixel values ​​greater than T are screened out and recorded as the first position set. Each feature in the output feature map that is in the first position set is used as a positive sample, and the features in the output feature map other than the positive sample are used as negative samples.

[0081] The center offset module predicts the coordinates of the center point of each vertebra in the original image based on the center point coordinates of each vertebra predicted by the center positioning module;

[0082] In this embodiment, the center offset module predicts the coordinates of the center point of each vertebra in the original image. The specific calculation process is as follows:

[0083] Suppose the coordinates of a pixel in the original image are (x, y). After n times of downsampling, the coordinates mapped on the output feature map are Assume that the center point offset of each vertebra is (△x,△y), and the calculation formulas of △x and △y are: The coordinates (x′, y′) on the output feature map are added with the calculated offset of the center point of each vertebra to obtain x″=x′+△x, y″=y′+△y. The predicted position of the vertebral center point on the original image can be obtained by scaling (x″, y″) by n times.

[0084] The angle positioning module predicts the coordinates of each vertebral end point in the original image based on the center point coordinates of each vertebra predicted by the center offset module;

[0085] The specific calculation process of the angular positioning module in this embodiment predicting the coordinates of each vertebral endpoint in the original image is as follows:

[0086] Calculate the predicted offsets from the center point of each vertebra to the upper left, upper right, lower left, and lower right endpoints of the same vertebra. k ,b k ) represents the predicted offset from the center point of a vertebra to the k-th end point of the vertebra. Add the offsets of the four positions to the coordinates of the center point of the vertebra to obtain x′ k =a k +x′,y′ k =b k +y′,(x′ k ,y′ k ) is the coordinate of the k-th endpoint of the vertebra on the feature map. By magnifying it by n times, the predicted positions of the endpoints of each vertebra on the original image can be obtained.

[0087] Step 4: Input all sample images in the training set into the network model constructed in step 3 in batches for training, and use all sample images in the validation set to verify the performance of the trained network model; after multiple training and validation, select the optimal network model;

[0088] Each time the network model is trained, the total loss function L is calculated all , and according to the total loss function L all Reversely update the network parameters in the network model to obtain a network model after the training is completed;

[0089] L all The calculation formula is:

[0090] L all =L hm +L hc +L cenoff +L coroff

[0091] Among them, L hm is the first loss function calculated based on the output of the center positioning module, L hCis the second loss function calculated based on the output of the center contrast learning module, L cenoff is the third loss function calculated based on the output of the center offset module, L coroff为 A fourth loss function calculated according to the output result of the angular positioning module;

[0092] In this embodiment, the first loss function L hm The calculation formula is:

[0093]

[0094] Among them, N is the total number of pixels on the Gaussian heat map, i is the pixel position index on the Gaussian heat map, and y i is the pixel value at the i-th position on the original image, p i is the pixel value of the i-th position on the Gaussian heat map, and α and β are both hyperparameters;

[0095] The second loss function L hc The calculation formula is:

[0096]

[0097] Among them, P i is the position set of positive samples in the output feature map, A is the total number of positive samples in the output feature map, i is the position of all samples in the output feature map, E i is the output feature map, i + is the position of the positive sample in the output feature map, is the feature map corresponding to the positive sample in the output feature map, τ is a hyperparameter; N i is the location set of negative samples in the output feature map, i - is the position of negative samples in the output feature map, is the feature map corresponding to the negative sample in the output feature map, sim(.) is the cosine similarity;

[0098] The third loss function L cenoff The calculation formula is:

[0099]

[0100] Where M is the total number of vertebral center points, x′ k and y′ k To predict the horizontal and vertical coordinates of the center point of the kth vertebra in the original image, x k and k is the theoretical horizontal and vertical coordinates of the center point of the kth vertebra in the original image, n is the ratio of the original image size to the feature map restored by the decoder, is the floor rounding function;

[0101] The fourth loss function L coroff The calculation formula is:

[0102]

[0103] Where P is the set of four endpoints of each vertebra, x i ′ and y i ′ is the predicted horizontal and vertical coordinates of the center point of the i-th vertebra in the original image, x i and i is the theoretical horizontal and vertical coordinates of the center point of the i-th vertebra in the original image, x i,p and i,p To predict the horizontal and vertical coordinates of the endpoint P of the i-th vertebra in the original image;

[0104] Step 5: Input the image to be tested in the test set into the optimal network model to obtain the key point positioning result of the image to be tested; the key point positioning result of the image to be tested includes the coordinates of the center point of each vertebra in the image to be tested predicted by the center offset module and the coordinates of the endpoints of each vertebra in the image to be tested predicted by the angle positioning module.

[0105] This embodiment also involves a Cobb angle calculation method, which uses the above-mentioned thoracic and lumbar key point automatic identification method based on long-range and short-range features. The Cobb angle calculation is a prior art and will not be elaborated here.

[0106] The present embodiment also relates to a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for automatically identifying thoracic and lumbar key points and calculating Cobb angles based on long- and short-range features as described above is executed.

[0107] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An automatic identification method of thoracic and lumbar key points based on long- and short-range features, characterized by The steps include: Step 1, obtain a certain number of anterior-posterior X-ray images of the spine, and mark the upper left, upper right, lower left and lower right positions of each thoracic and lumbar vertebra in each X-ray image to form a sample set; Step 2: Divide the sample set into training set, validation set and test set; Step 3, constructing a network model, wherein the constructed network model includes an encoder having d encoding layers, a decoder having d decoding layers, d-1 vertebral position capture modules and a processing unit, where d is a positive integer greater than 1; The d encoding layers are connected in sequence, the d decoding layers are also connected in sequence, and the output end of the d encoding layer is connected to the input end of the 1st decoding layer; the 1st vertebral position capture module is arranged between the 2nd encoding layer and the 3rd encoding layer, the 2nd vertebral position capture module is arranged between the 3rd encoding layer and the 4th encoding layer, ..., the d-2nd vertebral position capture module is arranged between the d-1th encoding layer and the dth encoding layer, the d-1th vertebral position capture module is arranged between the dth encoding layer and the 1st decoding layer, the 1st vertebral position capture module is jump-connected to the dth decoding layer, the 2nd vertebral position capture module is jump-connected to the d-1th decoding layer, ..., the d-1th vertebral position capture module is jump-connected to the 2nd decoding layer; The vertebral position capture module includes a short-range feature connection module and a long-range feature connection module. The specific processing process of the vertebral position capture module is: the original feature map input to the vertebral position capture module is divided into blocks twice, one block division is to evenly divide the original feature map into q first feature blocks, and the other block division is to evenly divide the original feature map into q-1 second feature blocks; the image size of the second feature block is the same as that of the first feature block, and q is a positive integer greater than 1; Then, the short-range feature connection module is used to perform self-attention processing on each first feature block and each second feature block respectively, to obtain a first feature map corresponding to each first feature block and a second feature map corresponding to each second feature block, and all the first feature maps are spliced ​​in the order of the first feature blocks to obtain a first output feature map; in addition, all the second feature maps are spliced ​​in the order of the second feature blocks to obtain a second output feature map, and the first output feature map and the second output feature map have the same image size as the original feature map. Finally, the first output feature map and the second output feature map are fused to obtain a feature map output by the short-range feature connection module; In addition, the long-range feature connection module is used to perform global average pooling processing on each first feature block and each second feature block, respectively, to obtain a third feature map corresponding to each first feature block and a fourth feature map corresponding to each second feature block; then, self-attention processing is performed on all third feature maps to obtain a fifth feature map, and the fifth feature map is added to the original feature map to obtain a third output feature map; similarly, self-attention processing is performed on all fourth feature maps to obtain a sixth feature map, and the obtained sixth feature map is added to the original feature map block to obtain a fourth output feature map; finally, the third output feature map and the fourth output feature map are fused to obtain a feature map output by the long-range feature connection module; The feature map output by the short-range feature connection module, the feature map output by the long-range feature connection module and the original feature map are added together to obtain the output of the vertebral position capture module; The processing unit includes a center positioning module, a center contrast learning module, a center offset module and an angle positioning module, and the center positioning module, the center contrast learning module, the center offset module and the angle positioning module are all connected to the output end of the dth decoding layer; Step 4: Input all sample images in the training set into the network model constructed in step 3 in batches for training, and use all sample images in the validation set to verify the performance of the trained network model; after multiple training and validation, select the optimal network model; Step 5: Input the image to be tested in the test set into the optimal network model to obtain the key point positioning result of the image to be tested; the key point positioning result of the image to be tested includes the coordinates of the center point of each vertebra in the image to be tested predicted by the center offset module and the coordinates of the endpoints of each vertebra in the image to be tested predicted by the angle positioning module.

2. The method for automatic identification of thoracic and lumbar key points according to claim 1, characterized in that: Before the second feature map is divided, a distance of h / 2q is first left above and below the original feature map, and then the image except the left positions in the original feature map is evenly divided to obtain q-1 second feature map blocks, where h is the height of the original feature map.

3. The method for automatically identifying thoracic and lumbar key points according to claim 2, characterized in that: The jth feature block in the feature map output by the short-range feature connection module The calculation formula is: in, is the jth first feature map in the first output feature map, is the jth first feature map in the second output feature map, and Mean(.) is the mean processing.

4. The method for automatically identifying thoracic and lumbar key points according to claim 2, characterized in that: The j-th feature block F in the feature map output by the long-range feature connection module j inter The calculation formula is: in, is the jth feature map in the third output feature map, is the j-th feature map in the fourth output feature map.

5. The method for automatically identifying thoracic and lumbar key points according to any one of claims 1 to 4, characterized in that: Each time the network model is trained in step 4, the total loss function L is calculated. all , and according to the total loss function L all Reversely update the network parameters in the network model to obtain a network model after the training is completed; L all The calculation formula is: L all =L hm +L hc +L cenoff +L coroff Among them, L hm is the first loss function calculated based on the output of the center positioning module, L hc is the second loss function calculated based on the output of the center contrast learning module, L cenoff is the third loss function calculated based on the output of the center offset module, L coroff is the fourth loss function calculated according to the output result of the angular positioning module.

6. The method for automatically identifying thoracic and lumbar key points according to claim 5, characterized in that: The specific calculation process of the center positioning module is: by calculating the average value of the coordinates of the four endpoints of the upper left, upper right, lower left and lower right of each vertebra in the original image, and taking it as the coordinate of the center point of each vertebra, then converting the center point of each vertebra into a Gaussian heat map through Gaussian transformation, and finally using the Gaussian heat map to predict the coordinate of the center point of each vertebra; the above original image is a training sample in the training set; The first loss function L hm The calculation formula is: Among them, N is the total number of pixels on the Gaussian heat map, i is the pixel position index on the Gaussian heat map, and y i is the pixel value at the i-th position on the original image, p i is the pixel value at the i-th position on the Gaussian heat map, and α and β are both hyperparameters.

7. The method for automatically identifying thoracic and lumbar key points according to claim 6, characterized in that: The specific calculation process of the center contrast learning module is: The output end of the d-th decoding layer is passed through the feature extraction layer to obtain an output feature map, which has the same size as the Gaussian heat map. The pixel positions in the Gaussian heat map with pixel values ​​greater than T are screened out and recorded as the first position set. Each feature in the output feature map that is in the first position set is used as a positive sample, and the features in the output feature map other than the positive sample are used as negative samples. The second loss function L hc The calculation formula is: Among them, P i is the position set of positive samples in the output feature map, A is the total number of positive samples in the output feature map, i is the position of all samples in the output feature map, E i is the output feature map, i + is the position of the positive sample in the output feature map, is the feature map corresponding to the positive sample in the output feature map, τ is a hyperparameter; N i is the location set of negative samples in the output feature map, i - is the position of negative samples in the output feature map, is the feature map corresponding to the negative sample in the output feature map, and sim(.) is the cosine similarity.

8. The method for automatically identifying thoracic and lumbar key points according to claim 6, characterized in that: The center offset module predicts the coordinates of the center point of each vertebra in the original image based on the coordinates of the center point of each vertebra predicted by the center positioning module; The third loss function L cenoff The calculation formula is: Where M is the total number of vertebral center points, x′ k and y′ k To predict the horizontal and vertical coordinates of the center point of the kth vertebra in the original image, x k and k is the theoretical horizontal and vertical coordinates of the center point of the kth vertebra in the original image, n is the ratio of the original image size to the feature map restored by the decoder, is the floor function.

9. The method for automatically identifying thoracic and lumbar key points according to claim 8, characterized in that: The angle positioning module predicts the coordinates of each vertebral end point in the original image based on the center point coordinates of each vertebra predicted by the center offset module; The fourth loss function L coroff The calculation formula is: Where P is the set of four endpoints of each cone, x′ i and y′ i To predict the horizontal and vertical coordinates of the center point of the i-th vertebra in the original image, x i and i is the theoretical horizontal and vertical coordinates of the center point of the i-th vertebra in the original image, x i，p and i，p To predict the horizontal and vertical coordinates of the endpoint p of the i-th vertebra in the original image.

10. A Cobb angle calculation method, characterized in that: The method for automatically identifying thoracic and lumbar key points based on long- and short-range features as described in any one of claims 1 to 9 above is applied.

Images