Spine three-column fracture classification and positioning system based on residual attention constraint mechanism

Abstract

The application relates to the fields of computer image processing and medical artificial intelligence, and discloses a spine three-column fracture classification and positioning system based on a residual attention constraint mechanism, which automatically classifies and positions the fracture of each vertebral anterior, middle and posterior column in a CT image by combining a deep learning method. Under the prompt of the automatic result, the analysis range can be reduced, and the overall work efficiency is improved. Furthermore, a deep learning model based on the residual attention constraint mechanism is proposed, which simultaneously obtains the category of each vertebra and the positioning map of the vertebra fracture under the multi-scale supervision constraint of the image scale and the voxel scale. In the whole process, the relationship between the positioning and classification tasks is fully utilized, so that the introduction of the positioning task can guide the model to pay more attention to the fracture area and thus obtain better classification performance, and the introduction of the classification task can influence the positioning result in a data-driven manner, thereby reducing the interference of the noise in the coarse label of the positioning map on the positioning performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of computer image processing and medical artificial intelligence, and in particular, to a deep learning algorithm for classification and positioning of spinal three-column fractures in CT images based on a residual attention constraint mechanism. BACKGROUND

[0002] Spinal fractures are caused by inappropriate axial loading in the context of trauma, osteoporosis, infection, metastasis or other bone diseases, which can lead to a decrease in the quality of life of patients and an increase in the risk of death.

[0003] Computed tomography (CT) is widely used in the diagnosis of spinal fractures and the selection of subsequent treatment plans in clinical practice. Patients with unstable spine after fracture usually need to undergo surgical treatment. In order to determine the stability of the spine after fracture, the D-F three-column theory (DENIS F. The three column spine and its significance in the classification of acute thoracolumbar spinal injuries [J]. Spine (Phila Pa 1976), 1983, 8(8):817-31) proposes that the stability of the spine is maintained by three columns: the anterior column, composed of the anterior two-thirds of the vertebral body and intervertebral disc, and the anterior longitudinal ligament; the middle column, composed of the posterior one-third of the vertebral body and intervertebral disc, and the posterior longitudinal ligament; and the posterior column, composed of the facet joints and the posterior longitudinal ligament complex. Fractures of the middle column or two columns and above often lead to mechanical instability of the spine. Due to the complex three-dimensional structure of the spine, doctors usually need to read CT images layer by layer in the sagittal and transverse planes, which is a tedious process and is prone to omissions. Using deep learning technology, the fractures of each vertebral anterior, middle and posterior column can be automatically classified and positioned before the doctor analyzes. Under the prompt of this automation result, the doctor can narrow the analysis range and directly locate the fracture area, and then analyze the stability of the patient's spine. In the whole process, the doctor's workload can be greatly reduced, and the work efficiency can be improved.

[0004] However, the existing deficiencies include: first, the process of analyzing the three-column fractures of the spine in the CT image by the doctor is tedious and requires a lot of time and effort; second, the existing deep learning model for classification of spinal fractures cannot provide fracture positioning at the same time; third, the classification and positioning method based on CAM does not fully utilize the relationship between the classification task and the positioning task, and the training process is easily disturbed by noise in the positioning label. SUMMARY

[0005] The purpose of the present application is to provide a three-column spinal fracture classification and positioning system based on a residual attention constraint mechanism, wherein doctors can narrow the analysis range and improve the overall work efficiency; and the category of each vertebra and the positioning map of the vertebral fracture can be obtained at the same time. Further, the noise in the rough label of the positioning map can be minimized to interfere with the positioning performance.

[0006] The present application discloses a three-column spinal fracture classification and positioning system based on a residual attention constraint mechanism, characterized in that it comprises:

[0007] A vertebra data set construction unit is configured to obtain a vertebra data set for model training and testing according to input data, wherein the input data is obtained after preprocessing the vertebra CT image data and its corresponding category label and positioning map label.

[0008] A classification and positioning model construction unit based on a residual attention constraint mechanism is configured to construct a classification and positioning model based on a residual attention constraint mechanism, which is a deep learning model for classifying and positioning the three-column fracture of each vertebra according to the vertebra data set; wherein the classification and positioning model construction unit based on a residual attention constraint mechanism is configured to generate a fracture classification and positioning map, guide the model to focus on the fracture area through image-level and voxel-level supervision constraints, and obtain a classification result and a positioning map.

[0009] A model training unit is configured to train the classification and positioning model based on a residual attention constraint mechanism; and

[0010] A detection unit is configured to obtain input data to be detected, and obtain a classification result and a positioning map corresponding to the input data to be detected based on the classification and positioning model based on a residual attention constraint mechanism.

[0011] In a preferred embodiment, the system further comprises:

[0012] A CT data set construction unit is configured to obtain vertebra CT image data and its corresponding category label and positioning map label for training and testing;

[0013] A data preprocessing unit is configured to obtain input data for the classification and positioning model based on a residual attention constraint mechanism according to the vertebra CT image data and its corresponding category label and positioning map label.

[0014] In a preferred embodiment, the system further comprises:

[0015] A model testing unit is configured to test the trained classification and positioning model based on a residual attention constraint mechanism.

[0016] In a preferred embodiment, in the CT data set construction unit, CT image data of a patient with spinal fracture is collected to construct a data set X, and the three-column fracture condition of each vertebra in the CT sequence is labeled to obtain a category label GT c and a positioning label GT s ; wherein the labeling process of the positioning label GT s first uses a two-dimensional or three-dimensional segmentation labeling brush to paint the regions of the anterior, middle and posterior column fractures of each vertebra, so that the background and the regions of the anterior, middle and posterior column fractures are covered by masks with values of 0, 1, 2 and 3, respectively, and then the labeled masks are saved as the positioning label GT s .

[0017] In a preferred embodiment, in the classification and positioning model construction unit based on the residual attention constraint mechanism, a classification and positioning network based on the residual attention constraint mechanism is constructed, and the network based on the residual attention constraint mechanism includes three branches: a backbone feature extraction network branch, a positioning map generation network branch, and a fusion classification network branch.

[0018] In a preferred embodiment, the feature extraction step of the backbone feature extraction network branch is: inputting data into the backbone feature extraction network branch, sequentially passing through the convolution layer C1, the maximum pooling layer M1, n residual attention constraint sub-networks and a global average pooling layer, and finally outputting a backbone feature vector F1 and m a three-dimensional spatial attention maps S, wherein m a is the number of residual attention modules in the entire model, and the spatial attention maps are generated by each residual attention module.

[0019] In a preferred embodiment, the positioning map generation step of the positioning map generation network branch is: first selecting m s three-dimensional spatial attention maps S extracted in 4-1 to construct a fracture positioning map, up-sampling them to the same size as the original input image, then concatenating the up-sampled maps along the feature channel, using a convolution layer with a 3×3×3 convolution kernel to fuse the multiple spatial attention maps into three spatial attention maps, and finally using a Sigmoid activation function to limit the values of the spatial attention maps to between 0 and 1 to obtain anterior, middle and posterior column fracture positioning maps SL, wherein m s is the number of selected spatial attention maps.

[0020] In a preferred example, the fusion classification network branch includes a positioning map feature extraction and a fusion classification prediction; the positioning map fusion part uses a neural network to extract the features of the fracture positioning map SL, and the fusion classification prediction part uses a full connection layer to fuse and classify the features of the fracture positioning map and the backbone feature vector F1.

[0021] In a preferred example, the step of obtaining the final classification result of the fusion classification network branch is: inputting the front, middle and rear column fracture positioning maps SL into the positioning map feature extraction network to extract a positioning map feature F2, then splicing the backbone feature vector F1 and F2 along the feature channel to obtain a fused feature vector F, and inputting F into the fusion classification prediction network to obtain the final classification result P.

[0022] In a preferred example, the positioning map feature extraction part first takes the average of the three positioning maps to obtain a fracture positioning map, then uses a maximum pooling layer and an average pooling layer to downsample the fracture positioning map to two 4x4x4 size feature maps, and then expands and splices the two feature maps to obtain a positioning map feature F2 with a dimension of 128.

[0023] In a preferred example, in the model training unit,

[0024]

[0025] wherein, L s a positioning adaptive mean square loss, is the label of the Kth fracture positioning map, SL K is the Kth fracture positioning map predicted by the model; K takes values of 0, 1 and 2, respectively corresponding to the front, middle and rear column fractures; i, j, k represent the three-dimensional coordinate indexes of the voxels; ε is a very small value, taking a value of 0.000001, used to ensure that the denominator is not zero.

[0026] In the embodiments of the present application, the method combining deep learning automatically classifies and locates the fractures of the front, middle and rear columns of each vertebra in the CT image. Under the prompt of this automatic result, the doctor can narrow the analysis range and improve the overall work efficiency; and the present application proposes a deep learning model based on a residual attention constraint mechanism, which simultaneously obtains the category of each vertebra and the positioning map of the vertebra fracture under the multi-scale supervision constraint of the image scale and the voxel scale. In the whole process, the relationship between the positioning and classification tasks is fully utilized, so that the introduction of the positioning task can guide the model to pay more attention to the fracture area and thus obtain better classification performance, and the introduction of the classification task can influence the positioning result in a data-driven manner, and try to reduce the interference of the noise in the coarse label of the positioning map on the positioning performance.

[0027] A large number of technical features are described in the specification of the present application, distributed in various technical solutions. If all possible combinations of technical features (i.e. technical solutions) of the present application are listed, the specification will be too long. In order to avoid this problem, each technical feature disclosed in the above summary of the application, each technical feature disclosed in the following embodiments and examples, and each technical feature disclosed in the drawings can be freely combined with each other to form various new technical solutions (which are considered to have been described in the specification), unless such combination of technical features is technically infeasible. For example, features A+B+C are disclosed in one example, features A+B+D+E are disclosed in another example, features C and D are equivalent technical means that play the same role, and can only be used at the same time, and feature E can be combined with feature C technically, then the solution of A+B+C+D should not be considered to have been described because it is technically infeasible, and the solution of A+B+C+E should be considered to have been described. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the structure of the spine three-column fracture classification and positioning system based on the residual attention constraint mechanism according to the first embodiment of the present application.

[0029] Figure 2 are display diagrams of two data coronal planes according to the first embodiment of the present application, each column corresponds to data of the same vertebra, wherein Figure 2 (a), Figure 2 (b) is a CT image input into the model, Figure 2 (c), Figure 2 (d) is a Gaussian positioning map, Figure 2 (e), Figure 2 (f) is a fracture positioning map label.

[0030] Figure 3 (a) is a structural schematic diagram of the residual module R according to the first embodiment of the present application.

[0031] Figure 3 (b) is a structural schematic diagram of the residual attention module RA according to the first embodiment of the present application.

[0032] Figure 4 is a schematic diagram of the classification and positioning model based on the residual attention constraint mechanism constructed according to the first embodiment of the present application.

[0033] Figure 5 is a schematic diagram of the fracture positioning map predicted by the six test set samples according to the first embodiment of the present application, wherein each column represents the same sample, and the labels represent whether the anterior column, middle column and posterior column are fractured in turn, 1 represents fracture, and 0 represents no fracture. DETAILED DESCRIPTION

[0034] In the following description, numerous specific details are set forth to provide a thorough understanding of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced without such specific details and that numerous variations and modifications from the descriptions are possible.

[0035] It should be noted that the technical solution of the present application is not a method for diagnosing and treating diseases. The present application is only a system for obtaining the category and position of three-column fractures of the spine, which automatically filters out the fractured vertebrae in the process of the doctor analyzing the stability of the patient's spine after the fracture, and provides a visual positioning diagram of the anterior, middle and posterior column fractures. The results obtained by the present application only serve as a prompt to automatically display the region of interest, and the doctor still needs to further analyze which treatment method to take.

[0036] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0037] Referring to Figure 1 , Figure 3 and Figure 4 , the first embodiment of the present application relates to a three-column fracture classification and positioning system of the spine based on a residual attention constraint mechanism, which has the structure as shown in Figure 1 The three-column fracture classification and positioning system of the spine based on a residual attention constraint mechanism includes the following units: a CT data set construction unit, a data preprocessing unit, a vertebra data set construction unit, a classification and positioning model construction unit based on a residual attention constraint mechanism, a model training step and a model testing unit. Specifically as follows:

[0038] CT data set construction unit

[0039] The CT data set construction unit is used to obtain the vertebra CT image data for training and testing and its corresponding category label and positioning label.

[0040] Specifically, the CT data set construction unit collects the vertebra CT image data of the spine fracture patient to construct a vertebra data set X, wherein the three-column fracture condition of each vertebra in the CT sequence is labeled to obtain a category label GT c and a positioning label GT s .

[0041] The category label GT c , characterized in that the labels 0, 1, 2 and 3 respectively represent the current vertebra health, anterior column fracture, middle column fracture and posterior column fracture.

[0042] The positioning label GTs The annotation process is characterized by first using segmentation annotation software and a 2D / 3D segmentation annotation brush of appropriate size to paint over the fracture areas of the anterior, middle, and posterior columns of each vertebra, so that the background and the fracture areas of the anterior, middle, and posterior columns are covered by masks with values ​​of 0, 1, 2, and 3, respectively. Then, the annotated masks are saved as the location map label GT. s The entire annotation process is simple and easy to operate, requiring minimal labor costs.

[0043] Data preprocessing unit

[0044] The data preprocessing unit is used to obtain input data for the classification and localization model based on the residual attention constraint mechanism, according to the vertebral CT image data and its corresponding category label and localization map label.

[0045] Specifically, the data preprocessing unit performs the following processes:

[0046] 2-1: Resample the CT images in the vertebral dataset X to obtain CT data D with uniform resolution and localization map labels GT. s .

[0047] 2-2: Locate the vertebrae in data D and obtain the geometric center coordinates C of each vertebra.

[0048] 2-3: Using coordinate C as the center, compare data D and GT s The data is cropped to obtain vertebral CT data V, which contains the entire target vertebra and has a size of s1×s2×s3, as well as the corresponding localization map label VGT. s .

[0049] 2-4: VGT s VGT converts tags to one-hot tags s K The feature is that K takes values ​​of 0, 1, and 2, which correspond to the location diagrams of fractures in the anterior, middle, and posterior columns, respectively.

[0050] 2-5: Adjust the window width and window level of the CT values ​​in data V to increase the contrast between bone tissue and other tissues. Then normalize the adjusted data V so that the mean of its gray values ​​is 0 and the variance is 1.

[0051] 2-6: Construct a Gaussian localization map G, characterized in that the Gaussian localization map has the same size as the data V and is used to indicate the current location of the center of the vertebra. The value of each voxel in the data G is calculated according to Formula 1.

[0052]

[0053] where i, j, k represent the coordinate index of the voxel in three directions, i c , j c , k c represent the center coordinates of the vertebrae, and σ represents the Gaussian smoothing index.

[0054] 2-7: Splicing data G and data V along the feature channel to obtain input data I of the classification and positioning model.

[0055] Vertebrae data set construction unit

[0056] The vertebrae data set construction unit is used to obtain a vertebrae data set for model training and testing according to the input data.

[0057] In other words, the vertebrae data set is used for subsequent deep learning model training and testing.

[0058] Preferably, the data of the vertebrae data set is the input data I obtained in 2-6 above, and the label is the positioning icon label VGT s and GT c obtained in 2-3 above.

[0059] Preferably, the vertebrae data set is divided, and the division process is performed on a patient-by-patient basis.

[0060] Preferably, the vertebrae data set is divided, and the ratio of the training set to the test set is about 3:1.

[0061] Classification and positioning model construction unit based on residual attention constraint mechanism

[0062] The classification and positioning model construction unit based on the residual attention constraint mechanism is used to construct a classification and positioning model based on the residual attention constraint mechanism (hereinafter also referred to as “classification and positioning model”), which is a deep learning model for classifying and positioning the three-column fracture of each vertebra according to the vertebrae data set.

[0063] Specifically, a classification and positioning network based on a residual attention constraint mechanism is constructed. The network based on the residual attention constraint mechanism includes three branches: a backbone feature extraction network branch, a positioning map generation network branch, and a fusion classification network branch. A batch normalization layer and a ReLu activation function are connected to each convolutional layer in the network. The backbone feature extraction network extracts basic features of the input data. The positioning map generation network is used to generate a fracture positioning map of the whole model, and the attention mechanism of the backbone feature extraction network is constrained by the positioning map. The fusion classification network fuses the basic features and the features of the positioning map, and obtains the final classification result after further processing. The three branches interact with each other during model training, and obtain high classification and positioning performance under the multi-scale supervision constraint of the image scale class label and the voxel scale positioning map label.

[0064] Specifically, the classification and positioning model construction unit based on the residual attention constraint mechanism performs the following processing:

[0065] 4-1: The backbone feature extraction network branch includes a convolutional layer C1, a maximum pooling layer MP1, n residual attention constraint sub-networks, and a global average pooling layer GAP connected in sequence.

[0066] The residual attention constraint sub-network includes a classic residual module R without a bottleneck and m residual attention modules RA without a bottleneck that introduce a spatial attention mechanism in the residual path. The convolution kernel step of the first convolutional layer in the R is greater than 1 or the hollow convolution rate is set to be greater than 1, which increases the receptive field in the whole backbone feature extraction network. The hyperparameters of the m residual attention modules are the same in the same residual attention constraint sub-network.

[0067] The feature extraction step of the backbone feature extraction network branch is as follows: the data is input into the backbone feature extraction network branch, sequentially passes through the C1, the maximum pooling layer MP1, the n residual attention constraint sub-networks, and the global average pooling layer GAP, and finally outputs a backbone feature vector F1 and m a three-dimensional spatial attention maps S, where m a is the number of residual attention modules in the whole model, and the spatial attention map is generated by each residual attention module.

[0068] 4-2: The positioning map generation network branch includes two convolutional layers with a convolution kernel of 3 and a Sigmoid activation layer.

[0069] The positioning map generation step of the positioning map generation network branch is as follows: first, m sThe three-dimensional spatial attention map S extracted in the fourth step is up-sampled to the same size as the original input image, then the up-sampled image is spliced along the feature channel, and then a convolution layer with a 3x3x3 kernel is used to fuse the multiple spatial attention maps into three spatial attention maps, and finally a Sigmoid activation function is used to limit the value of the spatial attention map to 0-1 to obtain the anterior, middle and posterior column fracture localization maps SL, wherein m s The number of selected spatial attention maps. By fusing spatial attention maps of different scales through two convolution layers, the localization of fractures of different scales can be realized, and the missed localization can be reduced.

[0070] 4-3: The fusion classification network branch includes a localization map feature extraction part and a fusion classification prediction part; the localization map feature extraction part uses a neural network to extract the features of the fracture localization map SL, and the fusion classification prediction part uses a fully connected layer to fuse and classify the features F2 of the fracture localization map and the backbone features F1. Extracting the features of the localization map and fusing them with the backbone features can on the one hand make full use of the constraints of the localization map to directly improve the classification performance of the model, and on the other hand can constrain the generation of the localization map through this fusion, and fine-tune the localization map in a data-driven manner.

[0071] The step of obtaining the final classification result by the fusion classification network branch is: inputting SL obtained in the above-mentioned 3-2 into the localization map feature extraction network to extract a localization map feature F2, then splicing F1 and F2 along the feature channel to obtain a fused feature vector F, and inputting F into the fusion classification prediction network to obtain the final classification result P. The P is a 3-dimensional vector, which respectively corresponds to the probabilities of anterior, middle and posterior column fractures. The whole process reuses the localization map features, which can obtain a feature vector with more performance, and thus improve the classification accuracy.

[0072] Model training unit

[0073] The model training unit is used to train the classification and localization model based on the residual attention constraint mechanism.

[0074] Preferably, in the training process, the parameters of the network are iteratively updated by a gradient descent algorithm to minimize the loss L on the training set.

[0075] Wherein, the loss L is composed of a classification cross-entropy loss L c and a localization adaptive mean square loss L s According to the proportion of 1:λ, the value of λ is 0-10. L c is a classic binary cross-entropy loss, L sThe calculation method is shown in formula 2. Through the combination of the two loss functions, the model parameters are updated while applying multi-scale constraints of image scale and voxel scale, and finally a model with good classification and positioning performance is obtained.

[0076]

[0077] wherein L s the positioning adaptive mean square loss, is the K-th fracture positioning icon label, SL K is the K-th fracture positioning icon label predicted by the model; K takes values of 0, 1, and 2, respectively corresponding to the anterior, middle, and posterior column fractures; i, j, and k represent the three-dimensional coordinate indexes of the voxels; and ε is a very small value, taking a value of 0.000001, and is used to ensure that the denominator is not zero.

[0078] The L s loss function, characterized in that, compared with the traditional mean square error, a denominator composed of the fracture area is divided, realizing dynamic adjustment of the influence effect of different samples on the loss function.

[0079] The L s loss function, characterized in that, only the loss of samples with fractures is calculated.

[0080] The model testing unit

[0081] The model testing unit is used to test the trained classification and positioning model based on the residual attention constraint mechanism.

[0082] Specifically, the CT data to be tested is preprocessed in step 2, and then each vertebra is sequentially sent to the trained classification and positioning model of the model training unit, and finally the category and fracture positioning map of each vertebra contained in the current CT data are obtained.

[0083] Preferably, in the CT data set construction unit, the positioning map annotation process can use a spherical brush with a radius of 2 in ITK-Snap for annotation; the annotation using the spherical brush with a radius of 2 can obtain a positioning map label containing less noise while spending relatively less annotation time.

[0084] In this embodiment, the following specific manner can be further implemented:

[0085] Preferably, the resolution in 2-1 above is unified to 1mmx1mmx1mm.

[0086] Preferably, the scheme for positioning the vertebrae in 2-2 above can select a pre-trained deep learning model SCNet for vertebra positioning.

[0087] Preferably, s1xs2xs3 in 2-3 above is 64x128x128.

[0088] Preferably, the convolution kernel size of the convolution layer C1 in 4-1 above is 7x7x7, and the number n of the residual attention constraint sub-networks is 4.

[0089] Preferably, in 4-1 above, the convolution step of the first convolution layer on the R-in residual path in the four residual attention constraint sub-networks is 2, 2, 1, and 1, respectively, and the hollow convolution rate is 1, 1, 2, and 4, respectively; in the above setting, the first two residual modules both down-sample the output feature map to half the size of the input feature map, and the last two residual modules expand the receptive field without changing the size of the feature map through the form of hollow convolution.

[0090] Preferably, in 4-1 above, the number m of residual attention modules in the four residual attention constraint sub-networks is 2, 3, 5, and 2, respectively.

[0091] Preferably, in 4-1 above, the number of feature channels of the four residual attention constraint sub-networks is 32, 64, 128, and 256, respectively.

[0092] Preferably, in 4-1 above, the spatial attention mechanism added is CBAM attention mechanism, which sequentially applies attention to the channel and spatial features in a series form, and the attention map generated by the spatial attention part is S described in 4-1.

[0093] Preferably, in 4-2 above, all spatial attention maps generated by the first n-1 residual attention constraint sub-networks are selected to construct the fracture positioning map.

[0094] Preferably, in 4-2 above, the up-sampling method of the spatial positioning map is trilinear interpolation.

[0095] Preferably, in 4-3 above, the positioning map feature extraction part first takes the average of the three positioning maps to obtain a fracture positioning map, and then uses a maximum pooling layer and an average pooling layer to down-sample the fracture positioning map to two 4x4x4 size feature maps, and then the two feature maps are expanded and spliced to obtain a positioning map feature F2 with a dimension d2 of 128; directly through two pooling layers to obtain features, on the one hand, it can reduce the parameter quantity of the model and speed up the calculation speed, on the other hand, the pooling operation can extract features containing relative position information.

[0096] Preferably, in 4-3 above, the fusion classification part is composed of a fully connected layer with an input of 384 dimensions and an output of 3 dimensions and a Sigmoid activation layer.

[0097] Preferably, in the above model training unit, lambda is set to 0.5, and the value mainly adjusts the influence degree of the positioning task and the classification task in the training process.

[0098] The detection unit

[0099] The input data to be detected is obtained, and based on the classification and positioning model based on the residual attention constraint mechanism, the classification result and the positioning map corresponding to the input data to be detected are obtained.

[0100] The above embodiment has the following technical effects:

[0101] The above embodiment can automatically classify and position the fracture of the anterior, middle and posterior columns of the vertebra.

[0102] The labeling scheme of the fracture positioning map proposed in the above embodiment is simple and practical.

[0103] The classification and positioning model based on the residual attention constraint mechanism proposed in the above embodiment can reduce the interference of background signals and obtain better classification performance than ordinary residual networks without constraints by using the residual attention constraint mechanism, i.e. adding an attention module to the main feature extraction network branch while combining the positioning loss to guide the model to pay more attention to the features of the fracture area during the training process.

[0104] The positioning adaptive mean square loss function of the classification and positioning model based on the residual attention constraint mechanism proposed in the above embodiment can automatically adjust the loss value according to the size of the fracture range in the sample, so that fractures of different ranges can fully participate in the training process, and better positioning results can be obtained compared with simple mean square error.

[0105] In the classification and positioning model based on the residual attention constraint mechanism proposed in the above embodiment, the fusion classification network branch fuses the features of the positioning map and the basic features to fully utilize the positioning information of the fracture and obtain more expressive feature vectors. At the same time, through this branch, the classification loss also constrains the positioning map, and adjusts the positioning map in a data-driven manner.

[0106] In order to better understand the technical solutions of the present application, a specific example is given below, and the details listed in the example are mainly for understanding, and do not limit the protection scope of the present application.

[0107] The CT data set construction unit

[0108] Specifically, the CT data set construction unit collects CT image data of 200 patients with spinal fractures to construct a vertebra data set X, and labels the three-column fracture of each vertebra in the CT sequence to obtain a class label GT cand the positioning label GT s .

[0109] Preferably, for the category label L c , the labels 0, 1, 2 and 3 respectively represent the current vertebral health, anterior column fracture, middle column fracture and posterior column fracture.

[0110] Preferably, in the labeling process of the positioning label L l , first, in the ITK-Snap software, a spherical brush with a radius of 2 is used to paint the regions where the anterior, middle and posterior column fractures of each vertebra are located, so that the regions where the fractures of the background, anterior, middle and posterior columns are located are covered by masks with values of 0, 1, 2 and 3 respectively, and then the labeled mask is saved as the positioning label GT s .

[0111] Data preprocessing unit

[0112] Specifically, the data preprocessing unit performs the following processing:

[0113] 2-1': resample the CT image in the vertebral data set X to obtain CT data D with a resolution of 1mm x 1mm x 1mm and a positioning label GT s .

[0114] 2-2': use a pre-trained deep learning model SCNet for vertebral positioning to locate the vertebrae in the data D, and obtain the center coordinates C of each vertebra.

[0115] 2-3': centering on the coordinates C, the data D and GT s are intercepted to obtain vertebral CT data V with a size of 64 x 128 x 128 that can contain the entire target vertebra, and a positioning label VGT s corresponding to the data V.

[0116] 2-4': convert the VGT s label to a one-hot label VGT s K , characterized in that K takes values of 0, 1, 2 respectively corresponding to the positioning of the anterior, middle and posterior column fractures.

[0117] 2-5': adjust the window width of the CT value in the data V to 1100 and the window level to 500 to increase the contrast between bone tissue and other tissues, and then normalize the adjusted data V so that the mean value of its gray value is 0 and the variance is 1.

[0118] 2-6': constructing a Gaussian localization map G, characterized in that the Gaussian localization map is the same size as the data V, and is used to indicate the position of the center of the current vertebra. The value of each voxel in the Gaussian localization map G is calculated according to formula 1:

[0119]

[0120] where i, j, k represent the coordinate index of the voxel in three directions, i c , j c , k c represent the center coordinates of the vertebra, and σ represents the Gaussian smoothing index, which is 30.

[0121] 2-7': obtaining input data I of the classification and localization model along the feature channel by G and V.

[0122] Vertebra data set construction unit

[0123] The vertebra data set is used for subsequent deep learning model training and testing, wherein the data of the data set is the input data I obtained in 2-6', and the label is the localization map label VGT s and GT c obtained in 2-3'.

[0124] Preferably, the vertebra data set is divided, and the division process is divided according to patients.

[0125] Preferably, the vertebra data set is divided, and the ratio of the training set to the test set is about 3:1.

[0126] Finally, the number of vertebrae in the training set is 742, and the number of vertebrae in the test set is 247.

[0127] Classification and localization model construction unit based on residual attention constraint mechanism

[0128] Specifically, the classification and localization model construction unit based on residual attention constraint mechanism constructs a classification and localization network based on residual attention constraint mechanism, and the network based on residual attention constraint mechanism includes three branches: a backbone feature extraction network branch, a localization map generation network branch, and a fusion classification network branch. The convolution layer used in the network is followed by a batch normalization layer and a ReLu activation function unless otherwise specified.

[0129] Specifically, the classification and localization model construction unit based on residual attention constraint mechanism performs the following processing:

[0130] 4-1': the main feature extraction network branch is composed of a convolutional layer C1 with a convolution kernel size of 7x7x7, a maximum pooling layer MP1, four residual attention constraint sub-networks and a global average pooling layer GAP in sequence.

[0131] The residual attention constraint sub-network is composed of a classic residual module R without a bottleneck and m residual attention modules RA without a bottleneck in the residual path, which are connected in sequence. The hyperparameter settings of the m residual attention modules are the same in the same sub-network.

[0132] In 4-1', the convolution step lengths of the first convolutional layer in the residual path in the four residual attention sub-networks are 2, 2, 1 and 1 in sequence, and the hole convolution rates are 1, 1, 2 and 4 respectively; the number of residual attention modules m in the four residual attention sub-networks is 2, 3, 5 and 2 respectively; the feature channel numbers of the four residual attention sub-networks are 32, 64, 128 and 256 respectively.

[0133] The feature extraction step of the main feature extraction network branch is: inputting data into the main feature extraction network branch, sequentially passing through the C1, the maximum pooling layer MP1, the four residual attention constraint sub-networks and the global average pooling layer, and finally outputting a 256-dimensional main feature vector F1 and 12 three-dimensional spatial attention maps S, which are generated by each residual attention module.

[0134] 4-2': the positioning map generation network branch includes two convolutional layers with a convolution kernel of 3 and a Sigmoid activation layer.

[0135] The positioning map generation step of the positioning map generation network branch is: first, upsample the first 10 three-dimensional spatial attention maps S extracted in 4-1' to the same size as the original input image, then concatenate the upsampled maps along the feature channel, and then use two convolutional layers with a convolution kernel of 3x3x3 to fuse the multiple spatial attention maps into three spatial attention maps, wherein the output feature channel number of the first convolution kernel is 32, and finally use a Sigmoid activation function to limit the value of the spatial attention map to 0-1 to obtain the front, middle and rear column fracture positioning maps SL.

[0136] 4-3': the fusion classification network branch includes a positioning map feature extraction and fusion classification prediction part; the positioning map fusion part uses a neural network to extract the features of the fracture positioning map SL, and the fusion classification prediction part uses a fully connected layer to fuse and classify the features of the fracture positioning map and the main feature F1.

[0137] The positioning map feature extraction part in 4-3' first obtains a fracture positioning map by averaging three positioning maps, and then uses a maximum pooling layer and a mean pooling layer to down-sample the fracture positioning map to two 4x4x4 size feature maps, and then the two feature maps are expanded and spliced to obtain a positioning map feature F2 with a dimension of 128.

[0138] The fusion classification part in 4-3' is composed of a full connection layer with an input of 384 dimensions and an output of 3 dimensions and a Sigmoid activation layer.

[0139] The step of obtaining the final classification result by the fusion classification network branch is: inputting the SL obtained in 3-2' into the positioning map feature extraction network to extract a 128-dimensional SL feature F2, then splicing F1 and F2 along the feature channel to obtain a 384-dimensional feature vector F, and inputting F into the fusion classification prediction network to obtain the final classification result P. The P is a 3-dimensional vector, which respectively corresponds to the probabilities of anterior, middle and posterior column fractures.

[0140] Figure 4 The classification and positioning model based on the residual attention constraint mechanism in the classification and positioning model construction unit based on the residual attention constraint mechanism is shown in the schematic diagram.

[0141] Model training unit

[0142] Preferably, in the training process, the parameters of the network are iteratively updated by the Adam gradient descent algorithm to minimize the loss L on the training set.

[0143] The loss L is obtained by weighting the classification cross-entropy loss L c and the positioning adaptive mean square loss L s in a ratio of 1:0.5; L c is a classic binary cross-entropy loss, L s The calculation method of L

[0144]

[0145] L s The positioning adaptive mean square loss, is the K-th fracture positioning map label, SL K is the K-th fracture positioning map predicted by the model; K takes values of 0, 1 and 2, respectively corresponding to anterior, middle and posterior column fractures; i, j and k represent three-dimensional coordinate indexes of voxels; ε is a very small value, which takes a value of 0.000001, to ensure that the denominator is not zero.

[0146] The L sThe loss function is characterized in that, compared with a traditional mean square error, a denominator composed of a fracture area is divided, so as to realize dynamic adjustment of the influence effect of different samples on the loss function.

[0147] The L s The loss function is characterized in that only the loss of a sample with a fracture is calculated.

[0148] The model testing unit

[0149] Among them, 247 CT data to be tested are preprocessed by the data preprocessing unit, and then each vertebra is sequentially sent to the classification and positioning model trained by the model training unit, and finally the category and fracture positioning map of each vertebra contained in the current vertebra CT data are obtained.

[0150] Table 1 is the evaluation index of the anterior, middle and posterior column fracture classification on the test set. The results show that the scheme can obtain better classification results on the test set, and can realize automatic classification of the three-column damage of the vertebra.

[0151] Table 1:

[0152]

[0153] In Table 1, ACC represents accuracy, AUC represents the area under the receiver operating characteristic curve, and AP represents average precision. AUC and AP are used to measure the comprehensive classification performance of the model.

[0154] The detection unit

[0155] For obtaining input data to be detected, and based on the classification and positioning model based on the residual attention constraint mechanism, obtaining a classification result and a positioning map corresponding to the input data to be detected.

[0156] Figure 5 Six test set samples predicted fracture positioning maps are shown, wherein each column represents the same sample, and the labels represent whether the anterior column, middle column and posterior column are fractured in turn, 1 represents fracture, and 0 represents no fracture. The results show that the fracture positioning map provided by the application can position and display the fractures of the anterior column, middle column and posterior column.

[0157] It should be noted that those skilled in the art should understand that the functions of each module shown in the above-described embodiments of the spinal three-column fracture classification and localization system based on residual attention constraint mechanism can be implemented by a program (executable instructions) running on a processor, or by specific logic circuits. If the spinal three-column fracture classification and localization system based on residual attention constraint mechanism described in this application is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0158] Accordingly, this application also provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the various method implementations of this application.

[0159] Furthermore, this application also provides a spinal three-column fracture classification and localization device based on a residual attention constraint mechanism, including a memory for storing computer-executable instructions and a processor; the processor is used to implement the steps in the above-described method embodiments when executing the computer-executable instructions in the memory. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The aforementioned memory can be read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or solid-state drive, etc. The steps of the methods disclosed in the various embodiments of this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0160] It has to be noted that, in the present patent application, the terms first and second etc. are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any actual relationship or order between such entities or operations. Moreover, the terms "comprises / comprising" or "includes / including" when used in this patent application are used to specify the presence of stated features, integers, steps or components but do not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. Furthermore, it is to be understood that the expression "at least one of A and B" or "at least one of A or B" or "at least one of A, B and C" or "at least one of A, B or C" or "one or more of A, B and C" or "one or more of A, B or C" includes variations such as: A alone; B alone; C alone; two of A, B and C; three of A, B and C; A and B together; A and C together; B and C together; or A, B and C together, and so forth. The expression "one or more of A, B and C" or "one or more of A, B or C" includes variations such as: A alone; B alone; C alone; two of A, B and C; three of A, B and C; A and B together; A and C together; B and C together; or A, B and C together, and so forth. The expression "one of A, B and C" or "one of A, B or C" includes variations such as: A alone; B alone; C alone; two of A, B and C; or three of A, B and C.

[0161] All documents mentioned in the present patent application are incorporated by reference in their entirety in the disclosure of the present patent application in order to more fully describe the application and the state of the art to which the application pertains. Further, it is recognized that variations of the present application are possible and it is intended that the application encompass such variations as long as they do not exceed the scope of the claims.

Claims

1. A spine three-column fracture classification and localization system based on residual attention constraint mechanism, characterized in that, The system comprises: a vertebra data set construction unit configured to obtain a vertebra data set for model training and testing according to input data, wherein the input data is obtained after preprocessing of vertebra CT image data and corresponding category labels and localization map labels; a classification and localization model construction unit based on a residual attention constraint mechanism, configured to construct a classification and localization model based on a residual attention constraint mechanism, which is a deep learning model for classifying and localizing three-column fractures of each vertebra according to the vertebra data set; wherein the classification and localization model based on the residual attention constraint mechanism comprises three branches: a backbone feature extraction network branch, a localization map generation network branch, and a fusion classification network branch; the backbone feature extraction network branch comprises a convolutional layer C1, a maximum pooling layer M1, n residual attention constraint sub-networks, and a global average pooling layer connected in series, for outputting a backbone feature vector F1 and ma three-dimensional spatial attention maps S, wherein ma is the number of residual attention modules in the entire model; the localization map generation network branch is configured to select ms spatial attention maps S, upsample them to the same size as the original input image, then concatenate them along the feature channel, use a convolutional layer to fuse multiple spatial attention maps into 3 spatial attention maps, and use a Sigmoid activation function to limit the values of the spatial attention maps to 0-1 to obtain the anterior, middle, and posterior column fracture localization maps SL; the fusion classification network branch comprises a localization map feature extraction part and a fusion classification prediction part, configured to extract features of the fracture localization maps SL, and fuse the localization map features with the backbone feature vector F1 before classification; the classification and localization model construction unit based on the residual attention constraint mechanism guides the model to focus on the fracture area through image-level and voxel-level supervision constraints, to obtain a classification result and a localization map; a model training unit configured to train the classification and localization model based on the residual attention constraint mechanism; and a detection unit configured to obtain input data to be detected, and obtain a classification result and a localization map corresponding to the input data to be detected based on the classification and localization model based on the residual attention constraint mechanism.

2. The system of claim 1, wherein, Further comprising: a CT data set construction unit configured to obtain vertebra CT image data and corresponding category labels and localization map labels for training and testing; a data preprocessing unit configured to obtain input data for the classification and localization model based on the residual attention constraint mechanism according to the vertebra CT image data and corresponding category labels and localization map labels.

3. The system of claim 1, wherein, Further comprising: a model testing unit configured to test the trained classification and localization model based on the residual attention constraint mechanism.

4. The system of claim 1, wherein In the CT data set construction unit, CT image data of a patient with spinal fracture is collected to construct a data set X, and the three-column fracture condition of each vertebra in the CT sequence is labeled to obtain a category label GT c and a positioning icon label GT s ; wherein the positioning icon label GT s is annotated as follows: first, a software capable of segmentation annotation is used to respectively paint the areas where the anterior, middle and posterior column fractures of each vertebra are located using a two-dimensional or three-dimensional segmentation annotation brush, so that the areas where the background, anterior, middle and posterior column fractures are located are covered by masks with values of 0, 1, 2 and 3 respectively, and then the annotated masks are saved as the positioning icon label GT s .

5. The system of claim 1, wherein, The positioning map feature extraction part first averages three positioning maps to obtain a fracture positioning map, then uses a maximum pooling layer and a mean pooling layer to down-sample the fracture positioning map to two 4x4x4 size feature maps, and then expands and splices the two feature maps to obtain a positioning map feature F2 with a dimension of 128.

6. The system of claim 1, wherein, The step of obtaining the final classification result by the fusion classification network branch is: inputting the front, middle and rear column fracture positioning maps SL into the positioning map feature extraction network to extract a positioning map feature F2, then splicing the backbone feature vector Fl and F2 along the feature channel to obtain a fused feature vector F, and inputting F into the fusion classification prediction network to obtain the final classification result P.

7. The system of claim 1, wherein, In the model training unit, Among them, L s Location-adaptive mean square loss, Labeling for fracture localization map of type K, SL K This is the location map of the Kth type of fracture predicted by the model; K takes values ​​of 0, 1, and 2, corresponding to anterior, middle, and posterior column fractures, respectively; i, j, and k represent the three-dimensional coordinate indices of the voxels; ε is a minimum value of 0.000001, used to ensure that the denominator is not 0.

Images