Deep learning-based thoracolumbar fracture rapid typing auxiliary judgment system

Abstract

The invention discloses a thoracolumbar vertebral fracture rapid typing auxiliary judgment system based on deep learning. The thoracolumbar vertebral fracture rapid typing auxiliary judgment system comprises a data preprocessing module, an s-type model, a z-type model and an integrated decision unit. The s-type model and the z-type model are respectively used for performing feature extraction on the s-type image and the z-type image in the instance package through a deep learning model, and outputting a corresponding prediction result through a multi-instance learning aggregation method; and the integrated decision-making unit is used for training the s-type model and the z-type model and outputting the predicted thoracolumbar fracture type. According to the method, prediction is carried out by combining the s-type model and the z-type model, so that the accuracy of thoracolumbar fracture type prediction can be improved, and the model can obtain better performance in a complex medical image classification task.

Description

Technical Field

[0001] The present invention belongs to the field of CT image recognition, and in particular relates to a rapid classification auxiliary judgment system for thoracolumbar fractures based on deep learning. Background Art

[0002] Thoracolumbar fracture is a common violent injury. The thoracolumbar segment is located in the transition zone between the relatively fixed thoracic vertebrae and the movable lumbar vertebrae with spinal cord and cauda equina. This special anatomical structure determines the high incidence of thoracolumbar fracture and spinal cord injury, and the diagnosis of thoracolumbar fracture is of decisive significance for the subsequent treatment and prognosis of patients. Therefore, accurate diagnosis is the key to the clinical diagnosis and treatment of thoracolumbar fracture. According to the new AO spine classification system for thoracolumbar fracture, the diagnosis of thoracolumbar fracture should be comprehensively evaluated from the aspects of fracture morphology and neurological function. The morphological diagnosis of fracture should be based on the degree of injury to the anterior, middle and posterior columns of the spine. Therefore, the morphological evaluation of fracture must rely on imaging data such as X-ray, CT, and MRI. In addition, due to the irregular shape of the vertebral body, the complex and diverse paravertebral soft tissues, the low resolution of imaging data, and the different clinical capabilities of doctors, it is not easy for imaging and clinical doctors to make a correct diagnosis of thoracolumbar fractures, and it is time-consuming and labor-intensive, with a high misdiagnosis rate. In most primary medical units, due to the relative lag of medical information and the low imaging level of medical equipment, these problems are particularly prominent. It is unrealistic to fully identify the fracture site and accurately and standardizedly classify and diagnose the thoracolumbar fracture. Therefore, how to quickly and accurately make a correct morphological diagnosis of thoracolumbar fractures is a difficult problem in clinical work; it is necessary to propose a deep learning algorithm model for morphological diagnosis and classification of thoracolumbar fractures for computer-aided diagnosis of clinical thoracolumbar fractures. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a rapid classification and auxiliary judgment system for thoracolumbar fractures based on deep learning.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] A deep learning-based rapid classification and auxiliary judgment system for thoracolumbar fractures, including

[0006] A data preprocessing module, used for taking the patient's CT image as an instance package and a single CT image as an instance, wherein the coronal CT image in the instance package is pre-marked as a z-type image, and the cross-sectional CT image is pre-marked as an s-type image;

[0007] The s-type model is used to extract features from the s-type images in the instance package through a deep learning model, and output a first prediction result of the thoracolumbar fracture type through a multi-instance learning aggregation method;

[0008] A z-type model is used to extract features from the z-type image in the instance package through a deep learning model, and output a second prediction result of the thoracolumbar fracture type through a multi-instance learning aggregation method; and

[0009] The integrated decision unit is used for training the S-type model and the Z-type model, and outputting the predicted thoracolumbar fracture type by integrating the first prediction result and the second prediction result.

[0010] Furthermore, the thoracolumbar fracture types include non-fracture and multiple compression fracture types, and each of the thoracolumbar fracture types corresponds to a sub-classification label. For each instance package in the data set, the corresponding sub-classification label is set as the package label according to the diagnosis result, and for each instance in the instance package, the corresponding sub-classification label is set as the instance label according to the diagnosis result.

[0011] Furthermore, when training the s-type model, the fine classification labels of the instance bag and the fine classification labels of the instances of the cross-sectional CT images are used for training;

[0012] When training the z-type model, the detailed classification labels of the instance package and the detailed classification labels of the instances of the coronal CT images are used for training, and before training, the package labels of the compression fracture types among the multiple compression fracture types that are difficult to identify through the coronal CT images and the instance labels of the coronal CT images in the instance package are set to the label category of no fracture.

[0013] Furthermore, the thoracolumbar fracture types include A0 type, A1 type, A2 type, A3 type, A4 type and Az type, wherein the A0 type is a spinous process or transverse process fracture without vertebral fracture; it is a left transverse process fracture without clinical significance; the A1 type is a compression fracture involving a single end plate, and the posterior wall of the vertebra is intact; the A2 type is a splitting or clamp fracture with the broken line involving the upper and lower end plates but not the posterior wall of the vertebra; the A3 type is an incomplete burst fracture. The fracture involves the posterior wall of the upper end plate or the lower end plate; the A4 type is a complete burst fracture, and the fracture involves the posterior wall of the upper and lower end plates at the same time; the Az type is a thoracolumbar spine without fracture. Among them, the A0 type is a compression fracture type that is not easy to identify through coronal CT images.

[0014] Furthermore, when the instance package is identified as a non-fracture type by the z-type model, the integrated decision unit uses the predicted value of the s-type model as the output value; otherwise, the integrated decision unit uses the predicted value of the z-type model as the output value.

[0015] Furthermore, the compression fracture types with similar CT images among the multiple compression fracture types are divided into the same major category, and each major category corresponds to a major category label; when training the z-type model, the instance package and the major category label of the instance are also input, and the second prediction result also includes the recognition result of the major category label.

[0016] Furthermore, the s-type model includes a first feature extraction module and a first classification module; the first feature extraction module uses a convolutional neural network as a basic feature extractor to extract s-type features from the s-type image in the instance package, and the first classification module is used to directly predict the output instance label based on the features extracted by the first feature extraction module, and uses a multi-instance learning aggregation method to aggregate the features extracted by the first feature extraction module and then predict the output package label;

[0017] The z-type model includes a second feature extraction module and a second classification module; the second feature extraction module uses a convolutional neural network as a basic feature extractor to extract z-type features from the z-type image in the instance package, and the second classification module is used to directly predict and output instance labels based on the features extracted by the second feature extraction module, and uses a multi-instance learning aggregation method to aggregate the features extracted by the second feature extraction module and then predict and output major category labels and package labels.

[0018] Furthermore, the first classification module includes a first multi-instance learning aggregation unit, a first fully connected layer, and a second fully connected layer. The first feature extraction module replaces the fully connected layer of the AlexNet model with the first feature mapping layer as a basic feature extractor; the first feature mapping layer is used to map the features of the extracted instances to the first fully connected layer for prediction, and map the features of the extracted packages to the first multi-instance learning aggregation unit. The first multi-instance learning aggregation unit aggregates the features of the packages using a gated attention mechanism and predicts through the second fully connected layer; the loss function of the first classification module is optimized for the prediction of package labels and instance labels respectively;

[0019] The second classification module includes a second multi-instance learning aggregation unit, a third fully connected layer, a fourth fully connected layer and a fifth fully connected layer. The second feature extraction module replaces the fully connected layer of the AlexNet model with the second feature mapping layer as a basic feature extractor; the second feature mapping layer is used to map the features of the extracted instances to the third fully connected layer for prediction, and map the extracted features of the major categories and the features of the packages to the second multi-instance learning aggregation unit. The second multi-instance learning aggregation unit uses a gated attention mechanism to aggregate the features of the major categories and the features of the packages, and then predicts them through the fourth fully connected layer and the fifth fully connected layer respectively; the loss function of the second classification module is optimized for the prediction of major category labels, package labels and instance labels respectively.

[0020] Furthermore, the method for training the s-type model and the z-type model includes the following steps:

[0021] S1. Collect spinal CT image data and diagnostic results of patients with thoracolumbar fractures and non-patients as a data set, take all CT images of the same patient in the data set as an instance package, take a single CT image as an instance, and mark the coronal CT image in each instance package as a z-type image, and mark the cross-sectional CT image as an s-type image;

[0022] S2, setting corresponding package labels and instance labels for instance packages and instances according to the type of thoracic and lumbar fractures;

[0023] S3. A first-level folder is created for each thoracolumbar fracture type in the data set. Each instance package is used as a second-level folder and is merged into the first-level folder of the corresponding thoracolumbar fracture type according to the diagnosis results. Each second-level file includes multiple z-type images, multiple s-type images and a classic folder. The classic folder stores the specific fracture pictures of the patient.

[0024] S4. Randomly divide the instance packages in the dataset into training set and validation set, and perform data augmentation on the CT images in the training set and validation set.

[0025] S5. Train the s-type model and z-type model using the training set and validation set respectively.

[0026] Furthermore, each of the example packages includes 5 z-type images and 5 s-type images, the training set accounts for 80%, and the validation set accounts for 20%.

[0027] In the present invention, by combining the s-type model and the z-type model for prediction, the accuracy of thoracic and lumbar fracture type prediction can be improved, which helps the model to achieve better performance in complex medical image classification tasks. The fully connected layer of the AlexNet model is replaced by a custom feature mapping layer as a basic feature extractor, which can improve the feature extraction capability and reduce the number of parameters. The multi-instance learning method is used for classification, which improves the accuracy and robustness of classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 This is a structural block diagram of an embodiment of a deep learning-based thoracolumbar fracture rapid classification auxiliary judgment system of the present invention.

[0030] Figure 2 This is a schematic diagram of the working process of the rapid typing auxiliary judgment system. DETAILED DESCRIPTION

[0031] The following describes the implementation methods of the present invention through specific examples. The illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present invention. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0032] See also Figure 1 , Figure 1 This is a structural block diagram of an embodiment of a deep learning-based rapid classification and auxiliary judgment system for thoracolumbar fractures of the present invention. The deep learning-based rapid classification and auxiliary judgment system for thoracolumbar fractures of this embodiment includes a data preprocessing module, an s-type model, a z-type model, and an integrated decision unit. The data preprocessing module is used to take the patient's CT image as an instance package and a single CT image as an instance to meet the data loading requirements of MIL; the coronal CT image in the instance package is pre-marked as a z-type image, and the cross-sectional CT image is pre-marked as an s-type image. The data preprocessing module also processes the CT images in the instance package through image processing technology to improve the generalization ability and robustness of the model.

[0033] The s-type model is used to extract features from the s-type images in the instance package through a deep learning model, and output a first prediction result of the thoracolumbar fracture type through a multi-instance learning aggregation method. In this embodiment, the s-type model includes a first feature extraction module and a first classification module; the first feature extraction module uses a convolutional neural network as a basic feature extractor to extract s-type features from the s-type images in the instance package. The first classification module includes a first multi-instance learning aggregation unit, a first fully connected layer, and a second fully connected layer, which is used to predict the output instance label through the first fully connected layer according to the features extracted by the first feature extraction module, and after aggregating the features extracted by the first feature extraction module using a multi-instance learning (MIL) aggregation method through the first multi-instance learning aggregation unit, the output package label is predicted through the second fully connected layer.

[0034] The z-type model is used to extract features from the z-type image in the instance package through a deep learning model, and output a second prediction result of the thoracolumbar fracture type through a multi-instance learning aggregation method. In this embodiment, the z-type model includes a second feature extraction module and a second classification module; the second feature extraction module uses a convolutional neural network as a basic feature extractor to extract z-type features from the z-type image in the instance package. The second classification module includes a second multi-instance learning aggregation unit, a third fully connected layer, a fourth fully connected layer, and a fifth fully connected layer, which is used to predict and output instance labels through the third fully connected layer according to the features extracted by the second feature extraction module, and after aggregating the features extracted by the second feature extraction module through the second multi-instance learning aggregation unit using a multi-instance learning aggregation method, the fourth fully connected layer and the fifth fully connected layer are used to predict and output major category labels and package labels respectively.

[0035] The integrated decision unit is used to train the s-type model and the z-type model, and to output the predicted thoracolumbar fracture type by synthesizing the first prediction result and the second prediction result during prediction. In this embodiment, the thoracolumbar fracture type includes no fracture and multiple compression fracture types. Specifically, the thoracolumbar fracture types in this embodiment include A0 type, A1 type, A2 type, A3 type, A4 type and Az type. Among them, the A0 type is a spinous process or transverse process fracture, and there is no vertebral fracture; it is a left transverse process fracture with no clinical significance. The A1 type is a compression fracture involving a single end plate, and the posterior wall of the vertebral body is intact. The A2 type is a split or clamp fracture in which the broken line involves the upper and lower end plates but does not involve the posterior wall of the vertebral body. The A3 type is an incomplete burst fracture. The fracture involves the posterior wall of the upper end plate or the lower end plate. The A4 type is a complete burst fracture, and the fracture involves the posterior wall of the upper end plate and the lower end plate at the same time. The Az type is a thoracolumbar vertebra without fracture. Among them, type A0 is a compression fracture type that is not easy to identify through coronal CT images. In order to facilitate the training of the model, the compression fracture types with similar CT images among the multiple compression fracture types can also be divided into the same category, and each category corresponds to a category of labels. In this embodiment, type A1 and type A2 are divided into the first category, type A3 and type A4 are divided into the second category, and type A0 and type Az are divided into the third category.

[0036] The method for training the s-type model and the z-type model includes the following steps:

[0037] S1. The spinal CT images and diagnostic results of patients with thoracolumbar fractures and non-patients were collected as a data set. The inclusion criteria for patients with thoracolumbar fractures in the data set were as follows:

[0038] (1) Acute thoracolumbar fractures (thoracolumbar vertebrae 10 to lumbar vertebrae 2) in patients aged 18-65 years

[0039] (2) Complete CT data (including 5 mm bone window and sagittal reconstruction), except for the following five cases:

[0040] a. Osteoporotic fractures and pathological fractures;

[0041] b. History of thoracic and lumbar spine surgery;

[0042] c. Obvious spinal degeneration;

[0043] d. Incomplete CT data;

[0044] e. Multiple vertebral fractures).

[0045] Afterwards, all CT images of the same patient (i.e., patients with thoracic and lumbar fractures and non-patients) in the dataset are taken as an instance package, a single CT image is taken as an instance, and the coronal CT image in each of the instance packages is marked as a z-type image, and the cross-sectional CT image is marked as an s-type image.

[0046] S2. Set corresponding package labels and instance labels for instance packages and instances according to the type of thoracolumbar fracture. Each of the thoracolumbar fracture types corresponds to a sub-classification label. For example, type A0 may correspond to sub-classification label 0, type A1 may correspond to sub-classification label 1, type A2 may correspond to sub-classification label 2, type A3 may correspond to sub-classification label 3, type A4 may correspond to sub-classification label 4, and type Az may correspond to sub-classification label 5; and sub-classification label 1 and sub-classification label 2 may correspond to general category label 11, and sub-classification label 3 and sub-classification label 4 may correspond to general category label 12, and sub-classification label 0 and sub-classification label 5 may correspond to general category label 13. For each instance package in the data set, set the corresponding sub-classification label and general category label as the package label according to the diagnosis result, and for each instance in the instance package (i.e., CT image), set the corresponding sub-classification label and general category label as the instance label according to the diagnosis result.

[0047] S3. A first-level folder is established in the data set corresponding to each type of thoracolumbar fracture, including an A0 folder, an A1 folder, an A2 folder, an A3 folder, an A4 folder and an Az folder. Each instance package is used as a second-level folder and is incorporated into the first-level folder of the corresponding thoracolumbar fracture type according to the diagnosis results. Each second-level file includes a plurality of z-type images, a plurality of s-type images and a classic folder, and the classic folder stores the specific fracture pictures of the patient. In this embodiment, each of the instance packages includes 5 z-type images and 5 s-type images. In addition, because Az is a non-fracture type, there is no classic folder for Az. It is worth noting that A0 type fractures are generally judged by observing s-type images, so there is no z-type image in the A0 folder. Through the above processing, the data set meets the data loading requirements of MIL.

[0048] S4. Randomly divide the instance packages in the data set into a training set and a validation set. In this embodiment, the training set accounts for 80% and the validation set accounts for 20%. Afterwards, data enhancement processing is performed on the CT images in the training set and the validation set. In the data preprocessing stage, this embodiment adopts a series of image processing techniques to improve the generalization ability and robustness of the model. For the training set images, the image is first adjusted to 256×256 size, then the diversity of the data is increased by random affine transformation (rotation, translation, scaling, etc.), and then randomly cropped to 224×224 size, and randomly flipped and rotated horizontally. Finally, the image is converted into a tensor and standardized (mean is 0.5, standard deviation is 0.5). For the validation set images, the preprocessing step includes adjusting the image to 256×256 size, center cropping to 224×224 size, and then standardizing. The final data loader will return an image information with a shape of [1,5,3,224,224], that is, 1 batch, 5 instance images, 3 channels, and each image size is 224×224.

[0049] S5. The s-type model and the z-type model are trained using the training set and the validation set, respectively. When training the s-type model, the fine classification labels of the instance package and the fine classification labels of the instance of the cross-sectional CT image are used for training. That is, the training of the s-type model does not input the labels of the major categories, and the first prediction result does not include the recognition results of the major category labels.

[0050] When training the z-type model, the major category labels and detailed category labels of the instance package and the major category labels and detailed category labels of the instances of the coronal CT images are used for training, and before training, the package labels of the compression fracture types among the multiple compression fracture types that are difficult to identify through the coronal CT images and the instance labels of the coronal CT images in the instance package are set to the label category of no fracture (including major category labels and detailed category labels).

[0051] The label mapping of this embodiment uses the method of multiple instance learning. Multiple instance learning (MIL) is a typical weakly supervised learning. The characteristic of classical supervised learning is to use data with clear labels to train the model to predict the label of new data; MIL is a special supervised learning method, in which the label is assigned to an instance bag, rather than a single instance, and the model needs to extract information from multiple instances to determine the label of the entire instance bag. In other words, in the multiple instance learning problem, the bag will be used as the basic unit for training. There is neither dependency nor ordering relationship between the instances in the instance bag. If all instances in an instance bag are judged to be negative (ie: negative examples), the bag label will be judged to be negative, otherwise the bag label will be judged to be positive (ie: positive example).

[0052] In the s-type model, label categories are directly mapped according to normal classification rules (i.e., detailed classification labels), but in the z-type model, labels are divided into large-category labels and detailed classification labels through a hierarchical label mapping method. In the actual training process, since there is no A0 type fracture data in the z-type dataset, the z-type type in the A0 folder will be marked as Az (non-fracture type). Finally, the data loader will return a two-dimensional label with a shape of [1,5] for the classification task of the s-type model, where the first dimension is the package label and the second dimension is the instance label. For the z-type model, a three-dimensional label will be returned with a shape of [1,5,1]. The first two are the same as the s-type model, and the last dimension stores the large-category label.

[0053] The integrated decision unit will reprocess the package label when finally merging the s-type model with the z-type model for model prediction. That is, as long as there is a fracture image in classic, the package label will be set as the label of the current fracture type. The design of this label mapping strategy not only simplifies the complexity of the classification task, but also improves the accuracy and robustness of the classification.

[0054] In the present embodiment, the first feature extraction module and the second feature extraction module are designed based on the AlexNet model. AlexNet is a classic deep convolutional neural network architecture, which significantly improves the accuracy of image classification by introducing deeper network structure, ReLU activation function, overlapping maximum pooling, local response normalization and data enhancement techniques. The design of the model includes five convolutional layers and three fully connected layers, which realizes efficient classification of complex images by extracting low-level, medium-level and high-level features of images layer by layer. The success of AlexNet marks the widespread application of deep learning in the field of computer vision and promotes the development of subsequent deep neural networks. In the present embodiment, AlexNet is selected as the basic feature extractor and is appropriately modified. By replacing the original fully connected layer of AlexNet with a custom feature mapping layer as the basic feature extractor, a more general feature representation can be generated to adapt to medical image classification tasks.

[0055] Among them, the first feature extraction module replaces the fully connected layer of the AlexNet model with a customized first feature mapping layer as a basic feature extractor, the first feature mapping layer is used to map the features of the extracted instances to the first fully connected layer for prediction, and maps the features of the extracted packages to the first multi-instance learning aggregation unit to aggregate the features of the packages, and then predicts through the second fully connected layer. The second feature extraction module replaces the fully connected layer of the AlexNet model with a customized second feature mapping layer as a basic feature extractor, the second feature mapping layer is used to map the features of the extracted instances to the third fully connected layer for prediction, and maps the features of the extracted major categories and packages to the second multi-instance learning aggregation unit to aggregate the features of the major categories and packages, and then predicts through the fourth fully connected layer and the fifth fully connected layer respectively. AlexNet's powerful feature extraction capability provides rich visual information for the subsequent layers of the model, which helps the model achieve better performance in complex medical image classification tasks.

[0056] When the fully connected layer is removed and a custom feature map layer is introduced, the custom feature map layer is usually placed at the position of the original fully connected layer, that is, at the end of the network. This replacement is usually for the following purposes:

[0057] a. Improve the ability of feature extraction: Custom feature mapping layers can introduce more flexibility and extract more representative features instead of directly outputting fixed classification results.

[0058] b. Reduce the number of parameters: The fully connected layer has a large number of parameters. Replacing it with a feature mapping layer can significantly reduce the model parameters, so that the features of the instance do not need to be aggregated through the attention mechanism, thereby improving the computational efficiency and training speed, and better adapting to new tasks: Through the custom feature mapping layer, the network can adapt to different task requirements, such as fine-grained classification or multi-task learning. Therefore, although the position is the same, the custom feature mapping layer has different functions from the original fully connected layer. The former usually replaces the fully connected layer with a convolutional layer or other types of layers to obtain a more detailed feature representation.

[0059] The multi-instance learning aggregation method of the first multi-instance learning aggregation unit and the second multi-instance learning aggregation unit can adopt a gated attention mechanism. Traditional MIL generally uses mean-pooling or max-pooling for aggregation. In this embodiment, a gated attention aggregation mechanism is adopted. The gated attention mechanism is an effective attention mechanism that enhances the model's ability to capture key information by selectively focusing on different parts of the input features. The core idea is to adjust the contribution of features in a weighted manner, thereby improving the expressive power of the model. The gated attention mechanism usually consists of two parallel attention paths, which use different activation functions to calculate the attention weights, and finally generate the final attention score through multiplication operations. The specific principles are as follows:

[0060] Assume that the input feature matrix is ​​H∈R M×D , where M is the dimension of the feature and D is the length of the feature vector. First, the input feature matrix H is transformed through two different linear transformations to generate two intermediate feature representations V and U:

[0061] V = tanh(HW v +b v )

[0062] U=σ(HW u +b u )

[0063] Among them, W v ∈R D×L and W u ∈R D×L is the learnable weight matrix, b v and b u is the bias term, L is the dimension of attention. Tanh is the hyperbolic tangent activation function, and σ is the sigmoid activation function. After obtaining V and U, we multiply them element-wise to merge V and U together to form the final attention weight matrix A:

[0064] A=V·U

[0065] Here, · represents the element-by-element multiplication operation. The result of this step represents the attention weight of each feature dimension, which combines linear and nonlinear information, allowing the model to more finely adjust the degree of attention to different feature dimensions. Finally, the attention weight matrix A is transposed and normalized by the Softmax function, and the weighted sum of the original feature matrix H is performed to obtain the final feature representation Z. Z will eventually be used in the classification of major categories and packages. The specific formula is as follows:

[0066] A T =Softmax(A)

[0067] Z=A T H

[0068] In general, the gated attention mechanism can significantly enhance the model's feature selectivity and expressiveness by dynamically adjusting the degree of attention paid to different features. Its adaptability enables the model to accurately capture key information in complex, high-noise environments while effectively filtering out irrelevant or interfering information, thereby improving the model's robustness and overall prediction performance. This mechanism is particularly suitable for processing diverse medical image data and can demonstrate higher accuracy and stability in classification tasks.

[0069] The loss function of the s-type model and the z-type model is designed to effectively integrate the hierarchical classification requirements in multi-task learning and the unique needs of MIL; the loss function consists of multiple parts, which are optimized for large-category classification, package classification, and instance-level prediction. In this embodiment, each sub-loss function uses the cross-entropy loss function, which is a commonly used function in classification tasks and will not be described in detail here.

[0070] In the z-type model, the loss function comprehensively considers the classification loss of the super-class, the classification loss of the bag-class, and the prediction differences between multiple image instances, thereby effectively guiding the model to learn at multiple levels. By weightedly combining these three parts of loss, the model can improve the overall classification performance while balancing the requirements of tasks at different levels.

[0071] For the s-type model, its main goal is to make up for the problem that the z-type model cannot identify A0 fractures due to the lack of A0 data. During the experiment, we found that after adding large-category classification, the recognition accuracy of s-type for A0 type was reduced. Therefore, when training the s-type model, the large-category classification was removed, and the loss function only included the package classification loss and instance-level loss. In addition, the loss function of multi-instance learning further enhances the robustness of the model when processing multiple image instances, especially when some instances may be noisy or inconsistently labeled, the model can still accurately predict the package label.

[0072] In this embodiment, since deep learning models for the s-type and z-type tasks are trained and their performance characteristics in specific classification tasks are utilized, the integrated decision unit adopts a combined prediction strategy based on conditional judgment.

[0073] First, the trained z-type model is used to make a preliminary prediction of the input image. Since the z-type model performs well in identifying non-A0 categories, we give priority to using the z-type model to identify the category of the input image. Since z-type does not have data of the A0 category, the label of the z-type data in the A0 folder will be marked as the Az type during the data loading process. Based on the above reasons, when the image is identified as the Az type by the z-type model, the s-type model will be further used for secondary judgment, and the prediction of the s-type model will be the final prediction value. When the image is identified as a non-A0 type by the z-type model, the final prediction value is still based on the prediction value of the z-type model. This combination strategy makes full use of the s-type model to make up for the lack of the z-type model in the A0 type data, and effectively improves the classification accuracy and robustness through the conditional judgment mechanism. Of course, in order to improve timeliness, this embodiment adopts the method of simultaneous prediction of the z-type model and the s-type model. When the instance package is identified as a non-Az type by the z-type model, the integrated decision unit uses the prediction value of the z-type model as the output value. When the instance package is identified as a non-fracture type (ie, Az type) by the z-type model, the integrated decision unit uses the predicted value of the s-type model as the output value.

[0074] To evaluate the performance of the combined model, this embodiment uses a variety of commonly used classification indicators, including accuracy, precision, recall, and F1 score. These indicators evaluate the performance of the model in the classification task from different angles, and can fully reflect the actual application effect of the model.

[0075] Accuracy: It measures how well the model’s predictions match the true labels, and represents the proportion of all correct predictions.

[0076] Precision: It indicates the proportion of samples predicted as positive by the model that are actually positive, reflecting the model's ability to reduce false positives.

[0077] Recall: It indicates the proportion of samples that are successfully predicted as positive among all samples that are actually positive, reflecting the model's ability to reduce false negatives.

[0078] F1 Score: The harmonic mean of precision and recall, which takes into account the performance of both and is suitable for datasets with imbalanced categories.

[0079] In addition, this embodiment further analyzes the classification effect of the model through a confusion matrix. The confusion matrix can show the matching of the actual label and the predicted label of each category, which helps to identify which categories are prone to misjudgment and deeply analyze the possible reasons.

[0080] In the validation set of the s-Type model, the overall classification accuracy reached 75.2%. From the confusion matrix, the model performed well on categories A0, A1, and A4, reaching 80%, 84%, and 88% accuracy, respectively, which shows that the model has high reliability in identifying these categories. However, the performance on categories A2 and A3 was significantly poor, especially on category A2, which only achieved an accuracy of 20%. This may be due to the small number of samples in category A2 (the entire data set has a total of 596 data, of which category A2 has only 28 data), resulting in the model's insufficient performance in learning the features of this category. In addition, the accuracy of category A5 is 78%. This result shows that the model still has room for improvement in identifying this category, especially in the ability to distinguish between different categories. There is a certain confusion.

[0081] The overall validation set accuracy of the z-Type model is 88.0%, which is significantly better than the s-Type model. As can be seen in the confusion matrix, the model performs particularly well in categories A1 and A5, reaching 89% and 98% accuracy, respectively. This shows that the z-Type model is very stable in identifying these categories. However, category A2 is also constrained by a small number of samples, with an accuracy of 0%. In addition, the accuracy on category A3 is 86%, while category A4 reaches 94%, which shows that the model has a strong ability to distinguish when identifying these categories, especially in category A4, where the z-Type model performs significantly better than the s-Type model. Overall, the z-Type model performs better than the s-Type model in categories A1, A3, A4, and A5, but due to the missing data of category A0, the s-Type model still needs to assist, which is why the integrated model uses the S-Type model to predict A0, while other types of fractures are predicted using the z-type model. The overall accuracy of the model after integrating the S-Type model and the z-type model is 83.8%, which is significantly higher than the S-Type model, and the recognition ability of the A0 category is increased compared with the z-type model.

[0082] The working principle of this embodiment is as follows:

[0083] As shown in the figure above, after stacking 5 s-type images in a group, a CNN (i.e., the first feature extraction module) is used to extract feature H to obtain s-type features (i.e., cross-sectional features). After stacking 5 z-type images in a group, another CNN (i.e., the second feature extraction module) is used to extract feature H to obtain z-type features (i.e., coronal features). After that, the features of the bag in the extracted feature s-type features are sent to a MIL aggregator (i.e., the first multi-instance learning aggregation unit), and after aggregation based on the aggregation method of the gated attention mechanism, the s-type bag preds (i.e., the prediction result of the instance bag of the s-type model) is obtained through the second fully connected layer; in addition, the features of the instance in the s-type features will also be directly used for the prediction of the instance through the first fully connected layer to obtain s-type instance preds (i.e., the prediction result of the instance of the s-type model). At the same time, the features of the large categories and the features of the bag in the extracted feature z-type features are sent to another MIL aggregator (i.e., the second multi-instance learning aggregation unit). After aggregation based on the aggregation method of the gated attention mechanism, the fourth fully connected layer predicts the z-type bag preds (i.e., the prediction results of the instance bag of the z-type model), and the fifth fully connected layer predicts the z-type super preds (i.e., the prediction results of the large categories of the z-type model). In addition, the features of the instances in the z-type features will also be directly used for instance prediction through the third fully connected layer to obtain z-type instance preds (i.e., the prediction results of the instances of the z-type model). Finally, the predictions of the s-type model and the z-tpye model are combined to obtain the final preds (i.e., the final bag label). The s-typeinstance preds and z-type instance preds calculated by the model will be used for subsequent loss function calculations.

[0084] In this embodiment, by combining the s-type model and the z-type model for prediction, the accuracy of the prediction of the thoracic and lumbar fracture types can be improved, which helps the model to achieve better performance in complex medical image classification tasks. The fully connected layer of the AlexNet model is replaced with a custom feature mapping layer as a basic feature extractor, which can improve the feature extraction capability and reduce the number of parameters. The multi-instance learning method is used for classification, which improves the accuracy and robustness of the classification.

[0085] The above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A rapid classification and auxiliary judgment system for thoracolumbar fractures based on deep learning, characterized by: include A data preprocessing module, used for taking the patient's CT image as an instance package and a single CT image as an instance, wherein the coronal CT image in the instance package is pre-marked as a z-type image, and the cross-sectional CT image is pre-marked as an s-type image; The s-type model is used to extract features from the s-type images in the instance package through a deep learning model, and output a first prediction result of the thoracolumbar fracture type through a multi-instance learning aggregation method; The z-type model is used to extract features from the z-type image in the instance package through a deep learning model, and output a second prediction result of the thoracolumbar fracture type through a multi-instance learning aggregation method; as well as The integrated decision unit is used for training the S-type model and the Z-type model, and outputting the predicted thoracolumbar fracture type by integrating the first prediction result and the second prediction result.

2. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 1 is characterized by: The thoracolumbar fracture types include non-fracture and multiple compression fracture types. Each thoracolumbar fracture type corresponds to a sub-classification label. For each instance package in the data set, the corresponding sub-classification label is set as the package label according to the diagnosis result. For each instance in the instance package, the corresponding sub-classification label is set as the instance label according to the diagnosis result.

3. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 2 is characterized by: When training the s-type model, the fine classification labels of the instance bag and the fine classification labels of the instances of the cross-sectional CT images are used for training; When training the z-type model, the detailed classification labels of the instance package and the detailed classification labels of the instances of the coronal CT images are used for training, and before training, the package labels of the compression fracture types among the multiple compression fracture types that are difficult to identify through the coronal CT images and the instance labels of the coronal CT images in the instance package are set to the label category of no fracture.

4. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 3 is characterized by: The thoracolumbar fracture types include A0, A1, A2, A3, A4 and Az, wherein the A0 type is a spinous process or transverse process fracture without vertebral fracture; it is a left transverse process fracture without clinical significance; the A1 type is a compression fracture involving a single end plate, and the posterior wall of the vertebra is intact; the A2 type is a splitting or clamp fracture with a broken line involving the upper and lower end plates but not the posterior wall of the vertebra; the A3 type is an incomplete burst fracture. The fracture involves the posterior wall of the upper end plate or the lower end plate; the A4 type is a complete burst fracture, and the fracture involves the posterior wall of the upper and lower end plates at the same time; the Az type is a thoracolumbar spine without fracture. Among them, the A0 type is a compression fracture type that is not easy to identify through coronal CT images.

5. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 1, characterized in that: When the instance package is identified as an unfractured type by the z-type model, the integrated decision unit uses the predicted value of the s-type model as an output value; Otherwise, the integrated decision unit uses the predicted value of the z-type model as the output value.

6. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 1, characterized in that: The compression fracture types with similar CT images among the multiple compression fracture types are divided into the same major category, and each major category corresponds to a major category label; when training the z-type model, the instance package and the major category label of the instance are also input, and the second prediction result also includes the recognition result of the major category label.

7. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 1 is characterized by: The s-type model includes a first feature extraction module and a first classification module; the first feature extraction module uses a convolutional neural network as a basic feature extractor to extract s-type features from the s-type image in the instance package, and the first classification module is used to directly predict and output instance labels based on the features extracted by the first feature extraction module, and to predict and output package labels after aggregating the features extracted by the first feature extraction module using a multi-instance learning aggregation method; The z-type model includes a second feature extraction module and a second classification module; the second feature extraction module uses a convolutional neural network as a basic feature extractor to extract z-type features from the z-type image in the instance package, and the second classification module is used to directly predict and output instance labels based on the features extracted by the second feature extraction module, and uses a multi-instance learning aggregation method to aggregate the features extracted by the second feature extraction module and then predict and output major category labels and package labels.

8. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 7, characterized in that: The first classification module includes a first multi-instance learning aggregation unit, a first fully connected layer, and a second fully connected layer. The first feature extraction module replaces the fully connected layer of the AlexNet model with the first feature mapping layer as a basic feature extractor; the first feature mapping layer is used to map the features of the extracted instances to the first fully connected layer for prediction, and map the features of the extracted packages to the first multi-instance learning aggregation unit. The first multi-instance learning aggregation unit aggregates the features of the packages using a gated attention mechanism and then predicts through the second fully connected layer. The loss function of the first classification module is optimized for the prediction of the package label and the instance label respectively; The second classification module includes a second multi-instance learning aggregation unit, a third fully connected layer, a fourth fully connected layer and a fifth fully connected layer. The second feature extraction module replaces the fully connected layer of the AlexNet model with the second feature mapping layer as a basic feature extractor; the second feature mapping layer is used to map the features of the extracted instances to the third fully connected layer for prediction, and map the extracted features of the major categories and the features of the packages to the second multi-instance learning aggregation unit. The second multi-instance learning aggregation unit aggregates the features of the major categories and the features of the packages using a gated attention mechanism, and predicts them through the fourth fully connected layer and the fifth fully connected layer respectively. The loss function of the second classification module is optimized for the prediction of major category labels, package labels, and instance labels.

9. The deep learning-based thoracolumbar fracture rapid classification auxiliary judgment system according to any one of claims 1 to 8, characterized in that: The method for training the s-type model and the z-type model includes the following steps: S1. Collect spinal CT image data and diagnostic results of patients with thoracolumbar fractures and non-patients as a data set, take all CT images of the same patient in the data set as an instance package, take a single CT image as an instance, and mark the coronal CT image in each instance package as a z-type image, and mark the cross-sectional CT image as an s-type image; S2, setting corresponding package labels and instance labels for instance packages and instances according to the type of thoracic and lumbar fractures; S3. A first-level folder is created for each thoracolumbar fracture type in the data set. Each instance package is used as a second-level folder and is merged into the first-level folder of the corresponding thoracolumbar fracture type according to the diagnosis results. Each second-level file includes multiple z-type images, multiple s-type images and a classic folder. The classic folder stores the specific fracture pictures of the patient. S4. Randomly divide the instance packages in the dataset into training set and validation set, and perform data augmentation on the CT images in the training set and validation set. S5. Train the s-type model and z-type model using the training set and validation set respectively.

10. The deep learning-based rapid classification auxiliary judgment system for thoracolumbar fractures according to claim 9, characterized in that: Each of the example packages includes 5 z-type images and 5 s-type images, the training set accounts for 80%, and the validation set accounts for 20%.

Images