A spine CT image classification method and system based on bidirectional covariance

By adopting a bidirectional covariance method in spinal CT image classification, combining the splicing and adaptive weight network of channel and spatial covariance features, the problem of underutilization of feature relationships and spatial layout in the existing technology is solved, and significantly improved Accuracy of spinal CT image classification.

CN118552799BActive Publication Date: 2024-11-26SHANDONG FIRST MEDICAL UNIV & SHANDONG ACADEMY OF MEDICAL SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411025314.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-26
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

Existing deep learning methods fail to fully consider the complex relationships and spatial layout between features in spinal CT image classification, resulting in poor classification accuracy.

Method used

Using a two-way covariance-based method, the spinal CT images are feature extracted through the SBCFormer network, channel covariance and spatial covariance are calculated, and they are spliced ​​and jointly predicted, and classified predictions are combined with an adaptive weight network.

Benefits of technology

The accuracy of spinal CT image classification is improved, the detailed relationship and complexity between features in higher-order feature space is captured, the model's comprehensive understanding of channel and spatial features is enhanced, and the classification accuracy is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118552799B_ABST
    Figure CN118552799B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of image processing technology, and provides a spine CT image classification method and system based on bidirectional covariance, including obtaining a spine CT image, inputting the spine CT image into an SBCFormer network for feature extraction, respectively calculating the channel covariance and spatial covariance of the extracted features, obtaining a channel covariance feature representation and a spatial covariance feature representation, splicing the channel covariance feature representation and the spatial covariance feature representation to obtain a spliced ​​feature representation, combining the channel covariance feature representation, the spatial covariance feature representation and the spliced ​​feature representation to perform image classification prediction, and outputting a spine CT image classification result. The present invention not only focuses on the correlation between feature channels, but also considers spatial structural information, so as to more comprehensively capture image content and improve image classification accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to a spine CT image classification method and system based on bidirectional covariance. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Spine CT images are often used to check the health of the spine, aiming to comprehensively evaluate the health of the spine and its surrounding structures. This technology can accurately present important tissues such as the bone structure of the spine, intervertebral discs, intervertebral foramina and nerve roots, and observe and detect abnormalities. In the diagnosis of spinal diseases and injuries, spinal CT images play a key role, providing doctors with important anatomical information and diagnostic basis. With this technology, doctors can accurately diagnose and effectively treat spinal diseases, thereby improving the treatment effect of patients. With the intensification of the aging population trend and changes in lifestyle, the incidence of spinal fractures continues to rise, posing major challenges to the healthcare system. Accurate classification of spinal fracture types is crucial to formulating appropriate treatment plans and predicting patient treatment and recovery.

[0004] Deep learning-based techniques have achieved remarkable results in image classification, but in these methods, global features are only captured through the global average pooling layer (GAP) at the end of the network, without considering the detailed relationship and complexity between features. Covariance pooling is a technique used for feature extraction in deep learning. It not only focuses on the features within a single channel, but also captures the correlation between features by calculating the covariance between different channels. This method considers the feature distribution within a single channel while also utilizing the statistical relationship between multiple channels, which can provide a richer feature representation. However, the existing covariance pooling only captures the correlation between different channels in the high-order feature space, without considering the relationship between features in spatial layout. When processing complex medical images, the classification accuracy is poor. Summary of the invention

[0005] In response to the above-mentioned defects, the present invention proposes a spinal CT image classification method and system based on bidirectional covariance. By modeling channel and spatial features, the model can better understand the intrinsic structure and complexity of the data, combine channel correlation and spatial layout, and improve the classification accuracy of spinal CT images.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] The first aspect of the present disclosure provides a spine CT image classification method based on bidirectional covariance, comprising the following steps:

[0008] Obtaining CT images of the spine;

[0009] The spinal CT image is input into the SBCFormer network for feature extraction, and the channel covariance and spatial covariance of the extracted features are calculated respectively to obtain the channel covariance feature representation and the spatial covariance feature representation;

[0010] Concatenate the channel covariance feature representation and the spatial covariance feature representation to obtain a concatenated feature representation;

[0011] Combine channel covariance feature representation, spatial covariance feature representation and concatenated feature representation for image classification prediction;

[0012] Output the classification results of spine CT images.

[0013] As a further implementation method, before calculating the channel covariance of the extracted features, the feature dimensions of the extracted features are reshaped to obtain the extracted features of the reshaped feature dimensions.

[0014] As a further implementation, the extracted features of the reshaped feature dimensions are transposed before computing the spatial covariance.

[0015] As a further implementation method, before concatenating the channel covariance feature representation and the space covariance feature representation, a matrix square root normalization method is used to normalize the channel covariance matrix and the space covariance matrix.

[0016] As a further implementation method, the channel covariance feature representation, the spatial covariance feature representation and the spliced ​​feature representation are combined to perform image classification prediction, specifically:

[0017] An adaptive weight network is set up and six learnable parameters are introduced. The classification prediction results of the three features are fused in a weighted manner through these six learnable parameters.

[0018] As a further implementation, the Adam optimizer with a cosine learning rate adjustment strategy with 5 epochs warm-up is used to optimize the learnable parameters of the network.

[0019] As a further implementation method, the acquired spine CT image is also preprocessed, specifically: the acquired spine CT image is processed using data enhancement technology.

[0020] The second aspect of the present disclosure provides a spine CT image classification system based on bidirectional covariance, comprising:

[0021] The data acquisition module is configured to: acquire a spine CT image;

[0022] The bidirectional covariance calculation module is configured to: input the spine CT image into the SBCFormer network for feature extraction, respectively calculate the channel covariance and spatial covariance of the extracted features, and obtain the channel covariance feature representation and the spatial covariance feature representation;

[0023] The feature concatenation module is configured to: concatenate the channel covariance feature representation and the spatial covariance feature representation to obtain a concatenated feature representation;

[0024] A joint prediction module is configured to: perform image classification prediction by combining channel covariance feature representation, spatial covariance feature representation and concatenated feature representation;

[0025] The output module is configured to: output the classification results of the spine CT image.

[0026] The third aspect of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the spine CT image classification method based on bidirectional covariance described in the first aspect of the present disclosure are implemented.

[0027] The fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a spinal CT image classification method based on bidirectional covariance described in the first aspect of the present disclosure are implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The spinal CT image classification method based on bidirectional covariance proposed in the present disclosure takes into account the long-distance dependence and second-order statistical modeling problems that were not covered in the previous classification of spinal CT image fractures using traditional convolutional network methods. SBCFormer is used as the backbone, and bidirectional covariance pooling operations are performed on the features extracted by SBCFormer. The feature expression is fully learned, and the geometric characteristics of the covariance matrix space are utilized, which provides stronger information expression capabilities compared to first-order statistical modeling. The detailed relationship and complexity between features are captured along the two dimensions of channel and feature in the high-order feature space. After that, the spatial covariance features and channel covariance features are predicted separately, and the category prediction is performed on the splicing of the two features at the same time, which assists the model in enhancing the comprehensive understanding of channel and spatial features. In addition, a weight adaptive network is set up to dynamically adjust the weight ratio of the three category predictions, providing the model with greater flexibility and adaptability, thereby achieving higher accuracy in the final category prediction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0031] Figure 1 is a flow chart of a spine CT image classification method based on bidirectional covariance disclosed in the present invention;

[0032] Figure 2 It is a framework diagram of the spine CT image classification method based on bidirectional covariance disclosed in the present invention;

[0033] Figure 3 It is a two-way covariance pooling calculation flow chart disclosed in the present invention. DETAILED DESCRIPTION

[0034] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0036] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0037] Embodiment 1

[0038] like Figure 1-2 As shown, the first embodiment of the present disclosure provides a spine CT image classification method based on bidirectional covariance, comprising the following steps:

[0039] S1 obtains spine CT images;

[0040] S2 inputs the spine CT image into the SBCFormer network for feature extraction, calculates the channel covariance and spatial covariance of the extracted features, and obtains the channel covariance feature representation and the spatial covariance feature representation;

[0041] S3 performs a splicing operation on the channel covariance feature representation and the spatial covariance feature representation to obtain a spliced ​​feature representation;

[0042] S4 combines channel covariance feature representation, spatial covariance feature representation and concatenated feature representation for image classification prediction;

[0043] S5 outputs the spine CT image classification result.

[0044] Among them, S1 obtains the spine CT image specifically as follows:

[0045] A publicly available spine CT image dataset was collected, which contains 3800 spine CT images, including 1900 spine fracture CT images and 1900 normal spine CT images. The dataset division in the present invention is shown in Table 1.

[0046] Table 1 Number of training and testing samples of spine CT images

[0047]

[0048] The acquired spinal CT images are preprocessed and data augmentation techniques are used to increase the diversity of data and improve the robustness of the model. In this disclosure, three data augmentation methods are used: random horizontal flipping of images, random vertical flipping of images, and random rotation. The input size of the image is adjusted to Input into the backbone to extract features, and record the input as .

[0049] S2 is as follows: the preprocessed spine CT image is input into the SBCFormer network for feature extraction. Here, the SBCFormer-XS network is used as the feature extraction backbone network, and the backbone is trimmed after the last SBCFormer layer, that is, the features extracted by the SBCFormer backbone are used as the input of the bidirectional covariance pooling. The feature extraction process can be expressed using the following formula:

[0050]

[0051] In the formula, represents the input image, represents the features extracted by the backbone network, Indicates the backbone network. , where B is the batch size, N=288 represents the number of channels, and H=W=7 represents the height and width respectively.

[0052] The bidirectional covariance pooling disclosed in this invention is divided into two parts, such as Figure 3 As shown in Figure 1, the covariance calculation related to the channel and the covariance calculation related to the spatial features are described below. It should be noted that for the convenience of explanation, we ignore the batch size B here.

[0053] Before computing the channel covariance, a reshape operation is first used to The feature dimensions are reshaped to adjust the features to the dimensions required to calculate the channel features. The calculation process of the reshaping operation is as follows:

[0054]

[0055] In the formula, by reshaping (in the formula with Indicates) operation, The dimension is reshaped, and the reshaped features are recorded as In the specific implementation of this embodiment, N=288, M=49.

[0056] Calculate the channel covariance matrix:

[0057]

[0058]

[0059] In the formula, Indicates a The all-one matrix, express The identity matrix of for After completing the second-order pooling calculation, the channel features .

[0060] The dynamic covariance pooling mechanism is introduced on the calculated channel covariance matrix to make the pooling process more flexible and adaptive. The specific calculation process is as follows:

[0061]

[0062] In the formula, is a nonlinear covariance pooling function, which uses a fully connected layer and an activation function to perform a nonlinear transformation on the covariance matrix to enhance the feature representation capability. The processed features are recorded as .

[0063] However, since the covariance matrix may contain extremely large or extremely small eigenvalues, this will lead to numerical instability during model training. Therefore, we perform matrix power normalization on the covariance matrix to adjust the scale of the eigenvalues ​​so that all eigenvalues ​​are within a reasonable range, thereby improving the stability and efficiency of the calculation.

[0064] We calculate the covariance matrix by Normalize the covariance matrix by raising it to the power of the square root of the covariance matrix. By solving the square root of the covariance matrix, we can amplify the smaller values ​​and appropriately suppress the larger values, which helps to increase the distinction between different features. This increase in distinction is particularly important for improving model performance. In addition, this can maintain the positive definiteness of the covariance matrix, which is an important mathematical property for many machine learning and optimization algorithms. Positive definiteness ensures the stability and convergence of the optimization process.

[0065] However, in the process of matrix square root normalization, since it relies on eigendecomposition or singular value decomposition, the support for calculating the square root of the matrix is ​​limited on the GPU. Therefore, the Newton-Schultz iteration method is used to iteratively calculate the square root of the matrix. The formula is expressed as:

[0066]

[0067]

[0068] Given and ,for , is the identity matrix. After a certain number of iterative calculations (set to three), Approximately converges to We connect its upper triangular entries to form the final feature representation of the channel covariance and submit it to the subsequent classifier. Its final representation is recorded as , whose size is In the specific implementation, is a vector of length 41616.

[0069] Before calculating the feature space covariance, we first Perform feature transposition operation to meet the needs of spatial covariance matrix calculation. The calculation process of the transposition operation is as follows:

[0070]

[0071] In the formula, by converting (in the formula Indicates) operation, The dimension is reshaped, and the reshaped features are recorded as , in the specific implementation, M=49,N=288.

[0072] Calculate the spatial covariance matrix, the formula is:

[0073]

[0074]

[0075] in, Indicates a The all-one matrix, express The identity matrix of for Similarly, the dynamic covariance mechanism is introduced. After completing the second-order pooling calculation, the channel features , and then use the matrix square root normalization method to normalize the covariance matrix, the formula is:

[0076]

[0077]

[0078] Given and ,for , is the identity matrix. After a certain number of iterative calculations (set to three), Approximately converges to We connect its upper triangular entries to form the final feature representation of the channel covariance and submit it to the subsequent classifier. Its final representation is recorded as , whose size is In the specific implementation, is a vector of length 1225.

[0079] S3 is specifically:

[0080] After calculating the spatial covariance and channel covariance, the two features are concatenated in the feature dimension. The calculation process is as follows:

[0081]

[0082] In the formula, by splicing (in the formula, The final covariance representation of the channel and space is represented by and Splicing to get representation In the specific implementation, is a vector of length 42841.

[0083] S4 is specifically:

[0084] The image classification prediction is performed by combining the channel covariance feature representation, the spatial covariance feature representation, and the concatenated feature representation. The process of joint prediction is as follows:

[0085]

[0086] in represents the weight of the classification layer, which is learned by the adaptive network, and denote the final covariance representation of channel and space respectively, It is the feature representation of the concatenation of the two. Represents learnable parameters To further adjust its contribution to the final classification decision, we introduce an adaptive weight generation network to generate adaptive weight parameters. , , After obtaining these three parameters, they are weighted and fused with the classification results represented by each feature. All weight parameters and , , These are all learnable parameters, which are continuously optimized through the back-propagation algorithm during the training process.

[0087] In the specific implementation, the adaptive network consists of two fully connected layers and a The layers are composed as follows:

[0088]

[0089]

[0090]

[0091] The features after splicing As input, and are the weight matrices of the two fully connected layers, is matrix multiplication, is the activation function, The layer normalizes the output of the fully connected layer into a probability distribution, thereby obtaining three weights whose sum is guaranteed to be 1, thus ensuring that they can be reasonably assigned to different feature representations.

[0092] The network parameters are updated through back-propagation for learning. During the network learning process, the initial learning rate is set to 5e-3 and the batch size is 32. The network is learned within 50 epochs, and the Adam optimizer with a cosine learning rate adjustment strategy with 5 epochs of warm-up is used to optimize the learnable parameters of the network.

[0093] After each training epoch, the validation set was used for verification, and the effect of the model was evaluated by the accuracy. Finally, the method described in the present invention achieved a recognition accuracy of 98.86% on the used dataset.

[0094] The bidirectional covariance-based spinal CT image classification method disclosed in the present invention can also be used in conjunction with other backbone networks, including other advanced convolutional neural network models or Transformer models, or some more advanced high-order methods can be used to further improve the method described in the present invention.

[0095] Embodiment 2

[0096] The second embodiment of the present disclosure provides a spine CT image classification system based on bidirectional covariance, including:

[0097] The data acquisition module is configured to: acquire a spine CT image;

[0098] The bidirectional covariance calculation module is configured to: input the spine CT image into the SBCFormer network for feature extraction, respectively calculate the channel covariance and spatial covariance of the extracted features, and obtain the channel covariance feature representation and the spatial covariance feature representation;

[0099] The feature concatenation module is configured to: concatenate the channel covariance feature representation and the spatial covariance feature representation to obtain a concatenated feature representation;

[0100] A joint prediction module is configured to: perform image classification prediction by combining channel covariance feature representation, spatial covariance feature representation and concatenated feature representation;

[0101] The output module is configured to: output the classification results of the spine CT image.

[0102] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0103] Embodiment 3

[0104] Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the spine CT image classification method based on bidirectional covariance as described in Embodiment 1 of the present disclosure are implemented.

[0105] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0106] Embodiment 4

[0107] Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a spinal CT image classification method based on bidirectional covariance as described in Embodiment 1 of the present disclosure are implemented.

[0108] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0109] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A spine CT image classification method based on bidirectional covariance, It is characterized in that The following steps are involved: Obtaining CT images of the spine; The spinal CT image is input into the SBCFormer network for feature extraction, and the channel covariance and spatial covariance of the extracted features are calculated respectively to obtain the channel covariance feature representation and the spatial covariance feature representation; Before calculating the channel covariance of the extracted features, a feature dimension of the extracted features is reshaped to obtain extracted features of the reshaped feature dimension; before calculating the spatial covariance, a transposition operation is performed on the extracted features of the reshaped feature dimension; The channel covariance feature representation and the space covariance feature representation are concatenated to obtain a concatenated feature representation; before the channel covariance feature representation and the space covariance feature representation are concatenated, the channel covariance matrix and the space covariance matrix are normalized using a matrix square root normalization method; The image classification prediction is performed by combining the channel covariance feature representation, the spatial covariance feature representation and the concatenated feature representation; the process of joint prediction is as follows: in represents the weight of the classification layer, which is learned by the adaptive network, and denote the final covariance representation of channel and space respectively, It is the feature representation of the concatenation of the two; Represents the exponent of the learnable parameter logit_scale to further adjust its contribution in the final classification decision; introduces an adaptive weight generation network to generate adaptive weight parameters , , After obtaining these three parameters, they are weighted and fused with the classification results represented by each feature; all weight parameters and , , These are all learnable parameters, which are continuously optimized by the back-propagation algorithm during the training process; Furthermore, the Adam optimizer with a cosine learning rate adjustment strategy with 5 epochs of warm-up is used to optimize the learnable parameters of the network; Output the classification results of spine CT images.

2. A spine CT image classification method based on bidirectional covariance as claimed in claim 1, It is characterized in that The method also includes preprocessing the acquired spine CT image, specifically, processing the acquired spine CT image using data enhancement technology.

3. A spine CT image classification system based on bidirectional covariance, It is characterized in that include: The data acquisition module is configured to: acquire a spine CT image; The bidirectional covariance calculation module is configured to: input the spinal CT image into the SBCFormer network for feature extraction, respectively calculate the channel covariance and spatial covariance of the extracted features, and obtain the channel covariance feature representation and the spatial covariance feature representation; before calculating the channel covariance of the extracted features, reshape the feature dimension of the extracted features to obtain the extracted features of the reshaped feature dimension; before calculating the spatial covariance, transpose the extracted features of the reshaped feature dimension; The feature concatenation module is configured to: concatenate the channel covariance feature representation and the space covariance feature representation to obtain a concatenated feature representation; before concatenating the channel covariance feature representation and the space covariance feature representation, normalize the channel covariance matrix and the space covariance matrix using a matrix square root normalization method; The joint prediction module is configured to perform image classification prediction by combining channel covariance feature representation, spatial covariance feature representation, and concatenated feature representation. The process of joint prediction is as follows: in represents the weight of the classification layer, which is learned by the adaptive network, and denote the final covariance representation of channel and space respectively, It is the feature representation of the concatenation of the two; Represents the exponent of the learnable parameter logit_scale to further adjust its contribution in the final classification decision; introduces an adaptive weight generation network to generate adaptive weight parameters , , After obtaining these three parameters, they are weighted and fused with the classification results represented by each feature; all weight parameters and , , These are all learnable parameters, which are continuously optimized by the back-propagation algorithm during the training process; Furthermore, the Adam optimizer with a cosine learning rate adjustment strategy with 5 epochs of warm-up is used to optimize the learnable parameters of the network; The output module is configured to: output the classification results of the spine CT image.

4. A medium having a program stored thereon, It is characterized in that When the program is executed by a processor, the steps in the spine CT image classification method based on bidirectional covariance as described in any one of claims 1 and 2 are implemented.

5. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps in the spine CT image classification method based on bidirectional covariance as described in any one of claims 1 and 2 are implemented.