Spine scoliosis ct segmentation method based on multi-view encoder and attention mechanism

Abstract

The present application relates to a spine scoliosis CT segmentation method based on a multi-view encoder and an attention mechanism, adopts a multi-view encoder method to extract features of CT images of different views to extract more complete three-dimensional spine structure features, and adopts a Transformer attention mechanism to learn long-range dependency relationships and global information of the extracted features, on the basis of a U-Net model, a multi-view encoder is constructed, and an attention mechanism is also introduced, an improved U-Net network based on a multi-view encoding and an attention mechanism is constructed, the improved U-Net network based on the multi-view encoding and the attention mechanism is trained, and a required segmentation network is obtained, and compared with the U-Net network, the method can extract more complete three-dimensional structure features, and the accuracy and robustness of segmentation are improved.

Description

Technical Field

[0001] This invention relates to a CT segmentation method for scoliosis based on a multi-view encoder and attention mechanism, belonging to the field of medical image analysis technology. Background Technology

[0002] my country has a large population of sedentary individuals and manual laborers. Prolonged sitting and excessive physical activity can lead to various spinal diseases, and in recent years, spinal diseases have shown a trend towards affecting younger people. Among those with scoliosis, adolescents account for approximately 75%-80% of all cases. Currently, the analysis of medical images is primarily performed by radiologists and attending physicians. However, the interpretation of medical images can vary subjectively depending on the attending physician. Furthermore, my country suffers from a severe shortage of medical resources; training qualified radiologists requires considerable time, and the time-consuming and laborious diagnostic process further burdens doctors. Delayed or incorrect diagnoses can miss the optimal time for patient treatment.

[0003] For clinicians, CT images provide more detailed information about the spinal structure than X-ray images, enabling them to accurately assess a patient's condition and make a diagnosis and treatment plan. However, manually segmenting the anatomical structures of interest on CT images of adolescents with idiopathic scoliosis is time-consuming, and the abnormal lateral curvature and torsion of the spine in these patients further complicates the segmentation process.

[0004] Current research on spine segmentation in CT images mainly focuses on using 2D and 3D convolutional neural networks. However, the mainstream network models currently perform poorly in segmenting CT images of the spine of adolescent idiopathic scoliosis patients due to the abnormal lateral curvature and torsion of the spine, easily leading to undersegmentation and missegmentation. Summary of the Invention

[0005] To overcome the shortcomings of existing research, this invention provides a CT segmentation method for scoliosis based on a multi-view encoder and attention mechanism. The method employs a multi-view encoder to extract features from CT images from different perspectives, thereby extracting more complete three-dimensional spinal structural features. Furthermore, a Transformer attention mechanism is used to learn long-range dependencies and global information from the extracted features. Experiments were conducted on spinal CT images of adolescent patients with idiopathic scoliosis, demonstrating that the proposed method outperforms mainstream network models in segmentation performance.

[0006] The specific steps of the CT segmentation method for scoliosis based on multi-view encoders and attention mechanisms are as follows:

[0007] Includes the encoding stage and the decoding stage:

[0008] The encoding stage includes: using three independent encoders, inputting the CT image into the three encoders from three perspectives: View 1, View 2 and View 3 respectively. The encoder corresponding to View 3 is used as the main encoder and is connected to the decoder through skip connections. The other two encoders are used as auxiliary encoders, and only the extracted high-level semantic features are concatenated with the high-level features extracted by the encoder corresponding to View 3. Then, the Transformer attention mechanism is used to learn the global context and long-range dependencies.

[0009] A patch containing multiple vertebrae is used as the input to the network. The patch is rotated and then fed into three independent encoders for feature extraction. Each encoder extracts key features that distinguish the vertebrae in a specific direction. The extracted features from different perspectives are then concatenated, and a 1×1×1 convolution is used to change the number of channels and adjust the dimension of the feature map to adapt to subsequent tasks or network structures.

[0010] The decoding stage includes: the number of decoding blocks in the decoder corresponds to the number of blocks in the encoder; the decoder upsamples the deep semantic features extracted by the encoder layer by layer to restore the size of the original input; the upsampled feature maps are concatenated with the low-level feature maps extracted from the corresponding encoder blocks; information is fused through skip connections; and then the concatenated features are sent to the next decoding block for upsampling, finally obtaining an output of the same size as the input.

[0011] The improved U-Net network based on multi-view encoding and attention mechanism is trained to obtain the desired segmentation network model.

[0012] The specific calculation process of the Transformer attention mechanism is shown in formula (1):

[0013]

[0014] Where x0 is the output of the multi-view encoder, L represents the index of the Transformer encoder layer, and Norm represents layer normalization. It is the output of MSA (Multi-head Attention), x l It is the result of mapping through an MLP (Multilayer Perceptron);

[0015] The formula for calculating MSA (multi-head attention) in the Transformer attention mechanism is as follows:

[0016]

[0017] Where Q, K, and V represent the query, key, and value matrices of the input features, respectively, and R refers to the relative position encoding matrix.

[0018] The Dice coefficient and ASSD are used to evaluate the segmentation network model:

[0019]

[0020]

[0021] Where X and Y represent the actual segmentation result and the predicted segmentation result, respectively, and S X and S Y The surfaces representing the true segmentation result and the predicted segmentation result are dist(x,y), where dist(x,y) is the Euclidean distance between x and y.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] This invention constructs a multi-view encoder and introduces an attention mechanism on the basis of the U-Net model, thus building an improved U-Net network based on multi-view encoding and attention mechanism. The improved U-Net network based on multi-view encoding and attention mechanism is trained to obtain the desired segmentation network. Compared with the U-Net network, the method of this invention, which combines multi-view encoding and attention mechanism, can extract more complete three-dimensional structural features and improve the accuracy and robustness of segmentation. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic diagram of a CT segmentation method for scoliosis based on multi-view encoders and attention mechanisms;

[0026] Figure 2 This is a schematic diagram of the multi-view data enhancement of the present invention;

[0027] Figure 3 This is a structural diagram of the Transformer attention mechanism of the present invention;

[0028] Figure 4 This is a comparison diagram of the segmentation results of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Reference Figure 1 The specific steps of the CT segmentation method for scoliosis based on multi-view encoders and attention mechanisms are as follows:

[0031] Includes the encoding stage and the decoding stage:

[0032] The encoding stage includes: using three independent encoders, inputting the CT image into the three encoders from three perspectives: View 1, View 2 and View 3 respectively. The encoder corresponding to View 3 is used as the main encoder and is connected to the decoder through skip connections. The other two encoders are used as auxiliary encoders, and only the extracted high-level semantic features are concatenated with the high-level features extracted by the encoder corresponding to View 3. Then, the Transformer attention mechanism is used to learn the global context and long-range dependencies.

[0033] A patch containing multiple vertebrae is used as input to the network, and the patch is rotated, such as... Figure 2 As shown, the features are then input into three independent encoders for feature extraction. Each encoder extracts key features that distinguish vertebrae in a specific direction. The features extracted from different perspectives are then concatenated, and a 1×1×1 convolution is used to change the number of channels and adjust the dimension of the feature map to adapt to subsequent tasks or network structures. Finally, this method can obtain features of scoliosis images from different directions and combine the information from these different views.

[0034] The decoding stage includes: the number of decoding blocks in the decoder corresponds to the number of blocks in the encoder; the decoder upsamples the deep semantic features extracted by the encoder layer by layer to restore the size of the original input; the upsampled feature maps are concatenated with the low-level feature maps extracted from the corresponding encoder blocks; information is fused through skip connections; and then the concatenated features are sent to the next decoding block for upsampling, finally obtaining an output of the same size as the input.

[0035] The improved U-Net network based on multi-view encoding and attention mechanism is trained to obtain the desired segmentation network model.

[0036] To extract global information and long-range dependencies from images, we introduce a Transformer attention mechanism, such as... Figure 3 As shown, through the multi-head self-attention mechanism, the Transformer module can perform global interactions across the entire input sequence, thereby capturing long-range dependencies. Simultaneously, multiple attention heads are used to calculate different attention weights, enabling the model to learn different features at different locations in the input sequence, particularly local and global features. The specific calculation process of the Transformer attention mechanism is shown in formula (1):

[0037]

[0038] Where x0 is the output of the multi-view encoder, L represents the index of the Transformer encoder layer, and Norm represents layer normalization. It is the output of MSA (Multi-head Attention), x l It is the result of mapping through an MLP (Multilayer Perceptron);

[0039] The formula for calculating MSA (multi-head attention) in the Transformer attention mechanism is as follows:

[0040]

[0041] Where Q, K, and V represent the query, key, and value matrices of the input features, respectively, and R refers to the relative position encoding matrix.

[0042] During training, the loss function used is a combination of Dice loss and cross-entropy loss. It can be calculated using a voxel-based approach.

[0043]

[0044] Where I represents the number of voxels, J represents the number of classes, and Y represents the number of voxels. i,j and G i,j Let i represent the probability output of class j at voxel i and the corresponding one-hot encoded true label, respectively.

[0045] To evaluate the performance of the segmentation model, the Dice coefficient and ASSD are used to evaluate the segmentation network model:

[0046]

[0047]

[0048] Where X and Y represent the actual segmentation result and the predicted segmentation result, respectively, and S X and S YThe surfaces representing the true segmentation result and the predicted segmentation result are dist(x,y), where dist(x,y) is the Euclidean distance between x and y.

[0049] On a CT dataset of adolescent idiopathic scoliosis patients, the segmentation performance of different networks was compared, and the segmentation results are as follows: Figure 4 As shown, (a) is the original image, (b) is the ground truth label, (c) is the segmentation result of the U-Net model, (d) is the segmentation result of the Attention U-Net model, (e) is the segmentation result of the V-Net model, and (f) is the segmentation result of our proposed method. Figure 4 Comparison of segmentation results from different networks shows that the method proposed in this invention achieves the best segmentation effect on the dataset, with fewer segmentation errors, and the problem of insufficient segmentation is alleviated to some extent.

[0050] To fully demonstrate the effectiveness of the multi-view encoder and attention mechanism used in this invention, the two evaluation metrics mentioned above were used for testing. The U-Net model with the Transformer module was denoted as Network_1, and the U-Net model with the multi-view encoder was denoted as Network_2. The comparison results are shown in Table 1. Table 1 shows that both the multi-view encoder and the attention mechanism improve the model's segmentation performance, proving the effectiveness of the two modules.

[0051] Table 1 Comparison of segmentation performance of different networks (Metrics: Dice (%) / ASSD (mm))

[0052]

[0053] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A CT segmentation method for scoliosis based on multi-view encoders and attention mechanisms, characterized in that: Includes the encoding stage and the decoding stage: The encoding stage includes: using three independent encoders, inputting the input CT image into the three encoders from three perspectives: View 1, View 2 and View 3 respectively. The encoder corresponding to View 3 is used as the main encoder and is connected to the decoder through skip connections. The other two encoders are used as auxiliary encoders, which only concatenate the extracted high-level semantic features with the high-level features extracted by the encoder corresponding to View 3. Then, the Transformer attention mechanism is used to learn the global context and long-range dependencies. A patch containing multiple vertebrae is used as the input to the network. The patch is rotated and then fed into three independent encoders for feature extraction. Each encoder extracts key features that distinguish the vertebrae in a specific direction. The extracted features from different perspectives are then concatenated, and a 1×1×1 convolution is used to change the number of channels and adjust the dimension of the feature map to adapt to subsequent tasks or network structures. The decoding stage includes: the number of decoding blocks in the decoder corresponds to the number of blocks in the encoder; the decoder upsamples the deep semantic features extracted by the encoder layer by layer to restore the size of the original input; the upsampled feature maps are concatenated with the low-level feature maps extracted from the corresponding encoder blocks; information is fused through skip connections; and then the concatenated features are sent to the next decoding block for upsampling, finally obtaining an output of the same size as the input. The improved U-Net network based on multi-view encoding and attention mechanism is trained to obtain the desired segmentation network model.

2. The CT segmentation method for scoliosis based on multi-view encoder and attention mechanism according to claim 1, characterized in that: The specific calculation process of the Transformer attention mechanism is shown in formula (1): Where x0 is the output of the multi-view encoder, L represents the index of the Transformer encoder layer, and Norm represents layer normalization. This is the output of MSA, x l It is the result of MLP mapping; The formula for calculating MSA in the Transformer attention mechanism is as follows: Where Q, K, and V represent the query, key, and value matrices of the input features, respectively, and R refers to the relative position encoding matrix.

3. The CT segmentation method for scoliosis based on a multi-view encoder and attention mechanism according to claim 1 or 2, characterized in that: The Dice coefficient and ASSD are used to evaluate the segmentation network model. Where X and Y represent the actual segmentation result and the predicted segmentation result, respectively, and S X and S Y The surfaces representing the true segmentation result and the predicted segmentation result are dist(x,y), where dist(x,y) is the Euclidean distance between x and y.

Images