A brain tissue three-dimensional MRI image segmentation network, a storage medium and a processor thereof
By employing a sparse global attention mechanism and a multi-scale feature fusion method, the problem of global information neglect in existing technologies is solved, achieving efficient three-dimensional MRI image segmentation of brain tissue and improving segmentation accuracy and computational efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
- Filing Date
- 2024-03-08
- Publication Date
- 2026-07-14
AI Technical Summary
Existing image segmentation networks ignore global information during the encoding process, which makes it impossible for the encoder to effectively fuse multi-scale information and results in a large computational load.
We employ a sparse global attention mechanism and a multi-scale feature fusion method. By using a sparse global attention module to perform multi-scale queries in the encoder, we reduce computational cost and improve feature extraction efficiency. In the decoder, we use graph convolution and deformable attention mechanism to enhance feature information.
It improves the accuracy and efficiency of three-dimensional MRI image segmentation of brain tissue, reduces computational resource consumption, and enhances the effectiveness and accuracy of feature extraction.
Smart Images
Figure CN118072142B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer image processing technology, and in particular relates to a three-dimensional MRI image segmentation network for brain tissue, its storage medium, and processor. Background Technology
[0002] Existing image segmentation networks generally build their encoders based on 3DuNet. 3DuNet's encoder is typically a standard convolutional neural network (CNN). CNNs can only focus on information surrounding the target pixel, lacking an understanding of global information. To enable the encoder to focus on more information, some researchers have used context transformer blocks (COTs) to build encoders. This approach obtains more contextual information compared to standard CNNs. However, this type of encoder uses k*k*k encoding, which essentially changes the key-value lookup in the attention mechanism from scattered across all pixels to a lookup within k steps. This results in the continued neglect of global information during the encoding process. Existing technologies have shortcomings. Summary of the Invention
[0003] The purpose of this application is to provide a three-dimensional MRI image segmentation network for brain tissue, its storage medium and processor, which aims to solve the technical problem of not only being able to fuse queries from the surrounding k steps, but also being able to perform multi-size queries in the same encoder without significantly increasing the computational load through a sparse global attention mechanism.
[0004] On one hand, this application provides a three-dimensional MRI image segmentation network for brain tissue; including a sequentially connected input part, encoder part, skip connection part, decoder part and output part; the encoder part is composed of four sets of coding units connected in series, which output feature signals of different dimensions respectively; the decoder part is composed of four decoders connected in series.
[0005] The input data is input from the input section, passes through the first encoding unit, the second encoding unit, the third encoding unit and the fourth encoding unit of the encoder section in sequence, and is output to the decoder section by the jump connection section. After passing through the first decoder, the second decoder, the third decoder and the fourth decoder of the decoder section in sequence, it is output as output data by the output section.
[0006] The skip connection section simultaneously connects the third encoding unit to the first decoder, the second encoding unit to the second decoder, and the first encoding unit to the third decoder; the input section is simultaneously connected to the fourth decoder; so that the feature signals of different dimensions are spliced together in the decoder to update the feature value of the same pixel;
[0007] The encoding unit includes a downsampling layer as input and an encoder as output; the encoder includes two parallel encoding paths; the encoding path is composed of layer normalization, sparse global attention module, layer normalization and multilayer perceptron connected in sequence.
[0008] On the other hand, this application provides a storage medium storing program files capable of implementing the brain tissue three-dimensional MRI image segmentation network described in any of the above claims.
[0009] On the other hand, this application provides a processor for running a program, wherein the program executes the brain tissue three-dimensional MRI image segmentation network described in any of the above claims.
[0010] To fully leverage image information and further improve the accuracy of brain tissue segmentation, this application designs a brain tissue segmentation method based on multi-scale feature fusion and graph convolutional attention mechanisms. Specifically, the encoder is optimized by using a sparse global attention module. This ensures that the encoder does not calculate correlations with every single pixel, but rather with corresponding pixels along the central axis and diagonal in a three-dimensional space, including the pixel itself. Once the pixel distances for correlation calculation are selected, the sparse global attention can be computed. This approach not only focuses on information at multiple scales but also effectively reduces computational cost, making the features extracted by the encoder more efficient. Attached Figure Description
[0011] Figure 1 This is a schematic diagram of the structure of the three-dimensional MRI image segmentation network for brain tissue provided in Embodiment 1 of this application;
[0012] Figure 2 This is a schematic diagram of the optimized structure of the brain tissue three-dimensional MRI image segmentation network of this application;
[0013] Figure 3 This is a schematic diagram of the encoder structure in the brain tissue three-dimensional MRI image segmentation network of this application;
[0014] Figure 4 This is a schematic diagram of the sparse global attention module in the brain tissue three-dimensional MRI image segmentation network of this application;
[0015] Figure 5 This is a schematic diagram illustrating the use of different preset distance values in the sparse attention module of this application;
[0016] Figure 6 This is a schematic diagram of the graph convolution module in the brain tissue three-dimensional MRI image segmentation network of this application;
[0017] Figure 7This is a schematic diagram of the deformable attention gate in the brain tissue three-dimensional MRI image segmentation network of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0019] The specific implementation of this application will be described in detail below with reference to specific embodiments:
[0020] Example 1:
[0021] Figure 1 The structural composition of the three-dimensional MRI image segmentation network for brain tissue provided in Embodiment 1 of this application is illustrated. For ease of explanation, only the parts relevant to the embodiments of this application are shown, and are described in detail below:
[0022] On the one hand, as attached Figure 1-4 As shown, this application provides a three-dimensional MRI image segmentation network for brain tissue; it includes a sequentially connected input part, an encoder part, a skip connection part, a decoder part, and an output part; the encoder part is composed of four sets of coding units connected in series, which output feature signals of different dimensions respectively; the decoder part is composed of four decoders connected in series.
[0023] From the perspective of data flow, the input data is input from the input section, passes through the first encoding unit, the second encoding unit, the third encoding unit and the fourth encoding unit of the encoder section in sequence, and is output to the decoder section by the jump connection section. It then passes through the first decoder, the second decoder, the third decoder and the fourth decoder of the decoder section in sequence, and is output as output data by the output section.
[0024] The skip connection section simultaneously connects the third coding unit to the first decoder, the second coding unit to the second decoder, and the first coding unit to the third decoder; the input section is also directly connected to the fourth decoder; this allows feature signals of different dimensions to be spliced together in the decoder to update the feature value of the same pixel, which is beneficial for the decoder to obtain effective features at multiple scales.
[0025] The encoding unit includes a downsampling layer as input and an encoder as output; the encoder includes two parallel encoding paths; the encoding path is composed of layer normalization, sparse global attention module, layer normalization and multilayer perceptron connected in sequence, and the dual encoding paths can more effectively acquire encoding information at different preset distances.
[0026] Furthermore, the sparse global attention module obtains the Query, Key, and Value values needed for computation from the input image data. First, it multiplies the Query and Key values, then adds them to the location information, and transforms this using Softmax to obtain the attention matrix. Finally, it multiplies the attention matrix with the Value values to obtain the sparse global attention result. This approach focuses on the global information of the target pixel, avoiding increased resource consumption while simultaneously increasing attention requirements.
[0027] Specifically, in the Transformer model, Query, Key, and Value are components of the attention mechanism, used to process the input sequence, calculate attention weights, and then generate the output sequence. Their meanings and functions are as follows:
[0028] 1. Query:
[0029] • Meaning: Queries are used to specify which parts of the input sequence to focus on.
[0030] • Attribute: Usually a representation of a certain position in the input sequence, obtained through linear transformation.
[0031] • Purpose: The Query value determines the focus of attention, i.e., which parts of the input sequence the model should focus on. By calculating the similarity between the Query and the Key, the contribution of each input position to the output is determined.
[0032] 2. Key:
[0033] • Meaning: The key is used to provide information about the input sequence so that the similarity between the query and the input sequence can be calculated.
[0034] • Attribute: This is also a representation of a certain position in the input sequence, which is also obtained through linear transformation.
[0035] • Purpose: The Key value is used to measure the degree of relevance between each position in the input sequence and the current query. It determines the importance of different positions so that different weights can be assigned to each position.
[0036] 3. Value:
[0037] • Meaning: The value contains specific information about each position in the input sequence.
[0038] • Attributes: Similar to Query and Key, they are obtained through linear transformation.
[0039] • Function: The Value is weighted according to the calculated weights to generate the final output. It determines the specific contribution of the input sequence and plays an important role in the formation of the output sequence.
[0040] In this application, as attached Figure 5 As shown, the Q (Query) value is point A in the graph, and K (Key) and V (Value) refer to the location of A+B.
[0041] This application calculates the similarity between the Query and Key, and then applies these similarities as weights to the corresponding Values. This leverages the Transformer's attention mechanism, enabling the model to dynamically focus on different parts of the input sequence, thereby better capturing long-range dependencies within the sequence.
[0042] Furthermore, the sparse global attention module of this application calculates the correlation between target pixels and their corresponding pixels along the axis and diagonal in the 3D environment based on a preset distance value; in the two parallel encoding paths, one uses a preset distance value of 1, and the other uses a preset distance value of 3. By setting the preset distance value, the global information corresponding to the target pixel can be flexibly selected.
[0043] Specifically, the above design optimizes the encoder using a sparse global attention module. This allows the encoder to calculate correlations not with every single pixel, but with corresponding pixels along the central axis and diagonal of the 3D space, including the pixel itself. Once the pixel distances for which correlations are to be calculated are selected, the sparse global attention can be computed. This approach not only considers information at multiple scales but also effectively reduces computational cost, making the features extracted by the encoder more efficient.
[0044] As attached Figure 2 As shown, the skip connection part uses a multi-scale deformable attention mechanism to connect the feature signals of different dimensions output by the four coding units to the decoder part. The deeper the feature layer (data dimension), the more advanced the features extracted by the encoder, and the less data there is, thus reducing the computational load. Processing the data output by these encoder units through a multi-scale deformable attention mechanism to improve accuracy is more cost-effective, as it can fully utilize the information at the skip connections and enhance the features at those connections.
[0045] At the same time, there is another reason not to process the newly input data with deformable attention mechanism. When the input data is directly input, no features are extracted. It is directly used for decoding, which is similar to the idea of residuals, which can reduce the overfitting of the model.
[0046] Furthermore, the multi-scale deformable attention mechanism adds the input data and positional encoding to obtain the Query value; applies a linear transformation to the Query value to obtain the offset and attention weights; simultaneously calculates the position of the center point at each scale to obtain the reference point; normalizes the reference point and offset, then adds them together to obtain the sampling position; applies a Softmax operation to the attention weights to obtain the second attention weight; then performs matrix dimension permutation (Transpose and view) operations on the second attention weight to obtain the third attention weight. This approach utilizes multi-layered information, enhancing the interactivity between layers and increasing the influence of long-distance information. It avoids the shortcomings of existing technologies that only focus on surrounding information.
[0047] Returning to the input data, the multi-scale deformable attention mechanism first uses a linear layer to obtain the value, and then splits the value according to the dimensions of different encoders to obtain different layers of features (feature_maps) namely Layer1, Layer2, Layer3 and Layer4.
[0048] Because each layer represents a different scale, linear interpolation is needed to map it to a specific pixel. The sampling location matrix (Sampling_locations) provides the specific scale coordinates, and then multiplies it with the feature_map to obtain the sampling value matrix (Sampling_value) for the sampling location matrix.
[0049] Then, the third attention weight and the sampling value are added, multiplied, and expanded to obtain the first output. A linear layer is then applied to the first output to obtain the second output, which serves as the output of the multi-scale deformable attention mechanism. This is beneficial for enhancing features at skip connections.
[0050] Furthermore, the first decoder, second decoder, third decoder, and fourth decoder are connected to an information enhancement module; the information enhancement module performs cascaded feature enhancement on the input information.
[0051] Specifically, the two input ports of the decoder involved in this application are different. It can be understood that one input is enhanced using multi-scale deformable attention, and the other input is enhanced using an information enhancement module. The decoder will perform a convolution operation after splicing.
[0052] Furthermore, the information augmentation module includes a graph convolutional module as input and a deformable attention gate as output. The information augmentation module utilizes a combination of graph convolutional (GConv) neural networks and deformable attention mechanisms (DAS) to improve performance and reduce the impact of decoding errors caused by directly using upsampling.
[0053] As attached Figure 6 As shown, the graph convolution module uses sequentially connected convolution kernels to process the input data, followed by convolutional layers, batch_norm layers, graph convolutional networks, convolutional layers, and batch_norm layers. The result is then added point-by-point to the data input to the graph convolution module. The convolutional kernel size of the convolutional layer is 11, the dilation factor is 1, and GELU activation is used.
[0054] As attached Figure 7 As shown, the deformable attention gate processes the input data using sequentially connected depthwise separable convolution, instance normalization, GELU activation, deformable convolution, layer normalization, and sigmoid activation, and then multiplies the input data of the deformable attention gate point by point to obtain the result.
[0055] Specifically, existing attention mechanisms are still limited to spatial and channel attention mechanisms. These methods use techniques such as aggregation, subsampling, and pooling to include specific attention computations, which in turn makes it difficult to provide dense attention. Existing technologies treat channel and spatial attention as independent processes, and therefore do not consider the information in the features holistically, which may lead to potential information loss.
[0056] The deformable attention gate used in this application combines the ability of layer features to provide contextual information with the ability of deformable convolution to focus image information, making the features more prominent.
[0057] Furthermore, a segmentation header is also included between the decoder section and the output section;
[0058] The segmentation head includes a convolutional layer at the output of each decoder and an aggregation layer connecting the outputs of all convolutional layers.
[0059] Furthermore, the segmentation head first uses a convolutional layer to reduce the number of channels to the number of segmentation categories. Then, for features where D, H, and W are less than 96, trilinear interpolation is performed to (96, 96, 96). Thus, the dimensions of the four outputs of the features are all (category, 96, 96, 96). Finally, an aggregation layer is used to average the feature values of the four channels. The final segmentation result is then output to the output section.
[0060] Furthermore, the convolutional layer connected to the first decoder is followed by an upsampling layer of 8x; the convolutional layer connected to the second decoder is followed by an upsampling layer of 4x; and the convolutional layer connected to the third decoder is followed by an upsampling layer of 2x.
[0061] Specifically, existing image segmentation networks typically use a concatenation method to process the decoder output. This approach lacks a process of upsampling and re-aggregating the signal, resulting in poor utilization of the features produced by each layer. The segmentation head proposed in this application, however, can better utilize the features of each layer based on an upsampling and re-aggregation process. Experiments have demonstrated that this improvement in the segmentation head contributes to higher image segmentation accuracy.
[0062] Meanwhile, the upsampling at different magnifications used in the segmentation head of this application can unify the feature dimension and avoid the difference in feature size between different layers, which would result in the final output feature size being different from the size of the original image.
[0063] In the image segmentation network of this application, sparse global attention is first used for optimization in the encoder, focusing on multi-scale information and effectively reducing the computational cost, making the features extracted by the encoder more efficient. Further improvements are made by using a multi-scale variable attention mechanism at skip connections to enhance the features at these connections. Simultaneously, the information augmentation module is improved by combining a graph convolutional (GConv) neural network and a deformable attention mechanism (DAS) to reduce the impact of decoding errors caused by directly using upsampling.
[0064] Example 2:
[0065] This application also provides a storage medium storing program files of a brain tissue three-dimensional MRI image segmentation network capable of implementing any of the above.
[0066] Example 3:
[0067] This application provides a processor for running a program, wherein the program executes a brain tissue three-dimensional MRI image segmentation network according to any of the above-mentioned methods.
[0068] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for segmenting three-dimensional MRI images of brain tissue, used for segmenting three-dimensional MRI images of brain tissue; the method includes a three-dimensional MRI image segmentation network for brain tissue, the three-dimensional MRI image segmentation network for brain tissue comprising a sequentially connected input part, an encoder part, a skip connection part, a decoder part, and an output part; characterized in that, The encoder section consists of four sets of encoding units connected in series, each outputting feature signals of different dimensions; the decoder section consists of four decoders connected in series. The input data is input from the input section, passes through the first encoding unit, the second encoding unit, the third encoding unit and the fourth encoding unit of the encoder section in sequence, and is output to the decoder section by the jump connection section. After passing through the first decoder, the second decoder, the third decoder and the fourth decoder of the decoder section in sequence, it is output as output data by the output section. The skip connection section simultaneously connects the third encoding unit to the first decoder, the second encoding unit to the second decoder, and the first encoding unit to the third decoder; the input section is simultaneously connected to the fourth decoder; so that the feature signals of different dimensions are spliced together in the decoder to update the feature value of the same pixel; The encoding unit includes a downsampling layer as input and an encoder as output; the encoder includes two parallel encoding paths; the encoding path is composed of layer normalization, sparse global attention module, layer normalization and multilayer perceptron connected in sequence.
2. The brain tissue three-dimensional MRI image segmentation method as described in claim 1, characterized in that, The sparse global attention module obtains the Query value, Key value, and Value value required for calculation from the input image data. First, it multiplies the Query value and the Key value, then adds them to the location information, and transforms them through Softmax to obtain the attention matrix. Finally, it performs matrix multiplication of the attention matrix and the Value value to obtain the sparse global attention result.
3. The brain tissue three-dimensional MRI image segmentation method as described in claim 2, characterized in that, The sparse global attention module calculates the correlation between the target pixel and its corresponding pixels on the axis and diagonal in the three-dimensional environment based on a preset distance value; in the two parallel encoding paths, one uses the preset distance value of 1 and the other uses the preset distance value of 3.
4. The brain tissue three-dimensional MRI image segmentation method as described in claim 2, characterized in that, The skip connection section uses a multi-scale deformable attention mechanism to input the feature signals of different dimensions output by the four coding units into the decoder section respectively, so as to make full use of the information at the skip connection and enhance the features at the skip connection.
5. The brain tissue three-dimensional MRI image segmentation method as described in claim 4, characterized in that, The multi-scale deformable attention mechanism adds the input data and position encoding to obtain the Query value; applies a linear transformation to the Query value to obtain the offset and attention weight; calculates the position of the center point at each scale to obtain the reference point; normalizes the reference point and the offset, then adds them together to obtain the sampling position; and applies a Softmax operation to the attention weight to obtain the second attention weight. Then, the matrix dimension permutation operation is performed on the second attention weights to obtain the third attention weights; Returning to the input data, the multi-scale deformable attention mechanism first uses a linear layer to obtain the value, and then the value is split according to the dimensions of different encoders to obtain features at different levels, namely Layer1, Layer2, Layer3 and Layer4. Because each layer represents a different scale, linear interpolation is needed to map to specific pixels. The sampling position matrix provides specific scale coordinates, which are then multiplied with the features of the different layers to obtain the sampling value matrix for the sampling position matrix. Then, the third attention weight and the sample value matrix obtained for the sampling position matrix are added, multiplied, and expanded to obtain the first output; then, a linear layer is applied to the first output to obtain the second output as the output result of the multi-scale deformable attention mechanism.
6. The brain tissue three-dimensional MRI image segmentation method as described in claim 1, characterized in that, The first decoder, the second decoder, the third decoder, and the fourth decoder are connected to an information enhancement module. The information enhancement module performs cascaded feature enhancement on the input information.
7. The brain tissue three-dimensional MRI image segmentation method as described in claim 6, characterized in that, The information enhancement module includes a graph convolution module as input and a deformable attention gate as output.
8. The brain tissue three-dimensional MRI image segmentation method as described in claim 7, characterized in that, The graph convolution module uses sequentially connected convolution kernels to process the input data, followed by a batch_norm layer, a graph convolutional network, and then adds the data from the input graph convolution module point by point to obtain the result. The convolution kernels of the convolutional layers are 1*1 in size, with an inflation factor of 1, and are activated using GELU.
9. The brain tissue three-dimensional MRI image segmentation method as described in claim 7, characterized in that, The deformable attention gate processes the input data using sequentially connected depthwise separable convolution, instance normalization, GELU activation, deformable convolution, layer normalization, and sigmoid activation, and then multiplies the input data of the deformable attention gate point by point to obtain the result.
10. The brain tissue three-dimensional MRI image segmentation method as described in claim 1, characterized in that, A segmentation header is also included between the decoder section and the output section; The segmentation head includes a convolutional layer disposed at the output of each decoder and an aggregation layer connecting the outputs of all the convolutional layers.
11. The brain tissue three-dimensional MRI image segmentation method as described in claim 10, characterized in that, The segmentation head first uses the convolutional layer to reduce the number of channels to the number of segmentation categories. Then, it performs trilinear interpolation on the feature outputs where D, H, and W are less than 96 to (96, 96, 96), so that the dimensions of the four outputs of the features are all (category, 96, 96, 96). Finally, the aggregation layer is used to average the feature values of the four channels; the final segmentation result is then output to the output section.
12. The brain tissue three-dimensional MRI image segmentation method as described in claim 10, characterized in that, The convolutional layer connected to the first decoder is followed by an upsampling layer of 8x; the convolutional layer connected to the second decoder is followed by an upsampling layer of 4x; and the convolutional layer connected to the third decoder is followed by an upsampling layer of 2x.
13. A computer storage medium, characterized in that, The storage medium stores a program file capable of implementing the brain tissue three-dimensional MRI image segmentation method according to any one of claims 1 to 12.
14. A processor, characterized in that, The processor is used to run a program, wherein the program executes the brain tissue three-dimensional MRI image segmentation method according to any one of claims 1 to 12.