Multi-attention fusion weak supervision pathological segmentation method for brain MRI image
Through the multi-attention fusion weak supervision method, combined with hybrid convolution and TransformerFocus module, the accuracy problem of traditional MRI image segmentation method in the absence of complex structures and data is solved, and efficient brain pathological segmentation is achieved, reducing labeling dependence and improve segmentation accuracy.
Patent Information
- Application Number
- CN202410753711.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-12
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional MRI image segmentation method is not effective when processing complex, multi-layered structures or small grayscale changes. The supervised method based on deep learning requires a large amount of data labeling, which is time-consuming and laborious and involves data security issues, making it difficult to lack data. In the case, accurate brain pathological division can be achieved.
A multi-attention fusion weak supervision pathological segmentation method is proposed. By constructing a multi-attention fusion framework based on weak supervision, a multi-scale feature fusion module with hybrid convolution and a boundary attention module based on TransformerFocus are introduced, using pseudo-labels and unsupervised The segmentation network refines boundary information, reduces dependence on labels and improves segmentation accuracy.
It realizes reducing the time-consuming and labor-intensive labeling in brain MRI images, improves the segmentation accuracy of complex structures and irregular shape pathological areas, reduces the dependence on professionals, and achieves high-precision pathological segmentation in the absence of data.
Smart Images

Figure CN118918323A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field, and in particular to a multi-attention fusion weakly supervised pathology segmentation method for brain MRI images. Background Art
[0002] Traditional segmentation methods are usually based on the grayscale, texture and shape features of MRI images, and aim to accurately extract the boundaries and contours of the structures of interest with the help of computer vision technology. Therefore, traditional segmentation methods can achieve relatively good segmentation effects for images with clear structures and obvious grayscale differences, but when facing complex, multi-level structures or MRI images with small grayscale changes, the limitations of traditional methods gradually emerge, limiting their application in high-precision segmentation. At the same time, although the classic segmentation methods based on deep learning can achieve relatively accurate segmentation results for brain pathological areas in MRI images, such as FCN, UNet, etc., these methods are all based on supervised methods with complete image annotation. In practical applications, since the annotation process not only requires professional knowledge, but also consumes a lot of manpower and time, and even involves data security and patient privacy issues, a large amount of MRI image data cannot be fully annotated. Therefore, how to complete the accurate segmentation of brain pathological areas under the premise of lack of data has become the primary task that needs to be solved in this field. In addition, the irregular shape of the pathological area and the non-fixed location area, as well as the blurred boundary problem caused by noise and artifact interference, make it more difficult to accurately segment the pathological area.
[0003] To this end, a multi-attention fusion weakly supervised pathology segmentation method for brain MRI images was proposed. Summary of the invention
[0004] The purpose of the present invention is to provide a multi-attention fusion weakly supervised pathology segmentation method for brain MRI images to solve the problems raised in the above background technology.
[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a multi-attention fusion weakly supervised pathology segmentation method for brain MRI images, comprising the following steps: step one, constructing a brain MRI image pathology segmentation framework based on weakly supervised multi-attention fusion, and generating pathological area segmentation results through the brain MRI image pathology segmentation framework; step two, introducing a multi-scale feature fusion module based on hybrid convolution to capture the global information of the lesion area, and serially combining convolution kernels with different expansion rates through residual connection to improve the capture range of the receptive field and the ability to distinguish objects of different sizes and shapes; step three, proposing a boundary attention module based on TransformerFocus, by capturing the contextual information in the input sequence and forming a global dependency representation, introducing a set of learnable category labels, so that they constitute a similarity mapping with the spatial feature map and transfer it to the corresponding segmentation mask, so as to realize the refined connection between global information and edge perception information.
[0006] According to the above technical solution, the brain MRI image pathology segmentation framework in step 1 includes a pseudo-label generation technology based on class activation mapping, a boundary mapping technology based on unsupervised segmentation, and a boundary fitting technology based on dual-link images.
[0007] According to the above technical solution, the workflow of the brain MRI image pathology segmentation framework is as follows: input the brain MRI image to be segmented, and output the last layer of feature map through the Classification Backbone pre-trained with image-level labels in the upper half of the framework, and then generate the required pseudo-label (Pseudo Mask) through the Mask Generator module; the other way is to input the brain MRI image to be segmented into the unsupervised segmentation network (Unsupervised Segment Network, USN) based on the boundary mapping technology of unsupervised segmentation, and finally generate a boundary map (Edge Map); finally, the pseudo-label and boundary map generated above are jointly input into the boundary fusion module (BoundaryFit Moudule) composed of the boundary fitting technology based on dual-link images for boundary fitting, so as to further refine the boundary information and generate the final pathological area segmentation result.
[0008] According to the above technical solution, the multi-scale feature fusion module in step 2 is composed of three consecutive convolution modules and an attention module connected in series, and the convolution module is composed of hole convolution kernels with different expansion rates, and corresponding normalization functions (Norm) and activation functions (ReLU) connected in series.
[0009] According to the above technical solution, the workflow of the attention module is: decomposing the input two-dimensional feature map into information in two directions, namely horizontal coordinate information and vertical coordinate information, generating a pair of direction-aware feature maps, further processing the two feature maps, generating two attention maps, and the attention maps are used to help the model quickly focus on the key information in the input feature map.
[0010] According to the above technical solution, before the feature map is input into the boundary attention module, the input feature map is divided into small blocks (Patches) of size P×P. Each Patch is regarded as a Token. The dimension of the above Patch is stretched to the D dimension required by the TransformerFocus Encoder through a trainable linear projection layer (Linear Projection). Subsequently, the feature map suitable for the Transformer structure and the Token of its corresponding classification label are sent to the boundary attention module for the next step of calculation and processing.
[0011] According to the above technical solution, the boundary attention module based on TransformerFocus in step three is used to process the input feature sequence through layer normalization. The purpose of this step is to standardize the input data to ensure that the input sequence maintains a stable distribution during the transmission process in the module. The normalized sequence enters the multi-head attention mechanism for autocorrelation operation.
[0012] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: (1) In view of the complex structure of the pathological area in the brain MRI image, which leads to the time-consuming and labor-intensive labeling process and high reliance on the assistance of professionals, this paper proposes a semantic segmentation framework based on weakly supervised boundary refinement. The framework consists of a two-way structure. One way is to output feature maps through the pre-trained classification backbone, and generate corresponding pseudo labels through the class activation mapping method; the other way is to cluster similar pixels through the unsupervised segmentation network to generate an edge map that suppresses irrelevant details, and then further fuse the two to remove irrelevant information, thereby generating the final segmentation result, which greatly reduces the time-consuming and labor-intensive large-scale labeling and the reliance on the assistance of professionals;
[0013] (2) To address the intra-class heterogeneity problem caused by the random location and irregular shape of pathological areas in brain MRI images, this paper constructs a multi-scale feature fusion module based on hybrid convolution. By residually connecting convolution kernels with different dilation rates, the capture range of the receptive field and the distinguishability of objects of different sizes and shapes are improved without increasing the number of parameters and preventing "gridding" from occurring;
[0014] (3) In view of the complex structure of brain MRI images, the blurred boundaries of pathological areas, and the frequent interference of artifacts, this paper proposes a boundary attention module based on TransformerFocus. This method can capture the contextual information in the input sequence to form a global dependency representation. On this basis, a set of learnable category labels is introduced, which form a set of similarity mappings with the spatial feature map and transfer them to the corresponding segmentation mask, which enables the network to further refine the edge perception information based on the global information. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0016] Figure 1 It is a schematic diagram of the brain MRI image pathology segmentation framework based on weakly supervised multi-attention fusion of the present invention;
[0017] Figure 2 It is a schematic diagram of the structure of a multi-scale feature fusion module based on hybrid convolution of the present invention;
[0018] Figure 3 It is a schematic diagram of the structure of the coordinate attention block of the present invention;
[0019] Figure 4 It is a schematic diagram of the structure of the boundary attention module based on TransformerFocus of the present invention;
[0020] Figure 5 It is a schematic diagram of the TransformerFocus encoder structure of the present invention. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] See also Figure 1-5 The present invention provides a technical solution: a multi-attention fusion weakly supervised pathological segmentation method for brain MRI images, comprising the following steps: Step 1, constructing a brain MRI image pathological segmentation framework based on weakly supervised multi-attention fusion, and generating pathological region segmentation results through the brain MRI image pathological segmentation framework; the framework is composed of a two-way structure, one of which is a pre-trained classification trunk output feature map, and the corresponding pseudo label is generated through the class activation mapping method; the other is to aggregate similar pixels through an unsupervised segmentation network to generate an edge map that suppresses irrelevant details, and then further fuse the two to remove irrelevant information, thereby generating the final segmentation result, greatly reducing the time-consuming and labor-intensive problem of large-scale labeling and relying on professional assistance. Step 2, introducing a multi-scale feature fusion module based on hybrid convolution to capture the global information of the lesion area, and combining convolution kernels with different expansion rates in series through residual connections to improve the capture range of the receptive field and the ability to distinguish objects of different sizes and shapes; by residually connecting convolution kernels with different expansion rates, the capture range of the receptive field and the distinguishability of objects of different sizes and shapes are improved without increasing the number of parameters and preventing "rasterization". Step 3, a boundary attention module based on TransformerFocus is proposed. On the basis of capturing the contextual information in the input sequence and forming a global dependency representation, a set of learnable category tags is introduced to form a similarity mapping with the spatial feature map and transfer it to the corresponding segmentation mask, thus realizing the refined connection between global information and edge perception information; this method can capture the contextual information in the input sequence and form a global dependency representation. On this basis, a set of learnable category tags is also introduced, which forms a set of similarity mapping with the spatial feature map and transfers it to the corresponding segmentation mask, which enables the network to further refine the edge perception information on the basis of connecting the global information.
[0023] Specifically, most of the previous weakly supervised methods are implemented based on the class activation mapping method alone. However, this type of method has extremely high requirements on the feature extraction ability of the backbone network. In other words, if the image background is complex and the feature texture is not obvious enough, the segmentation effect of this type of method is often unsatisfactory. Therefore, in order to obtain a more detailed pathological area, this paper proposes a brain MRI image pathological segmentation framework based on weakly supervised multi-attention fusion. Figure 1 As shown in the figure, this is the overall structure of the semantic segmentation framework based on weakly supervised boundary refinement proposed in this paper. The framework can be divided into three parts: pseudo-label generation technology based on class activation mapping ( Figure 1The upper and middle branches: Classification Backbone + MaskGenerator, boundary mapping technology based on unsupervised segmentation, and boundary fitting technology based on dual-link images. The processing flow of this framework is as follows: First, the brain MRI image to be segmented is input, and the last layer of feature maps output by the Classification Backbone in the upper branch of the framework that has been pre-trained with image-level labels are used, and then the required pseudo labels (Pseudo Mask) are generated through the Mask Generator module; the other branch inputs the brain MRI image to be segmented into the unsupervised segmentation network (Unsupervised Segment Network, USN) based on the boundary mapping technology of unsupervised segmentation. This module removes irrelevant details in the input image by clustering similar pixels together, and finally generates a boundary map (Edge Map); finally, the pseudo labels and boundary maps generated above are jointly input into the boundary fusion module (BoundaryFit Moudule) composed of the boundary fitting technology based on dual-link images for boundary fitting, thereby further refining the boundary information and generating the final pathological area segmentation result.
[0024] Specifically, if Figure 2 As shown, the multi-scale feature fusion module in step 2 is composed of three consecutive convolution modules and an attention module in series. The convolution module is composed of hole convolution kernels with different expansion rates (expansion rates are r = 1, 2 and 3), as well as the corresponding normalization function (Norm) and activation function (ReLU) connected in series, so as to better capture multi-scale feature information. In addition, the multi-scale feature fusion module based on hybrid convolution in this paper adopts a residual connection method, which enables the model to further preserve the spatial feature information from the previous network layer. Therefore, it can not only enrich the spatial feature information of the receptive field, but also compensate for the disappearance of local features caused by sparse sampling in the hole convolution, thereby further avoiding the rasterization problem;
[0025] Specifically, in order to further refine the local detail information of the pathological area, a coordinate attention module is introduced after three consecutive dilated convolutions. Its corresponding framework structure is as follows: Figure 3As shown. Traditional attention mechanisms usually only focus on the importance of features and ignore their location information, which may cause the network to lose the location information in the area of interest, especially when processing sequence information or spatial information. The core idea of the coordinate attention mechanism is to embed the location information into the channel attention, so that the model can better process data with complex spatial relationships. The workflow of the attention module is: decompose the input two-dimensional feature map into information in two directions, namely horizontal coordinate information and vertical coordinate information, generate a pair of feature maps with direction awareness, further process the two feature maps, and generate two attention maps. The attention map is used to help the model quickly focus on the key information in the input feature map.
[0026] In this module, each attention map is designed along a specific spatial direction to capture long-range feature dependencies in the input feature map. Specifically, one of the attention maps is calculated along the horizontal direction (e.g., from left to right or from right to left), focusing on capturing long-range dependencies in the horizontal direction of the feature map, which is crucial for extracting horizontal extensions or continuity features that may exist in brain pathological areas; the other attention map is calculated along the vertical direction (e.g., from top to bottom or from bottom to top), focusing on capturing long-range feature dependencies in the vertical direction to extract vertical morphological features that may exist in brain pathological areas. In this way, position information in both directions is saved in the generated attention map. Therefore, this method can not only maintain spatial orientation information to a certain extent, but also capture long-range correlations in spatial information, so that the model can locate the target area more accurately.
[0027] In addition, thanks to the consistent input and output dimensions of the module, it can be flexibly inserted into any backbone extraction network structure. For the segmentation of brain MRI pathological regions, this paper introduces a multi-scale feature fusion module based on hybrid convolution to capture the global information of the lesion area, and makes full use of this effective information to further improve the capture range of the receptive field, thereby solving the intra-class heterogeneity problem caused by the non-fixed position and irregular shape of the pathological area;
[0028] Specifically, since the input expected by Transformer Encoder is sequence data, such as words or sentences, its input is usually required to have a dimension of [T, D], where T is the sequence length and D is the dimension of each word vector. However, the feature map input in this article is spatial data with a dimension of [C, H, W], which does not match the input expected by Transformer Encoder and cannot be directly input into Transformer Encoder for processing. Figure 4 As shown in the figure, before the feature map is input into the boundary attention module, the input feature map is divided into small blocks (Patches) of size P×P, and each patch is regarded as a token. Therefore, the dimensions of these original input feature maps are converted to, where C is the number of channels of each patch (the same as the number of channels of the original input feature map), representing each image feature block, and N represents the total number of patches. The formula is as follows:
[0029]
[0030] The dimension of the patch in the above is stretched to the D dimension required by the TransformerFocus Encoder through a trainable linear projection layer. Subsequently, the feature map suitable for the Transformer structure and the token of its corresponding classification label are sent to the boundary attention module for the next step of calculation and processing.
[0031] Specifically, the boundary attention module based on TransformerFocus in step 3 is used to process the input feature sequence through layer normalization. The purpose of this step is to standardize the input data and ensure that the input sequence maintains a stable distribution during the transmission process in the module. The normalized sequence enters the multi-head attention mechanism for autocorrelation operation. The multi-head attention mechanism is the core component of the Transformer model. It can capture different information in the input sequence on different attention heads, thereby improving the representation ability of the model. In addition, a set of learnable category tags is introduced to form a similarity mapping between it and the spatial feature map and transfer it to the corresponding segmentation mask to achieve a refined connection between global information and edge perception information. After being processed by the multi-head attention mechanism, the output sequence will be further output through a fully connected layer (MLP, i.e., Multi-Layer Perceptron). As a common component in a neural network, the fully connected layer can perform nonlinear transformations on the input data to extract higher-level features.
[0032] The present invention proposes a semantic segmentation framework based on weakly supervised boundary refinement. The framework consists of a two-way structure. One way is to output feature maps through pre-trained classification backbones, and generate corresponding pseudo labels through the method of class activation mapping; the other way is to aggregate similar pixels through an unsupervised segmentation network to generate an edge map that suppresses irrelevant details, and then further fuse the two to remove irrelevant information, thereby generating the final segmentation result, which greatly reduces the time-consuming and labor-intensive problem of large-scale annotation and the reliance on professional assistance; and in view of the intra-class heterogeneity problem caused by the random position and irregular shape of pathological areas in brain MRI images, a multi-scale feature fusion module based on hybrid convolution is constructed. By residually connecting convolution kernels with different expansion rates, the capture range of the receptive field and the distinguishability of objects of different sizes and shapes are improved without increasing the number of parameters and preventing "gridding" from occurring; finally, in view of the complex structure of brain MRI images, the blurred boundaries of pathological areas, and the frequent interference of artifacts, a boundary attention module based on TransformerFocus is proposed. This method can capture contextual information in the input sequence, thereby forming a global dependency representation. On this basis, a set of learnable category labels are introduced, which constitute a set of similarity mappings with the spatial feature maps and are transferred to the corresponding segmentation masks, which enables the network to further refine the edge-aware information based on the connection of global information.
[0033] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0034] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A multi-attention fusion weakly supervised pathology segmentation method for brain MRI images, characterized by: The method comprises the following steps: Step 1, constructing a brain MRI image pathology segmentation framework based on weakly supervised multi-attention fusion, and generating a pathology region segmentation result through the brain MRI image pathology segmentation framework; Step 2: A multi-scale feature fusion module based on hybrid convolution is introduced to capture the global information of the lesion area. Convolution kernels with different dilation rates are serially combined through residual connections to improve the capture range of the receptive field and the ability to distinguish objects of different sizes and shapes. Step three, a boundary attention module based on TransformerFocus is proposed. By capturing the contextual information in the input sequence and forming a global dependency representation, a set of learnable category labels is introduced to form a similarity mapping between them and the spatial feature map and transfer it to the corresponding segmentation mask, thereby realizing the refined connection between global information and edge perception information.
2. According to claim 1, a multi-attention fusion weakly supervised pathology segmentation method for brain MRI images is characterized by: The brain MRI image pathology segmentation framework in step 1 includes a pseudo-label generation technology based on class activation mapping, a boundary mapping technology based on unsupervised segmentation, and a boundary fitting technology based on dual-link images.
3. The multi-attention fusion weakly supervised pathology segmentation method for brain MRI images according to claim 2, characterized in that: The workflow of the brain MRI image pathology segmentation framework is as follows: input the brain MRI image to be segmented, and pass through the last layer of feature map output by the Classification Backbone in the upper half of the framework that has been pre-trained with image-level labels, and then generate the required pseudo labels (Pseudo Mask) through the Mask Generator module; the other way is to input the brain MRI image to be segmented into the unsupervised segmentation network (Unsupervised Segment Network, USN) based on the boundary mapping technology of unsupervised segmentation, and finally generate a boundary map (Edge Map); finally, the pseudo labels and boundary maps generated above are jointly input into the boundary fusion module (BoundaryFitMoudule) composed of the boundary fitting technology based on dual-link images for boundary fitting, so as to further refine the boundary information and generate the final pathological area segmentation result.
4. The multi-attention fusion weakly supervised pathology segmentation method for brain MRI images according to claim 3, characterized in that: The multi-scale feature fusion module in step 2 is composed of three consecutive convolution modules and an attention module connected in series, and the convolution module is composed of hole convolution kernels with different expansion rates, and corresponding normalization functions (Norm) and activation functions (ReLU) connected in series.
5. The multi-attention fusion weakly supervised pathology segmentation method for brain MRI images according to claim 4, characterized in that: The workflow of the attention module is as follows: decompose the input two-dimensional feature map into information in two directions, namely horizontal coordinate information and vertical coordinate information, generate a pair of direction-aware feature maps, further process the two feature maps, and generate two attention maps. The attention maps are used to help the model quickly focus on the key information in the input feature map.
6. The multi-attention fusion weakly supervised pathology segmentation method for brain MRI images according to claim 4, characterized in that: Before the feature map is input into the boundary attention module, the input feature map is divided into small blocks (Patches) of size P×P. Each patch is regarded as a token. The dimension of the patch is stretched to the D dimension required by the TransformerFocus Encoder through a trainable linear mapping layer (LinearProjection). Subsequently, the feature map suitable for the Transformer structure and the token of its corresponding classification label are sent to the boundary attention module for the next step of calculation.
7. The multi-attention fusion weakly supervised pathology segmentation method for brain MRI images according to claim 6, characterized in that: The TransformerFocus-based boundary attention module in step three is used to process the input feature sequence through layer normalization. The purpose of this step is to standardize the input data to ensure that the input sequence maintains a stable distribution during the transmission process in the module. The normalized sequence enters the multi-head attention mechanism for autocorrelation operation.
