A four-axis fusion method based on CNN and Transformer

Abstract

This invention relates to the field of image processing technology, and more particularly to a four-axis fusion method based on CNN and Transformer. The method includes cropping a medical image segmentation dataset and converting it to RGB format; extracting image features through CNN branch encoders and Transformer branch encoders; fusing the image features from the CNN branch encoders and Transformer branch encoders through a four-axis fusion module to output a feature map; and then feeding the output feature map into a decoder for reconstruction. This invention combines the advantages of CNN and Transformer, using CNN to alleviate the locality limitation of Transformer and using Transformer to alleviate the global limitation of CNN. The four-axis fusion method not only focuses on regions in more directions, which is more beneficial for the segmentation of lesion edges, but also greatly reduces the computational load, thereby improving segmentation performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a four-axis fusion method based on CNN and Transformer. Background Technology

[0002] In recent years, with the development of computer and artificial intelligence technologies in image processing and pattern recognition, image processing technology has also been applied to the field of medical image analysis, gradually forming a new interdisciplinary field, namely medical image processing. It mainly uses computer technology to perform operations such as lesion detection, classification and segmentation on medical images, which can assist doctors in making timely and effective diagnoses of lesion areas in images, making medical diagnosis more accurate and visualized. It has now attracted widespread attention from experts at home and abroad.

[0003] Since the introduction of U-Net, deep learning methods have become the dominant approach for solving this task. Despite the popularity of Convolutional Neural Networks (CNNs), they remain hampered by limited receptive fields and an inductive bias due to locality and weight sharing, failing to capture long-range dependencies. Many efforts have focused on expanding the receptive field of CNNs to improve their context modeling capabilities. Fisher Yu proposed dilated convolutions with adjustable dilation rates, demonstrating superior performance in semantic segmentation. Zhao Hengshuang used pyramid pooling at multiple feature scales to aggregate multi-scale global information. Wang Xiaolong proposed nonlocal operations embedded at the end of the encoder to capture long-range dependencies. While these models improve context modeling to some extent, they are still inevitably limited by the receptive field of CNN networks.

[0004] Transformer is a sequence-to-sequence prediction framework with a proven track record in machine translation and natural language processing due to its powerful long-range modeling capabilities. The self-attention mechanism in Transformer dynamically adjusts the receptive field based on the input content, thus outperforming convolutional operations in modeling long-range dependencies.

[0005] Transformers are considered an alternative architecture and have achieved competitive performance on many computer vision tasks, such as image recognition, semantic / instance segmentation, object detection, low-level vision, and image generation. A typical example is the Vision Transformer (ViT), which outperforms ResNet-based CNNs on recognition tasks, but at the cost of training with 300M datasets. Since large training datasets are not always available, recent research has attempted to combine CNNs and Transformers into a hybrid model. Chen Jieneng designed TransUNet, where a CNN and a Transformer are cascaded together to form a powerful encoder for 2D medical image segmentation. While TransUNet's design is interesting and its performance is good, the presence of self-attention makes optimizing the model challenging. First, it requires a significant amount of training time to focus attention on the salient locations of objects, especially in 3D scenes. Second, due to its high computational complexity, ordinary Transformers are costly in processing multi-scale and high-resolution feature maps, which plays a crucial role in image segmentation.

[0006] The specific references are as follows:

[0007] Fisher Yu "Yu F, Koltun V.Multi-scale context aggregation by dilatedconvolutions[J]."

[0008] Zhao Hengshuang, "Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network[C]".

[0009] Wang Xiaolong, "Wang X, Girshick R, Gupta A, et al. Non-local neural networks [C]".

[0010] Chen Jieneng "Chen J, Lu Y, Yu Q, et al. Transunet: Transformers make strongencoders for medical image segmentation[J]." Summary of the Invention

[0011] To address the shortcomings of existing algorithms, this invention combines the advantages of CNN and Transformer. CNN alleviates the Transformer's inability to capture local features, while Transformer overcomes the CNN's inability to acquire global features. The four-axis fusion method not only focuses on regions in more directions, which is more conducive to the segmentation of lesion edges, but also greatly reduces the amount of computation, thereby improving segmentation performance.

[0012] The technical solution adopted in this invention is: a four-axis fusion method based on CNN and Transformer, comprising the following steps:

[0013] Step 1: Crop the medical image segmentation dataset and convert it to RGB format; use cvtColor and resize to adjust the color and size of the image for medical image format conversion, and then save the adjusted image into a file with the .npy extension.

[0014] Step 2: Extract image features using CNN branch encoder and Transformer branch encoder, and then use a four-axis fusion module to stitch together the image features from the CNN branch encoder and Transformer branch encoder and output a feature map.

[0015] Furthermore, the CNN branch encoder uses a ResNet34 module to extract semantic features, comprising 5 blocks, each containing two convolutional layers and a 2x downsampling; the input image X∈R H×W×3 After convolution and max pooling operations, we obtain... Extract the second block from Resnet34 The third piece The fourth piece The output is fused with the features of the Transformer.

[0016] Furthermore, when the image is input into the Transformer branch encoder, it is first divided into several square patch blocks; the patch blocks are flattened, and the flattened sequence is passed to the linear embedding layer with an output dimension of D0 to obtain the original embedding sequence.

[0017] The Transformer branch encoder contains 8 Transformer blocks, each containing an L-layer multi-head self-attention and multi-layer perceptron, and updates each patch block by aggregating global information at each layer;

[0018] The output of the last Transformer block is normalized to obtain the encoded sequence.

[0019] The progressive upsampling in SETR is used to decode the encoded sequence obtained by the Transformer.

[0020] Furthermore, the four-axis fusion module combines four-axis attention, multi-scale methods, and residual structures with separable convolutions and Hadamard products.

[0021] Furthermore, the multi-scale method employs four grouped convolutions with kernel sizes of 3, 5, 7, and 9, and the number of channels after each convolution is only one-quarter of the input feature map.

[0022] Concatenate the outputs of the four convolutions;

[0023] The multi-scale method is expressed as follows:

[0024] X i =Concat(Ci,Ti) (1)

[0025]

[0026] Among them, X i It is the result of concatenating the i-th group of CNN and Transformer features, f pro This refers to the operation of ConvPro convolutional blocks, F p i ro It is f pro Output characteristics after the operation.

[0027] Furthermore, four-axis attention includes: axial attention in height and width, and axial attention in the main diagonal and secondary diagonal.

[0028] Furthermore, the axial attention in height and width consists of a multi-head height axial attention, a multi-head width axial attention, and a convolution with a kernel size of 3, as shown in the formula:

[0029]

[0030] Among them, F pro It is the output of the ConvPro block. and These represent multi-head height axis focus and multi-head width axis focus, respectively, with Conv representing a 3×3 convolution operation.

[0031] Furthermore, the formulas for axial attention on the main diagonal and secondary diagonal are:

[0032]

[0033] Among them, F H&W It is the output of the H&W block. and These represent the main diagonal axial focus and the secondary diagonal axial focus, respectively, with Conv indicating a 3×3 convolution operation.

[0034] Furthermore, the input of the residual structure with separable convolutions and Hadamard products consists of two parts, one of which is T i With G i The Hadar product P is one part, and the other part is the summation output after a 1×1 convolution. Represented as:

[0035]

[0036] Among them, F 1x1 These are the features after a 1x1 convolution operation;

[0037] The Hadamard product P is represented as:

[0038]

[0039] Among them, Conv 3×3 It is a 3×3 convolution operation. It is Hadamaji. and They are feature T i and G i The projection matrix; these two parts are concatenated and then placed into the residual structure, which is represented as:

[0040]

[0041] Among them, M i These are the output characteristics of Gi and Ti after passing through the FAFusion fusion module.

[0042] Step 3: Send the output feature map to the decoder for reconstruction.

[0043] Furthermore, the decoder includes upsampling and MixConv, which combines two features through convolution operations, as shown below:

[0044] M i =Conv 3×3 (Conv 1×1 (Conv 3×3 (Up i )))+Conv 3×3 (F i+1 (8)

[0045] Among them, Up i F is the feature map after the upsampling operation. i+1 Output features for the fusion module.

[0046] The beneficial effects of this invention are:

[0047] 1. The FAFuse framework is based on CNN and Transformer. Through FAFusion, fine-grained local features and high-level semantic context information can be effectively integrated, thereby improving the feature representation learning ability.

[0048] 2. Experimental data on three typical medical image segmentation tasks show that FAFuse of the present invention consistently outperforms other existing methods, especially in the polyp segmentation task. Attached Figure Description

[0049] Figure 1 This is a diagram of the four-axis fusion module based on CNN and Transformer of the present invention;

[0050] Figure 2 This is a four-axis attention diagram of the four-axis fusion module of the present invention;

[0051] Figure 3 These are three typical medical image segmentation comparison diagrams of the present invention and other methods. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0053] like Figure 1 As shown, a four-axis fusion method based on CNN and Transformer includes the following steps:

[0054] Step 1: Cropping the medical image segmentation dataset into 224×224 images and dividing it into three parts: training, validation, and testing;

[0055] Step 2: Build the network. The encoding part extracts medical image features, and the decoding part restores the image and outputs the final segmentation feature map.

[0056] Step 3: Load the training and validation sets of medical image segmentation data into the network for training and validation, and save the model with the best performance;

[0057] Step 4: Use the saved result model to test the images in the test set, obtain the test results, and complete the medical image segmentation; perform detection on the polyp and skin datasets, generate a training model, determine the lesion area, and output the medical image segmentation results.

[0058] Specific network setup plan:

[0059] I. This invention employs a U-shaped hierarchical encoder and decoder architecture. The encoder consists of two branches that process information in different ways; including:

[0060] 1) CNN branch encoder, the receptive field gradually increases, the feature encoding is from local to global, and it consists of input layer, convolutional layer, activation function, pooling layer and fully connected layer.

[0061] 2) Transformer branch encoder, which models global self-attention through 8 Transformer blocks and recovers local details through several upsampling.

[0062] Features of consistent size extracted from both branches are fed into the FAFusion module, which uses multi-axial attention (including the axes of height, width, main diagonal, and secondary diagonal) and double Hadamard product to selectively fuse the dual-branch information. The dual-branch structure offers several advantages: First, FAFusion leverages the strengths of CNNs and Transformers; specifically, it captures more effective global and local information while maintaining sensitivity to underlying contextual information. Second, the FAFusion module can simultaneously utilize different features extracted from CNN and Transformer branches, resulting in fused features containing more semantic information.

[0063] II. The CNN branch encoder employs a feature pyramid structure. The resolution of the feature maps decreases as network depth increases, while the receptive field and the number of channels gradually increase, with feature encoding progressing from local to global. The CNN branch encoder uses a ResNet34 pre-trained model, containing 5 blocks, each with two convolutional layers, and performs a 2x downsampling of the input features; specifically, the input image (X∈R...)... H×W×3 After convolution and max pooling operations, the result is obtained. Extract the second block from Resnet34 The third piece The fourth piece The output is fused with features from the Transformer; furthermore, the CNN branch encoder can adjust its depth to suit the requirements of other models.

[0064] Third, the Transformer branch encoder adopts the encoding structure in the encoder-decoder architecture. When an image is input into the Transformer branch encoder, it is first divided into several square patch blocks, where S is the side length of the square patch block, with a default value of 16. Then, these patch blocks are flattened, and the flattened sequence is passed to a linear embedding layer with an output dimension of D0 to obtain the original embedding sequence. Since a simple self-attention mechanism cannot capture the order of inputs, i.e., it cannot distinguish between blocks at different positions, learnable positional embeddings of the same dimension are added to e. The resulting sequence serves as the input to the Transformer branch encoder, which contains 8 Transformer blocks, each containing L layers of multi-head self-attention (MSA) and a multilayer perceptron (MLP). The entire Transformer structure is composed entirely of attention mechanisms, with self-attention at its core. It updates each patch block by aggregating global information at each layer. The formula for self-attention is:

[0065]

[0066] Where, [q,k,v]=zW qkv , It is the projection matrix, D k Let Variance be the variance.

[0067] MSA is an extended version of SA. Literally, SA has one header, while MSA has multiple headers; specifically, MSA requires multiple sets of W... qkv Multiplying the input yields multiple sets of [q,k,v], and finally, multiple SAs are concatenated. The MLP consists of a series of neuron layers, designed to extend and refine linear regression. Layer normalization is applied to the output of the last Transformer block to obtain the encoded sequence. Next, progressive upsampling in SETR is used to decode the encoded sequence obtained by the Transformer. Specifically, the sequence is first reshaped to... The reconstructed feature map is a feature map with D0 channels; then, the image size is restored through successive upsampling, resulting in... and T0, T1, and T2 are fused with G1, G2, and G3 of the CNN branch encoder respectively (Note: The size of the fused feature maps is consistent).

[0068] IV. Effective feature fusion is the core of representation learning; therefore, the remaining question is how to perform the fusion. A simple approach is to simply perform convolution operations; however, this method loses long-range dependencies and global contextual information, capturing only local information. Therefore, this invention proposes a four-axis fusion module (FAFusion), such as... Figure 1 As shown, combining multi-axis attention and multi-scale features allows CNNs and Transformers to complement each other's strengths. It's worth noting that the features from the two branches are first concatenated, and then subjected to convolutional operations to obtain multi-scale features, reducing the number of channels to half that of the concatenated version.

[0069] The multi-scale method Conv Pro employs four grouped convolutions with kernel sizes of 3, 5, 7, and 9, respectively, and the number of channels after each convolution is only one-quarter of the input feature map. The outputs of the four convolutions are then concatenated. The concatenated feature map retains the same number of channels and contains information at multiple scales; a complete ConvPro can be represented as:

[0070] X i =Concat(Gi,Ti) (2)

[0071]

[0072] Among them, X i It is the result of concatenating the i-th group of CNN and Transformer features, f pro This refers to the operation of ConvPro convolutional blocks. It is f pro The output features after the operation, Gi and Ti refer to the i-th CNN feature map and the i-th Transformer feature map, respectively.

[0073] Axial attention in height and width (H&W) consists of a multi-head height axial attention, a multi-head width axial attention, and a convolution with a kernel size of 3. Axial attention was proposed because it can significantly reduce computational complexity and improve the model's long-range modeling capability. The functionality of a complete height and width axial block can be described as follows:

[0074]

[0075] Among them, F pro It is the output of the ConvPro block. and These represent multi-head height axial attention and multi-head width axial attention, respectively. Conv represents a 3×3 convolution operation. Since axial attention involves block operations, the purpose of using 3×3 convolution is to integrate spatial information.

[0076] Axial attention (M&C) on the main diagonal and secondary diagonal is similar in structure to H&W, but the focus is different. First, the bottom row of pixels with a value of 0 is filled in; then a transpose operation is performed; finally, axial attention in the height direction is applied to the transposed image to achieve axial attention on the main diagonal. It's important to note that after axial attention on the main diagonal, a transpose and padding removal operation are performed; in other words, the feature map size remains the same before and after axial attention on the main diagonal.

[0077] For axial attention along the secondary diagonal, first, fill the rightmost row of the feature map with pixels of value 0; then, perform a transpose operation; finally, apply axial attention along the height direction to the transposed image to achieve axial attention along the secondary diagonal. It's worth noting that the feature map dimensions remain the same before and after the secondary diagonal attention.

[0078] The functionality of a complete main diagonal and secondary diagonal axial attention block can be described as follows:

[0079]

[0080] Among them, F H&W It is the output of the H&W block. and These represent the main diagonal axial focus and the secondary diagonal axial focus, respectively, with Conv indicating a 3×3 convolution operation.

[0081] The input to the residual structure consists of two parts, one of which is T. i With G i The Hadar product P is one part, and the other part is the summation output after a 1×1 convolution. It can be represented as:

[0082]

[0083] Among them, F 1x1 These are the features after a 1x1 convolution operation; the Hadamard product P can be represented as:

[0084]

[0085] Among them, Conv 3×3 It is a 3×3 convolution operation. It is Hadamaji. and They are feature T i and G i The projection matrix; these two parts are concatenated and then placed into the residual structure, which can be represented as:

[0086]

[0087] Among them, M i These are the output characteristics of Gi and Ti after passing through the FAFusion fusion module.

[0088] V. The decoder mainly consists of two stages: upsampling and MixConv. MixConv combines two features through convolution operations, which can be represented as:

[0089] M i =Conv 3×3 (Conv1×1 (Conv 3×3 (Up i )))+Conv 3×3 (F i+1 (9)

[0090] Among them, Up i F is the feature map after the upsampling operation. i+1 M is the output feature of the fusion module. i It is obtained through two steps.

[0091] The decoder is built with pure convolutional modules, consisting of multiple upsampling steps, decoding hidden features, and outputting the final segmentation mask. At the decoder's entry point, bilinear interpolation is first applied to the output feature map F0 of the bottom fusion module, increasing F0's resolution by a factor of two. Then, the resized feature map is concatenated with the output feature map F1 of the previous fusion module, and the concatenated image is fed into consecutive 3×3 convolutions to integrate spatial information. This process is repeated for all subsequent layers until the original input resolution is restored. The final output is fed into a 1×1 convolutional layer to generate pixel-level semantic predictions. The encoder and decoder fuse the encoder's semantic information through jump-and-concatenation operations to obtain more contextual information. Finally, upsampling is used to obtain the segmentation prediction map.

[0092] Experimental results:

[0093] Comparison of the effects of this invention with other methods: Figure 3 As shown in Tables 1-3, the data comparison results demonstrate that empirical evidence from the figures and tables shows that the present invention FAFuse consistently outperforms other existing state-of-the-art methods, especially in the polyp segmentation task.

[0094] Table 1: Kvasir-SEG (Total: 1000, Training: 880, Test: 120).

[0095]

[0096] Table 2: CVC-ClinicDB (Total: 612, Training: 550, Testing: 62), also known as CVC-612.

[0097]

[0098] Table 3: ISIC2018 (Total: 2594, Training: 2074, Test: 520)

[0099]

[0100] Note: For each column, the best result is highlighted in bold.

[0101] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A four-axis fusion method based on CNN and Transformer, characterized in that, Includes the following steps: Step 1: Cropping the medical image segmentation dataset and converting it to RGB format; Step 2: Extract image features using CNN branch encoders and Transformer branch encoders, and fuse the image features from the CNN branch encoders and Transformer branch encoders using a four-axis fusion module to output a feature map; The four-axis fusion module combines four-axis attention, multi-scale methods, and residual structures with separable convolutions and Hadamard products; The multi-scale method uses four grouped convolutions with kernel sizes of 3, 5, 7, and 9, and the number of channels after each convolution is only one-quarter of the input feature map. Concatenate the outputs of the four grouped convolutions; The multi-scale method is expressed as follows: (1) (2) in, It is the first The result after concatenating features from CNN and Transformer. This refers to the operation of ConvPro convolutional blocks. yes Output characteristics after operation; Four-axis attention includes: axial attention in height and width, and axial attention in the main diagonal and secondary diagonal; The input to the residual structure with separable convolutions and Hadamard products consists of two parts, one of which is... and Hadar P The other part is The summation output after convolution , Represented as: (5) in, yes Features after convolution operation; Hadamaji P Represented as: ， (6) in, yes Convolution operation, It is Hadamaji. and These are features and The projection matrix; these two parts are concatenated and then placed into the residual structure, which is represented as: (7) in, yes and Output characteristics after passing through the fusion module FAFusion; Step 3: Send the output feature map to the decoder for reconstruction.

2. The four-axis fusion method based on CNN and Transformer according to claim 1, characterized in that: The CNN branch encoder uses a ResNet34 module to extract semantic features, containing 5 blocks, each block consisting of two convolutional layers and a 2x downsampling; the input image... After convolution and max pooling operations, we obtain... Extract the second block from ResNet34 The third piece The fourth piece The output is fused with the features of the Transformer.

3. The four-axis fusion method based on CNN and Transformer according to claim 1, characterized in that: When an image is input into the Transformer branch encoder, it is first divided into several square patch blocks; the patch blocks are then flattened, and the flattened sequence is passed to the output dimension. A linear embedding layer is used to obtain the original embedding sequence. ; The Transformer branch encoder contains 8 Transformer blocks, each containing an L-layer multi-head self-attention and multi-layer perceptron, and updates each patch block by aggregating global information at each layer; The output of the last Transformer block is normalized to obtain the encoded sequence. The progressive upsampling in SETR is used to decode the encoded sequence obtained by the Transformer.

4. The four-axis fusion method based on CNN and Transformer according to claim 1, characterized in that, The axial attention in height and width consists of a multi-head height axial attention, a multi-head width axial attention, and a convolution with a kernel size of 3, as shown in the formula: (3) in, It is the output of the ConvPro block. and These are the bullish height axis focus and the bullish width axis focus, respectively. Indicates 3 3. Convolution operation.

5. The four-axis fusion method based on CNN and Transformer according to claim 1, characterized in that, The formulas for the axial inclinations on the main diagonal and secondary diagonal are: (4) in, It is the output of the H&W block. and These are the primary diagonal axis focus for bullish trends and the secondary diagonal axis focus for bullish trends. Indicates 3 3. Convolution operation.

6. The four-axis fusion method based on CNN and Transformer according to claim 1, characterized in that, The decoder includes upsampling and MixConv. MixConv combines two features through convolution operations, as shown below: (8) in, This is the feature map after the upsampling operation. Output features for the fusion module.

Images