An image segmentation method based on full-scale fusion and flow field attention

Abstract

This paper discloses an image segmentation method based on full-scale fusion and flow field attention, comprising the following steps: obtaining an image dataset and preprocessing the data images; constructing an image segmentation model using U-Net as the backbone network; inputting training images from the dataset into the image segmentation model for training; adjusting the model to optimal results by selecting appropriate parameters and loss functions and saving the results; and inputting verification images from the dataset into the trained image segmentation model to obtain segmentation prediction results. By combining a network structure with specific functions and improving the network structure, this paper improves the method's adaptability to different segmentation tasks and image segmentation accuracy. Furthermore, it reduces the semantic differences between features at different scales, highlighting key feature information in the image, and significantly improving the network's performance and robustness.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, and in particular to an image segmentation method based on full-scale fusion and flow field attention. Background Art

[0002] In recent years, deep learning-based segmentation methods have made significant progress in image segmentation. The success of deep learning segmentation methods stems from the powerful image feature extraction capabilities of deep neural networks, which process this information to produce refined segmentation results. Deep neural network models include convolutional neural networks (CNNs), multilayer perceptrons (MLPs), and transformers. Convolutional neural networks have achieved considerable success in image segmentation, with examples including fully convolutional networks (FCNs), U-Nets, and SegNets. U-Net is a representative network for image segmentation. U-Net utilizes hierarchical CNNs to extract image features during the encoding phase. During the decoding phase, deconvolution and skip connections are used to achieve interactive fusion of encoder and decoder features, resulting in a better image segmentation prediction map. Based on U-Net, researchers have proposed Attention U-Net with an attention mechanism and Deformation-based Function (DFM) with feature refinement, both of which have achieved improved segmentation accuracy. However, the inherent local nature of the convolution operations in convolutional neural network-based methods results in their inability to model global feature correlations. Transformers and multi-layer perceptrons utilize self-attention mechanisms and fully connected networks to capture global image information, extracting image features that are difficult for convolutional neural networks to capture. For example, TransUNet replaces the deepest features in the U-Net encoder with features extracted from ViT. UNetXt, a medical image segmentation network that integrates convolution and multi-layer perceptrons, improves segmentation prediction accuracy. However, most current deep learning-based image segmentation methods still have shortcomings: First, the network input image size requirements are limited. Most Transformer-based methods require datasets with training and test images of uniform and square dimensions. However, widely collected real-world image data is often of arbitrary dimensions, and image resizing can cause image distortion and affect segmentation results. Second, the image feature information mined by a single network during the encoding and decoding stages may be insufficient. Multi-layer perceptron networks extract global feature information, while convolutional neural networks focus on local information. Therefore, by rationally combining different types of networks, more comprehensive feature information can be obtained to improve network performance. Third, the simple skip connection structure of U-Net makes it difficult to effectively fuse the coarse-grained information and fine-grained information in the skip connection features of each scale. There are semantic differences between the skip connection features of each scale, which limits the segmentation performance of the network and has certain defects.

[0003] Therefore, it is urgent to propose an image segmentation method based on full-scale fusion and flow field attention to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to provide an image segmentation method based on full-scale fusion and flow field attention to improve the applicability and segmentation accuracy of image segmentation tasks while improving the robustness of image segmentation.

[0005] To solve the above technical problems, the present invention provides an image segmentation method based on full-scale fusion and flow field attention, comprising the following steps:

[0006] Obtain image data sets and preprocess the data images;

[0007] Using U-Net as the backbone network, build an image segmentation model;

[0008] Input the training images in the dataset into the image segmentation model for training;

[0009] Adjust the model to the optimal effect by selecting appropriate parameters and loss functions and save it;

[0010] Input the verification image in the dataset into the trained image segmentation model to obtain the segmentation prediction result.

[0011] Furthermore, the graphic segmentation model includes a feature encoder, a full-scale feature fusion module and a feature decoder connected in sequence.

[0012] Furthermore, the image segmentation model data processing includes the following steps:

[0013] The feature encoder extracts hierarchical feature information and global feature information of the image;

[0014] The full-scale feature fusion module interactively fuses the feature information of each level extracted by the feature encoder and the global feature information;

[0015] The feature decoder refines the feature maps at each scale to obtain a segmentation result.

[0016] Furthermore, the feature encoder includes a U-Net backbone network and a convolutional MLP module connected in sequence.

[0017] Furthermore, the U-Net backbone network is used to obtain the feature maps of 5 layers of different scales of the input image. The convolutional MLP module is used to extract the global feature information of the input image and concatenate it with the underlying feature map F1 to obtain the fused feature T1.

[0018] Furthermore, the full-scale feature fusion module performs feature fusion along the feature height, width and channel dimensions to generate the fusion features of each branch The formula for generating the fusion feature of the first layer is:

[0019]

[0020] in Indicates that the i-th layer convolution operation is used to adjust the number of channels of the feature map, dc and uc represent convolution downsampling and convolution upsampling respectively, and Cat represents the channel superposition operation.

[0021] Furthermore, the feature decoder combines the output T1 of the feature encoder with the output of the full-scale feature fusion module Input step by step, output feature maps that are refined step by step

[0022] Furthermore, the feature decoder processing flow includes the following steps:

[0023] The feature map P i-1 Upsampled to the same size fusion feature T i After the two are cascaded, the characteristic flow field is obtained through convolution operation To guide the feature map P i-1 Perform deformation;

[0024] The deformed feature map P i-1 With T i Cascade and input and input to the feature decoder, output feature P i ;

[0025] Map the number of channels of the feature map P5 to the number of segmentation categories to obtain the final segmentation result.

[0026] Furthermore, the loss function is constructed by fusing the GDL loss function and the cross entropy loss function. The loss function is:

[0027] L=L CE +1.1L GDL ;

[0028] Among them, L CE is the cross entropy loss function; L GDL is the GDL loss function.

[0029] Furthermore, the GDL loss function and the cross entropy loss function are respectively:

[0030]

[0031]

[0032] in: Indicates the weight of the mth category in M ​​categories, g mn represents the true value of the nth pixel of category m, and pmn Indicates the corresponding predicted value.

[0033] Compared with the prior art, the present invention has at least the following beneficial effects:

[0034] The present invention improves the encoder and decoder structures of the traditional network. By combining the network structure with specific functions and improving the network structure, it makes up for the problems of the general U-shaped network in its insufficient ability to capture the global feature information of the image and the distortion during the upsampling of image features. It improves the adaptability of the method to different segmentation tasks and the image segmentation accuracy.

[0035] At the same time, the full-scale feature fusion module proposed in the present invention reduces the semantic differences between features of different scales by fusing coarse-grained features and fine-grained features at all levels of skip connections, highlights the key feature information of the image, and significantly improves the performance and robustness of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of the image segmentation method based on full-scale fusion and flow field attention of the present invention;

[0037] Figure 2 Schematic diagram of the network structure of the image segmentation model of the image segmentation method based on full-scale fusion and flow field attention of the present invention;

[0038] Figure 3 Schematic diagram of the convolutional MLP module structure of the image segmentation method based on full-scale fusion and flow field attention of the present invention;

[0039] Figure 4 Schematic diagram of a single full-scale feature fusion module of the image segmentation method based on full-scale fusion and flow field attention of the present invention;

[0040] Figure 5 Schematic diagram of the flow field attention module structure of the image segmentation method based on full-scale fusion and flow field attention in the present invention. DETAILED DESCRIPTION

[0041] The following is a more detailed description of the image segmentation method based on full-scale fusion and flow field attention of the present invention, with reference to a schematic diagram. A preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art may modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being generally known to those skilled in the art and not intended to limit the present invention.

[0042] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are greatly simplified and not to exact scale, and are provided solely for the purpose of assisting in the description of the embodiments of the present invention.

[0043] like Figure 1 As shown, the embodiment of the present invention proposes an image segmentation method based on full-scale fusion and flow field attention, including the following steps:

[0044] Obtain image data sets and preprocess the data images;

[0045] Using U-Net as the backbone network, build an image segmentation model;

[0046] Input the training images in the dataset into the image segmentation model for training;

[0047] Adjust the model to the optimal effect by selecting appropriate parameters and loss functions and save it;

[0048] Input the verification image in the dataset into the trained image segmentation model to obtain the segmentation prediction result.

[0049] The following lists preferred embodiments of the image segmentation method based on full-scale fusion and flow field attention in medical image segmentation to clearly illustrate the content of the present invention. It should be clear that the content of the present invention is not limited to the following embodiments, and other improvements through conventional technical means of ordinary technicians in this field are also within the scope of the present invention.

[0050] S100: Acquire a medical image dataset and pre-process the data images.

[0051] Specifically, if the medical image dataset is 3D, 1mm slices can be taken along the image's axial direction to create 2D slices. If the medical image dataset is 2D, no slicing is performed. If the acquired brain medical image contains skull structures, the skull can be removed through algorithmic processing. After preprocessing, the data image is normalized to ensure that the input image pixel mean is 0 and the variance is 1. Random rotation and random flipping are used to expand the data volume. The dataset is divided into training and test sets in a 6:4 ratio.

[0052] S200, using U-Net as the backbone network, builds an image segmentation model that includes a convolutional MLP module, full-scale feature fusion, and flow field attention decoding module.

[0053] Specifically, such as Figure 2As shown in the figure, the image segmentation model of this method is divided into three processing stages: the first stage uses a feature encoder that integrates a U-Net backbone network and a convolutional MLP module to extract the hierarchical feature information and global feature information of the image; the second stage uses a full-scale feature fusion module to interactively fuse the information features of each level extracted in the first stage to generate feature maps for skip connections at all levels; the third stage uses a flow field attention module (Flow and Attention Decoding Unit, FADU) that includes flow field transformation and attention mechanism. The feature decoder constructed by the module refines the feature maps of each scale input and obtains the predicted segmentation result through channel mapping.

[0054] The feature encoder consists of two parts: the U-Net backbone network and the convolutional MLP module. Given an input image After U-Net, we get 5 layers of feature maps of different scales At the same time, the convolutional MLP module is used to extract the global feature information of the input image I and concatenate it with the underlying feature map F1 to obtain T1. Figure 3 As shown in Figure 1, the convolutional MLP module contains a set of convolutional blocks consisting of 3×3 convolution downsampling, 3×3 depth-separable convolution and pooling downsampling, as well as three sets of convolutional MLP blocks containing MLP and convolution downsampling. The feature encoder extracts local spatial feature information of the image while retaining the global feature information of the image.

[0055] The full-scale feature fusion module consists of four independent branches, which perform feature fusion according to specific feature height, width and channel dimensions to generate fusion features of each branch. That is, the feature maps at each level are used for skip connections. Figure 4 , to illustrate the workflow of the single-branch structure. Down and up represent downsampling and upsampling of the feature map to the height and width of feature F2, respectively. Null means no sampling operation. C represents the convolution operation, which is uniformly 3×3 convolution here, with 64 output channels. After the Cat operation, the 5-layer feature of the input is cascaded and superimposed into a single feature map according to the channel. (The number of channels is 320), The convolution operation is used to adjust the number of channels of the feature map, and the number of channels is adjusted to the channel size of the corresponding feature map F2, and finally added to the feature map F2 to obtain the skip connection feature T2. Therefore, the fusion feature T of the i-th layer is generated. i The formula is:

[0056]

[0057] In formula (1), Indicates that the i-th layer convolution operation is used to adjust the number of channels of the feature map, adjusting the number of channels to the corresponding feature map F iThe channel size, dC and uC represent convolution downsampling and convolution upsampling respectively, and Cat represents the channel superposition operation.

[0058] The feature decoder is composed of a flow field attention module (FADU), which combines the output T1 of the feature encoder with the output of the full-scale feature fusion module. Input step by step, output feature maps that are refined step by step Achieve feature refinement while avoiding information redundancy. Figure 5 As shown, the flow field attention decoding module works as follows: First, the input feature P i-1 Upsample to T i The same size as the two, the characteristic flow field is obtained by convolution operation after the two are cascaded. To guide feature P i-1 The deformation formula is as follows:

[0059] warp(P i-1 )=P i-1 (p x +φ(p) x ,p x +φ(p) x ) (2)

[0061] In formula (2), the subscripts x and y of p are the coordinates of the pixel points. The flow field deformation reduces the problem of feature map distortion during upsampling. Then the deformed feature P i-1 With T i Cascade input 3×3 convolution block output feature P i ′. The convolution operation formula is:

[0062] P i ′=σ(C(W P P i-1 )+C(W T T i )) (3)

[0064] In formula (3), σ represents the ReLU activation function, C represents the convolution operation, and W P and W T Represent the hidden state P respectively i-1 and skip connection feature T i Since simply fusing high-level features with shallow features usually leads to information redundancy and confusion, the feature Pi′ is input into the convolutional attention module (CBAM) to obtain P i , the attention mechanism improves the efficiency of the module in utilizing effective information. Overall, P i Can pass the input P layer by layer i-1 and Ti To generate, the formula is as follows:

[0065] P i =FADU(P i-1 ,T i ;φ) (4)

[0067] In formula (4), i ranges from 1 to 5, and P0 is a tensor initialized to 0. At the end of the encoding process, a 3×3 convolution is used to map the number of channels of the feature map P5 to the number of segmentation categories to obtain the final segmentation result.

[0068] S300: Input the training images in the data set into the image segmentation model for training.

[0069] Specifically, during training, the model uses the Adam optimization algorithm to minimize the loss function and update the network parameters. The initial learning rate is set to 0.0006, and the weight decay value is 0.0005. The batch size of each training data is set to 1, and the total number of iterations is 30,000.

[0070] S400: Adjust the model to the optimal effect by selecting appropriate parameters and loss function and save it.

[0071] Specifically, the factors that affect the overall performance of the image segmentation model include not only the network structure design, but also the loss function. The loss function of this network is constructed by integrating the cross entropy loss function and the GDL loss function, where the cross entropy loss function is where g mn represents the true value of the nth pixel of category m, and p mn Represents the corresponding predicted value, and the loss function of GDL is: in It represents the weight of the mth category in the M categories. The final fusion loss function formula is: L = L CE +1.1L GDL .

[0072] S500: Input the verification image in the data set into the trained image segmentation model to obtain the segmentation prediction result.

[0073] Compared with the prior art, the present invention has at least the following beneficial effects:

[0074] The present invention improves the encoder and decoder structures of the traditional network. By combining the network structure with specific functions and improving the network structure, it makes up for the problems of the general U-shaped network in its insufficient ability to capture the global feature information of the image and the distortion during the upsampling of image features. It improves the adaptability of the method to different segmentation tasks and the image segmentation accuracy.

[0075] At the same time, the full-scale feature fusion module proposed in the present invention reduces the semantic differences between features of different scales by fusing coarse-grained features and fine-grained features at all levels of skip connections, highlights the key feature information of the image, and significantly improves the performance and robustness of the network.

[0076] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An image segmentation method based on full-scale fusion and flow field attention, characterized in that: The steps include: Obtain image data sets and preprocess the data images; Using U-Net as the backbone network, build an image segmentation model; Input the training images in the dataset into the image segmentation model for training; Adjust the model to the optimal effect by selecting appropriate parameters and loss functions and save it; Input the verification image in the dataset into the trained image segmentation model to obtain the segmentation prediction result; The graph segmentation model includes a feature encoder, a full-scale feature fusion module, and a feature decoder connected in sequence; the feature encoder includes a U-Net backbone network and a convolutional MLP module connected in sequence; The U-Net backbone network is used to obtain 5 layers of feature maps of different scales of the input image. The convolutional MLP module is used to extract the global feature information of the input image and combine it with the underlying feature map Cascade to obtain fusion features ; The full-scale feature fusion module performs feature fusion along the feature height, width and channel dimensions to generate the fusion features of each branch. , where the formula for generating the fusion feature of the layer is: in Indicates the The layer convolution operation is used to adjust the number of channels of the feature map. and Represent convolution downsampling and convolution upsampling, respectively. Indicates channel superposition operation; The feature decoder includes a flow field attention module, and the decoder converts the output of the feature encoder into And the output of the full-scale feature fusion module Input step by step, output feature maps that are refined step by step ; The flow field attention decoding module takes the input features Upsample to The same size as the two, the characteristic flow field is obtained by convolution operation after the two are cascaded. To guide the features Perform deformation.

2. The image segmentation method based on full-scale fusion and flow field attention according to claim 1, characterized in that: The image segmentation model data processing comprises the following steps: The feature encoder extracts hierarchical feature information and global feature information of the image; The full-scale feature fusion module interactively fuses the feature information of each level extracted by the feature encoder and the global feature information; The feature decoder refines the feature maps at each scale to obtain a segmentation result.

3. The image segmentation method based on full-scale fusion and flow field attention according to claim 2, characterized in that: The feature decoder converts the output of the feature encoder And the output of the full-scale feature fusion module Input step by step, output feature maps that are refined step by step .

4. The image segmentation method based on full-scale fusion and flow field attention according to claim 1, characterized in that The feature decoder processing flow includes the following steps: The feature map Upsampled to the same size fusion features After the two are cascaded, the characteristic flow field is obtained through convolution operation To guide the feature map Perform deformation; The deformed feature map and Cascade and input and input to the feature decoder, output feature ; The feature map The number of channels is mapped to the number of segmentation categories to obtain the final segmentation result.

5. The image segmentation method based on full-scale fusion and flow field attention according to claim 1, characterized in that: The loss function is constructed by fusing the GDL loss function and the cross entropy loss function. The loss function is: ； in, is the cross entropy loss function; is the GDL loss function.

6. The image segmentation method based on full-scale fusion and flow field attention according to claim 5, characterized in that: The GDL loss function and the cross entropy loss function are: ； ； in: represents the weight of the mth category in M ​​categories, Represents the true value of the nth pixel at position m, and Indicates the corresponding predicted value.

Images