Small object segmentation method in medical images based on dual-branch feature fusion and attention

Abstract

The present invention proposes a method for segmenting small targets in medical images based on dual-branch feature fusion attention, and is particularly concerned with the segmentation of cancerous areas and target organs in medical images, mainly targeting the problem of low segmentation accuracy of small targets. The constructed segmentation model mainly includes two core modules: a dual-branch feature fusion module DFFM and a reverse attention context module RACM. First, a dual-branch feature encoder is used to extract multi-resolution features, and then a DFFM is constructed to aggregate global and local context information to achieve information complementarity between features and provide sufficient guidance for accurate segmentation of small targets. In order to alleviate the decrease in segmentation accuracy caused by blurred boundaries in medical images, a reverse attention context module is proposed to enhance the edge texture of features. The present invention uses six different data sets as training sets. Experiments have shown that the method of the present invention has better segmentation accuracy than existing methods with fewer parameters, faster inference speed and lower model complexity.

Description

Technical Field

[0001] The present invention relates to the field of small object segmentation in medical images, and in particular to a small object segmentation method in medical images based on dual-branch feature fusion attention. Background Art

[0002] Medical image segmentation is the process of segmenting certain meaningful parts of a medical image, providing a reliable basis for clinical treatment and case studies. Previously, image segmentation used threshold-based methods to classify image pixels and perform segmentation based on the classification results, such as the adaptive threshold method and the dual threshold method. Although these methods are computationally simple, they have poor segmentation performance for images with similar pixel values ​​and overlapping boundaries between different categories, and determining a reasonable threshold is also very challenging. In addition, a segmentation algorithm based on boundary tracking and edge detection was proposed, which uses the drastic changes in edge pixels in different regions for segmentation. However, the segmentation results lack a complete regional structure and are subject to strong noise interference. Subsequently, an area-based segmentation algorithm was proposed, which uses the average features of the same part of the image for segmentation, but it is easy to cause over-segmentation of the image.

[0003] With the widespread application of deep learning, medical image segmentation technology has rapidly advanced. For example, the FCN network model replaces the fully connected layers of CNNs with convolutional layers, achieving a leap from image-level classification to pixel-level classification. However, this model lacks attention to detail and consideration of inter-pixel relationships, resulting in rough segmentation results. As an improvement on FCN, the U-Net model connects contracting paths with expanding paths through short links, enabling interaction between shallow and deep feature information. This network significantly improves prediction accuracy. However, this model suffers from computational redundancy, resulting in reduced computational speed, and it cannot simultaneously achieve both segmentation accuracy and pixel localization precision, resulting in poor generalization across datasets. Subsequently, many extensions of these models have been proposed. For example, the Attention U-Net suppresses learning of irrelevant regions in the input image and highlights useful features for specific tasks by adding an attention mechanism to the U-Net. Unet++ adds dense convolutional blocks to the U-Net, making the encoder and decoder semantics more similar. However, these models lack smooth edge segmentation, increase network parameters, and consume a large amount of memory. The fast segmentation network Unext obtains feature location information by marking and projecting convolutional features, completing segmentation at a faster speed. However, this model performs poorly for images with small object areas and small amounts of data.

[0004] The widespread application of Transformers in visual tasks subsequently promoted progress in medical image segmentation. Hu Han et al. constructed the Swin Transformer network model. It uses a sliding window operation to compute self-attention across non-overlapping windows, restricting attention to a single patch and incorporating positional information from CNN convolutions. Inspired by this, Swin-UNet first combined the Transformer and U-Net. This model replaces convolutional blocks with Transformer modules, incorporating positional information into feature maps and enabling interaction between local and global features. Simultaneously, UTNet applies self-attention modules in both the encoder and decoder to capture long-range dependencies at varying scales with minimal overhead. These models incorporate the Transformer into the U-Net architecture, overcoming the difficulty of extracting long-range information through convolution and addressing the data-intensive nature of the Transformer's learning of visual biases. However, the large number of parameters generated by the multi-head attention mechanism, positional encoding, and sliding windows results in high computational complexity and poor data generalization.

[0005] Furthermore, BisNet and BisNetV2 extract multi-scale features through detail branches and semantic branches, respectively, and then use specific modules to fuse features to obtain rich feature information. This ultimately improves network performance. Fast-SCNN uses dense convolution as the semantic branch and deep convolution as the structural branch to extract multi-scale features, then fuses features to predict segmentation results. However, these bilateral networks have significant limitations in terms of task types. They are primarily targeted at real-time semantic segmentation tasks and perform poorly on pixel-level segmentation tasks, particularly class-imbalanced medical image segmentation and small object segmentation.

[0006] Deficiencies in existing technologies:

[0007] 1. The network's feature fusion integrates all the encoder outputs, and does not highlight shallow detail features during the decoding process.

[0008] Some image segmentation methods generally use multi-scale feature fusion, but most of them directly combine the outputs of each encoder layer for decoding, which limits the network's ability to obtain shallow image detail features. As a result, the final segmentation results often suffer from classification errors and large edge texture errors. Therefore, it is necessary to sequentially fuse the features of each layer of the two branches to provide multi-level semantic information to the decoder. At the same time, feature fusion differs from simple feature combination (such as pixel-by-pixel addition and channel splicing) by adaptively integrating local features and global dependencies to effectively filter out other useless information in the space.

[0009] 2. The network training time and computing resource space consumption are large.

[0010] Most existing image segmentation methods incorporate the Transformer architecture to overcome the difficulty convolutional neural networks (CNNs) have in extracting long-range information, thereby improving segmentation accuracy. However, multi-head attention and positional encoding generate a large number of parameters during implementation, increasing model training time and consuming significant space resources. Therefore, it is necessary to design networks that require less training time and lower computational cost to improve training efficiency.

[0011] 3. The network does not consider the weakening of edge features in the process of integrating global and local features, resulting in over-segmentation or segmentation failure.

[0012] Existing image segmentation networks neglect the enhancement of edge features across diverse tasks. Some networks consider using subtraction to highlight edge features, but this approach is only applicable to single-label tasks. Therefore, it is necessary to design a model that enhances deep-level category outline information to obtain edge-enhanced feature maps. This model, applicable to diverse tasks, can address the degradation of segmentation accuracy caused by blurred edges. Summary of the Invention

[0013] In view of the shortcomings of the existing technology, this paper proposes a small target segmentation method for medical images based on dual-branch feature fusion attention.

[0014] The proposed method constructs a segmentation network for small target areas in medical images. Six different datasets are used as training sets for learning. The constructed segmentation network includes a feature encoder, a feature fusion module, and a reverse attention context module. The feature encoder is used to extract multi-resolution features, the feature fusion module is used to aggregate global and local context information, and the attention context module is used to enhance the edge texture of features. Specifically, the following are the steps:

[0015] Step 1: Prepare the datasets required for medical image segmentation, including six datasets: 3D multi-class nasopharyngeal carcinoma CT images, 3D single-class nasopharyngeal carcinoma MRI images, 3D cardiac MRI images, and Polyp polyp mixed dataset.

[0016] Step 2: Perform the first step of preprocessing on the six different data sets obtained in step 1, and divide them into training sets and test sets according to the agreed ratio;

[0017] Step 3: Perform the second step of preprocessing on the training set and the test set. At the same time, construct and initialize a dual-branch feature fusion attention small object segmentation network, specifically including:

[0018] Step 31: For the three-dimensional single-category nasopharyngeal carcinoma MRI dataset and the Polyp polyp dataset, convert the original images into three-channel RGB images, convert the corresponding label images into binary grayscale images, and perform data enhancement preprocessing on the binary grayscale images;

[0019] Step 32: Construct a small object segmentation network, which includes a dual-branch encoder, three feature fusion modules, and a decoder. The dual-branch encoder of the segmentation network is composed of two parallel independent classification backbone networks. The first branch encoder uses a ResNet34 network as a detail branch, and the second branch encoder uses a ShuffleNet network as a semantic branch to extract multi-scale feature information of the input image; the three feature fusion modules are respectively inserted between the dual-branch encoders to fuse the feature information of the same layer in the dual-branch encoder; the decoder of the segmentation network is composed of two stacked reverse attention context modules to recover the fused features of different scales;

[0020] Step 4: Input the training set processed in step 31 into the constructed segmentation network to train the network, specifically including:

[0021] Step 41: The same original image in the training set is fed into two parallel branch encoders of the encoder to extract feature information. Each branch outputs three layers of feature maps. The output of the detail branch is represented as a detail feature map R i , specifically including the first detail feature map R1, the second detail feature map R2, and the third detail feature map R3. The third detail feature map R3 is upsampled to generate the first auxiliary feature map f1. The output of the semantic branch is represented as a semantic feature map S i , specifically including a first semantic feature map S1, a second semantic feature map S2, and a third semantic feature map S3, wherein the third semantic feature map S3 is upsampled to generate a second auxiliary feature map f2, i=1, 2, 3;

[0022] Step 42: Then pass the detail feature map and the semantic feature map into the corresponding feature fusion module to obtain the fused feature map B i Specifically, R1 and S1 are sent to the third feature fusion module to obtain the third fused feature map B3, R2 and S2 are sent to the second feature fusion module to obtain the second fused feature map B2, and the second fused feature map B2 is upsampled to generate the fourth auxiliary feature map f4, R3 and S3 are sent to the first feature fusion module to obtain the first fused feature map B1, and the first fused feature map B1 is upsampled to generate the third auxiliary feature map f3, which adaptively integrates local features and global dependencies to output rich semantic information;

[0023] Step 43: The first fused feature map B1 and the second fused feature map B2 are passed to the decoder, the categories are strengthened by the first reverse attention context module, and the features are recovered using the shallow detail features to obtain a first feature recovery map RS1;

[0024] Step 44: Upsample the first fusion feature B1 to 2 times and perform channel concatenation with the first feature recovery map RS1, and then pass it into the double convolution combination to obtain the first stage feature prediction map f5;

[0025] Step 45: Then, the feature prediction map f5 of the first stage and the third fused feature map B3 output by the shallow third feature fusion module are used as the input of the second reverse attention context module, and step 43 is repeated to finally obtain the prediction result Pre;

[0026] Step 5: Calculate the loss of the prediction result Pre of step 45 and the label image, and respectively calculate the loss of the first auxiliary feature map f1, the second auxiliary feature map f2, the third auxiliary feature map f3, the fourth auxiliary feature map f4 and the label image, and the loss of the feature prediction map f5 and the label image, perform backpropagation on the loss value, and adjust the network parameters;

[0027] Step 6: After 10 training cycles of steps 4 and 5, the test set is passed to the trained image segmentation network for testing. It is determined whether the similarity Dice index of the current test of the image segmentation network is the highest. If so, the current model parameters are saved. Otherwise, jump to step 4 for the next round of training.

[0028] Step 7: Take out the network with the best performance during the test, and then feed the test set into the network in turn, calculate the average intersection over union (mIoU) and average similarity (mDice), and save the final predicted segmentation map.

[0029] According to a preferred embodiment, the fusion features of step 42 specifically include:

[0030] Step 421: First, adjust the input detail feature map R using the convolution kernel i resolution;

[0031] Step 422: Then, global average pooling is used to assign the actual category meaning to the detail feature map R i Each channel is convolved with the semantic feature map S i Perform channel splicing to complete the integration of multi-semantic information, emphasize the location characteristics of the target, and output the feature map F;

[0032] Step 423: Then perform Sigmoid activation on the feature map F, and i Get effective context information from the global attention feature map F s ;

[0033] Step 424: Finally, the convolved global attention feature map F s Adjust the resolution and inject it into the convolutional detail feature map Ri , adaptively integrate global and local dependencies, calculate their similarity, and output a multi-resolution fusion feature map B i .

[0034] According to a preferred embodiment, the feature recovery process in step 43 specifically includes:

[0035] Step 431: First, the first fused feature map B1 is bilinearly upsampled twice and then input into the first reverse attention context module with the second fused feature map B2. The channel space of the first fused feature map B1 is adjusted by convolution with a convolution kernel of 1X1. Then, the first fused feature map B2 and the second fused feature map B2 after convolution are channel-concatenated to reset the channel mapping relationship to obtain the cascade feature map F. r ;

[0036] Step 432: Then, the concatenated feature map F is changed by the Relu activation function and the 1X1 convolution operation. r The nonlinear features captured in the , get the detailed feature map F containing the category features m ;

[0037] Step 433: Further process the detailed feature map F m Perform Sigmoid activation to obtain a detailed pixel-by-pixel attention map, which is applied to the first fusion feature map B1 to enhance the edge features and obtain the edge feature weighted map F. w ;

[0038] Step 434: Finally, the edge weighted graph F after convolution w The pixel weight difference extracted from the second fusion feature B2 is injected to supplement the category feature and output the first feature recovery image RS1 of the first stage. Compared with the prior art, the beneficial effects of the present invention are:

[0039] 1. This paper designs an effective small object segmentation network (DFF-Net) based on dual-branch feature fusion attention to address the problem of low segmentation accuracy for small objects in medical images. With a low number of parameters, the proposed network can accurately segment the target region from medical images with blurred edges and class imbalance.

[0040] 2. The image segmentation network involved proposes a dual-branch feature fusion module (DFFM) to fuse multi-resolution feature maps to guide the accurate segmentation of small objects; and a reverse attention context module (RACM) to enhance the edge texture of features and alleviate the problem of accuracy loss caused by edge blur in medical images.

[0041] 3. After a large number of experiments, it is proved that the small target segmentation network with dual-branch feature fusion attention proposed in this invention has higher accuracy, fewer parameters, faster inference speed and better generalization in small target segmentation than the current segmentation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the network structure proposed by the present invention;

[0043] Figure 2 It is a structural diagram of the feature fusion module of the present invention;

[0044] Figure 3 Schematic diagram of the structure of the reverse attention context module of the present invention;

[0045] Figure 4 This is a comparison chart of experimental results on the right atrium dataset. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0047] The DFFM of the present invention refers to: Dual-branch Feature Fusion Module, dual-channel feature fusion module.

[0048] The RACM of the present invention refers to: Reverse Attention Context Module, reverse attention context module.

[0049] The present invention relates to the field of image semantic segmentation research, and in particular to the segmentation of cancerous areas and target organs in medical images. It primarily addresses the problem of low segmentation accuracy for small targets. Current convolutional neural networks have difficulty extracting long-range information and lack spatial information about the target, which can easily lead to over-segmentation of small targets. With the development of machine learning, the Transformer has been proposed to capture long-range dependencies of varying scales and to facilitate information exchange between global and local features. However, the multi-head attention, position encoding, and window sliding introduced by the Transformer generate a large number of parameters, resulting in a large amount of model computation and poor data generalization capabilities.

[0050] In view of the shortcomings of the existing solutions, this paper proposes a small target segmentation network DFF-Net based on dual-branch feature fusion attention. Figure 1This is a schematic diagram of the DFF-Net network structure proposed in this invention. Its input data is the original image, and its output is the segmentation prediction result Pre. Here, the resolution of X is CXWXH, where C represents the number of channels, H represents the image height, and W represents the image width. The network of this invention comprises two three-layer classification backbone networks (ResNet and ShuffleNet), arranged in parallel to form a feature extractor. It also includes a dual-channel feature fusion module (DFFM) for same-layer feature fusion and a reverse attention context module (RACM) to enhance edge features and provide accuracy compensation during the decoding process.

[0051] The core and main innovation of the image segmentation network of the present invention lies in the feature fusion module DFFM and the reverse attention context module RACM. Figure 2 This is a schematic diagram of the feature fusion module DFFM proposed in the present invention, which combines the output R i and S i (i=1,2,3) as input, fusion feature B i (i=1,2,3) as output. It is used to fuse the features of the same layer of the encoder, adaptively integrate local features and global dependencies, filter out other useless information in the space, and obtain rich and effective semantic information. In DFFM, adjust R i resolution and perform global average pooling to give R i Each channel of is assigned the actual category meaning. Then the convolution R is integrated through channel cascade. i and S i The multi-semantic information of the channel is used in this stage to complete the feature integration, which can better emphasize the position characteristics of small objects, and perform convolution operations and Sigmoid activation on the cascade feature map F for S i Get effective context information from the global attention feature map F s Finally, the convolution F s Injected into R after convolution i , adaptively integrate global and local dependencies, calculate their similarity, and output a multi-resolution feature aggregation map B i The feature maps processed by DFFM help focus attention on the target area during the decoding stage and prevent the loss of small target features during training.

[0052] Figure 3This is a structural diagram of the reverse attention context module of the present invention. In medical image segmentation, the accuracy of edge information often determines the effectiveness of feature recovery and the accuracy of segmentation results, and RACM is used to enhance edge features to improve segmentation accuracy. This module takes the outputs B1 and B2 of DFFM as input and the feature recovery maps RS1 and RS2 as output. First, the channel mapping relationship of B1 is adjusted and B1 and B2 are spliced ​​as feature maps. Then, the pixel weights are calculated through convolution and Sigmoid operations to increase the importance of edge features. Then, global features are filtered to highlight category information and increase the capture of nonlinear features. Finally, detailed appearance features are injected into high-level semantic features to restore feature information and obtain the prediction result RS i (i=1,2,3). In actual decoding, two RACMs are connected in series to complete segmentation prediction. The first RACM extracts feature information from B1 and B2 to generate the feature recovery map RS1. Skip links are added to ensure that the model fully utilizes spatial features from deeper layers. This allows the network to obtain a richer receptive field and generate a high-resolution feature map F with category information. F is then processed through DoubleConv and combined with B3 as the input to the second RACM. The internal RACM operations are repeated to obtain the final prediction result.

[0053] Existing medical image segmentation methods simply integrate local and global information for feature recovery, such as direct addition, which weakens the extraction of small target boundary features, resulting in excessive segmentation area or segmentation failure. In this paper, we consider using category information to highlight edge texture information to accurately segment cancerous areas or organs with blurred boundaries, including small targets. In addition, Dice loss and cross entropy loss (CE loss or BCE loss) are introduced during the training process to improve the network's prediction accuracy.

[0054] The method proposed in this paper constructs a segmentation network for small target areas in medical images. Six different datasets are used as training sets for learning. The constructed segmentation network includes a feature encoder, a feature fusion module DFFM, and a reverse attention context module RACM. The feature encoder is used to extract multi-resolution features, the feature fusion module is used to aggregate global and local context information, and the attention context module is used to enhance the edge texture of the features. Specifically, it includes:

[0055] Step 1: Prepare the datasets required for medical image segmentation, including six datasets: 3D multi-class nasopharyngeal carcinoma CT images, 3D single-class nasopharyngeal carcinoma MRI images, 3D cardiac MRI images, and Polyp polyp mixed dataset.

[0056] Step 11: Obtain 3D multi-class nasopharyngeal carcinoma CT image data Ml_NPC of 124 patients from the hospital's CT scanner;

[0057] Step 12: Obtain 3D single-category nasopharyngeal carcinoma MRI image data Sl_NPC of 120 patients from the hospital's Philips MRI scanner;

[0058] Step 13: Download the public dataset from the official website of the Automatic Cardiac Diagnosis Challenge, which contains 3D cardiac MRI image data of 100 patients. Extract the left atrial dataset (LACDC) and the right atrial dataset (RACDC).

[0059] Step 14: Download the divided Polyp datasets from the PraNet paper, including the Kvasir-SEG dataset and the CVC-ClinicDB dataset.

[0060] The resolution sizes of the 6 datasets are: Ml_NPC is 3X512X512, Sl_NPC is 3X160X192, LACDC is 1X128X128, RACDC is 1X128X128, Kvasir is 3X352X352, and CVC-ClinicDB is 3X352X352.

[0061] Step 2: Preprocess the six different data sets obtained in step 1 and divide them into training sets and test sets according to the agreed ratio.

[0062] The preprocessing operations for the six datasets include:

[0063] Step 21: Process the three-dimensional multi-category nasopharyngeal carcinoma data Ml_NPC, convert the three-dimensional CT image slices of each patient obtained by the CT scanner into two-dimensional CT images, and remove the unlabeled images, obtaining a total of 1622 512X512 two-dimensional CT images.

[0064] Among them, 908 images were randomly selected for training, 235 for validation, and 479 for testing. The dataset contains a total of 14 segmentation target classes, each with different shapes, categories, and region sizes.

[0065] Step 22: Process the single-class nasopharyngeal carcinoma data Sl_NPC. Because MRI images scan a large area from the head to the neck, while nasopharyngeal carcinoma occupies only a small portion of the head, only images containing nasopharyngeal carcinoma above the neck are used. Each patient's 3D MRI image slices acquired by the MRI are then converted to 2D images, and images with tumors having fewer than 100 pixels are removed. A total of 3111 160x192 2D images are obtained. 2489 of these images are randomly selected as the training set, and the remaining 622 images are used as the test set.

[0066] Step 23: Process the 3D cardiac MRI dataset. Each patient's downloaded 3D MRI image slices were converted to 2D images, and corresponding segmentation labels, including the left and right ventricles, were extracted. Since the image pixel size is 1.406 x 1.406 mm, the image size range is [0-255]. The images were cropped to a 128 x 128 resolution, and the pixel values ​​were normalized to [0-255]. Finally, a 3116-image 128 x 128 2D left atrium dataset (LACDC) was obtained, of which 2492 images were randomly selected as the training set and the remaining 624 as the test set; and a 3616-image 128 x 128 2D dataset (RACDC), of which 2892 images were randomly selected as the training set and 724 as the test set.

[0067] Step 24: Process the Polyp dataset. This paper uses the training set divided by the PraNet paper. The training set consists of 900 images from the Kvasir-SEG dataset and 550 images from the CVC-ClinicDB dataset, totaling 1450 samples. The test set consists of the remaining 100 images from Kvasir-SEG and 62 images from CVC-ClinicDB, with an image resolution of 352x352.

[0068] Step 3: Further process the training set and the test set, and at the same time, construct and initialize a dual-branch feature fusion attention small object segmentation network, specifically including:

[0069] Step 31: For the three-dimensional single-category nasopharyngeal carcinoma MRI dataset and the Polyp polyp dataset, the original images are converted into three-channel RGB images, the corresponding label images are converted into binary grayscale images, and the binary grayscale images are preprocessed for data enhancement.

[0070] The specific operations of step 31 data enhancement include:

[0071] Step 311: For six different data sets, the original images and the labeled images are randomly flipped horizontally, flipped vertically, and rotated to complete data enhancement.

[0072] Step 312: Convert the data-augmented image into a tensor format and normalize all pixel values ​​of the original image to [0-1].

[0073] Step 32: Construct a small object image segmentation network. The segmentation network includes a dual-branch encoder, three feature fusion modules, and a decoder. The dual-branch encoder of the segmentation network is composed of two parallel independent classification backbone networks. The first branch encoder uses a ResNet network as the detail branch, and the second branch encoder uses a ShuffleNet network as the semantic branch to extract multi-scale feature information of the input image. Three feature fusion modules (DFFMs) are inserted between the dual-branch encoders to fuse feature information from the same layer in the dual-branch encoder. The decoder of the segmentation network is composed of two stacked reverse attention context modules to recover features from fused features at different scales.

[0074] Step 4: Input the training set processed in step 31 into the constructed segmentation network to train the network, specifically including:

[0075] Step 41: The same original image in the training set is fed into two parallel branch encoders of the encoder to extract feature information. Each branch outputs three layers of feature maps. The output of the detail branch is represented as a detail feature map R i , specifically including the first detail feature map R1, the second detail feature map R2, and the third detail feature map R3. The third detail feature map R3 is upsampled to generate the first auxiliary feature map f1. The output of the semantic branch is represented as a semantic feature map S i , specifically including a first semantic feature map S1, a second semantic feature map S2, and a third semantic feature map S3. The third semantic feature map S3 is upsampled to generate a second auxiliary feature map f2, i=1, 2, 3; i represents the number of layers. When i is the same, it represents the feature map of the same layer in two parallel branches.

[0076] Step 42: Then pass the detail feature map and the semantic feature map into the corresponding feature fusion module to obtain the fused feature map B i Specifically, R1 and S1 are fed into the third feature fusion module to obtain the third fused feature map B3, R2 and S2 are fed into the second feature fusion module to obtain the second fused feature map B2, and the second fused feature map B2 is upsampled to generate the fourth auxiliary feature map f4, R3 and S3 are fed into the first feature fusion module to obtain the first fused feature map B1, and the first fused feature map B1 is upsampled to generate the third auxiliary feature map f3, adaptively integrating local features and global dependencies to output rich semantic information. The specific process includes:

[0077] Step 421: First, adjust the input detail feature map R using a convolution with a convolution kernel of 3X3 i resolution.

[0078] Step 422: Then, global average pooling is used to assign the actual category meaning to the detail feature map R i Each channel of the convolutional semantic feature map S i Perform channel splicing to complete the integration of multi-semantic information, emphasize the position characteristics of the target, and output the feature map F.

[0079] Step 423: Then perform Sigmoid activation on the feature map F, and i Get effective context information from the global attention feature map F s .

[0080] Step 424: Finally, the convolved global attention feature map F s Adjust the resolution and inject it into the convolutional detail feature map R i , adaptively integrate global and local dependencies, calculate their similarity, and output a multi-resolution fusion feature map B i .

[0081] Step 43: The first fused feature map B1 and the second fused feature map B2 are passed to the decoder, the categories are strengthened by the first reverse attention context module, and the features are restored using the shallow detail features to obtain the first feature recovery map RS1. Specifically:

[0082] Step 431: First, the first fused feature map B1 is bilinearly upsampled twice and then input into the first reverse attention context module with the second fused feature map B2. The channel space of the first fused feature map is adjusted by convolution with a convolution kernel of 1Î1. Then, the first fused feature map B2 and the second fused feature map B2 after convolution are channel-concatenated to reset the channel mapping relationship to obtain the cascade feature map F. r .

[0083] Step 432: Then, the concatenated feature map F is changed by the Relu activation function and the 1X1 convolution operation. r The nonlinear features captured in the , get the detailed feature map F containing the category features m .

[0084] Step 433: Further process the detailed feature map F m Perform Sigmoid activation to obtain a detailed pixel-by-pixel attention map, which is applied to the first fusion feature map B1 to enhance the edge features and obtain the edge feature weighted map F. w .

[0085] Step 434: Finally, the edge weighted graph F after convolution wThe pixel weight difference extracted from the second fusion feature B2 is injected to supplement the category feature and output the first feature recovery map RS1 of the first stage.

[0086] At the same time, skip links are added to ensure that the model fully utilizes spatial features from deeper layers, so that the network can obtain a richer receptive field and generate a high-resolution feature map F with category information.

[0087] Step 44: Upsample the first fusion feature B1 to 2 times and perform channel splicing with the first feature recovery map RS1, and then pass it into the double convolution combination to obtain the feature prediction map f5 of the first stage.

[0088] Among them, the double convolution combination stacks two convolution combinations, including a 3X3 convolution layer, a batch normalization layer and a Relu activation function.

[0089] Step 45: Then, the feature prediction map f5 of the first stage and the third fused feature map B3 output by the shallow third feature fusion module are used as the input of the second reverse attention context module, and step 43 is repeated to finally obtain the segmentation prediction result Pre.

[0090] Step 5: Calculate the loss between the prediction result Pre of step 45 and the label image, and respectively calculate the loss between the first auxiliary feature map f1, the second auxiliary feature map f2, the third auxiliary feature map f3, the fourth auxiliary feature map f4 and the label image, and the loss between the feature prediction map f5 and the label image, perform backpropagation on the loss values, and adjust the network parameters. Specifically, it includes:

[0091] Step 51: If the current training data is multi-category data, calculate the loss of the multi-category data and use the joint loss of Dice loss and cross entropy loss CE.

[0092] Step 52: If the current training data is single-category data, calculate the loss of the single-category data, using the joint weighted loss of IoU loss and binary cross entropy loss BCE, and Dice loss.

[0093] Step 53: In order to better train the model, the present invention combines the main loss and the auxiliary loss to constrain the degree of difference between the predicted image and the true label, and passes the loss calculation value into the network for back propagation to adjust the network parameters.

[0094] Step 6: After 10 training cycles of steps 4 and 5, the test set is passed to the trained image segmentation network for testing to determine whether the similarity Dice index of the segmentation network is the highest. If so, the current model parameters are saved; otherwise, the process jumps to step 4 for the next round of training.

[0095] Step 7: Take out the network with the best performance during the test, and then feed the test set into the network in turn, calculate the average intersection over union (mIoU) and average similarity (mDice), and save the final predicted segmentation map.

[0096] To verify the effectiveness of our method, we compared it with other existing methods. For a fair comparison, we used the officially released code from the other methods and followed their experimental setup. All methods were implemented in the same computing environment, with the same data augmentation method, and both quantitative and qualitative analysis were performed. The eight methods compared were:

[0097] Method 1: U-Net is a classic cell segmentation network that uses encoder-decoder combined with skip connections to achieve accurate semantic segmentation.

[0098] Method 2: UNet++ adds an attention mechanism to UNet to increase the accuracy of model segmentation;

[0099] Method 3: The HrNet method constructs multiple parallel branches and enhances semantic information and location information through multi-resolution information interaction to improve segmentation performance;

[0100] Method 4: DeepLabV3+ method is used to solve the problems of low accuracy of target segmentation and loss of boundary information at multi-scale. It reconsiders the characteristics of the dilated convolution and ASPP modules to improve segmentation accuracy.

[0101] Method 5: The BisNetV2 method is a classic two-branch network model that designs separate detail branches and semantic branches to achieve high-precision and high-efficiency real-time semantic segmentation;

[0102] Method 6: TransFuse is a powerful dual-branch segmentation network for medical images. It uses a Transformer network to extract global features, a CNN to obtain detailed features, and a feature fusion module to improve the final segmentation performance.

[0103] Method 7: The Polyp-PVT method is a network model for polyp segmentation, which mainly includes a cascade fusion module, a camouflage recognition module, and a similarity aggregation module, and has excellent segmentation performance;

[0104] Method 8: UNext is a medical image segmentation network based on convolutional multi-layer perceptron, which can achieve fast and accurate skin lesion segmentation.

[0105] Tables 1 and 2 respectively give the quantitative comparison results of 6 different data sets on 8 different methods with the average similarity index and average intersection-over-union index of the present invention.

[0106] Table 1 Comparison of average similarity indicators of different methods on different datasets

[0107]

[0108] Table 2 Comparison results of average intersection-over-union (IoU) indicators of different methods on different datasets

[0109]

[0110] The mDice metric is used to calculate the similarity between two samples, with a general range of values ​​between [0, 1]. The closer the mDice metric is to 1, the better. In Table 1, the percentage counting method is used to expand the measurement metric by 100 times. The mIoU metric is used to measure the correlation between the true and predicted values. The higher the correlation, the better. In Table 2, percentage counting is also used. A comparison of the results in Tables 1 and 2 shows that the segmentation results of the method of the present invention are more similar to the labeled image of the original image than all other methods, and the segmentation accuracy is higher.

[0111] In order to more intuitively illustrate the effectiveness of the method of the present invention, right atrial data is taken as an example to compare the segmentation effects of the existing method and the method of the present invention after processing the right atrial data. Figure 4 This is the qualitative result of the present invention on the right atrium dataset. Each column represents the segmentation results of different methods on different original images of the same dataset. Specifically, Figure 4 (a) represents the segmentation result of the U-Net method, Figure 4 (b) represents the segmentation result of the UNet++ method. Figure 4 (c) represents the segmentation result of the HrNet method. Figure 4 (d) shows the segmentation result of DeepLabV3+ method. Figure 4 (e) represents the segmentation result of the BesNetV2 method. Figure 4 (f) represents the segmentation result of the TransFuse method. Figure 4 (g) represents the segmentation result of the Polyp-PVT method. Figure 4 (h) represents the segmentation result of the UNext method. Figure 4 (i) represents the segmentation result of the method of the present invention, Figure 4 (j) represents the label image. Comparing the various methods, it can be seen that the method of the present invention is more accurate in classification than all other methods, and the target contour is more realistic and complete.

[0112] To investigate the effectiveness of the DFFM and RACM modules in improving performance, we conducted ablation experiments by removing both modules, then performing them separately. The results were compared with the full DFF-Net network model. Tables 3 and 4 present the mDice and mIoU results, respectively. Combining Tables 3 and 4, we can see that removing both modules results in a decrease in both metrics. This demonstrates the effectiveness of the DFFM and RACM modules.

[0113] Table 3 Average similarity index of ablation experiments

[0114]

[0115] Table 4 Average Intersection-over-Union (IoU) metrics of ablation experiments

[0116]

[0117] In order to study the relationship between the number of parameters and the computational speed of the present invention, the number of parameters and the computational speed of the present invention and other existing methods were calculated, as shown in Table 5. Params represents the number of model parameters, which is used to measure the complexity of the model. The smaller the index, the better. In Table 5, the unified parameter unit is M. Floating Points of Operations (FLOPs) refers to the number of floating-point operations, which is used to measure the computational complexity of the model. The lower the index, the better. GFLOPs represents 1 billion floating-point operations, that is, 1GFLPs = 10 9 FlOPs. Frames per second (FPS) refers to the number of frames per second and is used to measure the real-time performance of a model; a higher FPS is better. Table 5 shows that, with the exception of the lightweight networks BisNetV2 (Method 5) and UNext (Method 8), the proposed method has lower computational complexity, better real-time performance, and relatively fewer model parameters than other existing advanced network models.

[0118] Table 5 Comparison of parameters and calculation speed of different methods

[0119]

[0120] It should be noted that the above-described specific embodiments are illustrative only. Those skilled in the art may devise various solutions based on the disclosure of the present invention, and such solutions fall within the scope of the present invention and are intended to be protected by the present invention. Those skilled in the art should understand that the present description and its accompanying drawings are intended to be illustrative only and are not intended to limit the scope of the claims. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A small object segmentation method for medical images based on dual-branch feature fusion attention, characterized by: The method constructs a segmentation network for small target areas in medical images, using six different datasets as training sets for learning. The constructed segmentation network includes a feature encoder, a feature fusion module, and a reverse attention context module. The feature encoder is used to extract multi-resolution features, the feature fusion module is used to aggregate global and local context information, and the reverse attention context module is used to enhance the edge texture of the features. Specifically, Step 1: Prepare the datasets required for medical image segmentation, including six datasets: 3D multi-class nasopharyngeal carcinoma CT images, 3D single-class nasopharyngeal carcinoma MRI images, 3D cardiac MRI images, and Polyp polyp mixed dataset. Step 2: Perform the first step of preprocessing on the six different data sets obtained in step 1, and divide them into training sets and test sets according to the agreed ratio; Step 3: Perform the second step of preprocessing on the training set and the test set. At the same time, construct and initialize a dual-branch feature fusion attention small object segmentation network, specifically including: Step 31: For the three-dimensional single-category nasopharyngeal carcinoma MRI dataset and the Polyp polyp dataset, convert the original images into three-channel RGB images, convert the corresponding label images into binary grayscale images, and perform data enhancement preprocessing on the binary grayscale images; Step 32: Construct a small object segmentation network, which includes a dual-branch feature encoder, three feature fusion modules, and a decoder. The dual-branch feature encoder of the segmentation network is composed of two parallel independent classification backbone networks. The first branch encoder uses a ResNet network as a detail branch, and the second branch encoder uses a ShuffleNet network as a semantic branch to extract multi-scale feature information of the input image; the three feature fusion modules are respectively inserted between the dual-branch feature encoders to fuse feature information of the same layer in the dual-branch feature encoder; the decoder of the segmentation network is composed of two stacked reverse attention context modules to recover features of fused features of different scales; Step 4: Input the training set processed in step 31 into the constructed segmentation network to train the network, specifically including: Step 41: The same original image in the training set is fed into two parallel branch encoders of the encoder to extract feature information. Each branch outputs three layers of feature maps. The output of the detail branch is represented as a detail feature map R i , specifically including the first detail feature map R1, the second detail feature map R2, and the third detail feature map R3. The third detail feature map R3 is upsampled to generate the first auxiliary feature map f1. The output of the semantic branch is represented as a semantic feature map S i , specifically including a first semantic feature map S1, a second semantic feature map S2, and a third semantic feature map S3, wherein the third semantic feature map S3 is upsampled to generate a second auxiliary feature map f2, i=1, 2, 3; Step 42: Then pass the detail feature map and the semantic feature map into the corresponding feature fusion module to obtain the fused feature map B i Specifically, R1 and S1 are sent to the third feature fusion module to obtain the third fused feature map B3, R2 and S2 are sent to the second feature fusion module to obtain the second fused feature map B2, the second fused feature map B2 is upsampled to generate the fourth auxiliary feature map f4, R3 and S3 are sent to the first feature fusion module to obtain the first fused feature map B1, the first fused feature map B1 is upsampled to generate the third auxiliary feature map f3; Step 43: The first fused feature map B1 and the second fused feature map B2 are passed to the decoder, the categories are strengthened by the first reverse attention context module, and the features are recovered using the shallow detail features to obtain a first feature recovery map RS1; Step 44: upsample the first fusion feature map B1 to 2 times and perform channel concatenation with the first feature recovery map RS1, and then pass it into the double convolution combination to obtain the first stage feature prediction map f5; Step 45: Then, the feature prediction map f5 of the first stage and the third fused feature map B3 output by the shallow third feature fusion module are used as the input of the second reverse attention context module, and step 43 is repeated to finally obtain the prediction result Pre; Step 5: Calculate the loss of the prediction result Pre of step 45 and the label image, and respectively calculate the loss of the first auxiliary feature map f1, the second auxiliary feature map f2, the third auxiliary feature map f3, the fourth auxiliary feature map f4 and the label image, and the loss of the feature prediction map f5 and the label image, perform backpropagation on the loss values, and adjust the network parameters; Step 6: After 10 training cycles of steps 4 and 5, the test set is passed to the trained segmentation network for testing to determine whether the similarity Dice index of the segmentation network is the highest. If so, the current model parameters are saved; otherwise, the process jumps to step 4 for the next round of training. Step 7: Take out the network with the best performance during the test, and then feed the test set into the network in turn, calculate the average intersection over union (mIoU) and average similarity (mDice), and save the final predicted segmentation map Pre.

2. The method for segmenting small objects in medical images according to claim 1, wherein: The fusion features of step 42 specifically include: Step 421: First, adjust the input detail feature map R using a convolution with a convolution kernel of 3×3 i resolution; Step 422: Then, global average pooling is used to assign the actual category meaning to the detail feature map R i Each channel is convolved with the semantic feature map S i Perform channel splicing to complete the integration of multi-semantic information, emphasize the location characteristics of the target, and output the feature map F; Step 423: Then perform Sigmoid activation on the feature map F, and i Get effective context information from the global attention feature map F s ; Step 424: Finally, the convolved global attention feature map F s Adjust the resolution and inject it into the convolutional detail feature map R i , adaptively integrate global and local dependencies, calculate their similarity, and output a multi-resolution fusion feature map B i .

3. The method for segmenting small objects in medical images according to claim 2, wherein: The feature recovery process in step 43 specifically includes: Step 431: First, the first fused feature map B1 is bilinearly upsampled by two times and then input into the first reverse attention context module with the second fused feature map B2. The channel space of the first fused feature map B1 is adjusted by convolution with a convolution kernel of 1×1. Then, the first fused feature map B2 and the second fused feature map B2 after convolution are channel-concatenated to reset the channel mapping relationship to obtain the cascade feature map F. r ; Step 432: Then, the concatenated feature map F is modified by the ReLU activation function and the 1×1 convolution operation. r The nonlinear features captured in the , get the detailed feature map F containing the category features m ; Step 433: Further process the detailed feature map F m Perform Sigmoid activation to obtain a detailed pixel-by-pixel attention map, which is applied to the first fusion feature map B1 to enhance the edge features and obtain the edge feature weighted map F. w ; Step 434: Finally, the edge weighted graph F after convolution w The pixel weight difference extracted from the second fusion feature B2 is injected to supplement the category feature and output the first feature recovery map RS1 of the first stage.

Images