Image segmentation method and device, electronic equipment and computer storage medium

Abstract

The application relates to the fields of network models and image processing, and provides an image segmentation method and device, electronic equipment and a computer storage medium, the method comprising the following steps: performing feature extraction and feature compression on an input original feature map through a double residual structure module encoder to obtain a final output feature map of the original feature map; performing integration and mapping on the final output feature map through the double residual structure module encoder to obtain a first to-be-fused feature map and a second to-be-fused feature map; and performing feature fusion on the first to-be-fused feature map and the second to-be-fused feature map through a feature fusion enhancement module double decoder to obtain a final fused feature map. The image segmentation method provided in the application embodiment has strong feature extraction capability of the double residual structure module encoder, so that the extracted feature map has better semantic expression capability; in combination with the strong feature fusion capability of the feature fusion enhancement module double decoder, the accuracy of image segmentation is improved.

Description

Technical Field

[0001] This application relates to the fields of network models and image processing, and in particular to an image segmentation method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] With the application of computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound imaging technologies in medical imaging, an increasing number of medical images require faster and more accurate analysis and interpretation. However, existing traditional image segmentation methods require manual intervention. Due to disadvantages such as manually designed features, the accuracy of image segmentation cannot be guaranteed, and end-to-end automatic image segmentation without manual intervention is not possible. Existing convolutional neural network-based image segmentation methods face the following problems: 1. Due to the difficulty of annotation, the amount of medical image data is often relatively small, which is not conducive to network training; 2. Medical images are affected by the acquisition equipment, resulting in inconsistent data formats and difficulties in data preprocessing; 3. The tissues, cells, and other regions to be segmented in medical images have diverse structures and sizes, which is not conducive to feature extraction from targets of different sizes; 4. Some medical images, such as ultrasound images, are blurry, which is not conducive to feature extraction by the network. Therefore, the segmentation accuracy of existing image segmentation methods is relatively low. Summary of the Invention

[0003] This application provides an image segmentation method, apparatus, electronic device, and computer storage medium, aimed at improving the accuracy of image segmentation.

[0004] In a first aspect, this application provides an image segmentation method, comprising:

[0005] The input original feature map is processed by a dual residual structure module encoder to extract and compress features, resulting in the final output feature map.

[0006] The final output feature map is integrated and mapped by the dual residual structure module encoder to obtain the first feature map to be fused and the second feature map to be fused.

[0007] The first and second feature maps to be fused are fused using a dual decoder through a feature fusion enhancement module to obtain the final fused feature map.

[0008] In one embodiment, the step of extracting and compressing features from the input original feature map using a dual residual structure modular encoder to obtain the final output feature map of the original feature map includes:

[0009] Based on the convolution, batch normalization, and activation functions in the dual residual structure module encoder, feature extraction is performed on the original feature map to obtain the original features;

[0010] The original features are linearly added to the original feature map using residual edges to perform feature weighting, resulting in a weighted feature map.

[0011] The final output feature map is obtained by combining the weighted feature map with the attention mechanism in the dual residual structure module encoder.

[0012] The final output feature map is obtained by combining the weighted feature map with the attention mechanism in the dual residual structure module encoder, including:

[0013] The weighted feature map is passed through the activation function, and the average value of the feature map in the width and height dimensions is taken to obtain the weight vector of each channel.

[0014] The first weight coefficient of each channel is obtained by applying a linear mapping and the activation function to each of the weight vectors.

[0015] The final output feature map is obtained based on each of the first weight coefficients and the original feature map.

[0016] The process of obtaining the final output feature map based on each of the first weight coefficients and the original feature map includes:

[0017] Each of the first weight coefficients is element-wise multiplied with the original feature map to obtain the enhanced feature map.

[0018] The enhanced feature map is linearly superimposed with the original feature map using residual edges to obtain the final output feature map.

[0019] The process of integrating and mapping the final output feature map through the dual residual structure module encoder to obtain a first feature map to be fused and a second feature map to be fused includes:

[0020] The final output feature map is integrated by performing a convolution of a first preset size in the dual residual structure module encoder to obtain an integrated feature map.

[0021] The integrated feature map is feature-mapped by a preset number of convolutions of a second preset size in the dual residual structure module encoder, resulting in a first feature map to be fused with the same number of first original channels as the final output feature map, and a second feature map to be fused with twice the number of channels as the first original channels.

[0022] The feature fusion enhancement module performs feature fusion on the first and second feature maps to be fused using a dual decoder to obtain the final fused feature map, including:

[0023] The first feature map to be fused is obtained by fusing the feature fusion enhancement module with dual decoders and feature fusion enhancement method to obtain the first target fused feature map;

[0024] The second feature map to be fused is fused by the dual decoder of the feature fusion enhancement module combined with the feature fusion enhancement method to obtain the second target fused feature map;

[0025] The first target fusion feature map and the second target fusion feature map are fused based on a convolution of a preset size to obtain the final fusion feature map.

[0026] The step of fusing the first feature map to be fused using the feature fusion enhancement module's dual decoder combined with the feature fusion enhancement method to obtain the first target fused feature map includes:

[0027] The current layer features of the first feature map to be fused are combined with the upsampled features along the channel direction to obtain a fused feature map with twice the number of channels of the second original feature map to be fused.

[0028] The fused feature map is compressed by convolutional compression to obtain a compressed feature map with the same number of channels as the second original channel;

[0029] The compressed feature map is subjected to average pooling and max pooling in the spatial dimension, and the second weight coefficients of each channel are obtained through linear mapping, weighting and activation functions.

[0030] Each of the second weight coefficients is element-wise multiplied with the first feature map to be fused to obtain the channel-enhanced feature map.

[0031] The enhanced feature map is linearly superimposed with the first feature map to be fused using residual edges, and then channel compression is performed again through convolution to obtain the first target fused feature map with half the number of channels of the second original channel.

[0032] Secondly, this application provides an image segmentation apparatus comprising:

[0033] The feature extraction and compression module is used to extract and compress features from the input original feature map through a dual residual structure module encoder to obtain the final output feature map of the original feature map.

[0034] The feature integration and mapping module is used to integrate and map the final output feature map through the dual residual structure module encoder to obtain a first feature map to be fused and a second feature map to be fused.

[0035] The feature fusion module is used to perform feature fusion on the first feature map to be fused and the second feature map to be fused through the dual decoder of the feature fusion enhancement module to obtain the final fused feature map.

[0036] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image segmentation method described in the first aspect.

[0037] Fourthly, this application also provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including a computer program, which, when executed by the processor, implements the image segmentation method described in the first aspect.

[0038] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by the processor, implements the image segmentation method described in the first aspect.

[0039] The image segmentation method, apparatus, electronic device, and computer storage medium provided in this application improve the accuracy of image segmentation by leveraging the strong feature extraction capability of the dual residual structure module encoder during the image segmentation process, thereby enabling the extracted feature maps to have better semantic representation capabilities. Furthermore, by combining the strong feature fusion capability of the dual decoder of the feature fusion enhancement module, the accuracy of image segmentation is improved. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of this application, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the image segmentation method provided in this application;

[0042] Figure 2 This is the residual structure diagram provided in this application;

[0043] Figure 3 This is a schematic diagram of the dual residual structure module encoder provided in this application;

[0044] Figure 4 This is a schematic diagram of the dual decoder structure of the feature fusion enhancement module provided in this application;

[0045] Figure 5 This is a flowchart illustrating the global feature enhancement network of this application in an application scenario;

[0046] Figure 6 This is a schematic diagram of the image segmentation apparatus provided in this application;

[0047] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] Combination Figures 1 to 7 This application describes the image segmentation method, apparatus, electronic device, and computer storage medium provided. Figure 1 This is a flowchart illustrating the image segmentation method provided in this application; Figure 2 This is the residual structure diagram provided in this application; Figure 3 This is a schematic diagram of the dual residual structure module encoder provided in this application; Figure 4 This is a schematic diagram of the dual decoder structure of the feature fusion enhancement module provided in this application; Figure 5 This is a flowchart illustrating the global feature enhancement network of this application in an application scenario; Figure 6 This is a schematic diagram of the image segmentation apparatus provided in this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application.

[0050] This application provides embodiments of an image segmentation method. It should be noted that although the logical order is shown in the flowchart, under certain data conditions, the steps shown or described may be performed in a different order than that shown here.

[0051] This application uses an electronic device as an example to illustrate the implementation, and uses an image segmentation system as one of the manifestations of the electronic device, without imposing any limitations.

[0052] Reference Figure 1 , Figure 1 This is a flowchart illustrating the image segmentation method provided in this application. The image segmentation method provided in this application includes:

[0053] Step S10: The input original feature map is processed by a dual residual structure module encoder to extract and compress features, thereby obtaining the final output feature map of the original feature map.

[0054] It should be noted that the image segmentation method of this application embodiment can be applied to medical images and other images with higher complexity than medical images.

[0055] Furthermore, this application proposes a Global Feature Enhancement (GFE-UNet) network model, which can be simply referred to as the GFE-UNet model. Compared to the original UNet model, the GFE-UNet model improves both the encoder and decoder. The original UNet model's encoder consists of stacked convolutions, normalization, activation functions, and max pooling. The GFE-UNet model replaces the convolutional layers of the original UNet model with residual blocks (SE blocks) incorporating SE attention. The SE block consists of a dual residual structure, convolutions, and SE attention, giving it stronger generalization and feature extraction capabilities. This can be understood as the encoder in this application being a dual residual structure module encoder. Simultaneously, the GFE-UNet model uses max pooling downsampling for feature compression. Finally, the dual residual structure module encoder outputs five feature layers for subsequent upsampling and feature fusion. Furthermore, unlike the skip connections between a typical encoder and decoder, the GFE-UNet model in this application proposes a dual-decoder structure and performs feature fusion through two skip connections on the feature map at each scale. In addition, during the feature fusion stage, the convolutional layer of the original Unet model is replaced by a Feature Fusion Enhancement Module (FFEM) with stronger feature fusion capabilities. That is, the decoder in this application is a dual-decoder with a feature fusion enhancement module.

[0056] Furthermore, the dual residual structure module encoder includes, but is not limited to, SE Blocks, max pooling, 3x3 convolutions, and 1x1 convolutions. In one embodiment, the dual residual structure module encoder includes four SE Blocks, four max pooling operations, one 3x3 convolution, and two 1x1 convolutions. The SE Block is a crucial module for feature extraction in the dual residual structure module encoder. This module includes a dual residual structure, two sets of convolutions, batch normalization, activation functions, and SE attention. Residual structures are widely used in ResNet, see [reference]. Figure 2 As shown, Figure 2This is the residual structure diagram provided in this application. The residual structure uses a nonlinear transformation function to describe the input and output of a network, i.e., the input is X and the output is F(x). F usually includes convolution and activation operations. When an input is forcibly added to the output of the function, the output can be split into a linear superposition of F(x) and X. The linear superposition method not only solves the gradient vanishing problem, but more importantly, it slows down the degradation of the weight matrix and improves the network's representation ability and generalization ability.

[0057] Furthermore, in the SE Block, this embodiment of the application performs residual connections after convolution and after attention feature enhancement, which strengthens gradient propagation and enhances the semantic representation of features, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the dual residual structure module encoder provided in this application. Thanks to the dual residual structure and attention mechanism, the SE block has stronger generalization and feature extraction capabilities compared to simple convolution.

[0058] Further, the image segmentation system determines the input original feature map, which can be a convolutional feature map or an unconvolutional feature map. Further, the image segmentation system extracts features from the original feature map using the dual residual structure module encoder in the GFE-UNet model. Further, the image segmentation system compresses the extracted features from the original feature map using the dual residual structure module encoder in the global GFE-UNet model to obtain the final output feature map of the original feature map, as described in steps S101 to S103.

[0059] Further, steps S101 to S103 are described as follows:

[0060] Step S101: Based on the convolution, batch normalization and activation functions in the dual residual structure module encoder, feature extraction is performed on the original feature map to obtain the original features;

[0061] Step S102: The original features and the original feature map are linearly added together using residual edges to perform feature weighting, resulting in a weighted feature map;

[0062] Step S103: Based on the weighted feature map and the attention mechanism in the dual residual structure module encoder, the final output feature map is obtained.

[0063] Furthermore, in the SE Block, this embodiment of the application performs residual connections after convolution and after attention feature enhancement, which strengthens gradient propagation and enhances the semantic representation of features, such as... Figure 3 As shown, Figure 3This is a schematic diagram of the dual residual structure module encoder provided in this application. Thanks to the dual residual structure and attention mechanism, the SE block has stronger generalization and feature extraction capabilities compared to simple convolution, as detailed below:

[0064] Specifically, refer to Figure 3 As can be seen, the original feature map is first extracted by convolution, batch normalization and activation function in the dual residual structure module encoder to obtain the original features. Specifically, the image segmentation system extracts features from the original feature map by performing convolution, batch normalization and activation function twice to obtain the original features. Among them, convolution, batch normalization and activation function (ReLU) can be abbreviated as CBR.

[0065] Further, the extracted original features and the original feature map are linearly added together using residual edges to perform feature weighting, resulting in a weighted feature map X. Further, the weighted feature map X is combined with the attention mechanism in the dual residual structure module encoder to obtain the final output feature map, as described in steps S1031 to S1033.

[0066] The embodiments of this application enable the dual residual structure module encoder to have stronger feature extraction capabilities through residual structure blocks and SE attention mechanism, resulting in extracted feature maps with better semantic expression capabilities and improving the accuracy of image segmentation.

[0067] Further, steps S1031 to S1033 are described as follows:

[0068] Step S1031: Pass the weighted feature map through the activation function and take the average value of the width and height dimensions of the feature map to obtain the weight vector of each channel;

[0069] Step S1032: Obtain the first weight coefficient of each channel by applying a linear mapping and the activation function to each weight vector;

[0070] Step S1033: Based on each of the first weight coefficients and the original feature map, the final output feature map is obtained.

[0071] Specifically, refer to Figure 3 It can be seen that by passing the weighted feature map X through the ReLU activation function and performing global pooling in the width and height dimensions of the feature map, the weight vector ρ of each channel is obtained. w Wherein, the weight vector ρ of each channel w The weight vector ρ represents the influence of each channel on the feature. w The formula can be expressed as ρ w=X.mean(dim=[h, w]), where h and w represent height and width.

[0072] Furthermore, the weight vectors ρ w The first weight coefficient w1 of each channel is obtained through linear mapping and activation functions. In one embodiment, the linear mapping can be a multilayer perceptron, and the activation functions can be ReLU activation function and sigmoid activation function. Therefore, it can be understood that each weight vector ρ w After passing through a multilayer perceptron, ReLU activation function, and sigmoid activation function, the first weight coefficient w of each channel is obtained. The formula for the first weight coefficient w can be expressed as w = sigmoid(relu(W2·relu(W1·ρ)). w In the formula, W1 and W2 matrices represent multilayer perceptrons, which is a conventional concept, and · represents matrix dot product.

[0073] Further, based on each first weight coefficient w and the original feature map, the final output feature map is obtained, as described in steps S10331 to S10332.

[0074] The embodiments of this application utilize the SE attention mechanism to enable the dual residual structure module encoder to have stronger feature extraction capabilities, resulting in extracted feature maps with better semantic representation capabilities and improved image segmentation accuracy.

[0075] Further, steps S10331 to S10332 are described as follows:

[0076] Step S10331: Perform element-wise dot product between each of the first weight coefficients and the original feature map to obtain the enhanced feature map;

[0077] Step S10332: The enhanced feature map and the original feature map are linearly superimposed with residual edges to obtain the final output feature map.

[0078] Specifically, each first weight coefficient w is element-wise multiplied with the original feature map to obtain the enhanced feature map X. * Enhanced feature map X * It can be represented as X * =w·X. Further, the enhanced feature map X... * The residual edges are linearly superimposed on the original feature map to obtain the final output feature map.

[0079] In this embodiment, the enhanced features before output are weighted by linear superposition of residual edges and input features, thus ensuring the baseline level of the features.

[0080] In one embodiment, the SE Block-based dual residual structure module encoder has a stronger feature extraction capability. For the original feature map with an input of 512*512*3, its encoder structure and the output of each layer are shown in Table 1 below. Table 1 shows the encoder structure and the output of each layer.

[0081] Table 1 Encoder structure and output of each layer

[0082] Layers Channels Filter size Padding Stride Output size SE-Block 64 3*3 1 1 512*512*64 Max pooling - 2*2 0 2 256*256*64 SE-Block 128 3*3 1 1 256*256*128 Max pooling - 2*2 0 2 128*128*128 SE-Block 256 3*3 1 1 128*128*256 Max pooling - 2*2 0 2 64*64*256 SE-Block 512 3*3 1 1 64*64*512 Max pooling - 2*2 0 2 32*32*512

[0083] The original 512*512*3 feature map is passed through a 64-channel residual structured block SE-Block to output a 512*512*64 feature map; the 512*512*64 feature map is then max-pooled using 2*2 to output a 256*256*64 feature map; the 256*256*64 feature map is then passed through a 128-channel residual structured block SE-Block to output a 256*256*128 feature map; the 256*256*128 feature map is then max-pooled using 2*2 to output a 128*128*128 feature map. The feature map is processed as follows: a 128*128*128 feature map is passed through a 256-channel residual structure block SE-Block, and then output as a 128*128*256 feature map; the 128*128*256 feature map is then max-pooled using 2*2, and then output as a 64*64*256 feature map; the 64*64*256 feature map is then passed through a 521-channel residual structure block SE-Block, and then output as a 64*64*512 feature map; the 64*64*512 feature map is then max-pooled using 2*2, and then output as a 32*32*512 feature map.

[0084] Step S20: The final output feature map is integrated and mapped by the dual residual structure module encoder to obtain the first feature map to be fused and the second feature map to be fused.

[0085] Specifically, after the input original feature map undergoes feature extraction by four SE blocks and feature compression by max pooling, it needs to be integrated and mapped by two convolutions in the dual residual structure module encoder to obtain a first feature map to be fused and a second feature map to be fused, which have the same and different number of channels as the original channel map of the final output feature map. The first feature map to be fused and the second feature map to be fused are then used as inputs to the dual decoder of the feature fusion enhancement module for feature fusion, as described in steps S201 to S202.

[0086] Further, steps S201 to S202 are described as follows:

[0087] Step S201: The final output feature map is integrated by the first preset size convolution in the dual residual structure module encoder to obtain the integrated feature map.

[0088] Step S202: The integrated feature map is feature-mapped by a preset number of second preset-size convolutions in the dual residual structure module encoder to obtain a first feature map to be fused with the same number of first original channels as the final output feature map, and a second feature map to be fused with twice the number of channels as the first original channels.

[0089] Specifically, the final output feature map is integrated by a convolution of a first preset size in the dual residual structure module encoder to obtain an integrated feature map. In one embodiment, the first preset size convolution is a 3*3 convolution. Therefore, it can be understood that the final output feature map is integrated by a 3*3 convolution in the dual residual structure module encoder to obtain an integrated feature map.

[0090] Furthermore, the integrated feature map is feature-mapped by a preset number of convolutions of a second preset size in the dual residual structure module encoder, resulting in a first feature map to be fused with the same number of original channels as the final output feature map, and a second feature map to be fused with twice the number of channels as the first original channel map. In one embodiment, the preset number is 2, and the second preset size convolution is a 1*1 convolution. This can be understood as feature mapping performed sequentially by two 1*1 convolutions in the dual residual structure module encoder, mapping the number of channels of the feature map to a specific number. After the first 1*1 convolution, the number of channels is the same as the original number of channels in the final output feature map. After the second 1*1 convolution, the number of channels becomes half of the original number of channels in the final output feature map, resulting in a first feature map to be fused with the same number of original channels as the final output feature map, and a second feature map to be fused with twice the number of channels as the first original channel map.

[0091] In one embodiment, the final output feature map is a 32*32*512 feature map. The 32*32*512 feature map is combined by a 3*3 convolution, and then combined by two 1*1 convolutions to obtain a 32*32*512 feature map and a 32*32*256 feature map, as shown in Table 2. Table 2 shows the output of feature integration and feature mapping.

[0092] Table 2 Output of Feature Integration and Feature Mapping

[0093] Layers Channels Filter size Padding Stride Output size convolutional 512 3*3 1 1 32*32*512 convolutional 512 1*1 0 1 32*32*512 convolutional 256 1*1 0 1 32*32*256

[0094] Step S30: The first feature map to be fused and the second feature map to be fused are fused by the dual decoder of the feature fusion enhancement module to obtain the final fused feature map.

[0095] It should be noted that, to improve the network's ability to analyze deep features, the GFE-Unet model employs two parallel decoders. Not only do the encoder and decoder fuse features through skip connections, but the decoders also fuse features through skip connections. This dual-decoder structure not only enhances the ability to analyze deep features, but the repeated feature fusion also enables feature reuse for targets at different scales, strengthening the semantic representation of the features. The original UNet model uses skip connections to concatenate two feature maps along the channel dimension for feature fusion, and then extracts channel information through convolution to reduce the channel dimension of the fused features. However, applying this feature fusion method to a dual-decoder makes the decoding structure very redundant, and the extraction of channel information does not fundamentally change. Therefore, this embodiment employs a feature fusion enhancement module with a dual-decoder architecture.

[0096] Furthermore, the first feature map to be fused is fused using the dual decoder of the feature fusion enhancement module to obtain a first target fused feature map. Simultaneously, the second feature map to be fused is fused using the dual decoder of the feature fusion enhancement module to obtain a second target fused feature map. Further, the first target fused feature map and the second target fused feature map are fused using a convolution of a preset size to obtain a final fused feature map, as described in steps S301 to S303, wherein the preset size of the convolution is set according to the actual situation.

[0097] Further, steps S301 to S303 are described as follows:

[0098] Step S301: The first feature map to be fused is fused by the feature fusion enhancement module dual decoder combined with the feature fusion enhancement method to obtain the first target fused feature map;

[0099] Step S302: The second feature map to be fused is fused by the feature fusion enhancement module dual decoder combined with the feature fusion enhancement method to obtain the second target fused feature map;

[0100] Step S303: Perform feature fusion on the first target fusion feature map and the second target fusion feature map based on a convolution of a preset size to obtain the final fusion feature map.

[0101] Specifically, the first feature map to be fused is fused using a dual-decoder feature fusion enhancement module combined with a feature fusion enhancement method to obtain a first target fused feature map. Simultaneously, the second feature map to be fused is fused using the same dual-decoder feature fusion enhancement module combined with a feature fusion enhancement method to obtain a second target fused feature map. Further, the first target fused feature map and the second target fused feature map are fused using a convolution of a preset size to obtain a final fused feature map, as described in steps S3031 to S3035.

[0102] This application embodiment improves the accuracy of image segmentation by better fusing feature information of different sizes and structures through the strong feature fusion enhancement module dual decoder's strong feature fusion capability.

[0103] Further, steps S3031 to S3035 are described as follows:

[0104] Step S3031: The current layer features of the first feature map to be fused and the upsampled features are spliced ​​and fused along the channel direction to obtain a fused feature map with twice the number of channels of the second original feature map to be fused.

[0105] Step S3032: Channel compression is performed on the fused feature map through convolutional compression to obtain a compressed feature map with the same number of channels as the second original channel.

[0106] Step S3033: The compressed feature map is subjected to average pooling and max pooling in the spatial dimension, and the second weight coefficients of each channel are obtained through linear mapping, weighting and activation functions.

[0107] Step S3034: Perform element-wise point multiplication of each of the second weight coefficients with the first feature map to be fused to obtain the channel-enhanced feature map;

[0108] Step S3035: The enhanced feature map is linearly superimposed with the first feature map to be fused using residual edges, and then channel compression is performed again by convolution to obtain the first target fused feature map with half the number of channels of the second original channel.

[0109] Reference Figure 4 , Figure 4 This is a schematic diagram of the dual decoder structure of the feature fusion enhancement module provided in this application. The dual decoder of the feature fusion enhancement module can realize the feature fusion enhancement method. The dual decoder of the feature fusion enhancement module can selectively extract information of channels with higher importance according to the importance of the channels, and perform channel compression twice, as detailed below:

[0110] Specifically, the features of the current layer of the first feature map to be fused are concatenated and fused with the features upsampled by bilinear interpolation along the channel direction to obtain a fused feature map with twice the number of channels as the second original feature map. Further, the fused feature map is compressed by convolution (Conv) to obtain a compressed feature map with the same number of channels as the second original feature map. It should be noted that after channel compression by convolution (Conv), the number of channels in the compressed feature map is reduced by half compared to the number of channels in the fused feature map; however, the number of channels in the compressed feature map is the same as the number of channels in the first feature map to be fused.

[0111] Furthermore, the compressed feature map is subjected to average pooling (avg_pool) and max pooling (max_pool) in the spatial dimension, and second weight coefficients for each channel are obtained through linear mapping (MLP), weighted summation, and sigmoid activation function. Further, each second weight coefficient is element-wise multiplied with the first feature map to be fused to obtain the channel-enhanced feature map. Further, the channel-enhanced feature map and the first feature map to be fused are linearly superimposed with residual edges and then compressed again through convolution (Conv) to obtain a first target fused feature map with half the number of channels of the second original feature map.

[0112] The processing procedure for the second feature map to be fused is the same as that for the first feature map to be fused, and will not be repeated here.

[0113] In one embodiment, the first feature map to be fused is a 32*32*512 feature map, and the second feature map to be fused is a 32*32*256 feature map. The output results of each layer of the dual decoder of the feature fusion enhancement module are shown in Table 3. The 32*32*512 feature map, after four bilinear interpolations and four FFEM modules, outputs a 512*512*32 feature map. The 32*32*256 feature map, after four bilinear interpolations and four FFEM modules, outputs a 512*512*16 feature map. After feature fusion of the 512*512*32 and 512*512*16 feature maps, the final output is a 512*512*2 feature map.

[0114] Table 3 Output results of each layer of the dual decoder in the feature fusion enhancement module

[0115] Layers Output size of decoder1 Output size of decoder2 convolutional 32*32*512 32*32*256 Bilinear interpolate 64*64*512 64*64*256 FFEM 64*64*256 64*64*128 Bilinear interpolate 128*128*256 128*128*128 FFEM 128*128*128 128*128*64 Bilinear interpolate 256*256*128 256*256*64 FFEM 256*256*64 256*256*32 Bilinear interpolate 512*512*64 512*512*32 FFEM 512*512*32 512*512*16 convolutional - 512*512*2

[0116] This application embodiment improves the accuracy of image segmentation by combining the strong feature fusion capability of the dual decoder of the feature fusion enhancement module with the feature fusion enhancement method to better fuse feature information of different sizes and structures.

[0117] Furthermore, the embodiments of this application employ a joint loss function, namely, the focal loss function and the die loss function are used to jointly calculate the loss.

[0118] For the focal loss function: The focal loss function is a modification of the standard cross-entropy loss. The focal loss function reduces the weights of easily classified samples, allowing the model to focus more on difficult-to-classify samples during training, thus achieving hard sample discovery. The formula can be expressed as: FL(p t )=-0.25×(1-p t ) 2 log(p t ), where p is the probability of classifying a sample point. When p is larger, meaning the model has a higher probability of classifying a sample pixel, the sample point is easily distinguished and is considered a simple sample, with a relatively small focal loss. Conversely, when p is smaller, meaning the model has a lower probability of classifying a sample pixel, the sample point is difficult to distinguish and is considered a difficult sample, with a larger focal loss. This causes the model to focus more on difficult-to-classify samples during training.

[0119] The Dice loss function is a loss function that measures set similarity. It is typically used to calculate the similarity between two samples. The smaller the Dice loss, the greater the similarity between the two samples. Its formula can be expressed as: Where X is the actual segmentation map and Y is the predicted segmentation map, the closer Y is to X, the smaller the dice loss.

[0120] Therefore, the overall joint loss function expression for calculating the loss by combining the focal loss function and the dice loss function is Loss = 0.5 × Focal loss + 0.5 × Dice loss.

[0121] Furthermore, referring to Figure 5 , Figure 5This is a flowchart illustrating the global feature enhancement network of this application in an application scenario. The original feature map is a 512*512*3 feature map. After passing through a 64-channel residual structured block SE-Block, the original 512*512*3 feature map is output as a 512*512*64 feature map. After 2*2 max pooling, the 512*512*64 feature map is output as a 256*256*64 feature map. After passing through a 128-channel residual structured block SE-Block, the 256*256*128 feature map is output as a 256*256*128 feature map. After 2*2 max pooling, the 256*256*128 feature map is output as a 128... A 128*128 feature map is passed through a 256-channel residual structured block SE-Block, resulting in a 128*128*256 feature map. This 128*128*256 feature map is then max-pooled (2*2) to produce a 64*64*256 feature map. This 64*64*256 feature map is then passed through a 521-channel residual structured block SE-Block, resulting in a 64*64*512 feature map. Finally, this 64*64*512 feature map is max-pooled (2*2) to produce a 32*32*512 feature map. In other words, the original 512*512*3 feature map, after four SE-Block operations and four max-pooling operations, yields a final output feature map of 32*32*512.

[0122] Furthermore, the encoder and decoder are connected via skip connections, and the decoders are also connected via skip connections. Therefore, the feature map fused by the encoder and the first-layer decoder using FFEM is then fused again with the second-layer decoder using FFEM, specifically as follows:

[0123] The final output feature map of 32*32*512 is combined with features through a 3*3 convolution, and then successively passed through two 1*1 convolutions to obtain a feature map of 32*32*512 and a feature map of 32*32*256. The 32*32*512 feature map is then subjected to four bilinear interpolations and four FFEMs to output a 512*512*32 feature map. The 32*32*256 feature map is then subjected to four bilinear interpolations and four FFEMs to output a 512*512*16 feature map. The features of the 512*512*32 and 512*512*16 feature maps are then fused to output a 512*512*2 feature map.

[0124] The image segmentation method provided in this application improves the accuracy of image segmentation by leveraging the strong feature extraction capability of the dual residual structure module encoder, which in turn enables the extracted feature maps to have better semantic representation capabilities. Combined with the strong feature fusion capability of the dual decoder of the feature fusion enhancement module, the accuracy of image segmentation is improved.

[0125] Furthermore, the image segmentation apparatus provided in this application will be described below, and the image segmentation apparatus and the image segmentation method can be referred to each other accordingly.

[0126] like Figure 6 As shown, Figure 6 This is a schematic diagram of the image segmentation apparatus provided in this application. The image segmentation apparatus includes:

[0127] The feature extraction and compression module 601 is used to extract and compress features from the input original feature map through a dual residual structure module encoder to obtain the final output feature map of the original feature map.

[0128] The feature integration and mapping module 602 is used to integrate and map the final output feature map through the dual residual structure module encoder to obtain a first feature map to be fused and a second feature map to be fused.

[0129] The feature fusion module 603 is used to perform feature fusion on the first feature map to be fused and the second feature map to be fused through the dual decoder of the feature fusion enhancement module to obtain the final fused feature map.

[0130] Furthermore, the feature extraction and compression module 601 is also used for:

[0131] Based on the convolution, batch normalization, and activation functions in the dual residual structure module encoder, feature extraction is performed on the original feature map to obtain the original features;

[0132] The original features are linearly added to the original feature map using residual edges to perform feature weighting, resulting in a weighted feature map.

[0133] The final output feature map is obtained by combining the weighted feature map with the attention mechanism in the dual residual structure module encoder.

[0134] Furthermore, the feature extraction and compression module 601 is also used for:

[0135] The weighted feature map is passed through the activation function, and the average value of the feature map in the width and height dimensions is taken to obtain the weight vector of each channel.

[0136] The first weight coefficient of each channel is obtained by applying a linear mapping and the activation function to each of the weight vectors.

[0137] The final output feature map is obtained based on each of the first weight coefficients and the original feature map.

[0138] Furthermore, the feature extraction and compression module 601 is also used for:

[0139] Each of the first weight coefficients is element-wise multiplied with the original feature map to obtain the enhanced feature map.

[0140] The enhanced feature map is linearly superimposed with the original feature map using residual edges to obtain the final output feature map.

[0141] Furthermore, the feature integration mapping module 602 is also used for:

[0142] The final output feature map is integrated by performing a convolution of a first preset size in the dual residual structure module encoder to obtain an integrated feature map.

[0143] The integrated feature map is feature-mapped by a preset number of convolutions of a second preset size in the dual residual structure module encoder, resulting in a first feature map to be fused with the same number of first original channels as the final output feature map, and a second feature map to be fused with twice the number of channels as the first original channels.

[0144] Furthermore, the feature fusion module 603 is also used for:

[0145] The first feature map to be fused is obtained by fusing the feature fusion enhancement module with dual decoders and feature fusion enhancement method to obtain the first target fused feature map;

[0146] The second feature map to be fused is fused by the dual decoder of the feature fusion enhancement module combined with the feature fusion enhancement method to obtain the second target fused feature map;

[0147] The first target fusion feature map and the second target fusion feature map are fused based on a convolution of a preset size to obtain the final fusion feature map.

[0148] Furthermore, the feature fusion module 603 is also used for:

[0149] The current layer features of the first feature map to be fused are combined with the upsampled features along the channel direction to obtain a fused feature map with twice the number of channels of the second original feature map to be fused.

[0150] The fused feature map is compressed by convolutional compression to obtain a compressed feature map with the same number of channels as the second original channel;

[0151] The compressed feature map is subjected to average pooling and max pooling in the spatial dimension, and the second weight coefficients of each channel are obtained through linear mapping, weighting and activation functions.

[0152] Each of the second weight coefficients is element-wise multiplied with the first feature map to be fused to obtain the channel-enhanced feature map.

[0153] The enhanced feature map is linearly superimposed with the first feature map to be fused using residual edges, and then channel compression is performed again through convolution to obtain the first target fused feature map with half the number of channels of the second original channel.

[0154] The specific embodiments of the image segmentation apparatus provided in this application are basically the same as the embodiments of the image segmentation method described above, and will not be repeated here.

[0155] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an image segmentation method, which includes:

[0156] The input original feature map is processed by a dual residual structure module encoder to extract and compress features, resulting in the final output feature map.

[0157] The final output feature map is integrated and mapped by the dual residual structure module encoder to obtain the first feature map to be fused and the second feature map to be fused.

[0158] The first and second feature maps to be fused are fused using a dual decoder through a feature fusion enhancement module to obtain the final fused feature map.

[0159] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to perform the image segmentation methods provided by the above methods, the method including:

[0161] The input original feature map is processed by a dual residual structure module encoder to extract and compress features, resulting in the final output feature map.

[0162] The final output feature map is integrated and mapped by the dual residual structure module encoder to obtain the first feature map to be fused and the second feature map to be fused.

[0163] The first and second feature maps to be fused are fused using a dual decoder through a feature fusion enhancement module to obtain the final fused feature map.

[0164] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image segmentation methods provided above, the method comprising:

[0165] The input original feature map is processed by a dual residual structure module encoder to extract and compress features, resulting in the final output feature map.

[0166] The final output feature map is integrated and mapped by the dual residual structure module encoder to obtain the first feature map to be fused and the second feature map to be fused.

[0167] The first and second feature maps to be fused are fused using a dual decoder through a feature fusion enhancement module to obtain the final fused feature map.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image segmentation method, characterized in that, include: The input original feature map is processed by a dual residual structure module encoder to extract and compress features, resulting in the final output feature map. The final output feature map is integrated and mapped by the dual residual structure module encoder to obtain the first feature map to be fused and the second feature map to be fused. The first and second feature maps to be fused are fused using a dual decoder through a feature fusion enhancement module to obtain the final fused feature map. The feature fusion enhancement module performs feature fusion on the first and second feature maps to be fused using a dual decoder to obtain the final fused feature map, including: The current layer features of the first feature map to be fused are combined with the upsampled features along the channel direction to obtain a fused feature map with twice the number of channels of the second original feature map to be fused. The fused feature map is compressed by convolutional compression to obtain a compressed feature map with the same number of channels as the second original channel; The compressed feature map is subjected to average pooling and max pooling in the spatial dimension, and the second weight coefficients of each channel are obtained through linear mapping, weighting and activation functions. Each of the second weight coefficients is element-wise multiplied with the first feature map to be fused to obtain the channel-enhanced feature map. The enhanced feature map is linearly superimposed with the first feature map to be fused using residual edges, and then channel compression is performed again by convolution to obtain the first target fused feature map with half the number of channels of the second original channel. The second feature map to be fused is fused by the dual decoder of the feature fusion enhancement module combined with the feature fusion enhancement method to obtain the second target fused feature map; The first target fusion feature map and the second target fusion feature map are fused based on a convolution of a preset size to obtain the final fusion feature map.

2. The image segmentation method according to claim 1, characterized in that, The step of extracting and compressing features from the input original feature map using a dual residual structure module encoder to obtain the final output feature map includes: Based on the convolution, batch normalization, and activation functions in the dual residual structure module encoder, feature extraction is performed on the original feature map to obtain the original features; The original features are linearly added to the original feature map using residual edges to perform feature weighting, resulting in a weighted feature map. The final output feature map is obtained by combining the weighted feature map with the attention mechanism in the dual residual structure module encoder.

3. The image segmentation method according to claim 2, characterized in that, The final output feature map is obtained by combining the weighted feature map with the attention mechanism in the dual residual structure module encoder, including: The weighted feature map is passed through the activation function, and the average value of the feature map in the width and height dimensions is taken to obtain the weight vector of each channel. The first weight coefficient of each channel is obtained by applying a linear mapping and the activation function to each of the weight vectors. The final output feature map is obtained based on each of the first weight coefficients and the original feature map.

4. The image segmentation method according to claim 3, characterized in that, The process of obtaining the final output feature map based on each of the first weight coefficients and the original feature map includes: Each of the first weight coefficients is element-wise multiplied with the original feature map to obtain the enhanced feature map. The enhanced feature map is linearly superimposed with the original feature map using residual edges to obtain the final output feature map.

5. The image segmentation method according to claim 1, characterized in that, The process of integrating and mapping the final output feature map through the dual residual structure module encoder to obtain a first feature map to be fused and a second feature map to be fused includes: The final output feature map is integrated by performing a convolution of a first preset size in the dual residual structure module encoder to obtain an integrated feature map. The integrated feature map is feature-mapped by a preset number of convolutions of a second preset size in the dual residual structure module encoder, resulting in a first feature map to be fused with the same number of first original channels as the final output feature map, and a second feature map to be fused with half the number of channels as the first original channels.

6. An image segmentation apparatus, characterized in that, include: The feature extraction and compression module is used to extract and compress features from the input original feature map through a dual residual structure module encoder to obtain the final output feature map of the original feature map. The feature integration and mapping module is used to integrate and map the final output feature map through the dual residual structure module encoder to obtain a first feature map to be fused and a second feature map to be fused. The feature fusion module is used to perform feature fusion on the first feature map to be fused and the second feature map to be fused through the dual decoder of the feature fusion enhancement module to obtain the final fused feature map. The feature fusion module is specifically used for: The current layer features of the first feature map to be fused are combined with the upsampled features along the channel direction to obtain a fused feature map with twice the number of channels of the second original feature map to be fused. The fused feature map is compressed by convolutional compression to obtain a compressed feature map with the same number of channels as the second original channel; The compressed feature map is subjected to average pooling and max pooling in the spatial dimension, and the second weight coefficients of each channel are obtained through linear mapping, weighting and activation functions. Each of the second weight coefficients is element-wise multiplied with the first feature map to be fused to obtain the channel-enhanced feature map. The enhanced feature map is linearly superimposed with the first feature map to be fused using residual edges, and then channel compression is performed again by convolution to obtain the first target fused feature map with half the number of channels of the second original channel. The second feature map to be fused is fused by the dual decoder of the feature fusion enhancement module combined with the feature fusion enhancement method to obtain the second target fused feature map; The first target fusion feature map and the second target fusion feature map are fused based on a convolution of a preset size to obtain the final fusion feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image segmentation method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image segmentation method according to any one of claims 1 to 5.

Images