Image segmentation method and device combining feature difference recognition and detail enhancement

Abstract

The invention provides an image segmentation method and device combining feature difference recognition and detail enhancement, and the method comprises the following steps: obtaining a to-be-segmented image to construct a data set, and carrying out the marking and preprocessing of the data set; constructing an image segmentation model, wherein the image segmentation model comprises an encoder network, a jump connection network, a decoder network and an output network; the encoder network uses a pre-trained CNN (Convolutional Neural Network) and Transform hybrid network to carry out feature extraction; the jump connection network comprises a channel space interaction module for fusing encoder features and decoder features; the decoder network comprises a convolution-channel attention module for extracting key features and a convolution-up-sampling module for up-sampling; the output network comprises a multi-level feature fusion module for fusing the multi-scale features of the decoder; training the image segmentation model to obtain a trained image segmentation model; and segmenting the image data by adopting the trained image segmentation model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image segmentation method and device combining feature difference recognition and detail enhancement. Background Art

[0002] Image segmentation is a crucial basic task in image processing. Its goal is to extract several regions with semantic meaning from the image in order to conduct in-depth analysis and processing of the image content. Through accurate image segmentation, a detailed understanding of the image content can be achieved, thereby revealing the characteristics and attributes of each region in the image. This technology is not only crucial for object recognition, target tracking and scene understanding, but also has a wide range of applications in many fields such as medical image analysis, autonomous driving, and video surveillance. Such problems have the characteristics of complex backgrounds, numerous details, and similarities between the target and the background. Accurately segmenting the target area in a complex environment is of great significance for the practical application of image segmentation.

[0003] UNet is one of the most popular image segmentation architectures. This architecture achieves accurate image segmentation through a symmetrical encoder-decoder structure and skip connections. UNet has the characteristics of simple and effective architecture design, flexible encoder-decoder configuration and efficient performance, and can adapt to a variety of application scenarios. However, the original UNet network faces many challenges in image segmentation tasks. First, its simple encoder network has certain limitations in extracting image features with complex backgrounds. Secondly, the skip connection achieved by addition or splicing operations fails to effectively fuse the high-level detail information of the encoder's shallow layer with the semantic information of the decoder's deep layer, thus affecting the model's accurate recognition of details such as boundaries and textures. Finally, the decoder in the original UNet has insufficient recognition ability for important features and cannot effectively distinguish the feature differences between the target and the background. In addition, the upsampling operation in the decoding process is prone to loss of details, and relying solely on the output of the top layer of the decoder as the final segmentation result will seriously affect the segmentation quality. Therefore, improving UNet's ability to model important features and feature differences and enhancing its ability to recognize and retain details have become important development directions in image segmentation research, which will help improve the effectiveness and reliability of image segmentation technology in practical applications. Summary of the invention

[0004] In view of the above problems, the present invention provides an image segmentation method and device combining feature difference recognition and detail enhancement. This method improves the UNet modeling ability of feature differences and details, and achieves high-precision segmentation of target areas under complex image backgrounds, which is of great significance for the practical application of image segmentation technology.

[0005] The present invention adopts the following technical solutions:

[0006] An image segmentation method combining feature difference recognition and detail enhancement comprises the following steps:

[0007] S1: Obtain the image to be segmented and annotate it; use random rotation and random flipping operations to preprocess the image to be segmented, and divide it into training set, validation set and test set;

[0008] S2: Build an image segmentation model, including an encoder network, a skip connection network, a decoder network, and an output network;

[0009] The encoder network uses a pre-trained CNN and Transformer hybrid model; the jump connection network includes a channel-space interaction module, which includes a cascaded channel interaction unit and a spatial interaction unit; the decoder network includes a convolution-channel attention module and a convolution-upsampling module; the output network includes a multi-level feature fusion module;

[0010] S3: Design a multi-stage feature loss aggregation method to calculate multi-stage losses based on the joint loss function to enhance the convergence of the model;

[0011] S4: Specify the training rounds, perform model training, and save the weight of the optimal model;

[0012] S5: Test model performance.

[0013] Preferably, in S2, the CNN and Transformer hybrid network is specifically:

[0014] A hybrid model based on convolutional neural network (CNN) and Transformer modules is selected to build a multi-stage UNet encoder. The ImageNet dataset images are preprocessed to a uniform size and input into the CNN-Transformer hybrid model for pre-training. The ImageNet dataset contains more than 14 million annotated images covering more than 20,000 categories. The trained model is used as an encoder to extract features from the input image. Specifically, the input image is fed into the pre-trained encoder to obtain a feature map containing semantic information at different levels, which provides input for the subsequent skip connection network and decoder network.

[0015] Preferably, in S2, the channel interaction unit is specifically:

[0016] The channel-space interaction module is composed of cascaded channel interaction units and spatial interaction units. It is mainly used for skip connections in the UNet architecture, aiming to effectively fuse the feature maps of the encoder stage i (i=2,3,…,n-1) and the decoder stage i+1. This module uses a refined channel and spatial interaction mechanism to explore the relationship between semantics at different levels, achieve semantic alignment and complementarity, and thus improve the ability to capture details. Its output is then fed into the convolution-channel attention module to further assist the decoding process, ultimately supporting more accurate image reconstruction and semantic segmentation tasks.

[0017] The core of the channel interaction unit is to adjust the importance of the two input feature maps based on their mutual relationship to achieve semantic alignment in the channel dimension. Specifically, for the feature map X of the encoder stage i (i = 2, 3, ..., n-1) and the feature map Y of the decoder stage i+1, firstly, a one-dimensional channel vector is obtained by convolution 1×1, batch normalization and adaptive average pooling operations, and then the channel importance weight W of the two is obtained by using the Sigmoid activation function. x and W y , as shown in equations (1) and (2).

[0018] W x =σ(AAP(BN(Conv 1×1 (X)))) (1)

[0019] W y =σ(AAP(BN(Conv 1×1 (Y)))) (2)

[0020] Among them, Conv 1×1 represents a convolution operation with a kernel size of 1; BN represents a batch normalization operation; AAP represents an adaptive average pooling operation; σ represents a sigmoid activation function.

[0021] Next, feature map X and feature map Y are subjected to convolution 3×3, batch normalization, and ReLU activation function operations to further optimize the features, and then the optimized features are multiplied by the channel weights of another feature map to adjust the channel importance. Finally, the result is added to the original feature map to obtain the final channel interaction result, as shown in equations (3) and (4).

[0022] X : =ReLU(BN(Conv 3×3 (X))×W y +X (3)

[0023] Y : =ReLU(BN(Conv 3×3 (Y))×Wx +Y (4)

[0024] Among them, Conv 3×3 represents a convolution operation with a kernel size of 3; ReLU represents the rectified linear unit activation function; X : and Y : They represent the results of the interaction between the feature map X and feature map Y channels respectively.

[0025] Preferably, in S2, the spatial interaction unit is specifically:

[0026] The core of the spatial interaction unit is to comprehensively consider the relationship between the encoder and decoder feature map pixels after channel interaction and redistribute the importance weights of all pixels to achieve semantic complementarity. Specifically, the feature map X : and Y : First, the number of channels is mapped to 1 through convolution 1×1 and batch normalization operations to integrate global context information. Then, the two feature maps with the number of channels of 1 are concatenated together to obtain the initial weight matrix, and then convolution 1×1 is performed to obtain a weight matrix to fuse the input feature map information. This is shown in equations (5) and (6).

[0027] Q = Cat(BN(Conv 1×1 (X ～ )),BN(Conv 1×1 (Y ～ ))) (5)

[0028] P=Conv 1×1 (Q) (6)

[0029] Among them, Cat represents the splicing operation along the channel; Q represents the initial weight matrix; P represents the final correlation weight matrix.

[0030] Next, the Sigmoid activation function is used to map the weight values ​​to between 0 and 1. Finally, the feature maps X: and Y : Multiply them by the weights respectively, and add the two results to obtain the feature map after the final fusion of the spatial interaction module, as shown in equation (7).

[0031] Z=X ～ ×σ(P)+Y ～ ×σ(P) (7)

[0032] Where Z represents the feature map after the final fusion of the spatial interaction module.

[0033] Preferably, in S2, the convolution-channel attention module is specifically:

[0034] The convolution-channel attention module is mainly composed of convolution blocks, squeeze-and-excitation networks (SENet) and residual mechanisms. The convolution block contains two groups of 3×3 convolutions, batch normalization (BatchNormalization) and ReLU activation functions. This structure is designed to extract rich feature information while effectively maintaining the spatial structure. The SENet module guides the network to focus on features with greater discriminative power by adaptively adjusting the weights of feature channels, thereby effectively distinguishing the feature differences between the target and the background. This mechanism aims to highlight key information areas and suppress unimportant background information. In order to improve the transmission of information, the convolution-channel attention module introduces residual connections. Through the addition operation, the learned features are directly transferred to deeper network layers. This structure can effectively alleviate the problems of gradient vanishing and gradient exploding. In addition, residual connections can accelerate the convergence process and promote the flow of information, thereby further improving the performance of the network.

[0035] Specifically, the convolution-channel attention module takes the output of encoder stage n or the output of channel spatial interaction module or the output of decoder stage 2 as input, first extracts and optimizes features through convolution blocks, and applies residual connections to retain the original information, as shown in equations (8) and (9).

[0036] Z1=ReLU(BN(Conv 3×3 (Z))) (8)

[0037] Z2=ReLU(BN(Conv 3×3 (Z1)))+Z (9)

[0038] Among them, Z1 represents the intermediate output of the convolution block, and Z2 represents the output of the convolution block.

[0039] The obtained features are then fed into SENet, where the importance of each feature channel is adaptively adjusted through learnable attention weights. Finally, a residual connection is applied to add the input and output of SENet to obtain the output of the convolution-channel attention module, as shown in Equation (10).

[0040] S=SENet(Z2)+Z2 (10)

[0041] Among them, S represents the output of the convolution-channel attention module; SENet represents the compressed activation network.

[0042] Preferably, in S2, the convolution-upsampling module is specifically:

[0043] The convolution-upsampling module consists of an upsampling operation, a 3×3 convolution, a batch normalization, and a ReLU activation function. This module is designed to increase the resolution of the feature map.

[0044] In the (i=2,3,…,n-1) stage, the upsampling operation expands the resolution of the feature map output by the convolution-channel attention module to twice the original size, thereby effectively matching the feature map of the encoder stage i and providing input for the channel space interaction module. For the decoder stage 2, the upsampling operation expands the resolution of the feature map output by the convolution-channel attention module to twice the original size and then feeds it into the decoder stage 1. Subsequently, the 3×3 convolution operation is used to further adjust and optimize the upsampled feature map to improve its representation ability. By combining the batch normalization technique, this module can reduce internal covariate shift, thereby accelerating model training. Finally, the ReLU activation function is introduced to support nonlinear feature learning. The above process is shown in Equation (11).

[0045] U = ReLU(BN(Conv 3×3 (Up 2× (S)))) (11)

[0046] Among them, U represents the output of the convolution-upsampling module; Up 2× Indicates upsampling the feature map by two times.

[0047] Preferably, in S2, the multi-level feature fusion module is specifically:

[0048] In order to improve the accuracy of segmentation results, this patent proposes a multi-level feature fusion module, which aims to comprehensively consider the different scale features learned by the decoder at each stage. By interacting and fusing these features, the relationship between features at different levels is fully explored to make up for the details lost during the upsampling process, thereby promoting the model's prediction ability in complex scenarios.

[0049] First, the output feature maps of the last four layers of the decoder are upsampled by 2, 4, 8, and 16 times, respectively, to restore the same resolution as the original input image. Then, the number of channels of the upsampled feature map is mapped to the number of task-related categories through point-by-point convolution, where each channel represents a category feature. The above process can be expressed as Equation (12).

[0050] Y i =Conv 1×1 (Up j× (X i )),i=1,2,3,4.j=2,4,8,16. (12)

[0051] Among them, Xi represents the output of the decoder at stage i; Y i Indicates that X i The corresponding feature map after doubling the resolution and adjusting the number of channels; Up j× Indicates upsampling the feature map by j times.

[0052] Next, the corresponding channels of the feature maps are concatenated, that is, the features of each category are concatenated together. This process can be expressed as equation (13).

[0053] Y c =Cat(Cat(Y 11 ,Y 21 ,L,Y i1 ),L,Cat(Y 1m ,Y 2m ,L,Y im )),i=1,2,3,4 (13)

[0054] Among them, Y im Represents the feature map Y i The mth channel of Y; m represents the number of categories; c Represents the feature map after the corresponding channels are concatenated.

[0055] Finally, the intra-class and inter-class features are integrated through two-step convolution operations to generate the segmentation result. First, the feature channels of each class are convolved separately using group convolution with the number of groups equal to the number of categories, and a feature map with a channel number of 1 is generated for each class. Then, the learned features are fused using standard convolution to integrate the inter-class features, and finally the prediction output is obtained. This process can be expressed as Equation (14) and Equation (15).

[0056] G = ReLU(BN(GConv 3×3 (Y c ))) (14)

[0057] output = Conv 3×3 (G) (15)

[0058] Among them, G represents the feature map after the integration of intra-class features; GConv 3×3 Represents a group convolution operation with a kernel size of 3; output represents the final predicted output.

[0059] Preferably, in S3, the joint loss function is specifically:

[0060] The joint loss function is equal to the weighted sum of the classification loss and the segmentation loss. The segmentation loss is calculated using Dice Loss to evaluate the overlap between the prediction result and the true value label. The classification loss is calculated using the cross entropy loss to evaluate the category prediction of each pixel. The joint loss function can be expressed as formula (16).

[0061] L=λ1L DICE +λ2L CE (16)

[0062] Among them, L represents the joint loss function; λ1 and λ2 represent Dice Loss (L DICE ) and cross entropy loss (L CE ) is the weight in the joint loss function.

[0063] Preferably, in S3, the multi-stage feature loss aggregation method is specifically:

[0064] In addition to calculating the loss of the final segmentation result, the multi-stage feature loss aggregation method also performs deep supervision on the output of each stage of the decoder. Specifically, first take non-empty subsets of the feature maps Y1, Y2, Y3, and Y4 to obtain 15 sets. Then add the elements in each subset to obtain 15 prediction results. Finally, the total loss is the sum of the losses of these 15 prediction results and the final segmentation result (that is, the output of the multi-level feature fusion module) and the true value label. The above process can be expressed as the following formula.

[0065] Sets = nonsubset ([Y1,Y2,Y3,Y4]) (17)

[0066]

[0067] Among them, Sets represents a set of 15 non-empty subsets; the nonsubset function represents taking a non-empty subset of a list; n represents the number of elements in each set; R represents a set of 15 prediction results; loss represents the total loss; label represents the true value label.

[0068] Preferably, an image segmentation device combining feature difference recognition and detail enhancement comprises at least one processor and a memory; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an image segmentation method combining feature difference recognition and detail enhancement according to any one of claims 1 to 8. The beneficial effects of the present invention are:

[0069] 1. The present invention proposes an image segmentation method that combines feature difference recognition and detail enhancement, which can accurately segment images with complex backgrounds and images with similar foreground and background.

[0070] 2. The present invention proposes a channel space interaction module, which retains more features and improves the accurate recognition of details such as boundaries and textures by semantically aligning and complementing the features learned in different encoder and decoder stages.

[0071] 3. The present invention proposes a convolution-channel attention module and introduces a channel attention mechanism to effectively distinguish the feature differences between the target and the background, thereby highlighting the target area and suppressing irrelevant background areas.

[0072] 4. The present invention proposes a multi-level feature fusion module, which alleviates the problem of detail loss by fusing semantic information of different scales learned in the decoder stage to obtain more accurate and robust segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings of the embodiments are briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present invention, but are not intended to limit the present invention.

[0074] Figure 1 A flowchart of the method proposed by the present invention;

[0075] Figure 2 A network structure diagram of the method proposed in the present invention;

[0076] Figure 3 This is a network structure diagram of the channel space interaction module proposed in the present invention;

[0077] Figure 4 This is a network structure diagram of the convolution-channel attention module proposed in the present invention;

[0078] Figure 5 This is a network structure diagram of the multi-level feature fusion module proposed in the present invention; DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0081] An image segmentation method combining feature difference recognition and detail enhancement comprises the following steps:

[0082] S1: Get the image to be segmented;

[0083] A total of 3779 abdominal CT images of 30 patients were collected, and the resolution of all images was converted to 256×256. Eight organs in the images were labeled, namely aorta, gallbladder, left kidney, right kidney, pancreas, spleen, stomach and liver. Random rotation and random flipping operations were used to preprocess the images to be segmented and divided into training set, validation set and test set.

[0084] S2: Build an image segmentation model, including an encoder network, a skip connection network, a decoder network, and an output network;

[0085] The encoder network uses a pre-trained CNN and Transformer hybrid network; the jump connection network includes a channel-space interaction module, which includes a cascaded channel interaction unit and a space interaction unit; the decoder network includes a convolution-channel attention module and a convolution-upsampling module; the output network includes a multi-level feature fusion module;

[0086] The images of the ImageNet dataset are preprocessed into a uniform size of 256×256 and input into the CNN-Transformer hybrid network MaxViT for pre-training. MaxViT includes 5 stages. The resolution of each stage is halved layer by layer. The first stage maps the original input channel number to 96. The number of input and output channels in the second stage remains unchanged, and the number of channels in other stages is doubled. The ImageNet dataset contains more than 14 million annotated images, covering more than 20,000 categories. The trained MaxViT network is used as an encoder to extract features from the input image. Specifically, the input image with a dimension of 256×256×3 is fed into the pre-trained encoder to obtain feature maps containing semantic information at different levels in stages 2 to 5, namely the first feature map, the second feature map, the third feature map, and the fourth feature map, with dimensions of 64×64×96, 32×32×192, 16×16×384, and 8×8×768, respectively. The first feature map, the second feature map, and the third feature map are respectively fed into the channel space interaction module, and the fourth feature map is fed into the fifth stage of the decoder.

[0087] The channel-space interaction module is composed of cascaded channel interaction units and spatial interaction units. It is mainly used for skip connections in the UNet architecture, aiming to effectively fuse the feature maps of the encoder stage i (i=2,3,4) and the decoder stage i+1. This module uses a refined channel and spatial interaction mechanism to explore the relationship between semantics at different levels, achieve semantic alignment and complementarity, and thus improve the ability to capture details. Its output is then fed into the convolution-channel attention module to further assist the decoding process, ultimately supporting more accurate image reconstruction and semantic segmentation tasks.

[0088] The core of the channel interaction unit is to adjust the importance of the two input feature maps based on their mutual relationship to achieve semantic alignment in the channel dimension. Specifically, for the feature map X of the encoder stage i (i = 2, 3, 4) and the feature map Y of the decoder stage i+1, firstly, a one-dimensional channel vector is obtained by convolution 1×1, batch normalization and adaptive average pooling operations, and then the channel importance weight W of the two is obtained by using the Sigmoid activation function. x and W y , as shown in equations (1) and (2).

[0089] W x =σ(AAP(BN(Conv 1×1 (X)))) (1)

[0090] W y =σ(AAP(BN(Conv 1×1 (Y)))) (2)

[0091] Among them, Conv 1×1 represents a convolution operation with a kernel size of 1; BN represents a batch normalization operation; AAP represents an adaptive average pooling operation; σ represents a sigmoid activation function.

[0092] Next, feature map X and feature map Y are subjected to convolution 3×3, batch normalization, and ReLU activation function operations to further optimize the features, and then the optimized features are multiplied by the channel weights of another feature map to adjust the channel importance. Finally, the result is added to the original feature map to obtain the final channel interaction result, as shown in equations (3) and (4).

[0093] X : =ReLU(BN(Conv 3×3 (X))×W y +X (3)

[0094] Y : =ReLU(BN(Conv 3×3 (Y))×W x+Y (4)

[0095] Among them, Conv 3×3 Represents a convolution operation with a kernel size of 3; ReLU represents the rectified linear unit activation function; X: and Y: represent the results of the interaction of the feature map X and feature map Y channels, respectively.

[0096] The core of the spatial interaction unit is to comprehensively consider the relationship between the encoder and decoder feature map pixels after channel interaction, and redistribute the importance weights of all pixels to achieve semantic complementarity. Specifically, the feature maps X: and Y: are first mapped to 1 by convolution 1×1 and batch normalization operations to integrate global context information. Then the two feature maps with a channel number of 1 are concatenated together to obtain the initial weight matrix, and then convolution 1×1 is performed to obtain a weight matrix to fuse the input feature map information. This is shown in equations (5) and (6).

[0097] Q = Cat(BN(Conv 1×1 (X ～ )),BN(Conv 1×1 (Y ～ ))) (5)

[0098] P=Conv 1×1 (Q) (6)

[0099] Among them, Cat represents the splicing operation along the channel; Q represents the initial weight matrix; P represents the final correlation weight matrix.

[0100] Next, the Sigmoid activation function is used to map the weight values ​​to between 0 and 1. Finally, the feature map X : and Y : Multiply them by the weights respectively, and add the two results to obtain the feature map after the final fusion of the spatial interaction module, as shown in equation (7).

[0101] Z=X ～ ×σ(P)+Y ～ ×σ(P) (7)

[0102] Where Z represents the feature map after the final fusion of the spatial interaction module.

[0103] The dimensions of the feature maps after fusion by the channel-space interaction module are 64×64×96, 32×32×192 and 16×16×384, respectively, and the fused feature maps are sent to the convolution-channel attention module of the decoder stage i (i=2,3,4) for key feature extraction.

[0104] The convolution-channel attention module consists of a convolution block, a squeeze-and-excitation network (SENet—Squeeze-and-Excitation Network), and a residual mechanism. The convolution block is composed of two sets of convolutions 3×3, batch normalization, and ReLU activation functions connected in series. This structure is designed to extract rich feature information while effectively maintaining the spatial structure. The SENet module guides the network to focus on features with greater discriminative power by adaptively adjusting the weights of feature channels, thereby effectively distinguishing the feature differences between the target and the background. This mechanism aims to highlight key information areas and suppress unimportant background information. In order to improve the transmission of information, the convolution-channel attention module introduces residual connections. Through the addition operation, the learned features are directly transferred to deeper network layers. This structure can effectively alleviate the problems of gradient vanishing and gradient exploding. In addition, residual connections can accelerate the convergence process and promote the flow of information, thereby further improving the performance of the network.

[0105] Specifically, the convolution-channel attention module takes the fourth feature map, the output of the channel space interaction module, and the output of the decoder stage 2 as input, first extracts and optimizes the features through the convolution block, and applies the residual connection to retain the original information, as shown in equations (8) and (9).

[0106] Z1=ReLU(BN(Conv 3×3 (Z))) (8)

[0107] Z2=ReLU(BN(Conv 3×3 (Z1)))+Z (9)

[0108] Among them, Z1 represents the intermediate output of the convolution block, and Z2 represents the output of the convolution block.

[0109] The obtained features are then fed into SENet, where the importance of each feature channel is adaptively adjusted through learnable attention weights. Finally, a residual connection is applied to add the input and output of SENet to obtain the output of the convolution-channel attention module, as shown in Equation (10).

[0110] S=SENet(Z2)+Z2 (10)

[0111] Among them, S represents the output of the convolution-channel attention module; SENet represents the compressed activation network.

[0112] The output of the convolution-channel attention module is the output of each stage of the decoder, corresponding to decoder stages 1 to 5, which are the fifth feature map, the sixth feature map, the seventh feature map, the eighth feature map, and the ninth feature map, respectively, with dimensions of 128×128×96, 64×64×96, 32×32×192, 16×16×384, and 8×8×768. The fifth, sixth, seventh, and eighth feature maps are used as inputs to the multi-level fusion module, and the seventh, eighth, and ninth feature maps are simultaneously sent to the convolution-upsampling module to expand the resolution and improve the input for the channel space interaction module. The sixth feature map is sent to the decoder stage 1 through the convolution-upsampling module.

[0113] The convolution-upsampling module consists of an upsampling operation, a 3×3 convolution, a batch normalization, and a ReLU activation function. This module is designed to increase the resolution of the feature map.

[0114] In the (i=2,3,…,n-1) stage, the upsampling operation expands the resolution of the feature map output by the convolution-channel attention module to twice the original size, thereby effectively matching the feature map of the encoder stage i and providing input for the channel space interaction module. For the decoder stage 2, the upsampling operation expands the resolution of the feature map output by the convolution-channel attention module to twice the original size and then feeds it into the decoder stage 1. Subsequently, the 3×3 convolution operation is used to further adjust and optimize the upsampled feature map to improve its representation ability. By combining the batch normalization technique, this module can reduce internal covariate shift, thereby accelerating model training. Finally, the ReLU activation function is introduced to support nonlinear feature learning. The above process is shown in Equation (11).

[0115] U = ReLU(BN(Conv 3×3 (Up 2× (S)))) (11)

[0116] Among them, U represents the output of the convolution-upsampling module; Up 2× Indicates upsampling the feature map by two times.

[0117] After the convolution-upsampling module, the dimensions of the seventh, eighth, and ninth feature maps are 64×64×96, 32×32×192, and 16×16×384, respectively.

[0118] In order to improve the accuracy of segmentation results, this patent proposes a multi-level feature fusion module, which aims to comprehensively consider the different scale features learned by the decoder at each stage. By interacting and fusing these features, the relationship between features at different levels is fully explored to make up for the details lost during the upsampling process, thereby promoting the model's prediction ability in complex scenarios.

[0119] First, the fifth feature map, the sixth feature map, the seventh feature map, and the eighth feature map are upsampled and expanded by 2 times, 4 times, 8 times, and 16 times, respectively, to restore the same resolution as the original input image. Then, the number of channels of the upsampled feature map is mapped to the number of categories related to the task by point-by-point convolution, which is equal to 9 (8 target categories and 1 background category) in this embodiment. Among them, each channel represents a category feature. The above process can be expressed as equation (12).

[0120] Y i =Conv 1×1 (Up j× (X i )),i=1,2,3,4.j=2,4,8,16. (12)

[0121] Among them, X i represents the output of the decoder at stage i; Y i Indicates that X i The corresponding feature map after doubling the resolution and adjusting the number of channels; Up j× Indicates upsampling the feature map by j times.

[0122] Next, the corresponding channels of the feature maps are concatenated, that is, the features of each category are concatenated together. This process can be expressed as equation (13).

[0123] Y c =Cat(Cat(Y 11 ,Y 21 ,L,Y i1 ),L,Cat(Y 1m ,Y 2m ,L,Y im )),i=1,2,3,4 (13)

[0124] Among them, Y im Represents the feature map Y i The mth channel of Y; m represents the number of categories; c Represents the feature map after the corresponding channels are concatenated.

[0125] Finally, the intra-class and inter-class features are integrated through two-step convolution operations to generate the segmentation result. First, the feature channels of each class are convolved separately using group convolution with the number of groups equal to the number of categories, and a feature map with a channel number of 1 is generated for each class. Then, the learned features are fused using standard convolution to integrate the inter-class features, and finally the prediction output is obtained. This process can be expressed as Equation (14) and Equation (15).

[0126] G = ReLU(BN(GConv 3×3 (Yc ))) (14)

[0127] output = Conv 3×3 (G) (15)

[0128] Among them, G represents the feature map after the integration of intra-class features; GConv 3×3 Represents a group convolution operation with a kernel size of 3; output represents the final predicted output, whose dimension is 256×256×9.

[0129] S3: Design a multi-stage feature loss aggregation method to calculate multi-stage losses based on the joint loss function to enhance the convergence of the model;

[0130] The joint loss function is equal to the weighted sum of the classification loss and the segmentation loss. The segmentation loss is calculated using Dice Loss to evaluate the overlap between the prediction result and the true value label. The classification loss is calculated using the cross entropy loss to evaluate the category prediction of each pixel. The joint loss function can be expressed as formula (16).

[0131] L=λ1L DICE +λ2L CE (16)

[0132] Among them, L represents the joint loss function; λ1 and λ2 represent Dice Loss (L DICE ) and cross entropy loss (L CE ) is the weight in the joint loss function.

[0133] In addition to calculating the loss of the final segmentation result, the multi-stage feature loss aggregation method also performs deep supervision on the output of each stage of the decoder. Specifically, first take non-empty subsets of the feature maps Y1, Y2, Y3, and Y4 to obtain 15 sets. Then add the elements in each subset to obtain 15 prediction results. Finally, the total loss is the sum of the losses of these 15 prediction results and the final segmentation result (that is, the output of the multi-level feature fusion module) and the true value label. The above process can be expressed as the following formula.

[0134] Sets = nonsubset ([Y1,Y2,Y3,Y4]) (17)

[0135]

[0136] Among them, Sets represents a set of 15 non-empty subsets; the nonsubset function represents taking a non-empty subset of a list; n represents the number of elements in each set; R represents a set of 15 prediction results; L represents the joint loss function; loss represents the total loss; label represents the true value label.

[0137] S4: Train the model for 300 rounds and save the weight of the optimal model;

[0138] S5: Test model performance.

[0139] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. An image segmentation method combining feature difference recognition and detail enhancement, characterized in that: The following steps are involved: S1: Get the image to be segmented; S2: Build an image segmentation model, including an encoder network, a skip connection network, a decoder network, and an output network; The encoder network uses a pre-trained convolutional neural network and a Transformer hybrid network; the jump connection network includes a channel-space interaction module, which includes a cascaded channel interaction unit and a space interaction unit; the decoder network includes a convolution-channel attention module and a convolution-upsampling module; the output network includes a multi-level feature fusion module; S3: Design a multi-stage feature loss aggregation method to calculate multi-stage losses based on the joint loss function to enhance the convergence of the model; S4: Specify the training rounds, perform model training, and save the weight of the optimal model; S5: Test model performance.

2. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S2, the convolutional neural network and Transformer hybrid network are specifically: The images in the ImageNet dataset are preprocessed into a uniform size and input into a multi-stage hybrid network based on a convolutional neural network and a Transformer module for pre-training. The trained hybrid network is used as an encoder to extract features from the input image. The input image is sent to the pre-trained encoder to obtain a feature map containing semantic information at different levels. This provides input for the subsequent skip connection network and decoder network.

3. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S2, the channel interaction unit is specifically: The channel-space interaction module consists of cascaded channel interaction units and spatial interaction units, which are mainly used for skip connections in the UNet architecture to effectively fuse the feature maps of encoder stage i and decoder stage i+1; i = 2, 3, ..., n-1; The core of the channel interaction unit is that the two input feature maps adjust the importance of their own features based on their mutual relationship to achieve semantic alignment in the channel dimension; for the feature map X of the encoder stage i and the feature map Y of the decoder stage i+1, they are firstly subjected to convolution 1×1, batch normalization and adaptive average pooling operations to obtain a one-dimensional channel vector respectively, and then the sigmoid activation function is used to obtain the channel importance weight W of the two. x and W y , W x =σ(AAP(BN(Conv 1×1 (X)))); W y =σ(AAP(BN(Conv 1×1 (Y)))); Among them, Conv 1×1 represents a convolution operation with a kernel size of 1; BN represents a batch normalization operation; AAP represents an adaptive average pooling operation; σ represents a sigmoid activation function; Next, feature map X and feature map Y are subjected to convolution 3×3, batch normalization, and ReLU activation function operations to further optimize the features, and then the optimized features are multiplied by the channel weights of another feature map to adjust the channel importance; finally, the result is added to the original feature map to obtain the final result after channel interaction; X:=ReLU(BN(Conv 3×3 (X))×W y +X; And:=ReLU(BN(Conv 3×3 (Y))×W x +Y; Among them, Conv 3×3 represents a convolution operation with a kernel size of 3; ReLU represents the rectified linear unit activation function; X: and Y : They represent the results of the interaction between the feature map X and feature map Y channels respectively.

4. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S2, the spatial interaction unit is specifically: feature map X ～ and Y ～ First, the number of channels is mapped to 1 through convolution 1×1 and batch normalization operations to integrate global context information; Then the two feature maps with a channel number of 1 are concatenated together to obtain the initial weight matrix, and then a weight matrix is ​​obtained by convolution 1×1 to fuse the input feature map information; Q=Cat(BN(Conv 1×1 (X~)),BN(Conv 1×1 (Y~))); P=Conv 1×1 (Q); Among them, Cat represents the splicing operation along the channel; Q represents the initial weight matrix; P represents the final correlation weight matrix; Next, the Sigmoid activation function is used to map the weight value to between 0 and 1; finally, the feature map X ～ and Y ～ Multiply them by the weight respectively, and add the two results to get the feature map after the final fusion of the spatial interaction module; Z=X ～ ×σ(P)+Y ～ ×σ(P); Where Z represents the feature map after the final fusion of the spatial interaction module.

5. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S2, the convolution-channel attention module is specifically: The convolution-channel attention module consists of a convolution block, a compressed activation network, and a residual mechanism. The convolution block consists of two sets of 3×3 convolutions, batch normalization, and ReLU activation functions connected in series. SENet models the feature differences between the target and the background by adaptively adjusting the weights of the feature channels, thereby highlighting important areas and suppressing background areas. The residual connection applies an addition operation to pass the learned features to deeper layers, enhancing information propagation through additional paths, and effectively solving the gradient vanishing and exploding problems. Specifically, the convolution-channel attention module takes the output of encoder stage n or the output of channel space interaction module or the output of decoder stage 2 as input, first extracts and optimizes features through convolution blocks, and applies residual connections to retain the original information; Z1=ReLU(BN(Conv 3×3 (Z))); Z2=ReLU(BN(Conv 3×3 (Z1)))+Z; Among them, Z1 represents the intermediate output of the convolution block, and Z2 represents the output of the convolution block; The obtained features are then fed into SENet, and the importance of each feature channel is adaptively adjusted through learnable attention weights; finally, the input and output of SENet are added using residual connections to obtain the output of the convolution-channel attention module; S = SENet(Z2)+Z2; Among them, S represents the output of the convolution-channel attention module; SENet represents the compressed activation network.

6. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S2, the convolution-upsampling module is specifically: The convolution-upsampling module is composed of upsampling operation, convolution 3×3, batch normalization and ReLU activation function in series; for the decoder stage i+1, i=2,3,…,n-1, the upsampling operation expands the resolution of the feature map output by the convolution-channel attention module to twice the original size, thereby effectively matching the feature map of the encoder stage i and providing input for the channel space interaction module; for the decoder stage 2, the upsampling operation expands the resolution of the feature map output by the convolution-channel attention module to twice the original size, and then feeds it into the decoder stage 1; the convolution operation is intended to adjust the upsampled features; U =ReLU(BN(Conv 3×3 (Up 2× (S)))); Among them, U represents the output of the convolution-upsampling module; Up 2× Indicates upsampling the feature map by two times.

7. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S2, the multi-level feature fusion module is specifically: First, the output feature maps of the last four layers of the decoder are upsampled by 2, 4, 8, and 16 times, respectively, to restore the same resolution as the original input image; then the number of channels of the upsampled feature map is mapped to the number of categories related to the task through point-by-point convolution, where each channel represents a category feature; the above process is expressed as the equation: Y i =Conv 1×1 (Up j× (X i )),i=1,2,3,4.j=2,4,8,16.; Among them, X i represents the output of the decoder at stage i; Y i Indicates that X i The corresponding feature map after doubling the resolution and adjusting the number of channels; Up j× Indicates that the feature map is upsampled j times; Next, the feature maps are concatenated corresponding to the channels, that is, the features of each category are concatenated together. This process is expressed as an equation: Y c =Cat(Cat(Y 11 ,Y 21 ,L,Y i1 ),L,Cat(Y 1m ,Y 2m ,L,Y im )),i=1,2,3,4; Among them, Y im Represents the feature map Y i The mth channel of Y; m represents the number of categories; c Represents the feature map after the corresponding channels are concatenated; Finally, the intra-class and inter-class features are integrated through two-step convolution operations to generate the segmentation result; first, the feature channels of each class are convolved separately using group convolution with the number of groups equal to the number of categories, and a feature map with a channel number of 1 is generated for each class; then the learned features are fused using standard convolution to integrate inter-class features, and finally the prediction output is obtained; this process is expressed as the equation: G=ReLU(BN(GConv 3×3 (Y c ))); output=Conv 3×3 (G); Among them, G represents the feature map after the integration of intra-class features; GConv 3×3 Represents a group convolution operation with a kernel size of 3; output represents the final predicted output.

8. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S3, the joint loss function is specifically: The joint loss function is equal to the weighted sum of the classification loss and the segmentation loss. The segmentation loss is calculated using Dice Loss to evaluate the overlap between the prediction result and the true value label. The classification loss is calculated using cross entropy loss to evaluate the category prediction of each pixel. The joint loss function is expressed as the formula: L=λ1L DICE +λ2L CE ; Among them, L represents the joint loss function; λ1 and λ2 represent Dice Loss (L DICE ) and cross entropy loss L CE Weight in the joint loss.

9. The image segmentation method combining feature difference recognition and detail enhancement according to claim 1, characterized in that: In S3, the multi-stage feature loss aggregation method is specifically: In addition to calculating the loss of the final segmentation result, the multi-stage feature loss aggregation method also performs deep supervision on the output of each stage of the decoder; first, non-empty subsets of feature maps Y1, Y2, Y3 and Y4 are taken to obtain 15 sets; then the elements in each subset are added to obtain 15 prediction results; finally, the total loss is the sum of the losses of these 15 prediction results and the final segmentation result and the true value label; the above process is expressed as the following formula; Sets=nonsubset([Y1,Y2,Y3,Y4]); Among them, Sets represents a set of 15 non-empty subsets; the nonsubset function represents taking a non-empty subset of a list; n represents the number of elements in each set; R represents a set of 15 prediction results; L represents the joint loss function; loss represents the total loss; label represents the true value label.

10. An image segmentation device combining feature difference recognition and detail enhancement, characterized in that: It comprises at least one processor and a memory; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an image segmentation method combining feature difference recognition and detail enhancement as described in any one of claims 1 to 8.

Images