Image semantic segmentation method based on mixed context attention

Abstract

The invention discloses an image semantic segmentation method based on mixed context attention. Collected image data is preprocessed; dividing the image data into a training set and a test set without crossing; designing a loss function for segmentation, wherein the loss function selects mixed loss of DiceLoss and BCELoss; and designing a segmentation model based on mixed context attention, training the segmentation model based on the loss function, and executing an image semantic segmentation task by using the trained segmentation model. According to the invention, reasonable design and expansion are carried out by using a U-Net architecture, so that the model realizes adaptive learning for a segmentation target; in combination with an attention mechanism and a multi-task learning mode, the feature extraction capability of the model for difficult segmentation examples such as small target features and fuzzy boundary features is enhanced; and meanwhile, the parameter quantity of the model is remarkably reduced, the over-fitting possibility is reduced, and the lightweight requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image semantic segmentation method based on hybrid context attention. Background Art

[0002] Image segmentation is a key technology in image processing and an important task in the field of computer vision. The goal of image segmentation is to divide each pixel in the image into parts belonging to different objects, objects or regions based on the visual features and semantic information between different pixels in the image. Currently, image segmentation is mainly divided into semantic segmentation, instance segmentation and interactive segmentation. These segmentation technologies can be distinguished by different categories, different entities, and whether there are interactive operations.

[0003] Semantic segmentation is the most basic and important content in the field of image segmentation. It classifies pixels of the same type into one category and accurately classifies each pixel of the image. It is widely used in autonomous driving, automatic monitoring, medical image analysis, satellite remote sensing image processing and other digital image processing fields. Early semantic segmentation methods were mainly based on traditional methods, mainly including methods based on image processing, such as edge detection, region segmentation, color clustering, etc., followed by methods based on texture features, such as gray-level co-occurrence matrix or wavelet transform. And morphological segmentation, hybrid genetic algorithm, etc. There are also some methods combined with machine learning, such as: FCM clustering, regional level set, etc. Traditional semantic segmentation methods can achieve a certain segmentation accuracy, but they still rely on prior knowledge, cannot capture high-level semantic information, and have poor robustness for complex target segmentation, and cannot be well applied to real-life scenarios.

[0004] In recent years, deep learning has developed rapidly and has shown strong performance in the field of image segmentation. With the ability of automatic feature learning, convenient operation and efficient performance of deep learning models, high-precision segmentation results can be obtained while taking into account both time and space efficiency. The classic Fully Convolutional Networks (FCN) proposed in 2015 has the main feature of being able to accept input images of any size and generate output images of the same size, where each pixel is assigned a semantic label, thus achieving end-to-end pixel-level semantic segmentation. FCN loses its focus on image details due to multiple pooling stacking; then some improved networks based on FCN were proposed one after another, such as: SegNet based on encoder-decoder structure, U-Net, Deeplab series network based on dilated convolution, SwinUNet based on the combination of Transformer and U-Net, HRNet adapted to high-resolution images, etc., but these models have some defects: U-Net series network completes cross-layer information retention through residual skip connection, integrates high- and low-level feature information, breaks the situation of information loss and zero interaction of different layers of information, but brings more information redundancy and noise, these useless information greatly reduces the segmentation ability of the model; DeeplabV3 series network expands the model receptive field by adding dilated convolution, which often causes the model to extract discontinuous features and thus results in poor segmentation effect. Transformer series segmentation networks are mostly combined with U-Net to achieve segmentation tasks, and have achieved good results, but Transformer training takes a long time, has high computing cost, and requires a large amount of data, which is not suitable for the field of biomedical image segmentation where data sets are scarce. HRNet network structure is relatively complex and not suitable for general real-life scenarios. In summary, in the field of applied technology, segmentation tasks still have some limitations. For example, most segmentation algorithms are limited to a single field. The same segmentation model does not have good universality in remote sensing images, automatic cutouts, unmanned driving, and biomedical image segmentation. It cannot simultaneously present good segmentation performance for different modal 2D, 3D and other image information. Secondly, a common problem with current segmentation models is that the segmentation effects of segmentation targets with large morphological distribution and size differences are significantly different. Furthermore, for certain specific scenarios such as unmanned driving and satellite remote sensing observations, it is necessary to segment and evaluate targets in real time, which places high demands on the time, space cost and reasoning speed of the model. Most of the existing segmentation models cannot achieve a good balance between time efficiency and segmentation accuracy. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide an image semantic segmentation method with a wide range of applications, precise segmentation boundaries and high execution efficiency.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for image semantic segmentation based on hybrid contextual attention: comprising the following steps:

[0008] (1) Preprocessing the collected image data;

[0009] (2) Divide the image data into two parts: training set and test set without overlap;

[0010] (3) Design a loss function for segmentation, which is a hybrid loss of DiceLoss and BCELoss.

[0011] (4) Design a segmentation model based on hybrid contextual attention. The segmentation model includes a split attention encoder, a cascaded attention decoder, and a multi-stage auxiliary loss calculation module. The split attention encoder is used to extract semantic features of different layers and levels in the image. The cascaded attention decoder is used to fuse semantic features of different layers and enhance the expression of feature information. The multi-stage auxiliary loss calculation module can use the output of the cascaded attention decoder to enhance the classification ability of the model hidden layer by gradient backpropagation.

[0012] (5) Based on the loss function, the segmentation model is trained, and the trained segmentation model is used to perform the image semantic segmentation task.

[0013] As an optimization, in step (1), the pictures in the image data are converted into PNG format, and then the PNG format pictures are processed using normalization and standardization methods.

[0014] As an optimization, the loss function is calculated as follows:

[0015]

[0016]

[0017] L seg =α·L dice +β·L bce (3) Where α and β are the weight hyperparameters of the two losses, and their sum is 1. N represents the total number of pixels, and C represents the total number of categories. Indicates the category value corresponding to the c-th class and the i-th pixel in the binary GroundTruth, s i cIt represents the predicted probability value of the corresponding category, and ε=1e-5 represents the smoothing exponent.

[0018] As an optimization, the split attention encoder consists of a dual-branch convolutional stem block module and split attention encoding modules from stage 1 to stage 4;

[0019] The left branch of the dual-branch convolution stem block module stacks two 3*3 convolution blocks, and the right branch uses 1*1 convolution. The split attention encoding module from stage 1 to stage 4 consists of 3*3 convolution, split attention unit, and maximum pooling. The number of repetitions of the split attention unit in stages 1 to 4 are 1, 2, 3, and 3 respectively; the split attention unit consists of layer normalization, split attention, and a multi-layer perceptron.

[0020] As an optimization, the dual-branch convolutional stem block module takes the original image as input. The dual-branch convolutional stem block module adopts a dual-branch structure to extract important edge details of the image. The outputs of the two branches of the dual-branch convolutional stem block module are added and activated by the Relu function. The specific calculation of the dual-branch convolutional stem block module is shown below:

[0021] F output =Relu(Conv 3×3 (Conv 3×3 (F input ))+Conv 1×1 (F input )) (4)

[0022] Among them, F input Represents the original feature map of the input, F output Represents the output feature map.

[0023] As an optimization, the split attention encoding module includes a 3*3 convolution and a split attention unit and a maximum pooling module. The input feature map is first doubled in number of channels through a 3×3 convolution to improve the spatial information expression ability of the segmentation model, and then the result of the convolution is used as the input of the split attention unit;

[0024] The split attention unit is divided into two parts. The first part is to use the layer normalization module to normalize the input feature map to make the training more stable, and then use the split attention module to extract image features with receptive fields, and finally make residual connections so that the gradient propagation of the segmentation model is not affected by the intermediate neurons; the second part is to use layer normalization to extract global information through the multi-layer perceptron for the results obtained in the first part, and finally make residual connections. The specific process is calculated by the following formula:

[0025] F input′ =Conv 3×3 (Finput )

[0026]

[0027]

[0028]

[0029] F input″ =F input′ +Split Attention(LayerNorm(F input′ )) (8)

[0030] F input″′ =F input″ +MLP(LayerNorm(F input″ )) (9)

[0031] F output =MaxPool(F input″′ ) (10)

[0032] Split represents the splitting of channels, DepthConv 7×7 It represents the use of deep convolution with a convolution kernel size of 7*7 to extract image features, and MaxPool represents the use of 2*2 maximum pooling to reduce the size of the feature map and thus reduce the amount of computation of the segmentation model.

[0033] As an optimization, the cascaded attention decoder consists of four non-isomorphic feature fusion modules, four collaborative attention modules, and four 1*1 convolutions. The non-isomorphic feature fusion module integrates the outputs of different stages of the split attention encoder to achieve a reasonable fusion of high-level semantic information and low-level detail information, so as to enhance the segmentation model's ability to extract multi-scale features, thereby improving segmentation performance. The segmentation model uses the collaborative attention module to enhance the segmentation model's regional perception of the segmentation target, and uses 1*1 convolution to adjust the number of feature maps output by the dual-branch convolution stem block and the four stages of the cascaded attention decoder and upsample them to the original resolution in parallel.

[0034] As an optimization, the non-isomorphic feature fusion module is composed of linear layers, and the input end is composed of low-level features and high-level semantic features. It adopts information interaction to fuse low-level detail features and high-level semantic features.

[0035] The high-level semantic features are expanded into the original three parts through the linear layer. The first part is used for interaction of low-level detail features and suppresses some non-critical neurons through the activation function Sigmoid. The other two high-level semantic features are activated by the activation function Sigmoid and Tanh to suppress neurons and then multiply the feature maps to achieve the effect of attention.

[0036] The calculation process of the non-isomorphic feature fusion module is as follows:

[0037]

[0038]

[0039]

[0040]

[0041] Represents high-level semantic features, Represents low-level detail features, σ represents the activation function Sigmoid, T represents the activation function Tanh, and Concat represents the channel concatenation operation.

[0042] As an optimization, the collaborative attention module is a simple and effective attention module for feedforward convolutional neural networks. The collaborative attention module consists of three parts. The first part is the channel attention module, which is used to highlight the criticality of the channel to the segmentation model; the second part is the spatial attention module, which allows the segmentation model to improve its sensitivity to the specific spatial position of each channel, thereby focusing on important target positioning; the third part is a 3*3 convolution. The purpose of using convolution here is to perform enhanced feature extraction after obtaining the channel attention module and the spatial attention module, and adjust the number of channels to reduce the amount of segmentation model calculation. The calculation process of the collaborative attention module is as follows:

[0043] F ECA =Unsqueeze(σ(Squeeze(GAP(F input ))))*F input (15)

[0044] F PSA =σ(Conv 3×3 (Concat(GAP(F ECA ), GMP(F ECA ))))*F ECA (16)F output =Conv 3×3 (F PSA (F ECA (F input ))) (17)

[0045] Squeeze represents channel compression and dimensionality reduction, and UnSqueeze represents channel expansion and dimensionality increase. ECA represents the output of the channel attention module, F PSArepresents the output of the spatial attention module, GAP represents the global average pooling operation, and GMP represents the global maximum pooling operation.

[0046] As an optimization, the multi-stage auxiliary loss calculation module is used to perform auxiliary loss calculation on the probability maps generated at different stages in the cascaded attention decoder. The probability maps are then activated by applying Sigmoid activation or Softmax activation to obtain the final prediction output.

[0047] Dual-branch convolutional stem block module: Compared with the single-branch convolution path, the advantage of this module is that it uses a dual-branch structure to obtain low-level features with rich details. The left branch obtains the local features of the target by stacking two convolutions, and the right branch scans the single-point information through point-by-point convolution to ensure lightweight design.

[0048] Split attention encoding module: Compared with the common encoding module that stacks several convolutions to improve the model's receptive field, the SAEM module first uses convolution to double the number of feature maps, which helps improve the model's ability to express spatial information.

[0049] Secondly, based on the Transformer architecture, the self-attention mechanism is replaced with the invented split attention module to ensure that global features are modeled at each position, so as to better capture long-distance semantic relationships.

[0050] Non-isomorphic feature fusion module: Compared with commonly used feature fusion operations such as channel stacking and feature map addition, the innovation of the non-isomorphic feature fusion module lies in completing the feature fusion process by interactively integrating two image features with information asymmetry and information redundancy.

[0051] Collaborative attention module: The collaborative attention module is essentially a feature enhancement module. Its innovation lies in the serialization of channel information and spatial information, which helps the model to be more sensitive to key channels and further enhance the spatial features in key channels.

[0052] Multi-stage auxiliary loss calculation module: The currently popular semantic segmentation model only adjusts the model parameters through the segmentation head of the last layer of the model by taking the value of the loss function. The innovation of this module lies in adding a segmentation head in the middle layer of the model to adjust the neurons deep in the model through the value of the loss function, so that it has classification capabilities.

[0053] Compared with the prior art, the present invention has the following advantages: Compared with the traditional segmentation algorithm, the present invention does not require artificial prior knowledge, and relies on the powerful learning ability of the neural network to enable the model to have strong performance and more accurate segmentation results in different segmentation tasks. Compared with other deep learning segmentation models, the present invention is reasonably designed and expanded with the U-Net architecture, allowing the model to achieve adaptive learning for segmentation targets of different sizes and shapes, breaking the limitation of only extracting single-scale features in the inherent segmentation network; combining the attention mechanism and multi-task learning method to enhance the model's feature extraction ability for segmentation difficult examples such as small target features and fuzzy boundary features, and has better segmentation performance than other excellent segmentation models; it also has good performance for different tasks; at the same time, it significantly reduces the number of model parameters to reduce the possibility of overfitting and meet lightweight requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of image semantic segmentation in the present invention;

[0055] Figure 2 is a schematic diagram of the segmentation model in the present invention;

[0056] Figure 3 Schematic diagram of a double-branched convolutional stem block in the present invention;

[0057] Figure 4 Schematic diagram of splitting the attention encoding module in the present invention;

[0058] Figure 5 Schematic diagram of splitting the attention module in the present invention;

[0059] Figure 6 Schematic diagram of the non-isomorphic feature fusion module in the present invention;

[0060] Figure 7 is a schematic diagram of a collaborative attention module in the present invention;

[0061] Figure 8 This is a comparison chart of model parameters of different models on data set A in this embodiment;

[0062] Fig. 9 This is a visualization comparison diagram of segmentation masks of different models on data set B in this embodiment;

[0063] Fig.10 This is a visualization comparison diagram of segmentation masks of different models on data set C in this embodiment. DETAILED DESCRIPTION

[0064] like Figures 1 to 7 As shown, the image semantic segmentation method based on hybrid context attention in this specific implementation manner comprises the following steps:

[0065] (1) Preprocessing the collected image data;

[0066] (2) Divide the image data into two parts: training set and test set without overlap;

[0067] (3) Design a loss function for segmentation, which is a hybrid loss of DiceLoss and BCELoss.

[0068] (4) Design a segmentation model based on hybrid contextual attention. The segmentation model includes a split attention encoder, a cascaded attention decoder, and a multi-stage auxiliary loss calculation module. The split attention encoder is used to extract semantic features of different layers and levels in the image. The cascaded attention decoder is used to fuse semantic features of different layers and enhance the expression of feature information. The multi-stage auxiliary loss calculation module can use the output of the cascaded attention decoder to enhance the classification ability of the model hidden layer by gradient backpropagation.

[0069] (5) Based on the loss function, the segmentation model is trained, and the trained segmentation model is used to perform the image semantic segmentation task.

[0070] In this specific embodiment, in step (1), the picture in the image data is converted into a png format, and then the png format picture is processed using a normalization and standardization method.

[0071] In this specific embodiment, in order to cope with the characteristics of different tasks with large scale changes in segmentation targets (such as houses and pedestrians in remote sensing images), and some tasks with small targets (such as biomedical lesions), the target area changes are large and complex. Therefore, a mixed loss (Formula 3) of DiceLoss (Formula 1) and BCELoss (Formula 2) is selected, where α and β are the weight hyperparameters of the two losses, and the sum is 1. In this embodiment, it is set to [0.5, 0.5] to make the model training more balanced and stable. The calculation formula of the loss function is as follows:

[0072]

[0073]

[0074] L seg =α·L dice +β·L bce (3)

[0075] N represents the total number of pixels, C represents the total number of categories, Represents the category value corresponding to the c-th class and the i-th pixel in the binary GroundTruth, Represents the predicted probability value of the corresponding category, ε = 1e-5, represents the smoothing exponent. It is used to prevent the denominator from being predicted to be 0, which may lead to some extreme situations. In addition, it can also play a role in smoothing loss and gradient.

[0076] In this specific embodiment, the split attention encoder is composed of a dual-branch convolutional stem block module (Stem) and a split attention encoding module (SAEM) of stage 1 to stage 4;

[0077] The left branch of the dual-branch convolutional stem block module stacks two 3*3 convolution blocks, and the right branch uses 1*1 convolution. The split attention encoding module from stage 1 to stage 4 consists of 3*3 convolution, split attention unit (SAU), and maximum pooling. The number of repetitions of the split attention unit in stages 1 to 4 is 1, 2, 3, and 3 respectively; the split attention unit consists of layer normalization (LayerNorm), split attention (SplitAttention), and multi-layer perceptron (MLP). The design idea of ​​the split attention encoder is to use Stem to obtain fine edge details and use SAEM to extract semantic information at different levels of the image to highlight key areas.

[0078] In this specific embodiment, the dual-branch convolution stem block module takes the original image as input, and the Stem module composed of ordinary convolution is more advantageous in providing the model with low-level features with rich details. The dual-branch convolution stem block module uses a dual-branch structure to extract important edge details of the image, and the outputs of the two branches of the dual-branch convolution stem block module are added and activated by the Relu function. The specific calculation of the dual-branch convolution stem block module is shown below:

[0079] F output =Relu(Conv 3×3 (Conv 3×3 (F input ))+Conv 1×1 (F input )) (4)

[0080] Among them, F input Represents the original feature map of the input, F output Represents the output feature map.

[0081] In this specific embodiment, the split attention encoding module includes a 3*3 convolution (3x3 Conv) and a split attention unit and a maximum pooling (MaxPool) module. The input feature map is first doubled in number of channels through a 3×3 convolution to improve the spatial information expression ability of the segmentation model, and then the result of the convolution is used as the input of the split attention unit;

[0082] The split attention unit is divided into two parts. The first part is to use the layer normalization module to normalize the input feature map to make the training more stable, and then use the split attention module to extract image features with receptive fields, and finally make residual connections so that the gradient propagation of the segmentation model is not affected by the intermediate neurons; the second part is to use layer normalization to extract global information through the multi-layer perceptron for the results obtained in the first part, and finally make residual connections. The specific process is calculated by the following formula:

[0083] F input′ =Conv 3×3 (F input )

[0084]

[0085]

[0086]

[0087] F input″ =F input′ +Split Attention(LayerNorm(F input′ )) (8)

[0088] F input″′ =F input″ +MLP(LayerNorm(F input″ )) (9)

[0089] F output =MaxPool(F input″′ ) (10)

[0090] Split represents the splitting of channels, DepthConv 7×7 It represents the use of deep convolution with a convolution kernel size of 7*7 to extract image features, and MaxPool represents the use of 2*2 maximum pooling to reduce the size of the feature map and thus reduce the amount of computation of the segmentation model.

[0091] In this specific embodiment, the cascaded attention decoder is composed of four non-isomorphic feature fusion modules (NFFM), four collaborative attention modules (CAM), and four 1*1 convolutions. The non-isomorphic feature fusion module integrates the outputs of different stages of the split attention encoder to achieve a reasonable fusion of high-level semantic information and low-level detail information, so as to enhance the segmentation model's ability to extract multi-scale features, thereby improving the segmentation performance. In addition, considering that some segmentation targets cause feature information loss in the process of continuous pooling, the segmentation model uses the collaborative attention module to enhance the segmentation model's regional perception of the segmentation target, making the model more sensitive to the segmentation target and thus enhancing the model's discrimination ability. The output of the dual-branch convolution stem block and the feature map output of the four stages of the cascaded attention decoder are adjusted and upsampled to the original resolution in parallel using 1*1 convolution to prepare for the multi-stage auxiliary loss calculation module.

[0092] In this specific embodiment, the non-isomorphic feature fusion module is composed of a linear layer, and the input end is composed of low-level features and high-level semantic features. Considering the differences between low-level features and high-level features, an information interaction method is adopted to fuse low-level detail features and high-level semantic features;

[0093] The high-level semantic feature matrix has a large number of channels and a small resolution. It represents many key features that have a great impact on the model. The high-level semantic features are expanded into three original parts through a linear layer. The first part is used for interaction with low-level detail features and suppresses some non-critical neurons through the activation function Sigmoid. The other two high-level semantic features are activated by the activation function Sigmoid and Tanh to suppress neurons and then multiply the feature maps to achieve the effect of attention.

[0094] Low-level detail features are closely related to tiny feature points and boundary contour information. They have high resolution and small number of channels, which is suitable for the model to focus on important areas and spatial position information in the feature map of each channel: when processing, it is necessary to first add the input low-level detail features to a high-level semantic feature feature map to highlight the key areas. Residual skip connections are also used to retain important high-level semantic features. The calculation process of the non-isomorphic feature fusion module is as follows:

[0095]

[0096]

[0097]

[0098]

[0099] Represents high-level semantic features, Represents low-level detail features, σ represents the activation function Sigmoid, T represents the activation function Tanh, and Concat represents the channel concatenation operation.

[0100] In this specific embodiment, the collaborative attention module is a simple and effective attention module for a feedforward convolutional neural network. Given an intermediate feature map, the CAM module will infer the attention map in two independent dimensions (channel and space) in turn, and then multiply the attention map with the input feature map to perform adaptive feature optimization. The collaborative attention module consists of three parts. The first part is the channel attention module, which is used to highlight the criticality of the channel to the segmentation model; the second part is the spatial attention module, which allows the segmentation model to improve its sensitivity to the specific spatial position of each channel, thereby focusing on important target positioning; the third part is a 3*3 convolution. The purpose of using convolution here is to perform enhanced feature extraction after obtaining the channel attention module and the spatial attention module, and adjust the number of channels to reduce the amount of calculation of the segmentation model. The calculation process of the collaborative attention module is as follows:

[0101] F ECA =Unsqueeze(σ(Squeeze(GAP(F input ))))*F input (15)

[0102] F PSA =σ(Conv 3×3 (Concat(GAP(F ECA ), GMP(F ECA ))))*F ECA (16) F output =Conv 3×3 (F PSA (F ECA (F input ))) (17)

[0103] Squeeze represents channel compression and dimensionality reduction, and UnSqueeze represents channel expansion and dimensionality increase. ECA represents the output of the channel attention module, F PSA represents the output of the spatial attention module, GAP represents the global average pooling operation, and GMP represents the global maximum pooling operation.

[0104] In this specific embodiment, the original intention of designing the multi-stage auxiliary loss calculation module is that the hidden layer of the model may cause the hidden layer neurons to be inactivated due to the damage of gradient propagation; the purpose of the multi-stage auxiliary loss is to force the hidden layer of the model to also have a certain classification ability so as to obtain a more accurate segmentation result. This model uses Loss1~Loss5 in the training stage to improve the classification ability of the hidden layer of the model, and only uses Loss1 in the test stage; the auxiliary loss can act as a regular constraint term, forcing the neurons in the hidden layer of the model to also have classification ability, thereby avoiding the gradient vanishing problem caused by the excessive depth of the network to a certain extent; the multi-stage auxiliary loss calculation module is used to perform auxiliary loss calculation on the segmentation maps generated at different stages in the cascaded attention decoder, and the segmentation maps obtain the final prediction output by applying Sigmoid activation or Softmax activation; the multi-stage auxiliary loss calculation module helps to solve the gradient vanishing problem in the training process, while improving the accuracy and stability of the segmentation.

[0105]

[0106] The training is carried out through the loss function, and the gradient back propagation of the neural network is used to update the model weights during the training process. The segmentation effect on the validation set after the model training is used to judge whether the model is better, and the saved model weights are updated; finally, after the model training is completed, the segmentation effect is evaluated on the test set.

[0107] The calculation principle of the segmentation evaluation index is as follows: DSC (Dice similarity coefficient) and IoU (Intersection over union) reflect the overlap between the predicted image and the label. The larger the value, the better the segmentation effect. HD (Hausdorff distance) is a bidirectional HD, which means to find the smallest distance between the two maximum distances calculated from two directions between set X and set Y. The smaller the value, the smaller the boundary gap between the two. SE (Sensitivity) reflects the ability of the model to distinguish positive samples. The larger the value, the better.

[0108]

[0109]

[0110]

[0111] HD=max{d(x, y), d(y, x)} (22)

[0112]

[0113]

[0114] In the formula, sets x and y represent labels and predictions respectively; TP represents positive samples predicted by the model as positive, TN represents negative samples predicted by the model as negative; FP represents negative samples predicted by the model as positive; FN represents positive samples predicted by the model as negative. d(x, y) represents the distance from x to y.

[0115] All the test results of the models trained on the dataset are constructed with distributions of mean and standard deviation, and all indicators exceed the 95% confidence level.

[0116] Take ventricular MRI segmentation as an example. Three publicly recognized data sets were used for processing to verify the effectiveness of the model proposed in this invention. Among them, data set A obtained 1577 slices as a training set and 262 slices as a test set after data preprocessing; data set B obtained 603 slices as a training set and 202 slices as a test set after data preprocessing; data set C obtained 243 slices as a training set and 514 slices as a test set after data preprocessing. The following indicators are used to evaluate the segmentation model when verifying and testing the segmentation performance of the model: Dice coefficient (DSC), intersection over union (IoU), HD distance (HD), precision (PR), and recall rate (Rrecall).

[0117] The segmentation model of the present invention is compared with a variety of excellent segmentation algorithms. The segmentation results on data set A are shown in Table 1, where bold indicates the best and underline indicates the second best (the results of subsequent tables are consistent). The parameter comparison is as follows Figure 8 ; The segmentation results on dataset B are shown in Table 2, and the visual comparison results are shown in Fig. 9 By analyzing the overlap between the coils representing the predicted results and the coils representing the real results in the observation diagram, it can be concluded that the image segmented by the present invention has the best effect; the segmentation results on the dataset C are shown in Table 3, and the visual comparison effect is shown in Fig.10 By analyzing the overlap between the coils representing the predicted results and the coils representing the actual results in the observation graph, it can be concluded that the image segmented by the present invention has the best effect. In addition, a comparative experiment was conducted on the multi-stage auxiliary loss, and the most suitable Loss for training the segmentation model in the present invention was obtained. 1 ~Loss 5 , the comparison results are shown in Table 4. Finally, from the perspective of model complexity, the comparison is made, using the parameter amount (#.Params.) and common evaluation indicators to comprehensively evaluate the powerful segmentation ability of the segmentation model in the present invention. All experiments show that the segmentation model in the present invention has the best segmentation performance in this implementation case, and has the smallest model parameter amount, fast inference speed, and meets the lightweight requirements.

[0118]

[0119] Table 1

[0120]

[0121] Table 2

[0122]

[0123]

[0124] Table 3

[0125]

[0126] Table 4

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for image semantic segmentation based on hybrid contextual attention: characterized by: The following steps are involved: (1) Preprocessing the collected image data; (2) Divide the image data into two parts: training set and test set without overlap; (3) Design a loss function for segmentation, which is a hybrid loss of DiceLoss and BCELoss. (4) Design a segmentation model based on hybrid contextual attention. The segmentation model includes a split attention encoder, a cascaded attention decoder, and a multi-stage auxiliary loss calculation module. The split attention encoder is used to extract semantic features of different layers and levels in the image. The cascaded attention decoder is used to fuse semantic features of different layers and enhance the expression of feature information. The multi-stage auxiliary loss calculation module can use the output of the cascaded attention decoder to enhance the classification ability of the model hidden layer by gradient backpropagation. (5) Based on the loss function, the segmentation model is trained, and the trained segmentation model is used to perform the image semantic segmentation task.

2. The image semantic segmentation method based on hybrid contextual attention according to claim 1, characterized in that: In step (1), the image in the image data is converted into png format, and then the png format image is processed using normalization and standardization methods.

3. The image semantic segmentation method based on hybrid contextual attention according to claim 1, characterized in that: The loss function is calculated as follows: L seg =α·L dice +β·L bce (3) Among them, α and β are weight hyperparameters of two losses, and their sum is 1. N represents the total number of pixels, and C represents the total number of categories. Represents the category value corresponding to the c-th class and the i-th pixel in the binary GroundTruth, It represents the predicted probability value of the corresponding category, and ε=1e-5 represents the smoothing exponent.

4. The image semantic segmentation method based on hybrid contextual attention according to claim 1, characterized in that: The split attention encoder consists of a dual-branch convolutional stem block module and split attention encoding modules from stage 1 to stage 4; The left branch of the dual-branch convolution stem block module stacks two 3*3 convolution blocks, and the right branch uses 1*1 convolution. The split attention encoding module from stage 1 to stage 4 consists of 3*3 convolution, split attention unit, and maximum pooling. The number of repetitions of the split attention unit in stages 1 to 4 are 1, 2, 3, and 3 respectively; the split attention unit consists of layer normalization, split attention, and a multi-layer perceptron.

5. The image semantic segmentation method based on hybrid contextual attention according to claim 4, characterized in that: The dual-branch convolutional stem block module takes the original image as input. The dual-branch convolutional stem block module adopts a dual-branch structure to extract important edge details of the image. The outputs of the two branches of the dual-branch convolutional stem block module are added and activated by the Relu function. The specific calculation of the dual-branch convolutional stem block module is shown below: F output =Relu(Conv 3×3 (Conv 3×3 (Finput))+Conv 1×1 (F input )) (4) where F input Represents the original feature map of the input, F output Represents the output feature map.

6. The image semantic segmentation method based on hybrid contextual attention according to claim 4, characterized in that: The split attention encoding module includes a 3*3 convolution and a split attention unit and a maximum pooling module. The input feature map is first doubled in number of channels through a 3×3 convolution to improve the spatial information expression ability of the segmentation model, and then the result of the convolution is used as the input of the split attention unit; The split attention unit is divided into two parts. The first part is to use the layer normalization module to normalize the input feature map to make the training more stable, and then use the split attention module to extract image features with receptive fields, and finally make residual connections so that the gradient propagation of the segmentation model is not affected by the intermediate neurons; the second part is to use layer normalization to extract global information through the multi-layer perceptron for the results obtained in the first part, and finally make residual connections. The specific process is calculated by the following formula: F input′ =Conv 3×3 (F input ) F input″ =F input′ +Split Attention(LayerNorm(F input′ )) (8) F input″′ =F input″ +MLP(LayerNorm(F input″ )) (9) F output =MaxPool(F input″′ ) (10) Split represents the splitting of channels, DepthConv 7×7 It represents the use of deep convolution with a convolution kernel size of 7*7 to extract image features, and MaxPool represents the use of 2*2 maximum pooling to reduce the size of the feature map and thus reduce the amount of computation of the segmentation model.

7. The image semantic segmentation method based on hybrid contextual attention according to claim 4, characterized in that: The cascaded attention decoder consists of four non-isomorphic feature fusion modules, four collaborative attention modules, and four 1*1 convolutions. The non-isomorphic feature fusion module integrates the outputs of different stages of the split attention encoder to achieve a reasonable fusion of high-level semantic information and low-level detail information, so as to enhance the segmentation model's ability to extract multi-scale features, thereby improving segmentation performance. The segmentation model uses the collaborative attention module to enhance the segmentation model's regional perception of the segmentation target, and uses 1*1 convolution to adjust the number of feature maps output by the dual-branch convolution stem block and the four stages of the cascaded attention decoder and upsample them to the original resolution in parallel.

8. The image semantic segmentation method based on hybrid contextual attention according to claim 7, characterized in that: The non-isomorphic feature fusion module is composed of linear layers, and the input end is composed of low-level features and high-level semantic features. The information interaction method integrates low-level detail features and high-level semantic features; The high-level semantic features are expanded into the original three parts through the linear layer. The first part is used for interaction with low-level detail features and some non-critical neurons are suppressed through the activation function Sigmoid. The other two high-level semantic features are multiplied by the feature map after the activation function Sigmoid and Tanh are used to suppress neurons to achieve the effect of attention. The calculation process of the non-isomorphic feature fusion module is as follows: Represents high-level semantic features, Represents low-level detail features, σ represents the activation function Sigmoid, T represents the activation function Tanh, and Concat represents the channel concatenation operation.

9. The image semantic segmentation method based on hybrid contextual attention according to claim 7, characterized in that: The collaborative attention module is a simple and effective attention module for feedforward convolutional neural networks. The collaborative attention module consists of three parts. The first part is the channel attention module, which is used to highlight the criticality of the channel to the segmentation model; the second part is the spatial attention module, which allows the segmentation model to improve its sensitivity to the specific spatial position of each channel, thereby focusing on important target positioning; the third part is a 3*3 convolution. The purpose of using convolution here is to perform enhanced feature extraction after obtaining the channel attention module and the spatial attention module, and adjust the number of channels to reduce the amount of segmentation model calculation. The calculation process of the collaborative attention module is as follows: FOR ECA nUnsqueeze(σ(Squeeze(GAP(F). input ))))*FOR input (15) F PSA =σ(Conv 3×3 (Concat(GAP(F ECA ),GMP(F ECA ))))*F ECA (16) F output =Conv 3×3 (F PSA (F ECA (F input ))) (17) Squeeze represents the compression and dimensionality reduction of the channel, and Squeeze represents the expansion and dimensionality increase of the channel. ECA represents the output of the channel attention module, F PSA represents the output of the spatial attention module, GAP represents the global average pooling operation, and GMP represents the global maximum pooling operation.

10. The image semantic segmentation method based on hybrid contextual attention according to claim 1, characterized in that: The multi-stage auxiliary loss calculation module is used to perform auxiliary loss calculation on the segmentation maps generated at different stages in the cascaded attention decoder. The segmentation maps are finally predicted by applying Sigmoid activation or Softmax activation.

Images