A semantic segmentation model with encoder-decoder structure based on tensor and second-order covariance attention mechanism

Abstract

The present invention discloses a semantic segmentation model of a codec structure based on a tensor and second-order covariance attention mechanism. The present invention includes the following steps: 1. Model establishment, designing a semantic segmentation model of a codec structure based on a tensor and second-order covariance attention mechanism, using the attention mechanism in the encoder to learn the correlation between pixel image features and enhance deep semantic information, and fusing shallow and deep semantic information in the decoder to capture the contextual semantic information of the image to expand the inter-class differences of objects of different categories. 2. Model training, first preprocessing the data set, and then training the established model. 3. Model inference, inputting the image of the test set into the trained model, inferring the semantic category prediction value of the test image, and evaluating the accuracy of the test model inference. The present invention is a neural network model for image semantic segmentation, which achieves better segmentation effect in the field of semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a codec structure semantic segmentation model based on tensor and second-order covariance attention mechanism. Background Art

[0002] Image semantic segmentation is a fundamental problem in computer vision, widely used in scene understanding in fields such as autonomous driving, intelligent robotics, and human-computer interaction. Image semantic segmentation technology assigns consistent labels to pixels with similar semantic information in a visual image, interpreting different semantic categories. For example, in autonomous driving scenarios, we need to distinguish all pixels in an image belonging to cars, pedestrians, and so on, and label these pixels with semantic categories.

[0003] With the continuous development of deep learning in recent years, deep learning-based image semantic segmentation models have been increasingly applied in real-world scenarios. End-to-end problem modeling using convolutional neural networks (CNNs) and fully convolutional neural networks (FCNs) has become a mainstream research approach in computer vision. In image semantic segmentation, designing corresponding deep network models to achieve end-to-end modeling has significantly improved segmentation accuracy and speed, significantly improving segmentation results compared to traditional semantic segmentation methods.

[0004] The main methods for image semantic segmentation based on deep learning are: (1) using a fully convolutional neural network to process image semantic segmentation. Although the fully convolutional neural network can accept input images of any size and obtain outputs of the same size as the input, it cannot obtain sufficiently rich image context information, and the image features obtained are not very discriminative; (2) adding a post-processing module based on the fully convolutional neural network. In the past two years, the most commonly used method is to add an attention mechanism module, which can obtain rich context information. However, these attention-based methods mainly focus on the spatial dimension, assigning the same weight to each channel of the feature map, and ignoring the differences between the features of different channels.

[0005] Generally speaking, each feature channel can be viewed as a category-specific response and associated with different semantic responses. Therefore, we need to calculate a weight for each channel associated with a specific semantic feature, thereby aggregating information from the channel dimension and spatial dimension, obtaining global long-range context information, and enhancing the semantic segmentation effect.

[0006] References

[0007] 1.Long J,Shelhamer E,Darrell T.Fully convolutional networks forsemanticsegmentation[C] / / Proceedings of the IEEE conference on computervision and patternrecognition.2015:3431-3440.

[0008] 2.Chen LC, Papandreou G, Kokkinos I, et al. Deeplab: Semantic imagesegmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE transactions onpattern analysis and machineintelligence, 2017, 40(4):834-848.

[0009] 3.Wang Summary of the Invention

[0010] The present invention provides a semantic segmentation model of an encoding / decoding structure based on a tensor and second-order covariance attention mechanism. The semantic segmentation effect of the model based on the tensor and second-order covariance attention mechanism is good and easy to implement.

[0011] The technical solutions of the invention are as follows:

[0012] A semantic segmentation model with an encoder-decoder structure based on tensor and second-order covariance attention mechanism, characterized by including data preprocessing, model establishment, model training and verification;

[0013] Data preprocessing mainly involves dividing the data into training set, evaluation set, and test set for a given image I and the corresponding real label map GT data, providing a data basis for model training, evaluation, and testing;

[0014] The model is established by analyzing the image semantic enhancement process, designing specific submodules, and constructing an end-to-end semantic segmentation deep network model;

[0015] Model training, evaluation, and validation are to determine the parameters of the established model, and to evaluate and test the segmentation effect of the designed model;

[0016] The encoder-decoder semantic segmentation model based on tensor and second-order covariance attention mechanism is developed in the following steps:

[0017] Step 1: Model building;

[0018] Step 2: Model training;

[0019] Step 3: Model inference.

[0020] In step 1, a semantic segmentation model with an encoder-decoder structure based on tensors and second-order covariance attention mechanism is designed. In the encoder, the attention mechanism is used to learn the correlation between pixel image features and enhance deep semantic information. In the decoder, the shallow semantic information and deep semantic information are fused to capture the rich contextual semantic information of the image, thereby expanding the inter-class differences of objects of different categories and ultimately improving segmentation accuracy.

[0021] First, an encoder model is established to extract features from the image. The backbone network ResNet-101 is used to extract the shallow features F1 and deep features F4 of the image.

[0022] Then, a second-order covariance attention model SCAM is established to capture global context information in the channel dimension to enhance the image feature F4 and obtain the image feature X1 with enhanced semantic information;

[0023] A tensor attention model (TAM) is established to aggregate the contextual information of the image feature X1 from the spatial and channel dimensions to obtain an enhanced image feature map X2.

[0024] Finally, a decoder model is built to upsample X2 to obtain a feature map X3. The shallow feature F1 and the feature map X3 are fused to obtain a feature map with enhanced context relevance. The fused feature map is then upsampled and the semantic prediction map Y of the image is obtained through bilinear interpolation.

[0025] In step 2, the training set data is preprocessed first, and the image I is randomly scaled, horizontally rotated, and subjected to geometric transformations and color transformations for data enhancement. The image is then cropped into a fixed-size image I. C ;

[0026] Then train the model established in step 1 and transform the image IC The backbone network ResNet-101, the second-order covariance attention model SCAM in the encoder, and the tensor attention model TAM are input to obtain the feature map X2. Finally, the decoder model fuses the shallow features and deep features to perform semantic segmentation prediction and obtain the prediction result Y. The cross entropy loss function is used to calculate the loss between the predicted value and the true value GT to measure the error between the predicted value and the true value. The network model parameters defined in step 1 are iteratively optimized and trained using the backpropagation algorithm. During the iterative process, after executing the training set data, the evaluation set data is executed to evaluate the model segmentation performance. The training is carried out until the entire model converges.

[0027] In step 3, the images of the test set are input into the trained codec structure semantic segmentation model, the semantic category prediction value of the test image is inferred, the mean intersection over union (MIoU) of the predicted value and the true value is calculated, and the accuracy of the test model's inference prediction is evaluated.

[0028] The specific process of establishing the codec structure semantic segmentation model described in step 1 is as follows:

[0029] First, the encoder model Encoder is established to extract features of the image. The backbone network ResNet-101 is used to extract the shallow features F1 to the deep features F4 of the image. C1 and C2 are the number of channels of the image feature, H1, H2 and W1, W2 are the height and width of the image feature;

[0030] Then, a second-order covariance attention model (SCAM) is established, which uses second-order statistics and local cross-channel interaction strategies to enhance the image feature discrimination from the channel dimension. The specific operations are as follows:

[0031] First calculate the second-order statistic covariance between the two feature maps:

[0032]

[0033] Where Cov(.) represents the covariance operation;

[0034] Then, we use group convolution to design a local cross-channel interaction strategy, calculate the importance of semantic features of different channels, and finally weight them channel by channel by multiplication to the input image feature F4 to complete the recalibration of semantic features in the channel dimension:

[0035]

[0036] Where C1D k (.) represents a one-dimensional convolution with a convolution kernel length of k, Group c×1 (.) represents a grouped convolution with a kernel size of c×1. represents the scaling multiplication along the channel dimension, M is the second-order covariance matrix,

[0037] After completing the second-order covariance attention model SCAM operation, the tensor attention module TAM is established. By introducing the bias learnable parameter tensor A, the attention weight coefficients of different channels and spatial positions of the feature map are calculated, and the classic non-local attention matrix S is expanded to the attention tensor Z1 to better identify the semantic features of different channels at different positions. The specific operation is as follows:

[0038] Q=Conv(X1); K=Conv(X1); V=Conv(X1) (3)

[0039] S=Softmax(Q T ×K) (4)

[0040] A=Softmax(Conv(X1)) (5)

[0041]

[0042]

[0043] X2=V×Z1 T (8)

[0044] In the formula represents each row vector of the matrix S multiplied by an element of the learnable parameter A, Conv(.) represents a convolution with a kernel size of 1×1. represents matrix addition using broadcasting mechanism, where T is the transpose of the matrix;

[0045] After completing the encoder model Encoder operation, establish the decoder model Decoder. The specific process of the decoder model Decoder operation is as follows:

[0046] The shallow feature map F1 and feature map X2 are used as the input of the decoder, and the enhanced feature map X2 is upsampled to obtain a feature map X3 of the same size as the shallow feature map F1; the shallow feature map F1 and feature map X3 are spliced ​​along the channel to form aggregated feature information; then, the fused information is upsampled and the semantic prediction map Y of the image is obtained through bilinear interpolation;

[0047] The specific process of model training described in step 2 is as follows:

[0048] Input image I, and extract image features through the backbone network. First, the second-order covariance attention model SCAM is used to capture global context information in the channel dimension to enhance the image feature F4. A tensor attention model TAM is established to aggregate the context information of the image feature X1 from the spatial dimension and channel dimension to obtain the enhanced image feature map X2. Finally, the decoder is used to fuse shallow features and deep features for semantic segmentation prediction to obtain the prediction result Y. The cross entropy loss function is used to calculate the loss between the predicted value and the true value GT. The loss function is as follows:

[0049]

[0050] Where B refers to the number of images input to the model, and C is the number of categories;

[0051] The specific process of model reasoning in step 3 is as follows:

[0052] After completing the model training in step 2, fix the model parameters, input the test set images, infer the semantic category prediction values ​​of the test images, calculate the intersection of the test prediction values ​​and the true values, and continue until all test set images are tested. The final inference prediction accuracy is given.

[0053] The beneficial effects of the present invention are as follows:

[0054] The proposed method improves accuracy compared to other algorithms for image semantic segmentation. Specifically, it demonstrates the following: 1) Based on a fully convolutional end-to-end network, the proposed method introduces an improved attention mechanism model to aggregate contextual information of image features from both spatial and channel dimensions, enhancing the semantic discriminability of image features; 2) It introduces a codec structure to fuse deep and shallow features, achieving higher accuracy in image semantic segmentation tasks than previous models. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is a diagram showing the overall structure of the model of the present invention.

[0056] FIG2 is a second-order covariance attention model SCAM of the present invention.

[0057] FIG3 is a tensor attention model TAM of the present invention.

[0058] FIG4 is a visualization result of the model experiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose and technical solution of the present invention more clearly understood, the application principle of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments. However, the protection scope of the present invention is not limited to the following specific embodiments.

[0060] Unless otherwise defined, all technical terms used hereinafter have the same meanings as those generally understood by those skilled in the art. The technical terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention.

[0061] Example 1:

[0062] like Figure 1 Figure 2 Figure 3 As shown, the present invention provides a codec structure semantic segmentation model based on tensor and second-order covariance attention mechanism. The specific steps are as follows:

[0063] The model establishment described in step 1 is as follows:

[0064] This invention uses the PASCAL VOC 2012 dataset, which contains 21 categories, as training and test data;

[0065] First, ResNet-101 is used as the backbone network to extract image features. The specific process is as follows: the input image is uniformly scaled to 513×513 and input into the backbone network. The backbone network extracts image features and outputs the last layer feature map. As input, the encoder model is input to further enhance and extract features; then, the encoder model operation is performed as follows:

[0066] ① Input the feature map F4 into the second-order covariance attention model SCAM, and use the second-order statistics and local cross-channel interaction strategy to enhance the image feature discrimination from the channel dimension. The specific operations are as follows:

[0067] First calculate the second-order statistic covariance between the two feature maps:

[0068]

[0069] Where Cov(.) represents the covariance operation;

[0070] Then, we use group convolution to design a local cross-channel interaction strategy, calculate the importance of semantic features of different channels, and finally weight them channel by channel by multiplication to the input image feature F4 to complete the recalibration of semantic features in the channel dimension:

[0071] X1=sigmoid(C1D k (Group c×1 (M)))⊙F4 (2)

[0072] Where C1D k (.) represents a one-dimensional convolution with a convolution kernel length of k, Groupc×1 (.) represents a grouped convolution with a kernel size of c×1. represents the scaling multiplication along the channel dimension, M is the second-order covariance matrix,

[0073] ②Use the tensor attention module TAM to aggregate the context information of the image feature X1 from the spatial dimension and channel dimension to obtain the enhanced image feature map X2. The specific operation is:

[0074] Q=Conv(X1); K=Conv(X1); V=Conv(X1) (3)

[0075] S=Softmax(Q T ×K) (4)

[0076] A=Softmax(Conv(X1)) (5)

[0077]

[0078]

[0079] X2=V×Z1 T (8)

[0080] In the formula represents each row vector of the matrix S multiplied by an element of the learnable parameter A, Conv(.) represents a convolution with a kernel size of 1×1. represents matrix addition using broadcasting mechanism, where T is the transpose of the matrix;

[0081] Finally, the decoder module Decoder operation is executed to fuse deep features and shallow features. The decoder module Decoder operation is as follows:

[0082] The shallow feature map F1 and feature map X2 are used as the input of the decoder, and the enhanced feature map X2 is upsampled to obtain a feature map X3 of the same size as the shallow feature map F1; the shallow feature map F1 and feature map X3 are spliced ​​along the channel to form aggregated feature information; then, the fused information is upsampled and the semantic prediction map Y of the image is obtained through bilinear interpolation;

[0083] This completes the model building operation.

[0084] The model training described in step 2 is as follows:

[0085] Train the model established in step 1. Pass the image I through the backbone network ResNet-101, the encoder model module Encoder, and the decoder model module Decoder to obtain the semantic prediction map Y; compare it with the true label map of the training set data, and calculate the loss value between the predicted value and the true value by using the cross entropy loss function. Then, adjust the parameter values ​​of the entire network according to the loss value, and iterate the training. During the iterative process, execute the training set data and then execute the evaluation set data to evaluate the model segmentation performance until the network converges;

[0086] This completes the model training operation.

[0087] The model reasoning described in step 3 is as follows:

[0088] After the model is trained in step 2, the model parameters are fixed, and the images in the test set are input to obtain the test prediction values. The semantic category prediction values ​​of the test images are inferred, and the average intersection-over-union ratio of the test prediction values ​​to the true values ​​is calculated to evaluate the prediction accuracy.

[0089] This completes the model inference operation.

[0090] The following table shows the accuracy of the method proposed in this paper on Pascal VOC 2012. FCN is the pioneering work on deep learning-based image semantic segmentation. Our is the deep model proposed in this paper. "aero" and "bike" represent the categories to be semantically segmented in the dataset. The mean intersection over union (MIoU) is used to evaluate the average accuracy of image semantic segmentation.

[0091]

Claims

1. A semantic segmentation model based on tensor and second-order covariance attention mechanism, characterized by The following steps are involved: Step 1: Model building; Specifically, we designed a semantic segmentation model with an encoder-decoder structure based on tensors and a second-order covariance attention mechanism. In the encoder, the attention mechanism learns the correlation between pixel image features and enhances the semantic information of deep features. In the decoder, the shallow and deep semantic information are fused to capture the rich contextual semantic information of the image, thereby expanding the inter-class differences between different categories of objects and ultimately improving segmentation accuracy. First, the encoder model Encoder is established to extract features of the image. The backbone network ResNet-101 is used to extract the shallow features F1 to the deep features F4 of the image. C1 and C2 are the number of channels of the image feature, H1, H2 and W1, W2 are the height and width of the image feature; Then, a second-order covariance attention model SCAM is established to capture global context information in the channel dimension to enhance the image feature F4 and obtain the image feature X1 with enhanced semantic information; A tensor attention model (TAM) is established to aggregate the contextual information of the image feature X1 from the spatial and channel dimensions to obtain an image feature map X2 with stronger semantic discrimination. Finally, a decoder model is built to upsample X2 to obtain a feature map X3. The shallow feature F1 and the feature map X3 are fused to obtain a feature map with enhanced context relevance. The fused feature map is then upsampled and the semantic prediction map Y of the image is obtained through bilinear interpolation. Among them, the established second-order covariance attention model SCAM is characterized by: Using second-order statistics and local cross-channel interaction strategies, the image feature discrimination is enhanced from the channel dimension. The specific operations are as follows: First calculate the second-order statistic covariance between the two feature maps: Where Cov(.) represents the covariance operation; Then, we use group convolution to design a local cross-channel interaction strategy, calculate the importance of semantic features of different channels, and finally weight them channel by channel by multiplication to the input image feature F4 to complete the recalibration of semantic features in the channel dimension: X1=sigmoid(C1D k (Group c×1 (M)))⊙F4 (2) Where C1D k (.) represents a one-dimensional convolution with a convolution kernel length of k, Group c×1 (.) represents the grouped convolution with a kernel size of c×1, ⊙ represents the scaling multiplication along the channel dimension, M is the second-order covariance matrix, The established tensor attention model TAM is characterized by: By introducing the bias learnable parameter tensor A, the attention weight coefficients of different channels and spatial positions in the feature map are calculated, and the classic non-local attention matrix S is expanded to the attention tensor Z1 to better distinguish the semantic features of different channels at different positions. The specific operation is as follows: Q=Conv(X1); K=Conv(X1); V=Conv(X1) (3) S=Softmax(Q T ×K) (4) A=Softmax(Conv(X1)) (5) Z=A◎S (6) <h2 style=";text-align:left;direction:ltr">X2=V×Z1<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> (8) Where ◎ represents each row vector of the matrix S multiplied by an element of the parameter matrix A, Conv(.) represents a convolution with a kernel size of 1×1. represents matrix addition using broadcasting mechanism, where T is the transpose of the matrix; Step 2: Model training; Train the model established in step 1 until the entire model converges; Step 3: Model inference; Input the images in the test set into the trained model, infer the semantic category prediction value of the test image, and evaluate the prediction accuracy.

2. A codec structure semantic segmentation model based on tensor and second-order covariance attention mechanism according to claim 1, characterized in that The model training described in step 1 is as follows: Input image I, and extract image features through the backbone network. First, a second-order covariance attention model SCAM is established to capture global context information in the channel dimension to enhance the image feature F4. A tensor attention model TAM is established to aggregate the context information of the image feature X1 from the spatial dimension and channel dimension to obtain an image feature map X2 with stronger semantic discrimination. Finally, a decoder is used to fuse shallow features and deep features for semantic segmentation prediction to obtain the prediction result Y. The cross entropy loss function is used to calculate the loss between the predicted value and the true value GT. The loss function is as follows: Where B is the number of images fed into the model, and C is the number of categories.

Images