A semantic segmentation method based on dataset distribution and contextual information

Abstract

The present invention discloses a semantic segmentation method based on the context information of data set distribution. The method first preprocesses the image, uses a neural network to obtain its deep features, and uses a linear layer to decode the features to obtain a preliminary prediction result; then, based on the obtained preliminary prediction results and the global feature statistical information, a new classifier is formed to guide the subsequent feature extraction; at the same time, according to the distribution of different samples in the data set, the class centers of different categories are accumulated and acted on the existing attention mechanism. Based on the above steps, a semantic segmentation neural network based on data and distribution is built. Compared with the prior art, the present invention has the characteristics of simple method, low computational complexity, high classification accuracy, etc., which effectively alleviates the problem of low prediction accuracy of small category samples in semantic segmentation tasks, and can be applied in multiple scenarios such as autonomous driving, smart cities and medical segmentation.

Description

Technical Field

[0001] The present invention relates to the fields of machine vision, deep learning, neural networks, etc., and proposes a semantic segmentation method based on data set distribution and context information. Background Art

[0002] Semantic Segmentation: It is one of the basic tasks of computer vision and has a wide range of applications in autonomous driving, intelligent transportation, medical diagnosis and other fields. Its goal is to enable the computer to understand the semantic environment of the current image, so as to divide the entire image area according to the classification information. Different from general image classification tasks, semantic segmentation requires the output of prediction results that match the size of the original image, and is a pixel-to-pixel image classification task. Therefore, when parsing an image, it is necessary to comprehensively process the dependencies between different pixels and different categories.

[0003] Existing research on semantic segmentation generally believes that the network should capture the long-range information of the image as much as possible on the encoder and connector, including the spatial structural features of the pixels and the interdependence between categories. One important method is to calculate the covariance matrix between features as weights, and then use matrix multiplication to assign weights to features at different positions. This method is also called "attention mechanism". In the semantic segmentation task, research on attention mechanism is currently focused on three directions:

[0004] 1. Attention mechanism in spatial dimension. When using matrix multiplication to calculate the covariance matrix, this method will eliminate the channel information, thereby forming an HW*HW attention matrix to describe the correlation of the spatial structure of different pixels. Since the input images of the semantic segmentation task may be of different sizes, this method often brings a large amount of calculation. Some methods use multiple single-dimensional iterations to achieve the fitting of high-dimensional matrices, while others calculate local attention instead of global attention.

[0005] 2. Attention mechanism in channel dimension. When using matrix multiplication to calculate the covariance matrix, this method will weight and eliminate the spatial information, thus forming a Channels*Channels attention matrix to describe the dependencies between different categories and channels. These methods often generate a rough segmentation result first, and then use their probability distribution to weight the feature channel weights.

[0006] 3. The possibility of integrating the above two. That is, to study the coupling ability of the two information, and how to efficiently capture the spatial information correlation and channel information dependency in the same model.

[0007] However, the above methods often only focus on the information of the current input image, while ignoring the distribution consistency of the entire data set; in the process of counting long-range information, they often produce high computational complexity, resulting in too long inference time; at the same time, excessive focus on the information of a single image can easily cause overfitting of the algorithm, which can easily lead to adverse effects in the verification and testing stages. Summary of the invention

[0008] The purpose of the present invention is to provide a semantic segmentation method, the main process of which is: using the backbone network to perform preliminary feature extraction on the image, using the output of the shallow network to perform preliminary prediction on the image, and calculating the covariance matrix between the preliminary prediction result and the deep features; traversing all data, using the sliding average method to count the covariance matrix of all images, and obtaining the feature distribution of the entire data set; in the inference stage, combining the current image information and the accumulated data set distribution information, using the cross-attention mechanism algorithm in the channel dimension, processing the deep features of the network, and then using the full convolution classifier to predict the image pixel by pixel. Compared with the prior art, the present invention has the characteristics of simple method, low computational complexity, high classification accuracy, etc., which effectively alleviates the problem of low prediction accuracy of small category samples in semantic segmentation tasks.

[0009] The specific technical solution for achieving the purpose of the present invention is:

[0010] A semantic segmentation method based on data set distribution and context information includes the following specific steps:

[0011] Step A: Preprocess the dataset images and complete the classifier settings based on the statistical information of the dataset;

[0012] Step B: Build a neural network model, design a loss function, adjust parameters such as learning rate and optimizer, and train the network on the data set;

[0013] Step C: Use the trained neural network to evaluate its performance on the validation set and test set.

[0014] In step A, you need to complete the following steps in sequence:

[0015] Step A1: preprocessing of dataset images;

[0016] Since the present invention needs to classify input images of different sizes pixel by pixel and calculate the covariance matrix of the preliminary prediction results and the minimum feature map at the same time, it is different from the general classification task. When processing the input image, it is necessary to pad the edge of the image so that the number of pixels of its width and height is an integer multiple of the width and height of the minimum feature map; and for the padded pixels, its loss function is not calculated during training;

[0017] In the training phase, in order to unify data distribution and avoid overfitting, the input images are randomly cropped, scaled, horizontally flipped, and contrast adjusted, and a Gaussian filter with random intensity is used to enhance the generalization performance of the network. At the same time, in the specific implementation process of the present invention, all images are linearly interpolated to the same size for training and statistics;

[0018] In the test phase, for input images of different scales, edge padding is also used to process the validation set and test set images. After that, no image enhancement is needed and they are directly sent to the network for semantic information inference. After that, the predicted value of edge padding can be deleted.

[0019] Step A2: Setting of classifier output dimension;

[0020] The present invention needs to set the codec structure according to the classification requirements of the data set. Two different linear classifiers are used to process network features. Since the network needs to generate preliminary prediction results at the encoder end and output accurate prediction results at the decoder end, the output dimensions of the two classifiers need to be equal to the number of classification categories of the data set.

[0021] In step B, the following steps need to be completed in sequence:

[0022] Step B1: Build a neural network. This network consists of three parts: encoder, connector, and decoder, which are connected in sequence. These include:

[0023] Step B1-1: Encoder settings; The present invention can use most mainstream deep models as backbone networks, such as ResNet, Xception, HRNet, etc. However, unlike other technologies, a separate classifier is used, connected to the secondary deep layer of the backbone network, so that it produces the initial prediction results of the network;

[0024] Assume that the backbone network has n stages and its output is X backbone ,but:

[0025] X backbone = {X1, X2, ..., X n}

[0026] The classifier consists of a convolutional layer with a kernel of 1x1, and its input dimension is the same as X n-1 The same, the output dimension is the number of categories of the data set; let the classifier be f coarse , the initial prediction result of its output is Y coarse ,have:

[0027] Y coarse =f coarse (X n-1 )

[0028] In addition, when deep learning is used to process computer vision problems, it is often necessary to extract the spatial hierarchical structure information of the image, and such information is often widely distributed in different data sets. Therefore, the model parameters pre-trained on a large data set are used to initialize the encoder network parameters of the present invention. These pre-trained data can use images from classification or detection data sets such as ImageNet and COCO;

[0029] Step B1-2: Connector settings; In the present invention, the main function of the connector is to process the output of the encoder, extract long-range information as much as possible, and provide assistance for the decoder's classification. This process has three steps: calculating the center of the image class, accumulating the center of the data set class, and using the attention algorithm to fuse the class centers. The specific implementation methods of the above steps will be introduced in turn:

[0030] Step B1-2-1: Calculate the center of this image class; the output of the encoder is the input of the connector, which is recorded as the deep feature map X of the backbone network. n , and the preliminary classifier f coarse The output Y coarse First, the network will coarse Perform linear interpolation to make its size in the spatial direction and X n The data dimensions of different variables are:

[0031]

[0032] Among them, HW is the feature map size, d is the number of feature map channels, and k is the number of dataset categories.

[0033] Next, the network uses the softmax function to coarse The spatial dimension direction of is weighted and its difference with X is calculated. n The covariance matrix A is:

[0034]

[0035] The covariance matrix counts the correspondence between different categories and feature channels in this image, which is called the class center of this image.

[0036] Step B1-2-2: Accumulate the class centers of the dataset; in order to allow the network to take into account the class center information of similarly distributed images in the dataset during the inference process, a weight matrix is ​​initialized It is used to store the class center information of the data set. During the training process, the matrix is ​​updated using the class center of each image using a sliding average method. Let the update coefficient be μ. For any time t, the update process to time t+1 can be expressed as:

[0037] M t+1 =μM t +(1-μ)A

[0038] The matrix M is obtained by sliding average accumulation of the image class center A. As the training progresses, the matrix can count the class centers of different images in the entire data set, which is called the cumulative class center.

[0039] Step B1-2-3: Use the attention algorithm to fuse the class centers; in steps B1-2-1 and B1-2-2, the class centers based on a single image have been extracted. and class centers based on the dataset It is used to describe the correspondence between categories and feature channels. Here, the attention algorithm is used to fuse the class center and deep features:

[0040]

[0041] The usage weight is and The three matrices of complete the mapping of Q, K, y, and C is the number of feature channels of the attention algorithm. In practical applications, these matrices can be implemented by a convolution layer with a convolution kernel size of 1x1. The definitions of Q, K, and V are as follows:

[0042] Q=X n W Q ,

[0043] Center fuse =Concat[LN(A T ), LN(M T )],

[0044] K=Center fuse W K ,

[0045] V=Center fuse W V ,

[0046] Through LN, i.e. LayerNorm, the class centers from different sources are standardized. After the attention mechanism algorithm, the output is

[0047] At the end of the connector, the present invention designs a residual mechanism to combine the input and output of the attention mechanism to improve the stability and convergence speed of the network. Different from the general attention residual mechanism, the mapped feature map is used as the original input. Experimental verification shows that this scheme can improve the performance of the network. The specific implementation method is as follows:

[0048] F out =F′+Q,

[0049] The obtained F out This is the connector output.

[0050] Step B1-3: Setting of the decoder; In the present invention, the main function of the decoder is to use a neural network to predict the pixel-by-pixel classification result of the input image based on the output of the connector.

[0051] In the process of constructing the decoder, the output F of the connector is processed using convolutions with kernel sizes of 3x3 and 1x1 respectively. out ; The 3x3 convolution layer has C convolution kernels, and the 1x1 convolution layer has k convolution kernels, and its output is During the training process, in order to avoid overfitting, random dropout was added between the two convolutional layers;

[0052] for Each pixel contains k scores, and the category with the largest score is the network's prediction result Y for the pixel. pred :

[0053] Y pred = argmax(Y fine )

[0054] Step B2: Design loss function; In the present invention, the encoder outputs and Both need to be constrained using the cross entropy loss function. The final loss function is the weighted sum of the two:

[0055] Loss = L coarse +λL fine

[0056] L coarse = CrossEntropy(δ(Y coarse ), GT)

[0057] L fine = CrossEntropy(δ(Y fine ), GT)

[0058] Among them, λ is a hyperparameter, GT is the pixel-by-pixel label given by the dataset; δ is the Softmax function, and CrossEntropy is the cross entropy loss function, which is used to constrain the prediction results of multi-classification problems. For the input value z i , i∈[1,N], and probabilities p,q, the mathematical expressions of the softmax function and the CrossEntropy function are:

[0059]

[0060]

[0061] It is worth noting that for different data sets, the data distribution of different categories is also different. In order to balance the differences in size categories between samples and alleviate the long tail problem, the loss functions of different categories can be weighted using a priori weights, which can also be applied to the present invention.

[0062] Step B3: Learning rate and optimizer settings; The present invention uses a common learning rate decay mode in segmentation tasks, namely, exponential decay to control the learning rate, which is mathematically expressed as:

[0063]

[0064] Among them, lr is the current learning rate, lr base is the initial learning rate, iter is the iteration progress of the current training, iter max The total number of iterations required to complete the training, poly is a hyperparameter used to adjust the speed of learning rate decay.

[0065] At the same time, the network is optimized using stochastic gradient descent, and its parameters are set according to the actual usage scenario.

[0066] In step C, you need to complete:

[0067] Step C: In the inference stage, the input image is normalized and directly input into the neural network without any other data enhancement methods. The output of the decoder is obtained, and the index value after argmax is used as the classification result. The mean intersection over union (mIoU) is used to evaluate the model quality.

[0068] Beneficial Effects of the Invention

[0069] Under the same training environment, using the same backbone network and other hyperparameter settings, the current semantic segmentation system using convolutional networks has the highest average intersection-over-Unit ratio of 80.09 for OCRNet on the Cityscapes dataset. The highest intersection-over-Unit ratio of the present invention on the same dataset is 80.29, achieving an improvement of 0.2.

[0070] In the ADE20K dataset, the average IoU of OCRNet is 44.60, while the average IoU of the present invention can reach 44.89. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a flow chart of the present invention;

[0072] Figure 2 It is a structural relationship diagram of network encoder, connector and decoder;

[0073] Figure 3 This is the encoder structure diagram;

[0074] Figure 4 This is the connector structure diagram;

[0075] Figure 5 This is the decoder structure diagram. DETAILED DESCRIPTION

[0076] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0077] Example

[0078] Taking the city street view dataset Cityscapes as an example, a neural network is built to complete the semantic segmentation task.

[0079] As attached Figure 1 As shown, the process can be divided into three steps:

[0080] Step 1: Preprocess the dataset images and complete the classifier settings based on the statistical information of the dataset;

[0081] Cityscapes is a commonly used dataset for pixel-by-pixel prediction tasks. It mainly collects street scene images of 50 European cities and annotates each image with pixel-level categories. When currently applied to semantic segmentation tasks, 19 categories are often used for network training and model evaluation. These categories include background categories (such as sky, road, etc.) and foreground categories (pedestrians, cars, etc.), which have important academic and application value in the fields of autonomous driving and smart cities.

[0082] This embodiment uses its accurately annotated subset for training and testing. The training set contains 2975 pictures and the validation set contains 500 pictures. The resolution of each picture is 2048x1024, and each pixel is divided into 19 valid classes and 1 invalid class.

[0083] First, preprocess the training data:

[0084] 1. Each image input to the network will be randomly scaled, and the scaling factor is controlled between 0.5 and 2.0;

[0085] 2. Then crop the image at random positions, and the cropped size is 512x512;

[0086] 3. Randomly reverse the image with a probability of 0.5;

[0087] 4. Randomly adjust the brightness, contrast and saturation of the image, with the range of random fluctuations being 0.3;

[0088] 5. Use [123.675, 116.28, 103.53] as the mean and [58.395, 57.12, 57.375] as the standard deviation to standardize the RGB channels of the image;

[0089] Use 4 RTX2080ti graphics cards to train and test the model. In this embodiment, set the batch size to 8, allocate two images to each GPU, and use SYNC BN to standardize the middle layer;

[0090] In this scenario, the output of the two classifiers on the encoder and decoder sides is set to 19 channels, corresponding to the 19 valid classes of Cityscapes. For points in the invalid class, their loss function is not calculated.

[0091] Step 2: Build a neural network, design a loss function, adjust the learning rate and optimizer parameters, and train the network on the data set;

[0092] According to the description in the summary of the invention, according to the attached Figure 2 The structure of the semantic segmentation network is constructed. The encoder uses ResNet101 as the backbone network, and its structure is shown in the attached figure. Figure 3 The connector and decoder structure of the network are shown in Figure 4 , Figure 5 After the network is built, the hyperparameter λ of the loss function is set to 0.4, so that the weight ratio of the loss function between the preliminary prediction result and the final prediction result is 0.4:1; when updating the class center of the data set, the momentum hyperparameter μ is set to 0.9;

[0093] Use stochastic gradient descent with momentum (SGD+Momentum) as the network optimizer, where the momentum value is set to 0.9; the weight decay coefficient (Weight Decay) is set to 0.0005;

[0094] During the training phase, the initial learning rate of the network was set to 0.01, and as the training progressed, exponential decay was used to adjust the learning rate. The decay coefficient poly was set to 0.9, and the minimum learning rate was set to 0.0001.

[0095] In this embodiment, the maximum number of iterations of training is set to 160k.

[0096] Step 3: Use the trained neural network to evaluate performance on the validation set and test set;

[0097] After the training is completed, the validation set data is used to evaluate the model indicators. The average intersection over union (mIoU) of the model is 80.29, which is about 0.3 higher than the best convolutional neural network method OCRNet under the same training environment.

[0098] The above model building and training environment configuration are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, shall fall within the protection scope of the present invention.

Claims

1. A semantic segmentation method based on data set distribution and context information, characterized in that: The method comprises the following specific steps: Step A: Preprocess the dataset images and complete the classifier settings based on the statistical information of the dataset, including: Step A1: Preprocessing of dataset images When processing the input image, pad the edges of the image so that the number of pixels in its width and height is an integer multiple of the width and height of the minimum feature map; and for the padded pixels, their loss function is not calculated during training; During the training phase, the input images are randomly cropped, scaled, horizontally flipped, and contrast adjusted, and a Gaussian filter with random intensity is used to enhance the generalization performance of the network. At the same time, all images are linearly interpolated to the same size for training and statistics. In the test phase, for input images of different scales, edge padding is first used to process the validation set and test set images, and then they are directly sent to the network for semantic information inference; and the predicted value of edge padding is deleted; Step A2: Setting the output dimension of the classifier Set the encoder-decoder structure according to the classification requirements of the dataset; use two different linear classifiers to process network features, The encoder generates preliminary predictions and the decoder outputs precise predictions. The output dimensions of these two classifiers are equal to the number of categories in the dataset. Step B: Build a neural network, design a loss function, adjust the learning rate and optimizer parameters, and train the network on the data set, including: Step B1: Build a neural network, which consists of three parts: an encoder, a connector, and a decoder, which are connected in sequence, where: Step B11: The encoder is set up using a mainstream deep model as the backbone network, which is ResNet, Xception or HRNet; a separate classifier is used, connected to the second deep layer of the backbone network to generate the network's preliminary prediction results; Assume that the backbone network has n stages and its output is X backbone ,but: X backbone ={X1,X2,...,X n } The classifier consists of a convolutional layer with a kernel of 1x1, and its input dimension is the same as X n-1 The same, the output dimension is the number of categories of the data set; let the classifier be f coarse , the initial prediction result of its output is Y coarse ,have: Y coarse =f coarse (X n-1 ) Step B12: Connector settings. The connector is used to process the encoder output, extract long-range information, and assist the decoder classification. The process has three steps: calculating the image class center, accumulating the data set class center, and fusing the class center using the attention algorithm. Specifically: Step B12a: Calculate the center of this image class. The output of the encoder is the input of the connector, which is recorded as the deep feature map X of the backbone network. n , and the preliminary classifier f coarse The output Y coarse ; First, for Y coarse Perform linear interpolation to make its size in the spatial direction and X n The same; at this time the data dimensions of different variables are: Among them, HW is the feature map size, d is the number of feature map channels, and k is the number of dataset categories; Next, use the softmax function to coarse The spatial dimension direction of is weighted and its difference with X is calculated. n The covariance matrix A is: The covariance matrix counts the correspondence between different categories and feature channels in this image, which is called the class center of this image; where, δ represents the softmax function, and T represents the matrix transpose; Step B12b: Accumulate the data set class centers and initialize a weight matrix Class for storing datasets Center information; During the training process, the weight matrix is ​​updated using the class center of each image using a sliding average method; Let the update coefficient be μ, for any time t, the update process to time t+1 is expressed as: M t+1 =μM t +(1-μ)A The matrix M is obtained by sliding average accumulation of the image class center A. As the training progresses, the matrix can be statistically integrated. The class centers of different images in a dataset are called cumulative class centers; Step B12c: Use the attention algorithm to fuse the class centers. In steps B12a and B12b, the class centers based on a single image have been extracted. and class centers based on the dataset Used to describe the correspondence between categories and feature channels; use the attention algorithm to fuse the class center and deep features: The usage weight is and The three matrices of complete the mapping of Q, K, and V, and C is the number of feature channels of the attention algorithm; these matrices are implemented by the convolution layer with a convolution kernel size of 1x1; the definitions of Q, K, and V are as follows: Through LN, i.e. LayerNorm, the class centers from different sources are standardized. After the attention mechanism algorithm, the output is At the end of the connector, a residual mechanism is designed to use the mapped feature map as the original input, which can improve the performance of the network. The specific implementation method is as follows: The obtained F out That is the connector output; Step B13: Setting up the decoder. The decoder is used to predict the pixel-by-pixel classification result of the input image using a neural network based on the output of the connector. In the process of constructing the decoder, the output F of the connector is processed using convolutions with kernel sizes of 3x3 and 1x1 respectively. out ; The 3x3 convolution layer has C convolution kernels, and the 1x1 convolution layer has k convolution kernels, and its output is During training, random dropout is added between two convolutional layers; for Each pixel contains the probability scores of k categories, and the one with the largest score is the network's prediction result Y for that pixel. pred : AND pred =argmax(Y fine ); Step B2: Design the loss function, the encoder output and Both use the cross entropy loss function (Cross Entropy Loss) for constraints; the final loss function is the weighted sum of the two: Loss=L coarse +λL fine L coarse =CrossEntropy(δ(Y coarse ), GT) L fine =CrossEntropy(δ(Y fine ), GT) Among them, λ is a hyperparameter, GT is the pixel-by-pixel label given by the dataset; δ is the Softmax function, and CrossEntropy is the cross entropy loss function, which is used to constrain the prediction results of multi-classification problems. For the input value z i , i∈[1,N], and probabilities p,q, the mathematical expressions of the softmax function and the CrossEntropy function are: Step B3: Learning rate and optimizer settings, use exponential decay to control the learning rate, its mathematical expression is: Among them, lr is the current learning rate, lr base is the initial learning rate, iter is the iteration progress of the current training, iter max The total number of iterations required to complete the training. Poly is a hyperparameter used to adjust the speed of learning rate decay. At the same time, the network is optimized using stochastic gradient descent. Step C: Use the trained neural network to evaluate the performance on the validation set and the test set. Specifically, the input image is normalized and directly input into the neural network obtained in step B. The output result of the decoder is obtained, and the index value after argmax is used as the classification result.

Images