A method for image semantic segmentation based on configurable context path

Abstract

The invention relates to an image semantic segmentation method based on a configurable context path, comprising the following steps: extracting features from an input image to obtain a feature map F; using a long short-term memory network (LSTM) to calculate Q context paths for each pixel in the feature map F, each context path comprising T regions; propagating the features of each pixel in the feature map F along the context path according to the Q context paths, correspondingly obtaining Q appearance feature maps, and all the appearance feature maps forming an appearance feature map set R; performing back propagation for each pixel in the appearance feature map set R according to the Q context paths in the previous second step propagation, correspondingly obtaining Q enhanced feature maps, and all the enhanced feature maps forming an enhanced feature map set A; fusing the enhanced feature map set A to obtain a semantic feature map Y; and optimizing and training a neural network model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision, and relates to a semantic segmentation technology, in particular to an image semantic segmentation method based on a configurable context path. Background Art

[0002] Semantic segmentation aims to infer the category of each pixel in an image and is one of the basic tasks in computer vision. In recent years, with the development of deep learning, the accuracy of semantic segmentation on large-scale datasets has been effectively improved. In order to enhance the accuracy of dense semantic category prediction, some studies consider the contextual information near the pixel to better enhance the semantic information.

[0003] Context propagation studies the communication between pixels and describes this communication in the form of context representation. The current advanced context propagation method uses spatial pyramid pooling, which uses receptive fields of different scales to capture local and long-range context information. However, since the pooling layer is applied in the last step, the underlying detailed information is lost.

[0004] Context enhancement occurs when context information propagates between pixels. It captures high-order relationships between pixels to further enhance context information. Some current studies on semantic segmentation merge pixels in the middle of information paths to enhance context. Graph models densely connect pixels between information paths. In this process, the pixels in the middle capture long-range dependencies, thereby enhancing context. However, graph models require pre-trained paths and lack certain flexibility.

[0005] References:

[0006] [1]He K, Zhang X, Ren S, et al.Deep residual learning for imagerecognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778.

[0007] [2]Hochreiter S,Schmidhuber J.Long short-term memory[J].Neuralcomputation,1997,9(8):1735-1780. Summary of the invention

[0008] The present invention provides an image semantic segmentation method based on a configurable context path, which uses a configurable context path to guide the aggregation of image context information, effectively increasing the flexibility of image context information aggregation. At the same time, the configurable context path can enable pixels in the image to obtain context information of different scales, so that the pixels in the image can obtain more accurate semantic features and achieve more accurate image semantic segmentation. The present invention is implemented by the following technical solutions:

[0009] A method for image semantic segmentation based on a configurable context path comprises the following steps:

[0010] A method for image semantic segmentation based on a configurable context path comprises the following steps:

[0011] The first step is to input the image Perform feature extraction to obtain feature map F;

[0012] In the second step, the long short-term memory network LSTM is used to calculate Q context paths for each pixel in the feature map F. Each context path contains T regions. According to the Q context paths, the features of each pixel in the feature map F are propagated along the context path, and Q appearance feature maps are obtained. All appearance feature maps constitute the appearance feature map set R. The method is:

[0013] (1) The tth region in the qth context path of the nth pixel in the feature map F is recorded as initialization is the coordinate position of the nth pixel in the feature map F;

[0014] (2) Using the long short-term memory network LSTM and the differentiable ROI pooling module, the context path area of ​​all pixels is calculated according to the following formula:

[0015]

[0016]

[0017] in stands for Long Short-Term Memory Network, represents the differentiable ROI pooling module, F represents the feature map, represents the feature corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, Represents the hidden state of the long short-term memory network LSTM corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F. Set to 0;

[0018] (3) The appearance feature of the tth region obtained by propagating the nth pixel in the feature map F on its qth context path is recorded as initialization

[0019] (4) Use a fully connected neural network to calculate the propagation characteristics of all pixels in the context path area according to the following formula

[0020]

[0021]

[0022] in represents the feature corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, represents the parameters of the fully connected neural network corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, softmax(.) represents the softmax function, ⊙ represents the Hadamard product, σ(.) represents the sigmoid function; all Combined together, they form the appearance feature map set R;

[0023] In the third step, for each pixel in the appearance feature map set R, back propagation is performed according to the Q context paths in the previous second step, and Q enhanced feature maps are obtained. All enhanced feature maps constitute the enhanced feature map set A. The method is:

[0024] (1) For the qth context path of the nth pixel in the appearance feature map R, the end pixel of its back propagation is denoted as m. Then the enhanced feature of the tth region during its back propagation is expressed as initialization

[0025] (2) Using a differentiable ROI pooling module and a fully connected neural network, the enhanced features obtained by back-propagating all pixels in the context path region are calculated according to the following formula:

[0026]

[0027]

[0028]

[0029] in represents the parameters of the fully connected neural network corresponding to the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, represents the appearance feature of the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, Represents the location information of the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, Represents the final appearance feature of the m-th pixel on the q-th context path;

[0030] (3) Yes Summing to get The formula is as follows:

[0031]

[0032] What you get It is the enhanced feature of pixel m calculated according to the qth appearance feature map in the appearance feature map set R; there are Q enhanced feature maps in total, which together constitute the enhanced feature map set A;

[0033] The fourth step is to fuse the enhanced feature map set A to obtain the semantic feature map Y;

[0034] The fifth step is to obtain the predicted probability distribution according to the semantic feature map Y, calculate the cross entropy between the predicted probability distribution and the true segmentation value of the input image, obtain the loss value, and use the gradient descent method to optimize the entire neural network model;

[0035] Step 6: Neural network model training.

[0036] Furthermore, the method of the fourth step is:

[0037] (1) Use a fully connected neural network to calculate the fusion coefficient of each pixel in each enhanced feature map according to the following formula:

[0038]

[0039] in, is the fusion coefficient of pixel n in the qth enhanced feature map, W f,p represents the parameters of the fully connected neural network used for the feature map and the p-th enhanced feature map, W p,q represents the parameters of the fully connected neural network used by the p-th enhanced feature map and the q-th enhanced feature map, Represents the feature corresponding to the nth pixel of the feature map F, represents the enhanced feature of the nth pixel in the pth enhanced feature map, Represents the enhanced feature of the nth pixel in the qth enhanced feature map;

[0040] (2) According to the fusion coefficient, the enhanced feature map set A is fused to obtain the semantic feature Y of each pixel after fusion. n:

[0041]

[0042] in represents the enhanced feature of the nth pixel in the qth enhanced feature map, represents the weighting coefficient of the nth pixel in the qth enhanced feature map, all Y n It constitutes the semantic feature map Y.

[0043] The beneficial effects of the technical solution provided by the present invention are:

[0044] Existing semantic segmentation methods all use fixed target receptive fields to capture the visual contextual relationship between targets, which not only lacks flexibility but also makes it difficult to obtain multi-scale visual contextual relationships. The present invention uses a semantic segmentation method with a configurable context path to learn the pixel-level appearance and semantic features of targets and capture the visual contextual relationship between targets at different scales. By using the visual contextual relationship between targets, a multi-scale contextual feature expression is constructed to obtain a more accurate image semantic segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of an image semantic segmentation method based on a configurable context path;

[0046] Figure 2 The experimental results of the method of the present invention are compared with the existing best method; DETAILED DESCRIPTION

[0047] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Based on the technical solution of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0048] First, some concepts of the present invention are explained. ResNet is composed of 101 convolutional layers, which are divided into four convolutional blocks in total, and each convolutional block contains several convolutional layers. In each convolutional block, there is a residual connection between adjacent convolutional layers. The long short-term memory network LSTM takes a cell feature vector and a hidden vector as input, and takes a cell feature vector and a hidden vector as output. In the present invention, the cell feature vector and a hidden vector are both features learned from pixels. ROI, that is, the region of interest, is an area centered on a certain pixel. The ROI pooling module extracts features for the area, usually averaging all pixels in the area.

[0049] The data sets used for experimental feasibility verification of the present invention are Cityscapes data set, PASCAL VOC2012 data set, and COCO-Stuff data set. The Cityscapes data set has 19 semantic categories, including 2975 training images, 500 verification images, and 1525 test images; the PASCAL VOC 2012 data set has 21 semantic categories, including 10582 training images, 1449 verification images, and 1456 test images; the COCO-Stuff data set has 172 semantic categories, divided into two versions: one version includes 9000 training images and 1000 verification images; the other version includes 120000 training images and 5000 verification images.

[0050] The first step is to select the backbone network ResNet (refer to reference [1]) for extracting image features and train the image Perform data enhancement, including scaling and cropping. First, scale the image using random scaling: randomly select a floating point number from 0.5 to 2.0, multiply the length and width of the input image by the coefficient, and complete the scaling. Then crop: select a part of the scaled image and crop it, and change it to the specified length and width as the image input to the backbone network ResNet. Finally, pass the data-enhanced image through the backbone network ResNet to obtain the feature map F.

[0051] In the second step, the long short-term memory network LSTM (see reference [2]) and the differentiable ROI pooling module are used to calculate Q context paths for each pixel in the feature map F. Each context path contains T regions. Then, according to the Q context paths, the features of each pixel in the feature map F are propagated along the context path. The features propagated from the last region in the context paths of all pixels constitute the appearance feature map set R. The specific method is:

[0052] (1) We denote the tth region in the qth context path of the nth pixel in the feature map F as initialization is the coordinate position of the nth pixel in the feature map F.

[0053] (2) Since the LSTM network can learn the relationship between elements in a sequence and the ROI pooling module can capture the characteristics of a certain area of ​​the image, the LSTM network and the differentiable ROI pooling module are used here to calculate the area of ​​the context path of all pixels according to the following formula:

[0054]

[0055]

[0056] in stands for Long Short-Term Memory Network, represents the differentiable ROI pooling module, F represents the feature map, represents the feature corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, Denotes the hidden state of the LSTM network corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F. We will Set to 0.

[0057] (3) We denote the t-th appearance feature obtained by propagating the n-th pixel in the feature map F on its q-th context path as initialization

[0058] (4) Use a fully connected neural network to calculate the propagation characteristics of all pixels in the context path area according to the following formula

[0059]

[0060]

[0061] in represents the feature corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, represents the parameters of the fully connected neural network corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, softmax(.) represents the softmax function, ⊙ represents the Hadamard product, and σ(.) represents the sigmoid function. All Combined together, they form the appearance feature map set R.

[0062] In the third step, for each pixel in the appearance feature map set R, back propagation is performed according to the context path in the previous second step to obtain the enhanced feature map set A. The method is:

[0063] (1) For the qth context path of the nth pixel in the appearance feature map R, the end pixel of its back propagation is denoted as m. Then the enhanced feature of the tth region during its back propagation can be expressed as We initialize

[0064] (2) Using a differentiable ROI pooling module and a fully connected neural network, the enhanced features obtained by back-propagating all pixels in the context path region are calculated according to the following formula:

[0065]

[0066]

[0067]

[0068] in represents the parameters of the fully connected neural network corresponding to the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, represents the appearance feature of the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, softmax(.) represents the softmax function, represents a differentiable ROI pooling module, Represents the location information of the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, represents the final appearance feature of the m-th pixel propagated on the q-th context path, ⊙ represents the Hadamard product, and σ(.) represents the sigmoid function.

[0069] (3) Yes Summing to get The formula is as follows:

[0070]

[0071] What you get It is the enhanced feature of pixel m calculated based on the qth appearance feature map in the appearance feature map set R. So there are Q enhanced feature maps in total, which are collectively referred to as the enhanced feature map set A.

[0072] The fourth step is to fuse the enhanced feature map set A to obtain the semantic feature map Y. The method is:

[0073] (1) According to the relationship between different enhanced feature maps in the enhanced feature map set, a fully connected neural network is used to calculate the fusion coefficient of each pixel in each enhanced feature map according to the following formula:

[0074]

[0075] Where W f,p represents the parameters of the fully connected neural network used for the feature map and the p-th enhanced feature map, W p,q represents the parameters of the fully connected neural network used by the p-th enhanced feature map and the q-th enhanced feature map, Represents the feature corresponding to the nth pixel of the feature map F, represents the enhanced feature of the nth pixel in the pth enhanced feature map, Represents the enhanced feature of the nth pixel in the qth enhanced feature map.

[0076] (2) Then, the enhanced feature map set A is fused according to the fusion coefficient to obtain the semantic feature Y of each pixel after fusion. n :

[0077]

[0078] in represents the enhanced feature of the nth pixel in the qth enhanced feature map, represents the weight coefficient of the nth pixel in the qth enhanced feature map. All Y n It constitutes the semantic feature map Y.

[0079] The fifth step is to pass the semantic feature map Y through two convolutional layers to obtain the predicted probability distribution O∈R H×W×C , where H and W represent the length and width respectively, C represents the number of channels, which is equal to the number of semantic categories of the training image. g ∈R H×W×C Calculate the cross entropy and get the loss value loss. The specific formula is as follows:

[0080]

[0081]

[0082] Among them, n represents the nth pixel in O, I g The nth pixel in I g The cth channel of the nth pixel in O n,c represents the cth channel of the nth pixel in O, O n,j Represents the jth channel of the nth pixel in O, and then the gradient descent method is used to optimize the entire neural network model according to the loss.

[0083] Step 6: Repeat steps 1 to 5 200 times to complete the training of the entire neural network model.

[0084] In the seventh step, the test image is input into the trained model to obtain the predicted probability distribution, and the argmax operation is performed on the predicted probability distribution to obtain the final semantic segmentation result.

[0085] The feasibility of the method of the present invention is verified by combining specific examples, as described below:

[0086] We conducted comparative experiments on three public datasets, namely Cityscapes, PASCAL VOC 2012, and COCO-Stuff. The Cityscapes dataset has 19 semantic categories, including 2,975 training images, 500 validation images, and 1,525 test images; the PASCAL VOC 2012 dataset has 21 semantic categories, including 10,582 training images, 1,449 validation images, and 1,456 test images; the COCO-Stuff dataset has 172 semantic categories and is divided into two versions: one version contains 9,000 training images and 1,000 validation images; the other version contains 120,000 training images and 5,000 validation images.

[0087] The images in the Cityscapes dataset are mainly outdoor scenes, while the images in the PASCAL VOC 2012 and COCO-Stuff datasets include both outdoor and indoor scenes. Result Analysis We selected the mean intersection over union (mIoU) as the evaluation metric, which can well reflect the quality of the semantic segmentation results.

[0088] according to Figure 2 The experimental results of the method and the existing optimal image semantic segmentation method on different data sets show that the semantic segmentation accuracy of the method of the present invention is higher than that of the existing optimal method, whether in indoor or outdoor scenes, large or small data sets. Therefore, the feasibility and superiority of the method of the present invention can be demonstrated.

Claims

1. A method for image semantic segmentation based on a configurable context path, comprising the following steps: A method for image semantic segmentation based on a configurable context path comprises the following steps: The first step is to input the image Perform feature extraction to obtain feature map F; In the second step, the long short-term memory network LSTM is used to calculate Q context paths for each pixel in the feature map F. Each context path contains T regions. According to the Q context paths, the features of each pixel in the feature map F are propagated along the context path, and Q appearance feature maps are obtained. All appearance feature maps constitute the appearance feature map set R. The method is: (1) The tth region in the qth context path of the nth pixel in the feature map F is recorded as initialization is the coordinate position of the nth pixel in the feature map F; (2) Using the long short-term memory network LSTM and the differentiable ROI pooling module, the context path area of ​​all pixels is calculated according to the following formula: in stands for Long Short-Term Memory Network, represents the differentiable ROI pooling module, F represents the feature map, represents the feature corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, Represents the hidden state of the long short-term memory network LSTM corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F. Set to 0; (3) The appearance feature of the tth region obtained by propagating the nth pixel in the feature map F on its qth context path is recorded as initialization (4) Use a fully connected neural network to calculate the propagation characteristics of all pixels in the context path area according to the following formula in represents the feature corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, represents the parameters of the fully connected neural network corresponding to the t-th region in the q-th context path of the n-th pixel in the feature map F, softmax(.) represents the softmax function, ⊙ represents the Hadamard product, σ(.) represents the sigmoid function; all Combined together, they form the appearance feature map set R; In the third step, for each pixel in the appearance feature map set R, back propagation is performed according to the Q context paths in the previous second step, and Q enhanced feature maps are obtained. All enhanced feature maps constitute the enhanced feature map set A. The method is: (1) For the qth context path of the nth pixel in the appearance feature map R, the end pixel of its back propagation is denoted as m. Then the enhanced feature of the tth region during its back propagation is expressed as initialization (2) Using a differentiable ROI pooling module and a fully connected neural network, the enhanced features obtained by back-propagating all pixels in the context path region are calculated according to the following formula: in represents the parameters of the fully connected neural network corresponding to the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, represents the appearance feature of the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, Represents the location information of the T-t+1th region in the qth context path of the mth pixel in the appearance feature map R, Represents the final appearance feature of the m-th pixel on the q-th context path; (3) Yes Summing to get The formula is as follows: What you get It is the enhanced feature of pixel m calculated according to the qth appearance feature map in the appearance feature map set R; there are Q enhanced feature maps in total, which together constitute the enhanced feature map set A; The fourth step is to fuse the enhanced feature map set A to obtain the semantic feature map Y; The fifth step is to obtain the predicted probability distribution according to the semantic feature map Y, calculate the cross entropy between the predicted probability distribution and the true segmentation value of the input image, obtain the loss value, and use the gradient descent method to optimize the entire neural network model; Step 6: Neural network model training.

2. The image semantic segmentation method according to claim 1, characterized in that: The fourth step is: (1) Use a fully connected neural network to calculate the fusion coefficient of each pixel in each enhanced feature map according to the following formula: in, is the fusion coefficient of pixel n in the qth enhanced feature map, W f,p represents the parameters of the fully connected neural network used for the feature map and the p-th enhanced feature map, W p,q represents the parameters of the fully connected neural network used by the p-th enhanced feature map and the q-th enhanced feature map, Represents the feature corresponding to the nth pixel of the feature map F, represents the enhanced feature of the nth pixel in the pth enhanced feature map, Represents the enhanced feature of the nth pixel in the qth enhanced feature map; (2) According to the fusion coefficient, the enhanced feature map set A is fused to obtain the semantic feature Y of each pixel after fusion. n : in represents the enhanced feature of the nth pixel in the qth enhanced feature map, represents the weighting coefficient of the nth pixel in the qth enhanced feature map, all Y n It constitutes the semantic feature map Y.

Images