An image semantic segmentation method based on attention mechanism and knowledge distillation

Abstract

The present application relates to computer vision technology in the field of motor vehicles, aircraft, and specifically relates to an image semantic segmentation method based on attention mechanism and knowledge distillation.The present application solves the problem that the existing image semantic segmentation method is difficult to balance the reasoning speed and segmentation accuracy.A kind of image semantic segmentation method based on attention mechanism and knowledge distillation, the method is realized by the following steps: step one: construct a multi-branch feature extraction coding module based on hollow convolution;Step two: construct a multi-scale hollow space pyramid pooling module;Step three: construct a spatial attention mechanism module embedded with channel attention;Step four: construct an adaptive multi-scale feature fusion module;Step five: respectively construct a teacher network and a student network;Step six: construct an edge information knowledge distillation loss function;Step seven: guide the student network to carry out edge knowledge distillation training;Step eight: evaluate the student network.The present application is suitable for real-time image semantic segmentation.

Description

TECHNICAL FIELD

[0001] The application relates to computer vision technology in the field of motor vehicles and aircraft, and particularly relates to an image semantic segmentation method based on an attention mechanism and knowledge distillation. BACKGROUND

[0002] As a pixel-level classification task, image semantic segmentation can provide fine-grained and high-level semantic information for scene understanding, and is widely used in visual perception and understanding tasks in the fields of motor vehicles, aircraft and the like. In order to realize effective perception of application scenarios, the trade-off between inference speed and segmentation accuracy has become an important challenge in current image semantic segmentation tasks.

[0003] With the development of artificial intelligence technology, image semantic segmentation technology guided by deep learning has gradually become the mainstream method of current research. At present, the image semantic segmentation method based on deep learning can be divided into high-precision image semantic segmentation and lightweight image semantic segmentation. The high-precision image semantic segmentation is constructed based on a deep convolutional neural network, and can realize effective improvement of accuracy, but the parameter quantity of the model is too large, and it is difficult to realize the deployment of lightweight devices, resulting in slow inference speed; the lightweight image semantic segmentation is constructed based on lightweight convolution or model compression technology, and can realize the reduction of model parameter quantity, but the model is easy to cause a large loss of accuracy, resulting in poor segmentation accuracy. Based on this, it is necessary to invent an image semantic segmentation method based on an attention mechanism and knowledge distillation to solve the problem that the existing image semantic segmentation method is difficult to balance the inference speed and segmentation accuracy. SUMMARY

[0004] The application provides an image semantic segmentation method based on an attention mechanism and knowledge distillation to solve the problem that the existing image semantic segmentation method is difficult to balance the inference speed and segmentation accuracy.

[0005] The application is implemented by adopting the following technical solutions:

[0006] The image semantic segmentation method based on the attention mechanism and the knowledge distillation is implemented by adopting the following steps:

[0007] Step 1: constructing a multi-branch feature extraction encoding module based on a dilated convolution; inputting an image into the module to generate five-stage features feature1-feature5;

[0008] Step 2: constructing a multi-scale dilated spatial pyramid pooling module; the module takes the feature feature5 as input to generate a feature feature6;

[0009] Step three: constructing a spatial attention mechanism module embedded with channel attention; the module takes features feature3, feature4, and feature6 as input to generate features feature7, feature8, and feature9;

[0010] Step four: constructing an adaptive multi-scale feature fusion module; the module takes features feature1, feature2, feature7, feature8, and feature9 as input to generate feature10;

[0011] Step five: constructing a teacher network and a student network based on the modules constructed in steps one to four;

[0012] Step six: constructing an edge information knowledge distillation loss function; taking the features feature10 output by the teacher network and the student network as input, determining the next candidate boundary point of the predicted boundary point of the student network, and transferring the boundary information by increasing the distance between the predicted boundary point and the candidate boundary point;

[0013] Step seven: first, training the teacher network using the training set in the image semantic segmentation dataset, and then based on the pre-training weight of the teacher network, using the edge information knowledge distillation loss function to guide the student network to perform edge knowledge distillation training;

[0014] Step eight: using the test set in the image semantic segmentation dataset to evaluate the student network.

[0015] In step one, the construction steps of the multi-branch feature extraction encoding module based on the dilated convolution are as follows:

[0016] First, remove the full connection layer of RepVGG, and construct the backbone network based on the first five stages of RepVGG, denoted as stage1-stage5;

[0017] Secondly, set the convolution parameter stride of the first layer branch structure in stage4 and stage5 to 1 respectively;

[0018] Finally, introduce non-proportional serial dilated convolution in stage3-stage5, and set the dilation rate of stage3-stage5 to (1, 2, 5).

[0019] In step two, the construction steps of the multi-scale dilated spatial pyramid pooling module are as follows:

[0020] First, reduce the channel number of the input feature by constructing a 1x1 convolution branch to reduce the model calculation complexity;

[0021] Secondly, a parallel 3x3 hole convolution branch is added in the ASPP module, and the hole rates of the four parallel 3x3 hole convolution branches are set to (8, 12, 24, 36).

[0022] In the third step, the construction steps of the spatial attention mechanism module embedded with channel attention are as follows:

[0023] Firstly, the spatial attention mechanism based on self-attention mechanism is decomposed into row attention mechanism and column attention mechanism;

[0024] Secondly, the channel attention mechanism is calculated based on the row attention mechanism and the column attention mechanism respectively, and the row attention and the column attention are encoded into a vector through the construction of a 1x1 convolution, a batch normalization layer and an activation function LeakyReLu, and then the vector is input into a multi-layer perception containing a hidden layer, and then a Sigmoid function is designed to generate a row and column matrix with channel and spatial attention weights;

[0025] Finally, the row and column matrix with channel and spatial attention weights are fused.

[0026] In the fourth step, the construction steps of the adaptive multi-scale feature fusion module are as follows:

[0027] Firstly, the feature 2 is classified pixel by pixel, and the index corresponding to the class with the highest channel dimension probability is taken, and then whether a pixel point and its neighborhood pixel classification results are equal is judged to determine whether the pixel point is a target edge, thereby obtaining a target edge matrix;

[0028] Secondly, the corresponding target object matrix and target edge matrix of each stage are constructed by means of bilinear interpolation up-sampling and maximum pooling down-sampling, and the target object matrix and the target edge matrix are weighted and summed by setting learnable parameters, thereby obtaining a feature weight matrix of each stage;

[0029] Then, based on the feature weight matrix of each stage, the features of each stage are weighted;

[0030] Then, the same fusion mode is constructed for the features 7, 8 and 9, that is, a 1x1 convolution is designed to reduce the channel number of the output feature of each stage to the same channel number of the output feature of the last stage, and the high-level semantic features of two stages are fused by channel concatenation;

[0031] Then, the fused high-level semantic features are sequentially upsampled by bilinear interpolation to have the same resolution as the features feature1 and feature2, and the remaining fusion strategies are the same as those for features feature7-feature9.

[0032] Finally, based on the fusion results of the five features, the features are refined by designing a 3x3 convolution, the channel number is reduced to the number of segmentation classes by a 1x1 convolution, and then the feature map is restored to the same spatial dimension as the model input by bilinear interpolation upsampling, thereby obtaining the pixel-by-pixel classification result feature10.

[0033] In the fifth step, the teacher network has a deep network structure and a large number of channels in each layer, and the student network has a shallow network structure and a small number of channels in each layer.

[0034] In the sixth step, the edge information knowledge distillation loss function is constructed as follows:

[0035] First, the predicted boundary points of the teacher network and the predicted boundary points of the student network are detected.

[0036] Second, for the predicted boundary points of the student network, the pixel point j in the 3x3 region centered on the pixel point i is taken as the next candidate boundary point, wherein the distance from the pixel point j to the predicted boundary point of the teacher network is the smallest compared with the distances from the remaining pixel points in the 3x3 region to the predicted boundary point of the teacher network.

[0037] Then, the KL divergence between the predicted boundary point of the student network and the remaining pixel points in the 3x3 region centered on the pixel point i is calculated.

[0038] Finally, the candidate boundary point is taken as the true boundary point, and the cross-entropy loss function is used for backpropagation to increase the KL divergence between the class probabilities of the pixel point i and the pixel point j, and to reduce the KL divergence between the pixel point i and the remaining adjacent pixel points.

[0039] Compared with the existing image semantic segmentation method, the image semantic segmentation method based on the attention mechanism and knowledge distillation has the following advantages: first, the multi-branch feature extraction coding module based on the cavity convolution proposed in the application can improve the loss of image detail information as much as possible by avoiding excessive down-sampling of features. In addition, by constructing a non-proportional serial cavity convolution, the receptive field can be increased to extract rich context information, and the grid effect can be relieved. Second, the multi-scale cavity spatial pyramid pooling module proposed in the application can obtain feature information of different receptive field sizes by designing multiple scales of cavity rates, which helps to improve the learning ability of the network for different scale size targets. Third, the spatial attention mechanism module embedded with channel attention proposed in the application can effectively improve the problem of large model parameter quantity caused by the self-attention mechanism. In addition, embedding channel attention into row attention and column attention can effectively relieve the conflict between different dimensional attention. Fourth, the adaptive multi-scale feature fusion module proposed in the application can fully utilize the learning advantages of different stages for different information by adaptively weighting the target main body and target edge of the feature of each stage. In addition, the multi-scale feature fusion method can effectively relieve the information loss in the down-sampling process of the coding module. Fifth, the edge information knowledge distillation loss function proposed in the application can transfer the rich edge information learned by the teacher network to the student network, which can improve the segmentation accuracy of the edge information of the student network and improve the overall segmentation precision of the student network.

[0040] The application effectively solves the problem that the existing image semantic segmentation method is difficult to balance the inference speed and segmentation precision, and is suitable for real-time image semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is the overall schematic diagram of the application.

[0042] Figure 2 is the schematic diagram of the multi-scale cavity spatial pyramid pooling module in the application.

[0043] Figure 3 is the schematic diagram of the spatial attention mechanism module embedded with channel attention in the application.

[0044] Figure 4 is the schematic diagram of the adaptive multi-scale feature fusion module in the application.

[0045] Figure 5 is the schematic diagram of the edge information knowledge distillation loss function in the application. DETAILED DESCRIPTION

[0046] The application discloses an image semantic segmentation method based on an attention mechanism and knowledge distillation.

[0047] Step one: a multi-branch feature extraction coding module based on a cavity convolution is constructed; an image is input into the module to generate five-stage features feature1-feature5;

[0048] Step two: a multi-scale cavity space pyramid pooling module is constructed; the module takes the feature feature5 as input to generate a feature feature6;

[0049] Step three: a spatial attention mechanism module embedded with channel attention is constructed; the module takes the features feature3, feature4 and feature6 as input to generate features feature7, feature8 and feature9;

[0050] Step four: an adaptive multi-scale feature fusion module is constructed; the module takes the features feature1, feature2, feature7, feature8 and feature9 as input to generate a feature feature10;

[0051] Step five: a teacher network and a student network are respectively constructed based on the modules constructed in steps one to four;

[0052] Step six: an edge information knowledge distillation loss function is constructed; the features feature10 output by the teacher network and the student network are taken as input to determine a next candidate boundary point of a predicted boundary point of the student network, and knowledge migration of boundary information is performed by increasing the distance between the predicted boundary point and the candidate boundary point;

[0053] Step seven: the teacher network is trained by using a training set in an image semantic segmentation dataset, and the student network is guided to perform edge knowledge distillation training by using the edge information knowledge distillation loss function based on the pre-training weight of the teacher network;

[0054] Step eight: the student network is evaluated by using a test set in the image semantic segmentation dataset.

[0055] In the step one, the construction steps of the multi-branch feature extraction coding module based on the cavity convolution are as follows:

[0056] Firstly, the full connection layer of the RepVGG is removed, and a backbone network is constructed based on the first five stages of the RepVGG, denoted as stage1-stage5;

[0057] Secondly, the convolutional parameter stride of the first layer branch structure in stage4 and stage5 is set to 1 respectively;

[0058] Finally, the non-proportional serial hole convolution is introduced in stage3-stage5, and the hole rate of stage3-stage5 is set to (1, 2, 5).

[0059] In the step two, the construction steps of the multi-scale hole space pyramid pooling module are as follows:

[0060] Firstly, the channel number of the input feature is reduced by constructing a 1x1 convolution branch, so as to reduce the model calculation complexity;

[0061] Secondly, a parallel 3x3 hole convolution branch is added in the ASPP module, and the hole rate of the four parallel 3x3 hole convolution branches is set to (8, 12, 24, 36).

[0062] In the step three, the construction steps of the spatial attention mechanism module embedded with channel attention are as follows:

[0063] Firstly, the spatial attention mechanism based on self-attention mechanism is decomposed into row attention mechanism and column attention mechanism;

[0064] Secondly, the channel attention mechanism is calculated based on the row attention mechanism and the column attention mechanism respectively, and the row attention and the column attention are encoded into a vector by constructing a 1x1 convolution, a batch normalization layer and an activation function LeakyReLu, and then the vector is input into a multi-layer perception containing a hidden layer, and then a Sigmoid function is designed to generate a row and column matrix with channel and spatial attention weights;

[0065] Finally, the row and column matrix with channel and spatial attention weights are fused.

[0066] In the step four, the construction steps of the adaptive multi-scale feature fusion module are as follows:

[0067] Firstly, the feature2 is classified pixel by pixel, and the index corresponding to the class with the highest channel dimension probability is taken, and then whether a pixel point and its neighborhood pixel classification result are equal is judged to determine whether the pixel point is the target edge, so as to obtain the target edge matrix;

[0068] Secondly, the target main body matrix and the target edge matrix corresponding to each stage are constructed by means of bilinear interpolation upsampling and maximum pooling downsampling, and the target main body matrix and the target edge matrix are weighted and summed by setting learnable parameters, so as to obtain the feature weight matrix of each stage;

[0069] Then, the features of each stage are weighted based on the feature weight matrix of each stage;

[0070] Then, the same fusion mode is constructed for the features feature7, feature8 and feature9, that is, the channel number of the output features is reduced stage by stage by designing a 1x1 convolution to make it the same as the channel number of the output features of the previous stage, and the high-level semantic features of two stages are fused by channel splicing;

[0071] Then, the fused high-level semantic features are sequentially upsampled by bilinear interpolation to have the same resolution as the features feature1 and feature2, and the remaining fusion strategies are the same as the fusion strategies of the features feature7-feature9.

[0072] Finally, based on the fusion results of the five features, the features are refined by designing a 3x3 convolution, and the channel number is reduced to the number of segmentation classes by a 1x1 convolution, and then the feature map is restored to the same spatial dimension as the model input by bilinear interpolation upsampling, thereby obtaining the pixel-by-pixel classification result feature10.

[0073] In the fifth step, the teacher network has a deep network structure, and the channel number of each layer is large; the student network has a shallow network structure, and the channel number of each layer is small.

[0074] In the sixth step, the construction steps of the edge information knowledge distillation loss function are as follows:

[0075] First, the predicted boundary points of the teacher network and the predicted boundary points of the student network are detected;

[0076] Second, for the predicted boundary points of the student network, the pixel point j in the 3x3 region centered on the pixel point i is taken as the next candidate boundary point; wherein, compared with the remaining pixel points in the 3x3 region, the distance from the pixel point j to the predicted boundary point of the teacher network is the smallest;

[0077] Then, the KL divergence between the predicted boundary point of the student network and the remaining pixel points in the 3x3 region centered on the pixel point i is calculated;

[0078] Finally, the candidate boundary point is taken as the true boundary point, and the cross-entropy loss function is used for back propagation to increase the KL divergence between the class probabilities of the pixel point i and the pixel point j, and to reduce the KL divergence between the pixel point i and the remaining adjacent pixel points.

[0079] Although the specific embodiments of the present application have been described above, it is understood by those skilled in the art that these are merely illustrative, and the scope of protection of the present application is defined by the appended claims. Those skilled in the art can make various changes or modifications to the embodiments without departing from the principles and the essence of the present application, and such changes and modifications fall within the scope of protection of the present application.

Claims

1. An image semantic segmentation method based on attention mechanism and knowledge distillation, characterized in that: This method is implemented using the following steps: Step 1: Construct a multi-branch feature extraction and encoding module based on dilated convolution; input the image into this module to generate features in 5 stages: feature1-feature5; Step 2: Construct a multi-scale void spatial pyramid pooling module; this module takes feature 5 as input and generates feature 6; Step 3: Construct a spatial attention mechanism module with embedded channel attention; this module takes feature 3, feature 4, and feature 6 as input and generates feature 7, feature 8, and feature 9; Step 4: Construct an adaptive multi-scale feature fusion module; this module takes feature1, feature2, feature7, feature8, and feature9 as input and generates feature10; Step 5: Construct the teacher network and student network based on the modules built in Steps 1 to 4 respectively; Step 6: Construct the edge information knowledge distillation loss function; take the feature10 output by the teacher network and the student network as input, determine the next candidate boundary point of the predicted boundary point of the student network, and transfer the knowledge of the boundary information by increasing the distance between the predicted boundary point and the candidate boundary point. Step 7: First, use the training set in the image semantic segmentation dataset to train the teacher network, and then use the edge information knowledge distillation loss function to guide the student network to perform edge knowledge distillation training based on the pre-trained weights of the teacher network. Step 8: Evaluate the student network using the test set in the image semantic segmentation dataset.

2. The image semantic segmentation method based on attention mechanism and knowledge distillation according to claim 1, characterized in that: In step one, the construction steps of the multi-branch feature extraction and encoding module based on dilated convolution are as follows: First, the fully connected layers of RepVGG are removed, and the backbone network is built based on the first 5 stages of RepVGG, denoted as stage1-stage5; Secondly, the convolution parameter stride of the first branch structure in stage4 and stage5 is set to 1 respectively; Finally, non-proportional serial dilated convolutions are introduced in stages 3-5, and the dilation rate of stages 3-5 is set to (1, 2, 5).

3. The image semantic segmentation method based on attention mechanism and knowledge distillation according to claim 1, characterized in that: In step two, the construction steps of the multi-scale void space pyramid pooling module are as follows: First, the number of channels in the input features is reduced by constructing 1×1 convolutional branches, thereby reducing the computational complexity of the model. Secondly, add a parallel 3×3 dilated convolution branch to the ASPP module, and set the dilation rate of the four parallel 3×3 dilated convolution branches to (8, 12, 24, 36).

4. The image semantic segmentation method based on attention mechanism and knowledge distillation according to claim 1, characterized in that: In step three, the construction steps of the spatial attention mechanism module with embedded channel attention are as follows: First, the spatial attention mechanism Non-Local, based on self-attention, is decomposed into row attention mechanism and column attention mechanism; Secondly, channel attention mechanisms are calculated based on row attention and column attention mechanisms respectively. Row attention and column attention are encoded into vectors by constructing 1×1 convolution, batch normalization layers, and the activation function LeakyReLu. The vectors are then input into a multilayer perceptron containing one hidden layer. A sigmoid function is then designed to generate row and column matrices with channel and spatial attention weights. Finally, the row and column matrices with channel and spatial attention weights are fused.

5. The image semantic segmentation method based on attention mechanism and knowledge distillation according to claim 1, characterized in that: In step four, the construction steps of the adaptive multi-scale feature fusion module are as follows: First, feature2 is classified pixel by pixel, and the index corresponding to the category with the highest probability in the channel dimension is taken. Then, the pixel is determined to be a target edge by judging whether the classification result of a pixel is equal to that of its neighboring pixels, thereby obtaining the target edge matrix. Secondly, the target subject matrix and target edge matrix for each stage are constructed by bilinear interpolation upsampling and max pooling downsampling. The target subject matrix and target edge matrix are weighted and summed by setting learnable parameters to obtain the feature weight matrix for each stage. Then, based on the feature weight matrix of each stage, the features of each stage are weighted; Then, the same fusion method is constructed for features 7, 8, and 9, that is, by designing 1×1 convolution to reduce the number of channels of the output features in stages so that it is the same as the number of channels of the output features in the previous stage, and the high-level semantic features of the two stages are fused by channel concatenation. Then, for the fused high-level semantic features, bilinear interpolation upsampling is used to make them have the same resolution as features1 and features2. The remaining fusion strategies are the same as the fusion strategies for features7-9. Finally, based on the fusion result of the five features, the features are refined by designing a 3×3 convolution, and the number of channels is reduced to the number of segmentation categories by a 1×1 convolution. Then, the feature map is restored to the same spatial dimension as the model input by bilinear interpolation upsampling, thus obtaining the pixel-wise classification result feature10.

6. The image semantic segmentation method based on attention mechanism and knowledge distillation according to claim 1, characterized in that: In step five, the teacher network has a deep network structure with a large number of channels in each layer; the student network has a shallow network structure with a small number of channels in each layer.

7. The image semantic segmentation method based on attention mechanism and knowledge distillation according to claim 1, characterized in that: In step six, the specific steps for constructing the edge information knowledge distillation loss function are as follows: First, detect the predicted boundary points of the teacher network and the predicted boundary points of the student network; Secondly, for the predicted boundary point of the student network, pixel j in the 3×3 region centered on pixel i is selected as the next candidate boundary point; among them, pixel j has the smallest distance to the predicted boundary point of the teacher network compared with the other pixels in the 3×3 region. Then, calculate the KL divergence between the predicted boundary point of the student network and the remaining pixels in the 3×3 region centered on pixel i; Finally, using the candidate boundary points as the ground truth boundary points, backpropagation is performed using the cross-entropy loss function to increase the KL divergence between the class probabilities of pixel i and pixel j, while reducing the KL divergence between pixel i and its other neighboring pixels.

Images