A semantic segmentation method based on self-attention mechanism and cavity convolution pooling, a storage medium and a visual device

Abstract

The application relates to the technical field of computer vision, in particular to a semantic segmentation method based on a self-attention mechanism and a hollow convolution pooling, a storage medium and a visual device, the method comprises the following steps: S1, using an encoder to extract features of an input image, a plurality of layers of features are extracted from shallow to deep, the feature sizes of the plurality of layers of features are sequentially halved layer by layer, and the channel numbers of the plurality of layers of features are sequentially increased layer by layer; S2, using a self-attention structure to respectively calculate a self-correlation matrix of each layer of features X i , performing matrix multiplication on the self-correlation matrix and original input features X i , obtaining global self-attention information, and outputting features Y; S3, using a lightweight hollow convolution pooling structure miniASPP to perform receptive field enhancement on each layer of input features Y i , obtaining context information, and outputting features Z i ; S4, using a decoder to splice the features processed in S3 through cascaded upsampling to obtain a semantic segmentation result with the size of the original input image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a semantic segmentation method based on self-attention mechanism and dilated convolution pooling, a storage medium and a visual device. BACKGROUND

[0002] Image segmentation task is an important direction of scene understanding in computer vision, which provides pre-useful information acquisition and visual scene basic attribute understanding for artificial intelligence decision. Since the fully convolutional network (FCN) was proposed in 2014, the full connection layer is removed, the input and output are realized point-to-point correspondence through convolution and deconvolution, or coding and decoding, which has become the basis of the current semantic segmentation framework research. However, after removing the full connection layer, the features need to be restored to the original size after depth compression to obtain the pixel-level classification label map output, which inevitably produces another problem: one is the loss of shallow information in the down-sampling process with the increase of network layers, and the other is the loss of context-related information due to the limitation of convolution kernel receptive field. Therefore, it is urgent to improve and optimize the FCN technology. SUMMARY

[0003] One of the purposes of the present application is to avoid the shortcomings in the prior art and provide a semantic segmentation method based on self-attention mechanism and dilated convolution pooling, which can obtain context information rich, abstract features of different scales, and can effectively complete the lost shallow information.

[0004] To achieve one of the above purposes, the present application provides the following technical solutions:

[0005] A semantic segmentation method based on self-attention mechanism and dilated convolution pooling is provided, comprising the following steps

[0006] S1, using an encoder to extract features of an input image, respectively extracting X1, X2, X3……X from shallow to deep k Multi-layer features, the feature size of the multi-layer features is halved layer by layer in the order of X1, X2, X3……X k The number of channels of the multi-layer features is increased layer by layer in the order of X1, X2, X3……X k ;

[0007] S2, using a self-attention structure to obtain the autocorrelation matrix of each layer of features X i processed by S1 to transpose the features X i , and then X i is again combined with the features X iMatrix multiplication is performed to obtain global self-attention information, and output features Y i , the number of channels of Y i is consistent with the number of channels of X i .

[0008] S3, a lightweight hole convolution pooling structure miniASPP is used to enhance the receptive field of each layer of input features Y i , obtain context information, and output features Z i .

[0009] S4, a decoder is used to concatenate the features processed by S3 in the order of Z k , Z k-1 , …, Z3, Z2, Z1 by cascading upsampling, to obtain a semantic segmentation result of the original input image size.

[0010] In some embodiments, in S1, four layers of features X1, X2, X3, and X4 are extracted; the number of channels of X1, X2, X3, and X4 is 256, 512, 1024, and 2048, respectively.

[0011] In some embodiments, S2 is implemented by the following stages:

[0012] S21, feature integration stage: for the input feature X i , two 1X1 convolutions are performed to obtain two features Q and K, the dimensions except the channel number are merged, the complex relationship between the two features is captured by bilinear pooling, and the autocorrelation matrix formula (1) is obtained, as shown in formula (1) below:

[0013]

[0014] wherein Q=[q1,…,q wh ]∈R m×wh , K=[k1,…,k wh ]∈R n×wh , Q and K are homologous features obtained by two different 1X1 convolutions on the same feature, represented as K=θ(X;W θ ).

[0015] The output variable of the bilinear pooling operation is defined as G=[g1,…,g n ]∈R m×n , and the second feature is a wh-dimensional row vector, and the final output formula (2) is as shown in formula (2) below:

[0016]

[0017] Each of the base elements g in formula (2) represents a characteristic set i is obtained by collecting local features ; j is the position of the pixel point on the feature map, i is different attention maps, and finally, softmax is applied to k to map the attention weight to 0-1, and a second-order attention pooling process formula (3) is obtained, as shown in formula (3) as follows:

[0018] g i = Qsoftmax(k i ) T (3)

[0019] S22, feature distribution stage: a 1X1 convolution is used to obtain a feature V, which is used as the input of feature distribution, and the integrated features in the previous stage are adaptively distributed to each position v of the input i , so as to perceive global information and obtain each feature after self-attention formula (4), as shown in formula (4) as follows:

[0020]

[0021] Wherein V = [v1, …, v wh ] ∈ R n×wh , the feature V is the input feature X after 1x1 convolution, and then mapped to 0-1 through softmax, represented as V = softmax (p (X; W ρ )).

[0022] S23, combination stage: the feature integration stage and the feature distribution stage are combined, and the X i processed in S1 is combined with the feature X i processed by global attention to obtain the final attention module operation formula (5), as shown in formula (5) as follows:

[0023]

[0024] In some embodiments, in S3, the miniASPP module is composed of three parallel sub-branches, two of which apply 3x3 convolution, and the two branches use 128 output channels and expansion rates of 1 and 6 respectively, and the remaining one sub-branch uses 64x64 average pooling operation with a step of 1, then the boundary is copied and filled to restore the spatial resolution of the input, and then a 1x1 convolution is performed with 128 output channels; finally, the outputs of the three sub-branches are spliced in the channel number, and after a 3X3 convolution, the features of the three branches are fused and reduced to 256 channels.

[0025] In some embodiments, the encoder takes a ResNet50 network.

[0026] Also provided is a storage medium for storing a program, which, when executed by a processor, implements the semantic segmentation method based on the self-attention mechanism and the dilated convolution pooling.

[0027] Also provided is a visual device comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the processor implements the semantic segmentation method based on the self-attention mechanism and the dilated convolution pooling when executing the program.

[0028] The semantic segmentation method based on the self-attention mechanism and the dilated convolution pooling has the following advantages:

[0029] (1) The semantic segmentation method based on the self-attention mechanism and the dilated convolution pooling of the present application extracts deep multi-layer features from shallow to deep for an input image, respectively calculates the self-correlation matrix of each layer of features, multiplies the matrix with the original input features to obtain global self-attention information, and obtains a double attention module. The double attention module and a lightweight dilated convolution pyramid structure are connected in parallel to learn and expand the pixel interrelation properties of the image itself, and obtain more accurate pixel-level classification. In addition, during the upsampling of the deep semantic features back to the original image size, cascaded upsampling is adopted to complete some details lost in the downsampling process, and finally an effective semantic segmentation framework based on the full convolution network is obtained.

[0030] (2) The semantic segmentation method based on the self-attention mechanism and the dilated convolution pooling of the present application learns the pixel interrelation properties of the image itself through the self-attention module, and simultaneously integrates the soft attention idea of collecting and then distributing to optimize the parameter quantity of the self-attention structure and improve the segmentation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 FIG. 1 is a semantic segmentation algorithm framework diagram of an embodiment.

[0032] Figure 2 FIG. 2 is a semantic segmentation algorithm flowchart of an embodiment.

[0033] Figure 3 FIG. 3 is a self-attention structure diagram of an embodiment.

[0034] Figure 4 FIG. 4 is a lightweight dilated convolution pooling structure diagram of an embodiment. DETAILED DESCRIPTION

[0035] Preferred embodiments of the present application will be described herein below with reference to the accompanying drawings. While the preferred embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0036] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0037] It is to be understood that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the term "or" as used herein encompasses "and / or", unless the context clearly dictates otherwise. It is to be further understood that the terms "comprise", "comprising", "comprises" and / or "comprising" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0038] Embodiments

[0039] Since the fully convolutional neural network (FCN) was proposed in 2014, the idea of removing the fully connected layer, realizing point-to-point correspondence between input and output through convolution and deconvolution, or encoding and decoding, has become the basis of the current semantic segmentation framework research. However, after removing the fully connected layer, the features need to be restored to the original image size to obtain the pixel-level classification label map output, which inevitably leads to another problem: first, in the downsampling process, the loss of shallow information caused by the deepening of the network layers, and second, the limitation of the convolution kernel receptive field leads to the loss of context-related information. Therefore, it is urgent to improve and optimize the FCN technology.

[0040] To solve the above problems, the present embodiment discloses a semantic segmentation method based on self-attention mechanism and dilated convolution pooling, and the flowchart can be referred to Figure 2 ,

[0041] S1, as Figure 1The semantic segmentation algorithm framework shown, the encoder structure adopts the ResNet50 network, and the features of the input image are extracted from shallow to deep, and four layers of features X1, X2, X3 and X4 are extracted. In actual application, other number of features are also extracted according to the needs. The feature sizes w and h of X1, X2, X3 and X4 are halved layer by layer in the order of X1, X2, X3 and X4, and the channel numbers of X1, X2, X3 and X4 are increased layer by layer, that is, the channel numbers of X1, X2, X3 and X4 are 256, 512, 1024 and 2048 respectively. This step extracts features from shallow to deep, which can completely extract features.

[0042] S2, using the self-attention structure (SA) shown in Figure 3 , respectively, to obtain the autocorrelation matrix of each layer of features X i processed by S1 to transpose the features X i , and then multiply the transposed X i with the features X i processed by S1 again to obtain global self-attention information and output features Y i with the same channel number as X i . The detailed steps are divided into the following two stages:

[0043] S21, the feature integration (gather i ing) stage, for the input features X, respectively through two 1X1 convolution, get two features Q, k, merge the dimensions except the channel number, then through bilinear pooling, capture the complex relationship between the two features, obtain the autocorrelation matrix, as shown in equation (1).

[0044]

[0045] Where Q=[q1,…,q wh ]∈R m×wh , K=[k1,…,k wh ]∈R n×wh , both are homologous features obtained by two different 1X1 convolution of the same feature, which can be represented as K=θ(X;W θ ).

[0046] In equation (1), T represents the transpose operation, that is, the rows and columns of the matrix are interchanged; q i , k i represent the i-th column vector in the feature matrix Q and K respectively.

[0047] The formula (1) represents the specific calculation implementation of the bilinear pooling operation on Q and K, which is the multiplication of each column vector of Q and the transpose of each column vector of K, and finally the accumulation; the Q matrix is the description of w*h column vectors.

[0048] The output variable of the bilinear pooling operation is defined as G=[g1,…,g n ]∈R m×n , wherein mxn represents the real number space to which the feature G matrix belongs, that is, the feature size of G is mxn, that is, G has n column vectors g i , and the size of each column vector g i is 1xm, that is, there are m numerical values in g i ; the second feature is a wh-dimensional row vector, and the final output is shown in formula (2).

[0049]

[0050] In the above formula, g i is each base element of the feature integration, which is obtained by collecting local features with weighting; j is the position of the pixel point on the feature map, i and k is the attention map of the feature g i ; then the softmax is applied to k to map the attention weight to 0-1, and finally the second-order attention pooling process is shown in formula (3).

[0051] g i =Qsoftmax(k i ) T (3)

[0052] S22, the feature distribution (Distributing) stage, which obtains the feature V through a 1x1 convolution and uses it as the input of the feature distribution, and then adaptively distributes the integrated features in the previous stage to each position v i of the input to perceive the global information and obtain each feature after self-attention, as shown in formula (4).

[0053]

[0054] wherein V=[v1,…,v wh ]∈R n×wh , the feature V is mapped to 0-1 through the softmax after the input feature X is convolved by 1x1, which is represented as V=softmax(ρ(X;W ρ )), and y i in formula (4) represents each position v iDo feature assignment operations.

[0055] In formula (1)-(4), θ, ρ represent three convolution operations, W represents the parameters corresponding to each operation, and indicates that Q, K, and V come from the same X and are homologous features obtained through 1x1 convolution (the learned parameters of each convolution operation are different).

[0056] S23, combine the above two steps, and perform residual operation on the original input X and the feature subjected to global attention to obtain the final attention module operation step, as shown in formula (5).

[0057]

[0058] In formula (5), F represents the operation in the feature assignment stage, that is, the operation implemented in formula (4), and G represents the operation in the feature integration stage, that is, the operation described in formula (1)-(3), V, represent convolution operations, and V is the input in the feature assignment stage.

[0059] S3, a lightweight hole convolution pooling structure miniASPP as shown in i is used to enhance the receptive field of each layer of input feature Y i , output feature Z i , and the channel number of Z is 256. The miniASPP module is composed of three parallel sub-branches, two of which apply 3x3 convolution, and the two branches use 128 output channels and expansion rates of 1 and 6 respectively, and the remaining one sub-branch uses 64x64 average pooling operation with a step of 1, then the boundary is copied and filled to restore the spatial resolution of the input, and then a 1x1 convolution is performed with 128 output channels; finally, the outputs of the three sub-branches are spliced in the channel number, and after a 3x3 convolution, the features of the three branches are fused and reduced to 256 channels. Finally, the miniASPP module is output.

[0060] S4, a decoder is used to concatenate Z1, Z2, Z3, and Z4 four layers of features processed in S3 by cascading upsampling, that is, Z4, Z3, Z2, and Z1 are concatenated in series to obtain a semantic segmentation result of the original input image size as shown in Figure 1 .

[0061] The semantic segmentation method based on self-attention mechanism and cavity convolution pooling in the embodiment proposes a self-attention structure, learns the pixel interrelation of the graph itself through the self-attention module, simultaneously integrates the soft attention idea of pre-collection and redistribution, optimizes the parameter quantity of the self-attention structure, and improves the segmentation efficiency; and proposes a lightweight cavity convolution pyramid structure, which expands the receptive field without increasing the parameter quantity, and fuses multi-depth and multi-scale features through the cascaded up-sampling mode of the lightweight structure to complete the lost shallow information in the feature compression process.

[0062] That is, the embodiment firstly extracts the deep multi-layer features of the image from shallow to deep, respectively calculates the self-correlation matrix of each layer of features, multiplies the self-correlation matrix with the original input features to obtain global self-attention information, and obtains a double-attention module; the double-attention module and the lightweight cavity convolution pyramid structure are connected in parallel to learn the pixel interrelation of the graph itself and expand the receptive field, so that more accurate pixel-level classification is obtained. In addition, in the process of up-sampling the deep semantic features back to the original image size, cascaded up-sampling is adopted to complete some details lost in the down-sampling process, and finally an effective semantic segmentation framework based on the full convolution network is obtained.

[0063] The relative arrangement, numerical expressions, and numerical values of the components and steps set forth in the embodiments do not limit the scope of the present application unless otherwise specifically stated. Meanwhile, it should be understood that the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportional relationship for the convenience of description. The technology, methods, and devices known to those of ordinary skill in the related art can not be discussed in detail, but should be considered as part of the authorized description under appropriate circumstances. In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments can have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0064] In the description of the present application, it should be understood that the orientation words such as "front, back, up, down, left, right", "transverse, vertical, perpendicular, horizontal", and "top, bottom" and the like indicate the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate and imply that the indicated device or element must have a particular orientation or be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the scope of protection of the present application; the orientation words "inner, outer" refer to the inner and outer relative to the contour of the parts themselves.

[0065] For purposes of the description hereinafter, the terms "upper", "lower", "right", "left", "rear", "front", "vertical" and "horizontal" as can be perceived herein relative to the accompanying drawings refer to the orientation of the components being described. However, it is to be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device described herein relative to the other device or structure is inverted, then a spatially relative term such as "above" can be interpreted as meaning "below" or "below" can be interpreted as meaning "above". The device can also be oriented in other ways (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. The devices can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0066] In addition, it should be pointed out that the use of "first", "second" and the like words to define parts, only for the convenience of the corresponding parts to be distinguished, as no further declaration, the above words have no special meaning, therefore can not be understood as limiting the scope of the present application.

[0067] The preferred embodiments of the present application have been described above with the aid of drawing only and are not limited to those preferred embodiments, and as those skilled in the art will readily appreciate, changes and modification can be made thereto without departing from the spirit and scope of the present application. Any further modifications, changes, improvements, and the like that come within the spirit and scope of the present application should be secured to the application.

Claims

1. A semantic segmentation method based on self-attention mechanism and dilated convolution pooling, characterized in that: Comprising the following steps Said S2 is realized by the following stages: S1, using an encoder to extract features of an input image, and extracting X1, X2, X3, …, X from shallow to deep k Multi-layer features, the feature size of which is halved layer by layer in the order of X1, X2, X3, …, X k The number of channels of the multi-layer features is increased layer by layer in the order of X1, X2, X3, …, X k ​ S2, a self-attention structure is used to obtain each layer of features X processed by S1 i , and a transpose feature X i is obtained by transposing the self-correlation matrix of X i ; the transposed X i is multiplied again with the feature X processed by S1 to obtain global self-attention information and output a feature Y i ; the channel number of Y i is consistent with that of X i . S3, using a lightweight hollow convolutional pooling structure miniASPP to each layer input feature Y i Receptive field enhancement, get context information, output feature Z i ; S4, using a decoder to concatenate the up-sampled S3 processed features in Z k , Z k-1 …Z3, Z2, Z1 splicing in order to get the original input image size semantic segmentation results; To each feature formula (4) after self-attention, formula (4) is as follows: S21, feature integration stage: for the input feature X i , respectively, through two 1X1 convolution, get two features Q, K, merge the channel number outside the dimension, through the bilinear pooling to capture the complex relationship between the two features, to calculate the autocorrelation matrix formula (1), formula (1) is as follows: wherein Q = [q1, …, q wh ] ∈ R m×wh , K = [k1, …, k wh ] ∈ R n×wh , Q, K are homologous features of the same feature obtained by two different 1X1 convolutions, denoted as K = θ(X; W θ ); Let the output variable of the bilinear pooling operation be defined as G = [g1,..., g n ]∈R m×n , rewrite the second feature is a row vector of dimension wh, and the final output is given by equation (2) as follows: Each of the base elements g representing characteristic integration in formula (2) i is obtained by collecting local features Weighted; j is the position of the pixel point on the feature map, i is different attention maps, and finally, softmax is applied to k to map the attention weight to 0-1, and the second-order attention pooling process formula (3) is obtained, as shown in formula (3): g i = Qsoftmax(k i ) T (3) S22, feature assignment stage: get feature V by a 1X1 convolution, use it as the input of feature assignment, adaptively distribute the integrated feature of the last stage to each position v of the input i , and get Formula (5) is as follows: wherein V = [v1, …, v wh ] ∈ R n×wh , the feature V is the input feature X after 1x1 convolution, and then mapped to 0-1 by softmax, represented as V = softmax(p(X; W ρ )); S23, combination stage: combine the feature integration stage and the feature assignment stage, the X i with the global attention processing after the feature X i Do residual operation, get the final attention module operation step formula (5), In S1, X1, X2, X3 and X4 four-layer features are extracted; the channel numbers of X1, X2, X3 and X4 are 256, 512, 1024 and 2048 respectively.

2. The semantic segmentation method based on self-attention mechanism and atrous convolution pooling according to claim 1, characterized in that:

3. The semantic segmentation method based on self-attention mechanism and dilated convolution pooling according to claim 1, characterized in that: in S3, the miniASPP module is composed of three parallel sub-branches, two of which apply 3x3 convolution, and the two branches use 128 output channels and dilations of 1 and 6 respectively, and the remaining one sub-branch uses 64x64 average pooling operation with a step of 1, then the boundary is copied and filled to restore the input spatial resolution, and then a 1x1 convolution is performed with 128 output channels; finally, the outputs of the three sub-branches are concatenated in the channel number, and after a 3x3 convolution, the features of the three branches are fused and reduced to 256 channels. The encoder adopts a ResNet50 network. The storage medium is used to store a program, and the program is executed by the processor to realize the semantic segmentation method based on the self-attention mechanism and the dilated convolution pooling according to any one of claims 1-4.

4. The semantic segmentation method based on self-attention mechanism and atrous convolution pooling according to claim 1, characterized in that: The processor, the memory and the computer program stored on the memory and executable on the processor are included, and the processor executes the program to realize the semantic segmentation method based on the self-attention mechanism and the dilated convolution pooling according to any one of claims 1-4.

5. A storage medium characterized by: ​ 6. A vision device characterized by: ​

Images