Image semantic segmentation method and system based on dual-channel and self-attention mechanism

Abstract

The present invention discloses an image semantic segmentation method and system based on dual channels and self-attention mechanism, the method comprising the following steps: obtaining a picture to be segmented; extracting feature maps of two channels for the picture to be segmented respectively; wherein the first channel extracts a multi-scale context information feature map; the second channel extracts a pixel-level feature map; the multi-scale context information feature map and the pixel-level feature map are learned through matrix operation and self-attention mechanism to obtain a feature map in which each pixel is related to its corresponding context area; the feature map in which each pixel is related to its corresponding context area is input into a trained classifier, and the image semantic segmentation result is output.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular to an image semantic segmentation method and system based on a dual-channel and self-attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Semantic segmentation technology not only belongs to the research scope of computer science, but also has important application significance for image restoration. It is also a cutting-edge research field involving multiple disciplines such as medical imaging, autonomous driving, and satellite remote sensing. Semantic segmentation technology has very important research significance and application value.

[0004] Currently, semantic segmentation technology generally uses traditional convolutional neural networks, such as VGG, ResNet, etc., but their structure is complex and the amount of calculation is large, which is inconsistent with the real-time performance and application of semantic segmentation.

[0005] Traditional networks continuously extract features from input images. After multiple layers of convolution and downsampling operations, the resolution of the feature map will drop significantly, and the original information will be lost. Therefore, more and more network designs are trying to reduce the loss of resolution. For example, the Deeplab series of articles use the ASPP module composed of dilated convolutions to extract semantic information of multi-scale receptive fields. The encoder module incorporates dilated convolutions, or uses convolution operations with a step size of 2 instead of pooling, which solves the contradiction between the step size and the receptive field and the feature map information. Although the effect is improved, the amount of calculation is large, and the dilated convolution cannot generate dense contextual information. Excessive use of dilated convolutions will also produce a grid effect, so the problem of semantic segmentation is not fundamentally solved. Semantic segmentation is a pixel-level segmentation task, so the relationship between convolution, step size, resolution, receptive field and context should be considered comprehensively. Obviously, if only the ASPP module is used, a grid effect will occur, and the overall context information will be lost, which is not a good solution for intensive segmentation tasks.

[0006] PSPnet adds a pyramid pooling module after the extracted features to solve the problem of missing global information. Its performance is better than Deeplab, but it lacks the relationship between pixels and context. In addition, these two methods have a common disadvantage, which is the use of traditional networks, which leads to excessive number of parameters. The depth of the network will inevitably lead to resolution loss when extracting features. Therefore, it is necessary to introduce a network that maintains high-resolution feature extraction and has low computational complexity to improve the accuracy and speed of semantic segmentation.

[0007] In addition, semantic segmentation is a pixel-level segmentation task. Traditional networks either only consider the relationship between pixels, such as the Deeplab series of networks, or only consider the information between regions, like the pyramid pooling structure module proposed by PSPnet. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an image semantic segmentation method and system based on dual-channel and self-attention mechanism, comprehensively considering the balance between complexity and accuracy, using the HRNetV2 network as the backbone to extract features of the input image, and the self-attention mechanism combined with the pyramid pooling module in PSPNet to learn between pixels and regions on the feature map extracted by the HRNetV2 network, thereby improving the accuracy.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] The image semantic segmentation method based on dual channels and self-attention mechanism includes the following steps:

[0011] Get the image to be segmented;

[0012] The image to be segmented is subjected to two-channel feature map extraction respectively; the first channel extracts a multi-scale context information feature map; the second channel extracts a pixel-level feature map;

[0013] The multi-scale context information feature map and the pixel-level feature map are learned through matrix operations and self-attention mechanism to obtain a feature map that is related to each pixel and its corresponding context area;

[0014] The feature map related to each pixel and its corresponding context area is input into the trained classifier to output the image semantic segmentation result.

[0015] Furthermore, the first channel extracts a multi-scale context information feature map specifically including: inputting the image to be segmented into the HRNetV2_w18 network to obtain a first feature map of the first channel, performing convolution on the first feature map of the first channel to obtain a second feature map of the first channel, and performing multi-scale context information extraction on the second feature map of the first channel through a pyramid pooling model to obtain a third feature map of the first channel.

[0016] Furthermore, after the third feature map of the first channel is obtained, random inactivation dropout is added to randomly inactivate neurons, and a cross entropy loss function is added after convolution to assist network training to obtain the fourth feature map of the first channel, that is, the multi-scale context information feature map.

[0017] Furthermore, the second channel extracting the pixel-level feature map specifically includes: inputting the image to be segmented into the HRNetV2_w18 network to obtain the first feature map of the second channel, and extracting the second feature map of the second channel, that is, the pixel-level feature map, after convolution, accelerated neural network and Relu function.

[0018] Furthermore, the matrix operation of the multi-scale context information feature map and the pixel-level feature map specifically includes: matrix multiplication of pixels at each position of the multi-scale regional context information feature map and the pixel-level feature map, including:

[0019] The multi-scale regional context information feature map is preprocessed and output to obtain a first matrix;

[0020] The pixel-level feature map of the image is normalized to output the second matrix, the first matrix and the second matrix are multiplied to output the third matrix feature map, that is, the fused pixel-level regional context feature map.

[0021] Furthermore, the self-attention mechanism learning includes learning the relationship between the pixel at each position and the context area of ​​the pixel, and generating a corresponding feature map according to the relationship between the pixel at each position and its area;

[0022] Specifically, the fused pixel-level regional context feature map and pixel-level feature map are input into the attention mechanism to obtain three matrices Q matrix, K matrix and V matrix;

[0023] Among them, the Q matrix is ​​the matrix used to query the relationship of the K matrix generated by the pixel-level feature map; the K matrix is ​​the matrix generated by the fused pixel-level regional context feature map and waiting for the Q matrix to be queried; the V matrix is ​​the matrix with actual information and feature attributes generated by the fused pixel-level regional context feature map.

[0024] Furthermore, the feature map of each pixel and its corresponding context area specifically includes:

[0025] The pixel-level feature map queries the relationship between the pixel-level feature map and the fused pixel-level regional context feature map through the Q matrix, the fused pixel-level regional context feature map obtains the query result through the K matrix query, and converts the query result into a probability weight matrix of the correspondence between the pixel-level feature map and the fused pixel-level regional context feature map; the V matrix is ​​reconstructed according to the probability weight matrix, and probability weight parameters are assigned to pixel areas of the same category and pixel areas of different categories.

[0026] One or more embodiments provide an image semantic segmentation system based on dual-channel and self-attention mechanism, including:

[0027] An image acquisition module is configured to: acquire a picture to be segmented;

[0028] A dual-channel feature map extraction module is configured to: extract feature maps of two channels from the image to be segmented; wherein the first channel extracts a multi-scale context information feature map; and the second channel extracts a pixel-level feature map;

[0029] A feature fusion module is configured to: learn the multi-scale context information feature map and the pixel-level feature map through matrix operations and self-attention mechanism to obtain a feature map that relates each pixel to its corresponding context area;

[0030] The semantic segmentation module is configured to: input the feature map related to each pixel and its corresponding context area into the trained classifier, and output the image semantic segmentation result.

[0031] One or more embodiments provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the image semantic segmentation methods based on dual-channel and self-attention mechanisms as described above are implemented.

[0032] One or more embodiments provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described image semantic segmentation methods based on dual-channel and self-attention mechanisms.

[0033] One or more of the above technical solutions have the following beneficial effects:

[0034] (1) The present invention comprehensively considers the balance between network complexity and accuracy, uses the HRNetV2 network as the backbone to extract features from the input image, and combines the self-attention mechanism with the pyramid pooling module in PSPNet to learn between pixels and regions on the feature map extracted by the HRNetV2 network, thereby improving the accuracy.

[0035] (2) The semantic segmentation technology and system proposed in the present invention can optimize network performance during the training process, strengthen the relationship between pixels and contextual information, and make the network focus only on the relationship between regions and pixels, rather than mixing all the regions of the entire feature map together for segmentation. The above scheme will improve the segmentation accuracy and will not cause misclassification. We use HRNetV2 instead of ResNet as the backbone, mainly because it is a lightweight feature extraction structure with low computing power requirements and superior performance, and the model training speed is accelerated. In general, our network system model has a simple structure, small computational complexity and much improved accuracy.

[0036] (3) In order to prevent the network from overfitting, the present invention adds random inactivation dropout and CrossEntropyloss function to perform auxiliary training on the network. At this time, the feature map obtained has contextual semantic information, and the feature map obtained by multi-scale average pooling is fused. The fused feature map has hierarchical and diverse contextual global semantic information. The role of auxiliary training is to speed up the convergence of system model training and extract richer contextual semantic information areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0038] Figure 1 This is a flow chart of an image semantic segmentation method based on dual channels and self-attention mechanism in an embodiment of the present invention;

[0039] Figure 2 It is a structural diagram of an image semantic segmentation method based on dual channels and self-attention mechanism in an embodiment of the present invention;

[0040] Figure 3 It is an overall framework diagram of the pyramid scene analysis network in an embodiment of the present invention;

[0041] Figure 4 This is a structural diagram of the self-attention mechanism in an embodiment of the present invention;

[0042] FIG5 is a graph showing the loss value and learning rate decrease during the last 24 hours of training in an embodiment of the present invention;

[0043] Figure 6(a)-Figure 6(c) are original images of segmentation images in different scenes in the embodiments of the present invention;

[0044] Figure 7(a)-Figure 7(c) are segmentation map labels in different scenarios in the embodiments of the present invention;

[0045] Figure 8(a)-Figure 8(c) Schematic diagram of segmentation effects in different scenarios according to the embodiments of the present invention. DETAILED DESCRIPTION

[0046] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0048] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0049] Embodiment 1

[0050] like Figure 1 As shown, this embodiment discloses an image semantic segmentation method based on dual channels and self-attention mechanism, comprising the following steps:

[0051] S101: Obtain the image to be segmented;

[0052] S102: Extract feature maps of two channels from the image to be segmented;

[0053] Among them, the first channel extracts the multi-scale context information feature map; the second channel extracts the pixel-level feature map;

[0054] S103: The multi-scale regional context information feature map and the pixel-level feature map are learned through matrix operations and self-attention mechanism to obtain a robust feature map in which each pixel is related to its corresponding context area.

[0055] S104: Input the feature map related to each pixel and its corresponding context area into the trained classifier, and output the image semantic segmentation result.

[0056] like Figure 2-3 As shown, as one or more embodiments, in S102, extracting a multi-scale context information feature map from the first channel specifically includes:

[0057] Input the image to be segmented into the HRNetV2_w18 network to obtain the first feature map of the first channel The first feature map of the first channel is convolved by 1*1conv to obtain the second feature map of the first channel The second feature map of the first channel is extracted through the pyramid pooling model (PPM) to obtain the third feature map of the first channel.

[0058] Among them, the feature map is used Represents, where x represents the channel number and y represents the feature map of the corresponding channel.

[0059] In order to prevent the network from overfitting, the third feature map of the first channel is added with random inactivation dropout, and the neurons are randomly inactivated with a probability of 0.1. After the 1×1 convolution, the cross entropy loss function is added to assist the network training to obtain the fourth feature map of the first channel. That is, a multi-scale context information feature map;

[0060] The CrossEntropyloss function calculation formula is as follows:

[0061]

[0062] Among them, class represents the label belonging to a certain class, input represents the feature vector of the input feature map, and input[j] represents the score of a certain class.

[0063] The advantage of the above scheme is that the CrossEntropyloss function is added through random dropout to accelerate the convergence of the system model and extract richer areas.

[0064] HRNetV2_w18 is a lightweight backbone compared to ResNet. It always maintains the same original structure during feature extraction. Figure 1 The feature extraction process is a process of maintaining high resolution.

[0065] Traditional feature extraction requires loss of resolution, and the depth of the network requires high computing power. Considering computing power and accuracy, this embodiment selects HRNetV2_w18 as the backbone.

[0066] As one or more embodiments, the pyramid pooling (PPM) model outputs pooling of different sizes to extract features from the feature map, and the pooling uses AdaptiveAvgPool2D (I x ) indicates that AdaptiveAvgPool2D is the pooling function, I x Represents the feature maps of different sizes of pooled output height and width, I x =[1,1],I x =[2,2],I x =[3,3],I x =[6,6], the number of channels is 67.

[0067] Therefore, rich contextual regions are extracted through pooling at different scales.

[0068] In S102, extracting the pixel-level feature map from the second channel specifically includes:

[0069] Input the image to be segmented into the HRNetV2_w18 network to obtain the first feature map of the second channel The first feature map of the second channel is extracted after 1*1conv convolution, accelerated neural network BatchNorm and Relu function to obtain the second feature map of the second channel. That is, pixel-level feature map.

[0070] As one or more embodiments, the multi-scale region context information feature map and the pixel-level feature map are processed by matrix operation, specifically including:

[0071] The pixels at each position of the multi-scale regional context information feature map and the pixel-level feature map are matrix multiplied; the multi-scale regional context information feature map is preprocessed and output to obtain a first matrix; the pixel-level feature map is normalized and output to obtain a second matrix, the first matrix and the second matrix are multiplied to output a third matrix feature map, i.e., the fused pixel-level regional context feature map.

[0072] The specific implementation method is:

[0073] The fourth feature map of the first channel After reshape1, the output is [n,c,h×w], and then after tanspose, the output is [n,h×w,c], named U, where n represents the number of images input at one time, c represents the number of channels of the feature map, h represents the height of the feature map, and w represents the width of the feature map.

[0074] The second feature map of the second channel After reshape1 and Softmax normalization, the output is [n, k, h×w], named I.

[0075] Multiply the two matrices I and U. The result of the multiplication is passed through the transpose and unsqueeze functions to finally output [n,c,k1]. This feature map is named It is Through the feature map after multi-region feature extraction, and The pixels at each position of the feature map are multiplied by pixels to enhance The feature map obtained by the relationship between regional features has a more obvious distinction between regions and has prominent contextual information.

[0076] This will make the category information between regions more refined, and will also strengthen and aggregate the pixels between each region in the feature map, resulting in a richer feature map.

[0077] As one or more embodiments, Figure 4 As shown, the process of learning the self-attention mechanism includes: inputting the results of matrix operations on the multi-scale context information feature map and the pixel-level feature map into the self-attention mechanism, and learning the relationship between the pixel at each position and its region;

[0078] The self-attention mechanism learns the relationship between the pixel at each position and the context area of ​​the pixel, and generates the corresponding feature map according to the relationship between the pixel at each position and its area, specifically:

[0079] Will To input to the self-attention mechanism, use 1×1conv, the number of convolution kernels is 256, the feature map is obtained, and then flattened in the width and height dimensions of the feature map to obtain the Key matrix;

[0080] Similarly, the pixel-level feature map of the input image is fed into the attention mechanism, using 1×1 conv with 256 convolution kernels. The feature map is flattened in the height and width dimensions to obtain the Query matrix. At the same time, After 256 1×1 convs, the width and height dimensions are flattened to obtain the Value matrix. Here we refer to the Query, Key, and Value matrices as Q, K, and V. The Q matrix is ​​the matrix used to query the relationship of the K matrix generated by the pixel-level feature map; the K matrix is ​​the matrix generated by the fused pixel-level regional context feature map waiting for the Q matrix to be queried; the V matrix is ​​the matrix with actual information and feature attributes generated by the fused pixel-level regional context feature map.

[0081] This embodiment is to find Each pixel in The relationship matrix between the corresponding areas includes the following steps:

[0082] The pixel-level feature map queries the relationship between the pixel-level feature map and the fused pixel-level regional context feature map through the Q matrix, the fused pixel-level regional context feature map obtains the query result through the K matrix query, and converts the query result into a probability weight matrix of the correspondence between the pixel-level feature map and the fused pixel-level regional context feature map; the V matrix is ​​reconstructed according to the probability weight matrix, and probability weight parameters are assigned to pixel areas of the same category and pixel areas of different categories.

[0083] For example:

[0084] Step 1: You need to use your own query matrix Q to query The relationship between To provide your own K matrix, query the results using QK T Implementation: For pixels belonging to the same class label, the cosine angle of the angle is close to 0, so the value of their inner product is very large, and for pixels belonging to different classes of labels, their values ​​of the inner product are very small.

[0085] Step 2: QK T Using the normalized exponential function softmax, transform and The corresponding relationship probability weight matrix

[0086] Step 3: This step is to reconstruct the V matrix, assigning higher probability weight parameters to pixel areas of the same category and lower probability weight parameters to pixel areas of different categories, according to the formula: V is implemented, and the reconstructed feature map is named self-attention feature map.

[0087] At this time, the reconstructed feature map not only has its own features and information, but also carries Each pixel in Features and information about the relationship between the corresponding context areas.

[0088] The specific formula is as follows:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] Among them, s x is a set of pixels of a single channel feature map belonging to different feature maps, d x It indicates that different channels are generated during the transmission of the feature map, c represents 1×1 convolution, and reshape1 represents the conversion function, which is used to convert the three-dimensional feature map into a matrix composed of vectors.

[0095] As a specific example:

[0096] For example, the data set in this embodiment corresponds to 19 categories, so the corresponding number of vectors is 19. For example, for variable Q, input After 1×1conv, the output is [n, 19, w1, h1], then after reshape1, the output is [n, 19, w1×h1], and finally after tran, the output is [n, w1×h1, 19].

[0097] Variable K, for example, input After 1×1conv, the output is [n, 19, h2, w2], and then after reshape1, the output is [n, 19, h2×w2].

[0098] For V, the same operation as Q, input The final output is [n,w2×h2,19]. BMM represents the multiplication of two matrices, and Softmax represents the normalized exponential function.

[0099] For formula (6), reshape2 converts the input tran(BMM(V,SIM)) of dim=0, i.e., the pixel set of the h1×w1 channel, into a single-channel two-dimensional plane feature map, which has the same dimension as the label dimension, so as to perform loss with the label.

[0100] Through these five sets of formulas, we can derive the relationship between each pixel and its context, and we can also calculate the relationship between each area and the corresponding pixel.

[0101] in, It is a robust feature map with the relationship between pixels and context.

[0102] The advantage of the above scheme is that the robust feature map with the relationship between pixels and contexts finally obtained in this embodiment not only takes into account the relationship between regions, the relationship between pixels and their surroundings, but also the relationship between pixels and regions, so that pixels of similar classes can be aggregated.

[0103] Finally, CONT and Do concat, input to 1×1 convolution, and then output a feature map of the same size as the input image use The loss function is the same as formula (1) when comparing the actual label.

[0104] Preferably, this embodiment optimizes the system model through a main loss function and an auxiliary loss function, and the proportions of the main loss function and the auxiliary loss function are 1 and 0.4 respectively.

[0105] To ensure fairness in the experiment, we reproduced the PSPNet model and our system model on the Cityscape dataset for comparative experiments. The device we selected was a GPU Tesla V100 32GB. The training method was to input bachsize of 10 images and iterate 100 times at a time to ensure that the loss value changes stably. Figure 5(a)-Figure 5(b) shown.

[0106] Set the total number of iterations to 160,000. Select the optimizer as SGD, the learning rate is 0.0025, the momentum is 0.9, two CrossEntropyloss, and the loss function is the same as formula (1). The two loss functions assist each other in system model training. Here, PSPnet uses ResNet50 as the backbone, and we choose HRNetV2_w18 as the backbone. The mIoU index of the network system of this embodiment is 1.7% higher than that of the original model, and the accuracy of target segmentation is significantly improved. The larger the mIoU, the more obvious the system advantage, and the more accurate the target segmentation boundary box positioning. The mPA of this embodiment is 0.5% higher than that of PSPnet.

[0107] This embodiment uses the cityscape benchmark data set for experiments. After comparison of the PSPnet algorithm, the mIoU and IoU values ​​obtained are shown in the following table. It can be seen from Table 1 and Table 2 that our network performance is better.

[0108] Table 1

[0109]

[0110] Table 2

[0111]

[0112] Figure 5(a)-Figure 5(b) This is a graph showing the loss value and learning rate drop during the last 24 hours of training in the embodiment of the present invention; Figure 5(a)-Figure 5(b) As shown, the visual display of the accuracy of this embodiment is demonstrated, and scene samples in the verification set are randomly selected for test verification to intuitively demonstrate the practicality of the system.

[0113] like Figure 6(a)-Figure 6(c) are original images of segmentation images in different scenes in the embodiments of the present invention; Figure 7(a)-Figure 7(c) are segmentation map labels in different scenarios in the embodiments of the present invention; Figure 8(a)-Figure 8(c)The segmentation effect diagrams in different scenes in the embodiments of the present invention. The method of this embodiment can correctly segment different categories, such as pedestrians, vehicles, cyclists, and traffic lights, and can also divide different categories into different colors. The method of this embodiment uses HRNetV2_w18 as the backbone, with fewer parameters and stronger practicality. It can not only effectively segment different complex scenes, but also is not affected by the surrounding environment and has stronger anti-interference ability. It performs well in environments with complex and changeable data.

[0114] Those skilled in the art will appreciate that during the inspection process, the detection content for different types of equipment is different, and the detection content can be pre-set according to the equipment type. After the image of the equipment to be detected is acquired, the detection is performed according to the type of the equipment. Specifically, a deep learning target detection algorithm is used to realize automatic recognition of the equipment status. In this embodiment, the deep learning model is deployed in an embedded AI analysis module, which realizes front-end deployment and improves the real-time analysis of the inspection video.

[0115] Embodiment 2

[0116] This embodiment provides an image semantic segmentation system based on dual channels and self-attention mechanism, including:

[0117] An image acquisition module is configured to: acquire a picture to be segmented;

[0118] The dual-channel feature map extraction module is configured to: extract feature maps of two channels from the image to be segmented; wherein the first channel extracts a multi-scale context information feature map; and the second channel extracts a pixel-level feature map;

[0119] A feature fusion module is configured to: learn the multi-scale context information feature map and the pixel-level feature map through matrix operations and self-attention mechanism to obtain a feature map that relates each pixel to its corresponding context area;

[0120] The semantic segmentation module is configured to: input the feature map related to each pixel and its corresponding context area into the trained classifier, and output the image semantic segmentation result.

[0121] Embodiment 3

[0122] The embodiment of this specification provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the image semantic segmentation method based on dual-channel and self-attention mechanism in Example 1 are implemented.

[0123] Embodiment 4

[0124] The embodiment of this specification provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the image semantic segmentation method based on dual-channel and self-attention mechanism in Example 1 are implemented.

[0125] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0127] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An image semantic segmentation method based on dual-channel and self-attention mechanism, characterized in that: The steps include: Get the image to be segmented; The image to be segmented is subjected to two-channel feature map extraction respectively; the first channel extracts a multi-scale context information feature map; the second channel extracts a pixel-level feature map; The multi-scale context information feature map and the pixel-level feature map are learned through matrix operations and self-attention mechanism to obtain a feature map that is related to each pixel and the context area corresponding to the pixel; Input the feature map related to each pixel and the context area corresponding to the pixel into the trained classifier and output the image semantic segmentation result; The first channel extracts a multi-scale context information feature map specifically including: inputting the image to be segmented into the HRNetV2_w18 network to obtain a first feature map of the first channel, performing convolution on the first feature map of the first channel to obtain a second feature map of the first channel, and performing multi-scale context information extraction on the second feature map of the first channel through a pyramid pooling model to obtain a third feature map of the first channel; The second channel extracting the pixel-level feature map specifically includes: inputting the image to be segmented into the HRNetV2_w18 network to obtain the first feature map of the second channel, and extracting the second feature map of the second channel, that is, the pixel-level feature map, after the first feature map of the second channel is subjected to convolution, accelerated neural network and Relu function; The matrix operation of the multi-scale context information feature map and the pixel-level feature map specifically includes: matrix multiplication of the pixels at each position of the multi-scale regional context information feature map and the pixel-level feature map, including: The multi-scale regional context information feature map is preprocessed and output to obtain a first matrix; The pixel-level feature map is normalized to output the second matrix, the first matrix and the second matrix are multiplied to output the third matrix feature map, that is, the fused pixel-level regional context feature map; The self-attention mechanism learning includes learning the relationship between the pixel at each position and the context area of ​​the pixel, and generating a corresponding feature map according to the relationship between the pixel at each position and the context area of ​​the pixel; Specifically, the fused pixel-level regional context feature map and pixel-level feature map are input into the attention mechanism to obtain three matrices Q matrix, K matrix and V matrix; Among them, the Q matrix is ​​a matrix generated by the pixel-level feature map to query the relationship of the K matrix; the K matrix is ​​a matrix generated by the fused pixel-level regional context feature map waiting for the Q matrix to be queried; the V matrix is ​​a matrix with actual information and feature attributes generated by the fused pixel-level regional context feature map; The feature map of each pixel and the context area corresponding to the pixel specifically includes: The pixel-level feature map queries the relationship between the pixel-level feature map and the fused pixel-level regional context feature map through the Q matrix, the fused pixel-level regional context feature map obtains the query result through the K matrix query, and converts the query result into a probability weight matrix of the correspondence between the pixel-level feature map and the fused pixel-level regional context feature map; the V matrix is ​​reconstructed according to the probability weight matrix, and probability weight parameters are assigned to pixel areas of the same category and pixel areas of different categories.

2. The image semantic segmentation method based on dual-channel and self-attention mechanism as claimed in claim 1, characterized in that: After the third feature map of the first channel is obtained, random inactivation dropout is added to randomly inactivate neurons, and a cross entropy loss function is added after convolution to assist network training to obtain the fourth feature map of the first channel, that is, the multi-scale context information feature map.

3. Image semantic segmentation system based on dual-channel and self-attention mechanism, characterized by: The following steps are involved: An image acquisition module is configured to: acquire a picture to be segmented; A dual-channel feature map extraction module is configured to: extract feature maps of two channels from the image to be segmented; wherein the first channel extracts a multi-scale context information feature map; and the second channel extracts a pixel-level feature map; A feature fusion module is configured to: learn the multi-scale context information feature map and the pixel-level feature map through matrix operations and a self-attention mechanism to obtain a feature map that relates each pixel to the context area corresponding to the pixel; A semantic segmentation module is configured to: input a feature map related to each pixel and the context area corresponding to the pixel into a trained classifier, and output a semantic segmentation result of the image; The first channel extracts a multi-scale context information feature map specifically including: inputting the image to be segmented into the HRNetV2_w18 network to obtain a first feature map of the first channel, performing convolution on the first feature map of the first channel to obtain a second feature map of the first channel, and performing multi-scale context information extraction on the second feature map of the first channel through a pyramid pooling model to obtain a third feature map of the first channel; The second channel extracting the pixel-level feature map specifically includes: inputting the image to be segmented into the HRNetV2_w18 network to obtain the first feature map of the second channel, and extracting the second feature map of the second channel, that is, the pixel-level feature map, after the first feature map of the second channel is subjected to convolution, accelerated neural network and Relu function; The matrix operation of the multi-scale context information feature map and the pixel-level feature map specifically includes: matrix multiplication of the pixels at each position of the multi-scale regional context information feature map and the pixel-level feature map, including: The multi-scale regional context information feature map is preprocessed and output to obtain a first matrix; The pixel-level feature map is normalized to output the second matrix, the first matrix and the second matrix are multiplied to output the third matrix feature map, that is, the fused pixel-level regional context feature map; The self-attention mechanism learning includes learning the relationship between the pixel at each position and the context area of ​​the pixel, and generating a corresponding feature map according to the relationship between the pixel at each position and the context area of ​​the pixel; Specifically, the fused pixel-level regional context feature map and pixel-level feature map are input into the attention mechanism to obtain three matrices Q matrix, K matrix and V matrix; Among them, the Q matrix is ​​a matrix generated by the pixel-level feature map to query the relationship of the K matrix; the K matrix is ​​a matrix generated by the fused pixel-level regional context feature map waiting for the Q matrix to be queried; the V matrix is ​​a matrix with actual information and feature attributes generated by the fused pixel-level regional context feature map; The feature map of each pixel and the context area corresponding to the pixel specifically includes: The pixel-level feature map queries the relationship between the pixel-level feature map and the fused pixel-level regional context feature map through the Q matrix, the fused pixel-level regional context feature map obtains the query result through the K matrix query, and converts the query result into a probability weight matrix of the correspondence between the pixel-level feature map and the fused pixel-level regional context feature map; the V matrix is ​​reconstructed according to the probability weight matrix, and probability weight parameters are assigned to pixel areas of the same category and pixel areas of different categories.

4. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the image semantic segmentation method based on dual-channel and self-attention mechanism described in any one of claims 1-2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the image semantic segmentation method based on dual-channel and self-attention mechanism described in any one of claims 1-2 are implemented.

Images