A semantic alignment real-time semantic segmentation method and system and storage medium

Abstract

The application discloses a kind of real-time semantic segmentation methods, systems and storage medium of semantic alignment, belong to real-time semantic segmentation field, including: training phase: using training set to real-time semantic segmentation model training, each sample in the training set includes the image related to semantic segmentation task and corresponding label for indicating segmentation result;The real-time semantic segmentation model includes: convolution network, semantic extraction branch and semantic alignment network;The convolution network includes local convolution module, convolution attention module and semantic decoding module;Application phase: the image to be processed is input into the trained convolution network, and the segmentation result is obtained.The application can improve the accuracy of real-time semantic segmentation task, and by discarding semantic extraction branch and semantic alignment network during inference, simplify the main network as a single branch convolution network during inference, with high segmentation accuracy and low inference delay characteristics, improve the prospect of real-time semantic segmentation network in actual application deployment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of real-time semantic segmentation, and more particularly relates to a semantic alignment real-time semantic segmentation method, system and storage medium. BACKGROUND

[0002] Semantic segmentation is a basic visual understanding task, which plays an important role in fields such as autonomous driving, medical image processing, mobile phone applications, etc. Multi-scale and long-distance context information can significantly improve the accuracy of semantic segmentation, so many semantic segmentation models have proposed various methods to capture rich context information, such as expanding the receptive field, multi-scale fusion, self-attention mechanism, etc. Although these methods have brought significant improvement in segmentation accuracy, they have also brought high computational cost; especially the self-attention mechanism, although it has global modeling capability, but it has a square complexity with the image resolution, which significantly increases the inference delay of the semantic segmentation model. However, many real-world applications require the segmentation model to have real-time performance, and the above limitations affect the practical application of these methods.

[0003] To solve the above problems, some real-time semantic segmentation methods are proposed to obtain higher segmentation accuracy with smaller delay cost. The mainstream real-time semantic segmentation method adopts a dual-branch model structure, which divides the network from the early input stage into two branches, respectively extracts spatial detail information and long-distance semantic information, and fuses the outputs of the two branches at the tail of the network. Some other real-time semantic segmentation methods share features in the early stage of the network, simplify the early stage of the network into a single branch, and divide the deep stage of the network into a spatial branch and a semantic branch, and perform multi-stage dense interaction between the two branches.

[0004] Although these real-time semantic segmentation methods have a better balance between inference speed and segmentation accuracy compared to general multi-scale hierarchical semantic segmentation methods, in the actual application of the inference stage, the former has a larger computational delay due to the separate listing of the network branch, and the latter has an additional amount of calculation due to the dense interaction operation of the deep features, resulting in an increase in inference delay and a decrease in inference speed, which affects the performance of the deployment in practical applications. SUMMARY

[0005] In view of the defects of the prior art and the improvement needs, the present application provides a semantic alignment real-time semantic segmentation method, system and storage medium, which aims to reduce the delay of real-time semantic segmentation and improve the segmentation speed.

[0006] To achieve the above purpose, according to the first aspect of the present application, a semantic alignment real-time semantic segmentation method is provided, comprising:

[0007] The training phase: the training set is used to train the real-time semantic segmentation model, each sample in the training set includes an image related to the semantic segmentation task and a corresponding label indicating the segmentation result; the real-time semantic segmentation model includes a convolutional network, a semantic extraction branch and a semantic alignment network;

[0008] The convolutional network includes a local convolution module, a convolutional attention module and a semantic decoding module;

[0009] The local convolution module is used to extract the bottom features in the input sample to generate a spatial feature map; the convolutional attention module is used to perform convolutional attention operation on the spatial feature map to obtain a feature-enhanced feature map; the semantic decoding module is used to perform multi-scale pooling aggregation on the feature-enhanced feature map, and then splice the spatial feature map to obtain a spliced feature map, and point-by-point classification to obtain a predicted semantic segmentation map;

[0010] The semantic extraction branch is used to perform hierarchical Transformer feature coding operation on the input sample, and output semantic feature maps of different scales;

[0011] The semantic alignment network is used to perform shape alignment on the feature-enhanced feature map, the spliced feature map and the predicted semantic segmentation map respectively with the semantic feature map of the corresponding scale, so that the convolutional network captures the semantic information in the semantic feature map;

[0012] Application stage:

[0013] The image to be processed is input into the trained convolutional network to obtain the segmentation result.

[0014] Further, the convolutional attention module includes a convolutional attention layer and a feedforward neural network layer connected in series;

[0015] The convolutional attention layer is used to calculate a feature similarity matrix of the spatial feature map and a learnable kernel independent of the input sample image, and to perform convolution on the similarity matrix with the learnable kernel to obtain a feature map with long-distance dependence information enhanced in spatial dimension;

[0016] The feedforward neural network layer is used to integrate the dependence information in different channels of the feature map to obtain the feature-enhanced feature map.

[0017] Further, the convolutional attention layer includes two groups of complementary convolutional attention operations, residual connection and normalization operation;

[0018] Wherein, the convolutional attention operation is represented as X represents the spatial feature map, K represents the learnable kernel, theta represents the double normalization operation, K Tdenotes the transpose of K; the learnable kernels in the two groups of convolutional attention operations are respectively used to capture features of longitudinal and transverse patterns in the feature map, so that the two groups of convolutional attention operations form a complementary structure;

[0019] The residual connection is used to add the outputs of the two groups of complementary convolutional attention operations to the spatial feature map, and the normalized operation is performed to obtain the feature map with enhanced spatial dimension long-range dependency information.

[0020] Further, the semantic extraction branch includes a patch embedding layer, a plurality of serially connected Transformer encoding blocks, and a feature decoding module.

[0021] The patch embedding layer is used to perform down-sampling and patching operations on the input sample to realize image compression and match the input form requirements of the plurality of serially connected Transformer encoding blocks.

[0022] The plurality of serially connected Transformer encoding blocks are used to perform hierarchical Transformer feature coding operations on the compressed image to obtain a semantic feature map.

[0023] The feature decoding module is used to decode the semantic feature map to obtain a decoded feature map, and classify the decoded feature map to obtain a segmentation map.

[0024] Further, the semantic alignment network includes a feature alignment layer and a semantic alignment loss function.

[0025] The feature alignment layer is used to align the shapes of the feature-enhanced feature map and the semantic feature map, align the shapes of the spliced feature map and the decoded feature map, and align the shapes of the predicted semantic segmentation map and the decoded feature map to obtain a segmentation map.

[0026] The semantic alignment loss function is used to calculate the similarity of the feature channel level of the features after shape alignment by the feature alignment layer, and supervise the learning of the convolutional network parameters based on the similarity.

[0027] Further, the semantic alignment loss function is:

[0028]

[0029] wherein φ denotes an activation function, denotes the KL divergence, x T denotes different scale semantic feature maps output by the semantic extraction branch, x S denotes different scale feature maps corresponding to the convolutional network;

[0030] The activation function φ is: c represents a certain specific channel, i is a specific pixel point, H and W represent the height and width of the image, is a hyperparameter, x c Take x T or x S .

[0031] Further, the loss function in the training process comprises: the semantic alignment loss function and the segmentation loss of the semantic decoding module; the segmentation loss of the semantic decoding module is the cross-entropy loss between the predicted semantic segmentation map and the label.

[0032] Further, the semantic decoding module comprises a multi-scale pooling aggregation layer and a feature classification layer.

[0033] The multi-scale pooling aggregation layer is used for multi-scale pooling aggregation of the feature-enhanced feature map, so that the size of the aggregated feature map is consistent with the size of the spatial feature map.

[0034] The feature classification layer is used for performing a splicing operation on the aggregated feature map and the spatial feature map to obtain a spliced feature map, and performing point-by-point classification to obtain a predicted semantic segmentation map.

[0035] According to another aspect of the present application, a real-time semantic segmentation system with semantic alignment is provided, comprising a computer readable storage medium and a processor.

[0036] The computer readable storage medium is used for storing executable instructions.

[0037] The processor is used for reading the executable instructions stored in the computer readable storage medium to execute the method of any one of the first aspect.

[0038] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the method of any one of the first aspect.

[0039] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0040] (1) The real-time semantic segmentation method of the present application extracts different scale semantic feature maps through the convolution network and the semantic extraction branch respectively, and performs real-time semantic alignment of the corresponding scale feature maps through the semantic alignment network, so that the convolution network learns rich semantic features from the transformer semantic branch with rich long-distance dependence through the semantic alignment network, thereby improving the accuracy of the real-time semantic segmentation task. And discard the semantic extraction branch and the semantic alignment network during inference, simplify the main network to a single branch convolution network during inference, greatly reduce the inference delay, improve the segmentation accuracy, reduce the inference delay, and improve the segmentation speed, improve the prospect of deploying the real-time semantic segmentation network in practical applications.

[0041] (2) As a preferred, the convolution attention module designed by the present application includes a convolution attention layer and a feedforward neural network layer connected in series, and the convolution attention operation in the convolution attention layer has consistency with the spatial attention operation in the Transformer encoding block of the semantic extraction branch. Both enhance the long-distance dependence information of the feature map through attention operation, and then integrate the channel information through the feedforward network, so as to make the alignment effect of the convolution attention module and the high-performance semantic extraction branch better. At the same time, the convolution attention module of the present application is realized by convolution, and the long-distance dependence is captured through convolution operation, which reduces the accuracy gap with the standard self-attention mechanism with smaller calculation complexity. While realizing the structure matching with the semantic extraction branch, the efficiency is higher. At the same time, due to the characteristics of pure convolution operation, the performance of deployment in practical applications is improved.

[0042] (3) As a preferred, the present application adopts two complementary convolution attention operations, and the learnable kernels in the two convolution attention operations capture the features of the longitudinal and transverse modes in the feature map respectively. The long-distance dependence information in the input feature image is enhanced through the similarity matrix and the features of the learnable kernel.

[0043] (4) As a preferred, a plurality of series of Transformer encoding blocks are used to extract hierarchical features of the input image, and the global receptive field of the Transformer is used to obtain different scale feature maps with high quality semantics.

[0044] (5) As a preferred, the semantic alignment network designed by the present application adopts a channel-based alignment method, which aligns the channel probability distribution of the different scale feature maps extracted by the convolution network with the different scale semantic feature maps extracted by the semantic extraction branch, rather than pixel-by-pixel feature alignment, which can better align long-distance semantic information and is more suitable for segmentation tasks.

[0045] (6) As preferred, the semantic alignment loss function designed by the present application is a channel-independent loss, which calculates the similarity of the probability distribution of each channel between the feature maps corresponding to the convolutional network and the semantic extraction branch at the same scale, can better measure the similarity of the semantic information of each channel of the feature map, and will adaptively concentrate in the semantic high response area and avoid being affected by the low response semantic misjudgment area, and has better alignment effect than the spatial point-by-point semantic alignment function.

[0046] (7) In the training process of the real-time semantic segmentation model, the loss function of the whole model includes the semantic alignment loss and the segmentation loss of the semantic decoding module, and through the training of the two parts of the loss, the convolutional network captures more accurate semantic information from the semantic extraction branch.

[0047] (8) The resolution size of the aggregated feature map is consistent with the input spatial feature map resolution size, and since the spatial feature map resolution output by the convolutional attention module is small and the resolution is further reduced after pooling, the calculation cost of the multi-scale pooling aggregation layer is small.

[0048] In summary, the present application can learn rich semantic features from the transformer semantic branch with rich long-distance dependencies through the semantic alignment network, improve the accuracy of the real-time semantic segmentation task, and simplify the inference of the backbone network to a single branch convolutional network by discarding the semantic extraction branch and the semantic alignment network during inference, remove the calculation cost caused by the additional introduction of the branch structure and the dense interaction module during inference, and greatly reduce the inference delay. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The semantic alignment real-time semantic segmentation method provided by the embodiment of the present application is shown in the figure.

[0050] Figure 2 The semantic alignment real-time semantic segmentation model provided by the embodiment of the present application is shown in the figure.

[0051] Fig. 3(a) is a schematic diagram of the feature compression layer structure provided by the embodiment of the present application; Fig. 3(b) and Fig. 3(c) are schematic diagrams of two local feature extraction layers, respectively.

[0052] Fig. 4(a) and Fig. 4(b) are schematic diagrams of the convolutional attention module and two groups of complementary convolutional attention operation provided by the embodiment of the present application.

[0053] Fig. 5(a) and Fig. 5(b) are schematic diagrams of the multi-scale pooling aggregation layer and the feature classification layer in the embodiment of the present application, respectively.

[0054] Figure 6 The semantic alignment network in the embodiment of the present application is shown in the figure.

[0055] Figure 7 For the automatic driving scene, the visualization results of the existing technology and the model prediction results of the present application are as follows, Figure 7 (a)-(e) in (a)-(e) correspond to the input image, the true value label, two kinds of existing technology and the model prediction result of the present application.

[0056] Figure 8 For geographic information system, the visualization results of the existing technology and the model prediction results of the present application are as follows, Figure 8 (a)-(g) in (a)-(g) correspond to the input image, the true value label, four kinds of existing technology and the model prediction result of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0058] In the present application, the terms "first", "second" and the like in the present application and the drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0059] As Figure 1 shown, the real-time semantic segmentation method of semantic alignment of the present application comprises:

[0060] Training phase:

[0061] The semantic alignment real-time semantic segmentation model is trained by using a standard training set, wherein each sample in the standard training set is composed of an image related to the semantic segmentation task and its corresponding label true value, and the label true value is used to indicate the task segmentation result;

[0062] The semantic alignment real-time semantic segmentation model is used to capture high-quality semantic information at a low inference delay cost; comprising: a convolution network, a semantic extraction branch and a semantic alignment network, wherein the convolution network comprises a local convolution module, a convolution attention module and a semantic decoding module;

[0063] The local convolution module is used to locally model the input image, extract the low-level features in the image, and generate a spatial feature map for semantic decoding based on the low-level features;

[0064] The convolution attention module is configured to perform a convolution attention operation on the spatial feature map output by the local convolution module to obtain a feature-enhanced feature map. Specifically, a feature similarity matrix is obtained by calculating the similarity relationship between the spatial feature map and a plurality of groups of learnable kernels independent of the input image, and the feature similarity matrix records the similarity vector between each feature point in the spatial feature map and all learnable kernels. The initial values of the plurality of groups of learnable kernels independent of the input image are obtained by random initialization, and the values of the learnable kernels are updated in the subsequent network training process. The convolution attention module is further configured to perform a convolution attention operation on the similarity matrix and the learnable kernels, and after information integration, a feature-enhanced feature map is obtained.

[0065] The semantic decoding module is configured to perform multi-scale pooling aggregation on the feature-enhanced feature map output by the convolution attention module to realize multi-scale dependency learning, and concatenate the feature map output after multi-scale pooling aggregation with the spatial feature map output by the local convolution module to obtain a concatenated feature map, and perform point-by-point classification on the concatenated feature map by a classifier to obtain a predicted semantic segmentation map.

[0066] The semantic extraction branch is configured to perform hierarchical Transformer feature encoding operation on the input image to output semantic feature images of different scales containing rich long-distance dependency information and high-quality semantic information. Specifically, the semantic extraction branch further includes a decoded feature map obtained by decoding the output semantic feature image, and a segmentation map obtained by classifying the decoded feature map. The semantic feature image, the decoded feature map and the segmentation map are used to provide semantic information guidance to the convolution network through the semantic alignment network.

[0067] The semantic alignment network is configured to align the feature image extracted by the convolution network and the feature image output by the semantic extraction branch, so that the convolution network captures more accurate semantic information. Specifically, the feature-enhanced feature map output by the convolution attention module is aligned with the semantic feature image output by the semantic extraction branch, the concatenated feature map output by the semantic decoding module is aligned with the decoded feature map output by the semantic extraction branch, and the predicted semantic segmentation map output by the semantic decoding module is aligned with the segmentation map output by the semantic extraction branch. The alignment includes that the number of channels and the size of the images of the two feature maps are consistent.

[0068] Application stage:

[0069] The image to be processed is input into the trained convolution network to obtain a task segmentation result. Specifically, the image to be processed is input into the local convolution module in the trained convolution network, and the semantic decoding module outputs a task result.

[0070] That is, in the embodiment, the semantic extraction branch and the semantic alignment network are only used in the network training process, and are discarded in the test execution process.

[0071] In the embodiment, the feature maps of different scales extracted by the convolution network and the semantic extraction branch are real-time semantic aligned by the semantic alignment network, the convolution network learns rich semantic features from the transformer semantic branch with rich long-distance dependencies through the semantic alignment network, and the accuracy of the real-time semantic segmentation task is improved; and the semantic extraction branch and the semantic alignment network are discarded during inference, the backbone network is simplified to a single-branch convolution network during inference, and the inference delay is greatly reduced; the embodiment improves the prospects for deployment of the real-time semantic segmentation network in practical applications with high segmentation accuracy and low inference delay.

[0072] As an optional implementation, the structure of the semantic alignment real-time semantic segmentation model provided by the embodiment is as shown in Figure 2 . Figure 2 In the embodiment, the unlabeled rectangular box represents a feature map, the height of the rectangular box represents the number of channels of the feature map, and the width of the rectangular box represents the width / height of the feature map. From shallow to deep, the number of channels of each feature map is doubled, and the width / height is reduced by 1 / 2. In other embodiments, the number of channels and the width / height of each feature map can be other values, which can be set according to actual needs.

[0073] As shown in Figure 2 , in the embodiment, the local convolution module includes one feature compression layer and two locally connected local feature extraction layers.

[0074] The feature compression layer is used for fast downsampling the width and height of the input image to 1 / 4 of the width and height of the input image, extracting bottom layer features at the resolution, and mapping the channels of the extracted bottom layer feature map to the required number of channels.

[0075] The first local feature extraction layer is used for extracting local information of the bottom layer feature map in the spatial dimension, and enhancing the feature values of each point of the feature map according to the local information.

[0076] The second local feature extraction layer is used for expanding the number of channels of the feature map to 2 times and reducing the width / height to 1 / 2, to obtain a spatial feature map for semantic decoding.

[0077] Specifically, as shown in FIG. 3(a), the feature compression layer includes two groups of cascaded down-sampling structures for rapidly down-sampling the input image; each group of down-sampling structures includes cascaded 3x3 convolution layers, batch normalization and Relu activation functions. The first group of cascaded 3x3 convolution layers, batch normalization and Relu activation functions reduce the width / height of the input image by 1 / 2 and map the channels of the feature map to the required number of channels; the second group of cascaded 3x3 convolution layers, batch normalization and Relu activation functions reduce the width / height of the input image to 1 / 4.

[0078] As shown in FIG. 3(c), the first local feature extraction layer includes two groups of cascaded 3x3 convolution layers without down-sampling, batch normalization and Relu activation functions.

[0079] As shown in FIG. 3(b), the second local feature extraction layer includes a group of 3x3 convolution layers with down-sampling, batch normalization and Relu activation functions and a group of cascaded 3x3 convolution layers without down-sampling, batch normalization and Relu activation functions.

[0080] As Figure 2 As shown in FIG. 4(a) and FIG. 4(b), in the embodiment, the convolution attention module includes a convolution attention layer and a feedforward neural network layer connected in series.

[0081] The convolution attention layer is used to capture long-distance dependency information in the spatial dimension of the spatial feature map output by the local convolution module and includes two groups of complementary convolution attention operations, residual connection and normalization operation. The convolution attention operation is used to calculate a feature similarity matrix of the spatial feature map and a learnable kernel independent of the image, and the similarity matrix is convolved with the learnable kernel, that is, the long-distance dependency information in the input feature image is enhanced by the similarity matrix and the features of the learnable kernel. The learnable kernels in the two groups of complementary convolution attention operations are respectively used to focus on capturing the features of the vertical and horizontal modes in the feature map. The output results of the two groups of complementary convolution attention operations are respectively added to the spatial feature map output by the local convolution module through residual connection, normalized and then added to obtain a feature map with long-distance dependency information enhanced in the spatial dimension.

[0082] Specifically, the convolution attention operation in the embodiment of the application can be represented as X represents the input feature map as Query, which is the spatial feature map output by the local convolution module in the embodiment of the application; K represents a learnable kernel independent of the image as Key, which is convolved with Query as the convolution kernel to obtain a feature response map; θ represents double normalization, that is, softmax operation in the spatial dimension and L1 normalization in the channel dimension, and the feature response map obtains a similarity matrix through double normalization; KT denotes the transpose of K, Value is used as a convolution kernel to convolve the similarity matrix; wherein, Key1 and Key2 in Fig. 4(b) are learnable kernels that focus on capturing horizontal and vertical pattern features in the feature map respectively, and Value1 and Value2 are corresponding transposes. In the embodiment of the present application, the convolution attention operation enhances the long-distance dependency information in the input feature map by calculating a feature similarity matrix of the feature map and a learnable kernel independent of the image, and by the similarity matrix and the learnable kernel.

[0083] a feedforward neural network layer for integrating the dependency information in different channels of the spatial dimension long-distance dependency information enhanced feature map to obtain a long-distance dependency information enhanced feature map, i.e., a feature enhanced feature map.

[0084] Specifically, the feedforward neural network layer includes a convolution layer, a residual connection and a normalization operation in series, wherein the convolution layer includes two 3x3 convolution layers in series; the two 3x3 convolution layers in series are used to perform convolution operation on the spatial dimension long-distance dependency information enhanced feature map output by the convolution attention layer, and the result is added to the output of the convolution attention layer through residual connection and normalized to obtain the feature enhanced feature map.

[0085] As a preferred, the convolution attention layer and the feedforward neural network layer connected in series are two groups, which are connected in series between the two groups to further enhance the capture of long-distance dependency information in the feature map.

[0086] In addition, in the embodiment, there is a simple 3x3 convolution before the convolution attention module for downsampling operation, which expands the number of channels of the feature map to 2 times and reduces the width / height by 1 / 2.

[0087] As shown in Fig. 4(c), in the embodiment, the semantic decoding module includes a multi-scale pooling aggregation layer and a feature classification layer. Figure 2 The multi-scale pooling aggregation layer is used to perform multi-scale pooling operation on the feature enhanced feature map output by the convolution attention module, and aggregate the results obtained by multi-scale pooling, and the size of the aggregated feature map is consistent with the size of the spatial feature map output by the local convolution module.

[0088] The feature classification layer is used to perform a splicing operation on the aggregated feature map and the spatial feature map output by the local convolution module to obtain a spliced feature map, and to perform point-by-point classification on the spliced feature map to obtain a predicted semantic segmentation map.

[0089]

[0090] ​Specifically, as shown in FIG. 5(a), the multi-scale pooling aggregation layer includes pooling operations with different sampling multiples and corresponding 1x1 convolution up-sampling operations, 3x3 convolution operations, 1x1 convolution channel integration operations, residual branches and addition operations;

[0091] The feature-enhanced feature map output by the convolution attention module is step-by-step up-sampled and added through pooling operations with different sampling multiples and corresponding 1x1 convolution up-sampling operations, and is fused and channel-spliced through 3x3 convolution operations, and then is channel-integrated through 1x1 convolution channel integration operations to obtain an integrated feature map;

[0092] The residual branch is a residual branch including only 1x1 convolution, which is used to perform 1x1 convolution operation on the feature-enhanced feature map output by the convolution attention module.

[0093] The addition operation is used to add the integrated feature map and the output after the residual branch to obtain a feature map output after multi-scale pooling aggregation.

[0094] In the embodiment of the application, the pooling operations with different sampling multiples are pooling operations with sampling multiples of 1 (no pooling), 2, 4, 8 and global, respectively, and h, w and c in FIG. 5(a) represent the height, width and channel number of the input feature map.

[0095] The resolution size of the aggregated feature map is consistent with the resolution size of the input feature map (the output spatial feature map of the convolution attention module). Since the output spatial feature map of the convolution attention module has a small resolution and the resolution is further reduced after pooling, the calculation cost of the multi-scale pooling aggregation layer is not large.

[0096] As shown in FIG. 5(b), the feature classification layer includes 3x3 convolution operations and 1x1 convolution operations in series; H, W and c represent the height, width and channel number of the spliced feature image, L represents the number of categories, and the output feature image of the multi-scale pooling aggregation layer is restored to HxW in resolution through a simple up-sampling operation before splicing.

[0097] As shown in FIG. 5(b), the feature classification layer includes 3x3 convolution operations and 1x1 convolution operations in series; H, W and c represent the height, width and channel number of the spliced feature image, L represents the number of categories, and the output feature image of the multi-scale pooling aggregation layer is restored to HxW in resolution through a simple up-sampling operation before splicing. Figure 2 As shown in FIG. 5(b), the feature classification layer includes 3x3 convolution operations and 1x1 convolution operations in series; H, W and c represent the height, width and channel number of the spliced feature image, L represents the number of categories, and the output feature image of the multi-scale pooling aggregation layer is restored to HxW in resolution through a simple up-sampling operation before splicing.

[0098] The patch embedding layer is used for down-sampling and patching the input image to realize image compression and match the input form requirement of the transformer coding block.

[0099] Multiple cascaded Transformer coding blocks are used to sequentially encode the layered features of the compressed image, extract the layered features of the input image, and utilize the global receptive field of the Transformer to obtain feature maps of different scales with high-quality semantics.

[0100] The feature decoding module is composed of pure multilayer perceptrons. It is used to decode the multi-scale semantic feature map output by the Transformer coding block to obtain the feature map after feature decoding, and to use the decoded feature map to classify and obtain the segmentation map.

[0101] like Figure 6 As shown, the semantic alignment network in this embodiment of the invention includes: a feature alignment layer and a semantic alignment loss function;

[0102] The feature alignment layer is used to map the number of channels of feature maps of different scales output by the convolutional network to the number of channels of the feature maps output by the semantic extraction branch, and to adjust the size of feature maps of different scales output by the convolutional network to be the same as the feature maps output by the semantic extraction branch, so as to achieve semantic alignment.

[0103] Specifically, the feature map of the semantic extraction branch is restored to shape H. t ×W t ×C t Then adjust the size to H×W×C t Adjust the size of the feature map corresponding to the convolutional network to H×W×C. s Then, the number of channels is mapped to C through a 1x1 convolutional layer. t This achieves the goal of aligning the shapes of the convolutional network and semantic extraction branch feature images; where H t W t C t C represents the length, width, and number of channels of the feature map after feature map recovery in the semantic extraction branch. s This represents the number of channels corresponding to the feature map of the convolutional network.

[0104] The semantic alignment loss function is used to calculate the channel-level similarity of features after shape alignment in the feature alignment layer. The calculated similarity values ​​are then used to supervise the learning of the convolutional network parameters. During network training, the network parameters are adjusted to maximize the similarity of the probability distribution of each channel between the feature maps corresponding to the convolutional network and the semantic extraction branch at the same scale, until the loss converges.

[0105] Specifically, the semantic alignment loss function in the embodiments of the present invention The loss for channel independence is in the form of:

[0106]

[0107] wherein, φ represents an activation function, x T represents the different scale feature maps obtained by the semantic extraction branch, and specifically includes a semantic feature image, a decoding feature map and a segmentation map, x S represents the different scale feature maps corresponding to the convolutional network, and specifically includes a feature-enhanced feature map output by the convolutional attention module, a spliced feature map output by the semantic decoding module and a predicted semantic segmentation map;

[0108] The specific form of the activation function φ is: c represents a certain specific channel, i represents a specific pixel point, H and W represent the height and width of the image, exp represents an exponential function, is a hyperparameter, and the activation function φ is used to convert the feature map into a channel-independent probability distribution, x c x T or x S ;

[0109] represents the KL divergence, and the specific form is: C represents the number of channels, log represents a logarithmic function, and the KL divergence is calculated according to the values of two different activation functions φ, and the KL divergence value of two feature maps with similar channel probability distributions is smaller.

[0110] By using the semantic alignment loss function in the above form, the similarity of the semantic information of each channel of the feature map can be well measured, and the semantic high-response area can be adaptively concentrated and the influence of the low-response semantic misjudgment area can be avoided, so that the alignment effect is better than that of the spatial point-by-point semantic alignment function.

[0111] In the training process of the real-time semantic segmentation model, the loss function of the entire model includes: a semantic alignment loss and a segmentation loss of the semantic decoding module, wherein the segmentation loss of the semantic decoding module is the cross-entropy loss between the semantic segmentation map predicted by the semantic decoding module and the set semantic label.

[0112] The effective effects that can be achieved by the present application are further explained and described below in combination with some specific application scenarios:

[0113] Taking real-time semantic segmentation as the target task, the real-time semantic segmentation model constructed in the above embodiments removes the semantic extraction branch and semantic alignment network after the training process is completed, and only the convolutional network is retained. This convolutional network can not only maintain the high-quality long-distance semantic information extraction capability similar to the semantic extraction branch, but also benefit from its own local modeling capability, which can significantly improve the accuracy of real-time semantic segmentation task; at the same time, because only the convolutional network is retained, the inference latency is small.

[0114] The ADE20K dataset was used to train the real-time semantic segmentation model. The ADE20K dataset is an image segmentation dataset for complex scenes, containing 20,000 images as the training set, 2,000 images as the validation set, and 3,000 images as the test set. Each pixel is labeled with a predefined category, and the dataset has 150 predefined semantic labels. The training set was randomly divided into training subsets of equal size, specifically, each training subset was 8 pixels in size. Data augmentation was performed on each training subset to improve accuracy. The data augmentation methods included the following operations:

[0115] (1) Calculate the mean value of each channel of the images in the training set;

[0116] (2) Subtract the image mean from each image in the training subset;

[0117] (3) Randomly flip horizontally and scale randomly, with the scaling factor being any one of {0.5, 0.75, 1.0, 1.5, 1.75}.

[0118] During training, a training subset is trained each time, and the completion of training for all training subsets constitutes the end of one iteration; training is repeated until the number of iterations reaches the upper limit, resulting in a trained real-time semantic segmentation model; in actual training, the upper limit of the number of iterations is preferably 160,000.

[0119] In the iterative training described above, the training process in one iteration is as follows: The network parameters of the real-time semantic segmentation model are trained using forward propagation and backpropagation algorithms. Forward propagation calculates the loss function for each training subset, and backpropagation obtains the gradient for that training subset. The loss calculation uses the segmentation loss and semantic alignment loss from the semantic decoding module.

[0120] The trained real-time semantic segmentation model is used to perform target segmentation tasks in different scenarios. The prediction visualization results of real-time semantic segmentation in different scenarios are as follows: Figure 7 and Figure 8 As shown; among them, the results in the autonomous driving scenario are as follows: Figure 7 As shown, Figure 7 (a) in the image is the input image. Figure 7 (b) in the text represents the truth label; different shades of color represent different categories of truth values.Figure 7 Different color depths in (c)-(e) in (a) represent different category predictions, wherein, Figure 7 (c)-(d) in (a) are the segmentation prediction results of SeaFormer-B and STDC2 advanced real-time segmentation methods, respectively, Figure 7 (e) in (a) is the segmentation prediction result of the real-time semantic segmentation model in the embodiment of the application. It can be seen that, compared with the two existing real-time segmentation methods, in the prediction result of the real-time semantic segmentation model in the embodiment of the application, the large-area regions such as roads and sidewalks have more complete predictions, and the rearview mirrors of cars, power poles, signal lights, and signs have more accurate category edge predictions.

[0121] The results under the geographic information system are as shown in Figure 8 . Figure 8 (a) in (a) is an input image, Figure 8 (b) in (a) is a true value label, and different color depths represent different category true values, Figure 8 (c)-(g) in (a), different color depths represent different category predictions, Figure 8 (c)-(f) in (a) are the segmentation prediction results of SegFormer-B0, RTFormer-B, TopFormer-B, and SeaFormer-B four advanced real-time segmentation methods, respectively, Figure 8 (g) in (a) is the segmentation prediction result of the real-time semantic segmentation model in the embodiment of the application. It can be seen that, compared with other advanced real-time segmentation methods in the same field, in the prediction result of the real-time semantic segmentation model in the embodiment of the application, large-area regions such as trees, buildings, and roads have more complete predictions, and places such as curtains, pillows, chairs, and window edges have more accurate category edge predictions. Meanwhile, in the embodiment of the application, for the real-time semantic segmentation models of different model sizes established, the inference speed on a single RTX3090 can reach 105-451FPS, which is better than the other four methods in the same field.

[0122] According to another aspect of the application, a semantic alignment real-time semantic segmentation system is provided, comprising a computer readable storage medium and a processor;

[0123] The computer readable storage medium is used to store executable instructions;

[0124] The processor is used to read the executable instructions stored in the computer readable storage medium to execute each step of the semantic alignment real-time semantic segmentation method in the above embodiment.

[0125] According to another aspect of the application, a computer readable storage medium having a computer program stored thereon is provided, and the program is executed by a processor to implement the semantic alignment real-time semantic segmentation method in the above embodiment.

[0126] Overall, the application can align the semantics of high-quality semantic extraction branches during training, accurately capture long-distance dependency information, and effectively improve the accuracy of real-time segmentation tasks; in the inference process, single-branch and pure convolution operations are used, which effectively reduces the inference delay of the real-time segmentation model; the application improves the prospects for deployment of real-time semantic segmentation networks in practical applications with high segmentation accuracy and low inference delay. The application can be applied to the fields of autonomous driving, geographic information systems, video monitoring, medical image analysis, robots, mobile phone applications, etc., and can quickly and accurately perform semantic segmentation tasks.

[0127] Meanwhile, the convolution attention module designed by the application includes a convolution attention layer and a feedforward neural network layer connected in series, and the convolution attention operation in the convolution attention layer has structural consistency with the spatial attention operation in the Transformer encoding block of the semantic extraction branch, both of which enhance the long-distance dependency information of the feature map through attention operation, and then both integrate channel information through the feedforward network, which facilitates better alignment of the convolution attention module and the high-performance semantic extraction branch. Meanwhile, the convolution attention module of the application is realized by convolution, and the long-distance dependency is captured through convolution operation, which reduces the accuracy gap with the standard self-attention mechanism with smaller computational complexity, and at the same time, the efficiency is higher while realizing structural matching with the semantic extraction branch. At the same time, due to the characteristics of pure convolution operation, the performance of deployment in practical applications is improved.

[0128] The semantic alignment network designed by the application adopts a channel-based alignment method rather than pixel-by-pixel alignment, which can better align long-distance semantic information and is more suitable for segmentation tasks.

[0129] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the application and is not intended to limit the application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A method of semantic aligned real-time semantic segmentation, characterized in that, The application relates to a real-time semantic segmentation method and device. The training phase comprises: training a real-time semantic segmentation model by using a training set, wherein each sample in the training set comprises an image related to a semantic segmentation task and a corresponding label used for indicating a segmentation result; the real-time semantic segmentation model comprises a convolution network, a semantic extraction branch and a semantic alignment network. The convolution network comprises a local convolution module, a convolution attention module and a semantic decoding module. The local convolution module is used for extracting bottom features in an input sample to generate a spatial feature map; the convolution attention module is used for performing a convolution attention operation on the spatial feature map to obtain a feature-enhanced feature map; and the semantic decoding module is used for performing multi-scale pooling aggregation on the feature-enhanced feature map, splicing the spatial feature map and the feature-enhanced feature map, and performing point-by-point classification to obtain a predicted semantic segmentation map. The semantic extraction branch is used for performing hierarchical Transformer feature coding operation on an input sample to output semantic feature maps of different scales; the semantic extraction branch comprises a patch embedding layer, a plurality of serially connected Transformer coding blocks and a feature decoding module; the patch embedding layer is used for performing downsampling and patching operation on an input sample to realize image compression and match the input form requirement of the plurality of serially connected Transformer coding blocks; the plurality of serially connected Transformer coding blocks are used for performing hierarchical Transformer feature coding operation on the compressed image to obtain a semantic feature map; and the feature decoding module is used for decoding the semantic feature map to obtain a decoded feature map and classifying the decoded feature map to obtain a segmentation map. The semantic alignment network is used for shape aligning the feature-enhanced feature map, the spliced feature map and the predicted semantic segmentation map with the semantic feature maps of corresponding scales respectively, so that the convolution network captures semantic information in the semantic feature maps; the semantic alignment network comprises a feature alignment layer and a semantic alignment loss function; the feature alignment layer is used for shape aligning the feature-enhanced feature map with the semantic feature map, shape aligning the spliced feature map with the decoded feature map and shape aligning the predicted semantic segmentation map with the decoded feature map to obtain a segmentation map; and the semantic alignment loss function is used for calculating the similarity of feature calculation channel levels of the features after shape alignment of the feature alignment layer and supervising parameter learning of the convolution network based on the similarity. The application phase comprises: inputting an image to be processed into the trained convolution network to obtain a segmentation result. The convolution attention module comprises serially connected convolution attention layers and a feedforward neural network layer.

2. The method of claim 1, wherein, The convolution attention layer is used for calculating a feature similarity matrix of the spatial feature map and a learnable kernel independent of the input sample image, performing convolution on the similarity matrix by using the learnable kernel to obtain a feature map with long-distance dependence information of spatial dimensions enhanced, and integrating the dependence information in different channels of the feature map by using the feedforward neural network layer to obtain the feature-enhanced feature map. The convolution attention layer is used for calculating a feature similarity matrix of the spatial feature map and a learnable kernel independent of the input sample image, performing convolution on the similarity matrix by using the learnable kernel to obtain a feature map with long-distance dependence information of spatial dimensions enhanced, and integrating the dependence information in different channels of the feature map by using the feedforward neural network layer to obtain the feature-enhanced feature map. ​ 3. The method of claim 2, wherein, The convolution attention layer comprises two sets of complementary convolution attention operations, a residual connection, and a normalization operation; wherein the convolutional attention operation is represented as , X represents the spatial feature map, K represents the learnable kernel, represents a double normalization operation, represents a transpose of K; the learnable kernels in the two sets of convolutional attention operations are respectively used to capture features of longitudinal and transverse modes in the feature map, so that the two sets of convolutional attention operations form a complementary structure; The residual connection is configured to add outputs of the two sets of complementary convolution attention operations to the spatial feature map, and the normalization operation is configured to obtain a feature map with enhanced spatial dimension long-range dependency information.

4. The method of claim 1, wherein, The semantic alignment loss function is: wherein, denotes an activation function, denotes a KL divergence, denotes different scale semantic feature maps output by the semantic extraction branch, denotes different scale feature maps corresponding to the convolutional network; Activation function is: , c represents a certain specific channel, i is a specific pixel point, H and W represent the height and width of the image, is a hyperparameter, take or .

5. The method of claim 1, wherein, The loss function in the training process comprises the semantic alignment loss function and a segmentation loss of the semantic decoding module; the segmentation loss of the semantic decoding module is a cross-entropy loss between the predicted semantic segmentation map and a label.

6. The method of claim 1, wherein, The semantic decoding module comprises a multi-scale pooling aggregation layer and a feature classification layer. The multi-scale pooling aggregation layer is configured to perform multi-scale pooling aggregation on the feature-enhanced feature map, so that a size of an aggregated feature map is consistent with a size of the spatial feature map. The feature classification layer is configured to perform a splicing operation on the aggregated feature map and the spatial feature map to obtain a spliced feature map, and perform point-by-point classification to obtain a predicted semantic segmentation map.

7. A real-time semantic segmentation system with semantic alignment, characterized in that, The computer readable storage medium and the processor are included; The computer readable storage medium is configured to store executable instructions; The processor is configured to read the executable instructions stored in the computer readable storage medium to execute the method of any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-6.

Images