A multi-path real-time semantic segmentation method and system

Abstract

The present invention provides a multi-path real-time semantic segmentation method and system, wherein the method comprises: obtaining an image data set; constructing a multi-path model, wherein the multi-path model comprises a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path; using the image data set to train and test the multi-path model to obtain the multi-path real-time semantic segmentation model; obtaining a picture to be segmented; inputting the test picture into the multi-path real-time semantic segmentation model, wherein the picture to be segmented is processed through the spatial information extraction path to obtain a first feature map, the picture to be segmented is processed through the first attention and feature fusion path to obtain a second feature map, the picture to be segmented is processed through the second attention and feature fusion path to obtain a third feature map, and the first, second and third feature maps are feature fused to obtain a real-time semantic segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field related to deep learning and image processing, and in particular to a multi-path real-time semantic segmentation method and system. Background Art

[0002] Semantic segmentation is a technology that classifies images at the pixel level. Each pixel in the image is assigned to a corresponding category, and different categories are usually marked with different colors. Semantic segmentation technology has been widely used in autonomous driving, medical image analysis, geographic information systems and other fields.

[0003] The current mainstream semantic segmentation technology is a fully convolutional neural network based on an encoder and a decoder. The main function of the encoder is to encode the original input image, and obtain a feature map with a higher semantic level and a larger receptive field through a series of continuous convolution and pooling operations. The decoder then gradually restores the abstract feature map to the original input image size, and the final output will be the semantic segmentation map of the original input RGB image.

[0004] However, the inventors of this application found that the above technology has at least the following technical problems in the process of implementing the technical solution of the invention in the embodiment of this application:

[0005] In order to obtain higher segmentation accuracy, the fully convolutional neural network in the existing technology will perform a series of cropping, convolution, and pooling operations on the input image, which will lose the original spatial information of the image. In addition, the large number of superpositions of convolutional layers will introduce a large number of weight parameters, which will greatly increase the computational overhead of the model, resulting in the model's training and reasoning speed far from reaching the real-time level, and there are technical problems such as low segmentation accuracy, loss of original image information, and low segmentation efficiency. Summary of the invention

[0006] The embodiment of the present application provides a multi-path real-time semantic segmentation method and system, which is used to solve the problem that the fully convolutional neural network in the prior art will perform a series of cropping, convolution, and pooling operations on the input image to obtain higher segmentation accuracy, which will lose the original spatial information of the image. In addition, a large number of superpositions of convolutional layers will introduce a large number of weight parameters, which will greatly increase the computational overhead of the model, resulting in the training and reasoning speed of the model far from reaching the real-time level, and there are technical problems such as low segmentation accuracy, loss of original image information and low segmentation efficiency.

[0007] In view of the above problems, an embodiment of the present application provides a multi-path real-time semantic segmentation method and system.

[0008] In a first aspect of an embodiment of the present application, a multi-path real-time semantic segmentation method is provided, the method comprising: obtaining an image data set; constructing a multi-path model, wherein the multi-path model comprises a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path; using the image data set to train and test the multi-path model to obtain a multi-path real-time semantic segmentation model; obtaining a picture to be segmented; inputting the picture to be segmented into the multi-path real-time semantic segmentation model, wherein the picture to be segmented is processed through the spatial information extraction path to obtain a first feature map, the picture to be segmented is processed through the first attention and feature fusion path to obtain a second feature map, the picture to be segmented is processed through the second attention and feature fusion path to obtain a third feature map, the first, second, and third feature maps are input into an FFM fusion layer for feature fusion to obtain an output result, and the output result is a real-time semantic segmentation result of the picture to be segmented.

[0009] According to a second aspect of an embodiment of the present application, a multi-path real-time semantic segmentation system is provided, wherein the system includes: a first obtaining unit, the first obtaining unit is used to obtain an image data set; a first constructing unit, the first constructing unit is used to construct a multi-path model, wherein the multi-path model includes a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path; a first processing unit, the first processing unit is used to train and test the multi-path model using the image data set to obtain a multi-path real-time semantic segmentation model; a second obtaining unit, the second obtaining unit is used to obtain a picture to be segmented; a second processing unit, the second processing unit is used to input the picture to be segmented into the multi-path real-time semantic segmentation model, wherein the picture to be segmented is processed through the spatial information extraction path to obtain a first feature map, the picture to be segmented is processed through the first attention and feature fusion path to obtain a second feature map, the picture to be segmented is processed through the second attention and feature fusion path to obtain a third feature map, the first, second, and third feature maps are input into an FFM fusion layer for feature fusion to obtain an output result, and the output result is a real-time semantic segmentation result of the picture to be segmented.

[0010] A third aspect of an embodiment of the present application provides a multi-path real-time semantic segmentation system, comprising: a processor, the processor is coupled to a memory, the memory is used to store a program, when the program is executed by the processor, the system executes the steps of the method described in the first aspect.

[0011] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0012] The real-time semantic segmentation method provided by the embodiment of the present application includes multiple processing paths, and the multiple processing paths include a spatial information extraction path based on deep separable convolution, a first attention and feature fusion path, and a second attention and feature fusion path. The first feature map can be obtained through the spatial information extraction path based on deep separable convolution, and the second feature map and the third feature map can be obtained respectively through the first attention and feature fusion path and the second attention and feature fusion path. The first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain the real-time semantic segmentation result. The embodiment of the present application adopts a spatial information extraction path based on deep separable convolution. In the process of extracting the feature map, there is no need to significantly crop the original image, and a large amount of spatial information can be retained. It can also greatly reduce the amount of weight parameters involved in the calculation during the training and reasoning process, making the calculation cost of the model smaller; based on multiple attention and feature fusion paths, while obtaining the maximum receptive field, it provides a feature screening function, and can purposefully learn the features that are most helpful to the segmentation accuracy when building the model, which greatly increases the overall segmentation accuracy and robustness of the model, and achieves the technical effect of reducing the computational cost of semantic segmentation, improving computational efficiency, and improving segmentation accuracy.

[0013] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of a multi-path real-time semantic segmentation method provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of the spatial information extraction path structure in a multi-path real-time semantic segmentation method provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of the structure of attention and feature fusion paths in a multi-path real-time semantic segmentation method provided in an embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of a multi-path real-time semantic segmentation model in a multi-path real-time semantic segmentation method provided in an embodiment of the present application;

[0018] Figure 5 A schematic diagram of the structure of a multi-path real-time semantic segmentation system is provided for an embodiment of the present application;

[0019] Figure 6 It is a schematic diagram of the structure of an exemplary electronic device according to an embodiment of the present application.

[0020] Explanation of the reference numerals: first obtaining unit 11 , first constructing unit 12 , first processing unit 13 , second obtaining unit 14 , second processing unit 15 , electronic device 300 , memory 301 , processor 302 , communication interface 303 , bus architecture 304 . DETAILED DESCRIPTION

[0021] The embodiment of the present application provides a multi-path real-time semantic segmentation method and system, which is used to solve the problem that the fully convolutional neural network in the prior art will perform a series of cropping, convolution, and pooling operations on the input image to obtain higher segmentation accuracy, which will lose the original spatial information of the image. In addition, a large number of superpositions of convolutional layers will introduce a large number of weight parameters, which will greatly increase the computational overhead of the model, resulting in the training and reasoning speed of the model far from reaching the real-time level, and there are technical problems such as low segmentation accuracy, loss of original image information and low segmentation efficiency.

[0022] The real-time semantic segmentation method provided by the embodiment of the present application includes multiple processing paths, and the multiple processing paths include a spatial information extraction path based on deep separable convolution, a first attention and feature fusion path, and a second attention and feature fusion path. The first feature map can be obtained through the spatial information extraction path based on deep separable convolution, and the second feature map and the third feature map can be obtained respectively through the first attention and feature fusion path and the second attention and feature fusion path. The first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain the real-time semantic segmentation result. The embodiment of the present application adopts a spatial information extraction path based on deep separable convolution. In the process of extracting the feature map, there is no need to significantly crop the original image, and a large amount of spatial information can be retained. It can also greatly reduce the amount of weight parameters involved in the calculation during the training and reasoning process, making the calculation cost of the model smaller; based on multiple attention and feature fusion paths, while obtaining the maximum receptive field, it provides a feature screening function, and can purposefully learn the features that are most helpful to the segmentation accuracy when building the model, which greatly increases the overall segmentation accuracy and robustness of the model, and achieves the technical effect of reducing the computational cost of semantic segmentation, improving computational efficiency, and improving segmentation accuracy.

[0023] Application Overview

[0024] Semantic segmentation is a technique for pixel-level classification of images. Each pixel in the image is assigned to a corresponding category, and different categories are usually marked with different colors. Semantic segmentation technology has been widely used in fields such as autonomous driving, medical image analysis, and geographic information systems. The current mainstream semantic segmentation technology is a fully convolutional neural network based on an encoder and decoder. The main function of the encoder is to encode the original input image, and obtain a feature map with a higher semantic level and a larger receptive field through a series of continuous convolution and pooling operations. The decoder then gradually restores the abstract feature map to the original input image size, and the final output will be the semantic segmentation map of the original input RGB image. There are many possibilities for the specific design of the encoder and decoder, among which the U-shaped network structure has been widely studied and applied in the industry. That is, a network structure similar to the U-shape is designed, the original image is input from one side, the semantic map is output from the other side, and feature extraction and feature fusion are completed in the middle of the U-shape.

[0025] Most U-shaped networks of fully convolutional neural networks in the prior art seek to obtain a larger receptive field in order to achieve higher segmentation accuracy. A series of cropping, convolution, and pooling operations on the input image will lose the original spatial information of the image. In order to obtain higher segmentation accuracy, a series of cropping, convolution, and pooling operations will be performed on the input image, which will lose the original spatial information of the image. In addition, a large number of superpositions of convolutional layers will introduce a large number of weight parameters, which will greatly increase the computational overhead of the model, resulting in the training and reasoning speed of the model far from reaching the real-time level. A few excellent models will take into account both the segmentation accuracy and reasoning speed of the model, but such models often do not have strong generalization capabilities, and the performance fluctuates too much on different data sets. There are technical problems in the prior art such as low segmentation accuracy, loss of original image information, and low segmentation efficiency.

[0026] In response to the above technical problems, the overall idea of ​​the technical solution provided by this application is as follows:

[0027] An embodiment of the present application provides a multi-path real-time semantic segmentation method, the method comprising: obtaining an image data set; constructing a multi-path model, wherein the multi-path model comprises a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path; using the image data set to train and test the multi-path model to obtain a multi-path real-time semantic segmentation model; obtaining a picture to be segmented; inputting the picture to be segmented into the multi-path real-time semantic segmentation model, wherein the picture to be segmented is processed through the spatial information extraction path to obtain a first feature map, the picture to be segmented is processed through the first attention and feature fusion path to obtain a second feature map, the picture to be segmented is processed through the second attention and feature fusion path to obtain a third feature map, the first, second, and third feature maps are input into an FFM fusion layer for feature fusion to obtain an output result, and the output result is a real-time semantic segmentation result of the picture to be segmented.

[0028] After introducing the basic principles of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described here. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0029] Embodiment 1

[0030] like Figure 1 As shown, an embodiment of the present application provides a multi-path real-time semantic segmentation method, the method comprising:

[0031] S100: obtaining an image data set;

[0032] Specifically, the image data set is a historical image data set for training and testing the multi-path model in the embodiment of the present application, that is, the image data for constructing the multi-path real-time semantic segmentation model. Before training and testing, the image data set is divided into a training data set and a test data set. The training data set is used to train the above-mentioned multi-path real-time semantic segmentation model, and the test data set is used to cross-validate the model to complete hyperparameter tuning. The test data set and the training data set should maintain the same processing logic. Preferably, before using the image processing set for training and testing, the image processing set is preprocessed, wherein the preprocessing includes horizontally and vertically random scale cropping of the original input image data set, setting random numbers to randomly flip the image, and randomly changing the brightness of the image. Finally, these processed data sets are uniformly output as a fixed size, and then the preprocessed image data set is divided into a training data set and a test data set, and the multi-path model is trained and tested respectively.

[0033] To give an example without limitation, in the field of autonomous driving, it is necessary to perform semantic segmentation on traffic road image data. The image dataset is a traffic road map dataset collected by cameras and / or cameras, where the main objects (Object) include: roads, traffic lights, various vehicles, buildings, pedestrians, sky, and nearly hundreds of categories. By using the image dataset to train a multi-path real-time semantic segmentation model, the image data with the above objects can be semantically segmented.

[0034] S200: constructing a multi-path model, wherein the multi-path model includes a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path;

[0035] Specifically, the multi-path model includes a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path.

[0036] Among them, the spatial information extraction path in the embodiment of the present application is based on deep separable convolution for spatial information extraction. The principle of deep separable convolution is different from the traditional convolution of semantic segmentation in the prior art. It can effectively reduce the amount of weight parameters introduced in the convolution process, thereby reducing the computational overhead of the model, reducing the memory occupied by the model, and improving the convolution efficiency.

[0037] The first attention and feature fusion path and the second attention and feature fusion path are both extraction paths that combine the attention mechanism with the feature fusion mechanism. The attention mechanism is a technology used to extract more valuable feature layers and filter feature layers that are not very helpful for prediction tasks. It can improve the robustness and reasoning accuracy of the model while reducing some computational overhead. The feature fusion mechanism is a technology that improves the semantic richness of feature maps. Features at different levels are preprocessed and then fused to obtain semantically rich feature maps.

[0038] S300: Using the image data set to train and test the multi-path model to obtain a trained multi-path real-time semantic segmentation model;

[0039] Specifically, the above-mentioned image dataset is used to train and test the multi-path model. After the multi-path model is trained based on the training dataset in the image dataset, the test dataset in the image dataset is used to cross-validate the multi-path model to test the segmentation performance of the multi-path model. After verification that it meets the requirements, it can be used as a multi-path real-time semantic segmentation model.

[0040] S400: Obtaining a picture to be segmented;

[0041] Specifically, the image to be segmented is any image data that needs to be semantically segmented in real time. For example, in the field of autonomous driving, the image to be segmented is traffic road map data.

[0042] S500: Input the picture to be segmented into the multi-path real-time semantic segmentation model, wherein the picture to be segmented is processed through the spatial information extraction path to obtain a first feature map, the picture to be segmented is processed through the first attention and feature fusion path to obtain a second feature map, the picture to be segmented is processed through the second attention and feature fusion path to obtain a third feature map, the first, second and third feature maps are input into the FFM fusion layer for feature fusion to obtain an output result, and the output result is a real-time semantic segmentation result of the picture to be segmented.

[0043] Specifically, the image to be segmented, that is, the image that needs semantic segmentation, is input into a multi-path real-time semantic segmentation model that has been trained and tested. Based on the spatial information extraction path of deep separable convolution, a feature map with a weight parameter calculation amount equivalent to one tenth of the traditional convolution calculation amount can be obtained, which can effectively reduce the calculation amount of semantic segmentation, and will not drastically crop the original image, and can retain a large amount of spatial information.

[0044] Since the above-mentioned spatial information extraction path based on depthwise separable convolution has a small amount of computation, the computational efficiency is greatly improved, but at the same time a small part of the accuracy is lost. In the embodiment of the present application, the first attention and feature fusion path and the second attention and feature fusion path are combined to process the segmented image, and the first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain the output result, thereby improving the accuracy of the output result and the robustness of the model.

[0045] In addition, for the training data set and test data set of the multi-path real-time semantic segmentation model, the training data set and the test data set should be preprocessed in the same process as the training data set and the test data set to ensure the consistency of the data processing logic. For example, the test data set and the training data set are cropped horizontally and vertically at random scales, and the random numbers are set to randomly flip the pictures, and the brightness of the pictures is randomly changed. Finally, these processed data sets are uniformly output as a fixed size. Then the pictures to be segmented must also be cropped horizontally and vertically at random scales, and random numbers are set to randomly flip the pictures, and the brightness of the pictures is randomly changed. Finally, these processed data sets are uniformly output as a fixed size. This is the preprocessing process.

[0046] The embodiment of the present application adopts a spatial information extraction path based on deep separable convolution. In the process of extracting feature maps, there is no need to significantly crop the original image, so a large amount of spatial information can be retained, and the amount of weight parameters involved in the calculation during training and inference can be greatly reduced, making the computational overhead of the model smaller; based on multiple attention and feature fusion paths, while obtaining the maximum receptive field, a feature screening function is provided. When building a model, the features that are most helpful for segmentation accuracy can be purposefully learned, which greatly increases the overall segmentation accuracy and robustness of the model, achieving the technical effects of reducing the computational cost of semantic segmentation, improving computational efficiency, and improving segmentation accuracy.

[0047] like Figure 2 As shown, in step S200 of the embodiment of the present application, “processing the image to be segmented through the spatial information extraction path to obtain a first feature map” includes step S210, and step S210 includes:

[0048] S211: Input the image to be segmented into the front-end convolution block for downsampling to obtain a first convolution output;

[0049] S212: Inputting the first convolution output into a first depthwise separable convolution layer for downsampling to obtain a second convolution output;

[0050] S213: Inputting the second convolution output into a second depth-wise separable convolution layer for downsampling to obtain a third convolution output;

[0051] S214: Input the third convolution output into the back-end convolution block to perform cross-channel information fusion to obtain the first feature map.

[0052] Specifically, the spatial information extraction path based on depthwise separable convolution includes two consecutive depthwise separable convolution layers and a front-end convolution block and a back-end convolution block arranged at the front end and the back end of the two depthwise separable convolution layers, wherein the front-end convolution block and the back-end convolution block are both traditional convolution blocks.

[0053] After inputting the image to be segmented, Figure 2 As shown, the front-end convolution block samples the image to be processed to implement the first Convolution Block operation. A single Convolution Block includes two steps: convolution operation and rule activation. The image to be processed is sampled through the front-end convolution block to obtain the first convolution output.

[0054] After obtaining the first convolution output, the first convolution output is input into the continuous first depth-separable convolution layer and the second depth-separable convolution layer, and convolution sampling is performed in sequence to obtain continuous second convolution output and third convolution output. Finally, the third convolution output is input into the back-end convolution block for cross-channel information fusion to obtain the first feature map.

[0055] The stride of the front-end convolution block is 2, and it uses a 3x3 convolution kernel for 2x downsampling. The stride of the back-end convolution block is 1, and it uses a 1x1 convolution kernel for cross-channel information interaction, ensuring that the output of this path is not affected by the reduction of weight parameters.

[0056] The principle of depthwise separable convolution on which the first depthwise separable convolution layer and the second depthwise separable convolution layer are based is as follows: a 3x3 spatial convolution kernel is applied to the feature map output by the first convolution, where the number of channels of the spatial convolution kernel is equal to the number of layers of the feature map, and then multiple groups of 1x1 convolution kernels are used to fuse multiple feature layers and generate multiple new feature layers, where the number of layers of the new feature map is equal to the number of 1x1 convolution kernels.

[0057] The step sizes of the first depth-separable convolutional layer and the second depth-separable convolutional layer are both 2. With the above-mentioned front-end convolutional block and back-end convolutional block, the input image to be segmented is Figure 1 A total of 3 downsamplings will be performed to obtain a feature map of 1 / 8 size. Since the output size is not a large crop of the original image, a large amount of spatial information can be retained.

[0058] After outputting the first feature map, the ability of the spatial information extraction path based on depthwise separable convolution to reduce the amount of weight parameters compared to traditional convolution can be quantitatively analyzed, including:

[0059] Calculation of weight parameters introduced by traditional convolution:

[0060] Convolution kernel size: kxk xc (size is k, number of channels is c).

[0061] Number of convolution kernels: n.

[0062] The number of parameters introduced by convolution: k×k×c×n=k^2nc.

[0063] The weight parameter amount calculation introduced by the depthwise separable convolution in the embodiment of the present application is:

[0064] Convolution kernel size: kxkxc (size is k, number of channels is c).

[0065] Number of convolution kernels: n.

[0066] The number of parameters introduced by convolution:

[0067] (k×k×c)+(1×1×c×n)=(k^2+n)×c

[0068] The weight parameter amount introduced in the spatial information extraction path in the embodiment of the present application (assuming that the kernel size is always k):

[0069] The image with c channels is first processed by the first convolution block into a feature map with n1 channels, and the weight parameter introduced is k. 2 n1c;

[0070] The feature map with n1 channels is processed into a feature map with n2 channels by the first depth-wise separable convolution. The weight parameter introduced is (k 2 +n2)n1;

[0071] The feature map with n2 channels is processed into a feature map with n3 channels by the second depth-wise separable convolution. The weight parameter introduced is (k 2 +n3)n2;

[0072] The feature map with n3 channels is processed into a feature map with n4 channels by the second convolution block, and the weight parameter introduced is k. 2 n4n3;

[0073] The weight parameter introduced by the entire path is: k 2 n1c+(k 2 +n2)n1+(k 2 +n3)n2+k 2 n4n3;

[0074] The amount of weight parameters introduced by using traditional convolution to complete the same operation is: k 2 n1c+k 2 n2n1+k 2 n3n2+k 2 n4n3;

[0075] The weight parameter reduction brought by this path is: 2 n2n1+k 2 n3n2-(k 2 +n2)n1-(k 2 +n3)n2.

[0076] In summary, the spatial information extraction path in the embodiment of the present application adopts a deep separable convolution, combined with the front-end convolution block and the back-end convolution block, which can effectively reduce the weight parameters of the above-mentioned weight parameter reduction amount compared to the traditional convolution, and can greatly reduce the amount of weight parameters involved in the calculation during the training and reasoning process. For lightweight models, the parameter amount can also be reduced by up to dozens of times. This makes the computational overhead of the model smaller. On the one hand, the memory occupied by the model will be reduced, and on the other hand, it also greatly empowers the reasoning speed. In addition, in order to avoid the problem of weakening the model learning ability due to a significant decrease in the amount of weight parameters, the spatial information extraction path is connected to traditional convolution blocks at both ends of the deep separable convolution, which can ensure that the model learning ability is not affected while the parameter amount is greatly reduced.

[0077] In the method provided in the embodiment of the present application, the step S200 of “processing the image to be segmented through the first attention and feature fusion path to obtain a second feature map” includes step S220, and step S220 includes:

[0078] S221: Input the image to be segmented into the attention channel, calculate the feature importance vector of the image to be segmented, and obtain the feature importance vector;

[0079] S222: performing feature layer division based on the feature importance vector to obtain a main input feature layer and a fusion input feature layer;

[0080] S223: Input the trunk input feature layer into the trunk road for processing, use the feature importance vector as a weight to calculate a new feature layer, and obtain a trunk road output result;

[0081] S224: Inputting the fused input feature layer into a feature fusion channel, fusing the fused input feature layer with the feature layer weighted by the feature importance vector, and obtaining a fusion channel output result;

[0082] S225: Perform a fusion operation on the main road output result and the fusion channel output result to obtain the second feature map.

[0083] Specifically, if Figure 3 As shown in the figure, the image to be segmented is input as the attention channel of the main channel, and the feature importance vector is calculated to obtain the feature importance vector. Each value in the feature importance vector is used as the basis for dividing the feature layer, and then the trunk input feature layer and the fusion input feature layer are divided. The trunk input feature layer and the fusion input feature layer are respectively input into the main channel and feature fusion channel for processing.

[0084] Each value in the feature importance vector is used as the weight coefficient of each feature layer and multiplied by each feature layer respectively, and a new feature word is calculated to obtain a new feature layer. It can be considered that the values ​​of more important features in the new feature layer will be relatively large, while the values ​​of less important features will be smaller or even close to 0. Then, the main road output results and the fusion channel output results are obtained respectively, and finally the main road output results and the fusion channel output results are fused to achieve feature fusion and offset the uncertainty of the attention mechanism to obtain the second feature map.

[0085] The attention and feature fusion path in the embodiment of the present application provides a function similar to feature screening while obtaining the maximum receptive field. It can purposefully learn the features that are most helpful to the segmentation accuracy, and ignore or not learn those interfering features. The feature importance vector is automatically calculated based on the existing feature layer, reducing the intervention of manual feature engineering. Since feature fusion is performed on the basis of the attention mechanism, the overall output of the module is a feature without too many biases. The combination of the first attention and feature fusion path and the second attention and feature fusion path has greatly increased the overall segmentation accuracy and robustness of the model.

[0086] Step S221 in the embodiment of the present application includes:

[0087] S2211: Inputting the image to be segmented into the third convolution block for feature extraction to obtain a fourth convolution output;

[0088] S2212: Calculate the average importance of the fourth convolution output through a global average pooling layer to obtain the feature importance vector of each feature layer, and perform numerical mapping on the feature importance vector, where the numerical mapping interval is (0, 1).

[0089] Specifically, the attention mechanism in the embodiment of the present application is implemented by designing a channel that can generate a feature importance vector. The image to be segmented first passes through the third convolution block in the channel to extract more abstract features, obtain the fourth convolution output, and the number of channels does not change. Then, a global average pooling layer is used to calculate the average importance of the fourth convolution output, and the average importance of each feature layer is calculated. A single feature layer will get a scalar, so after performing the pooling operation of all feature layers, a feature importance vector corresponding to each feature layer will be obtained, and finally the values ​​in the vector are mapped to the (0,1) interval.

[0090] The purpose of introducing the feature fusion mechanism in the embodiment of the present application is to offset the uncertainty that may exist in the attention mechanism. The feature fusion mechanism is realized by connecting another input terminal, the input of which is a feature layer with a relatively low degree of abstraction, which contains a large number of low-order features, including some features with very small values ​​(low importance) in the feature importance vector. Fusion of these features with features weighted by the importance vector can obtain richer semantic information than the features simply screened by the attention mechanism.

[0091] The embodiment of the present application combines the attention mechanism and the feature fusion mechanism. The attention mechanism can enable the model to purposefully improve the segmentation accuracy and perform semantic segmentation. The feature fusion mechanism can ensure that the semantic segmentation output does not have too many biased features, which can effectively improve the accuracy of the semantic segmentation results output by the multi-path real-time semantic segmentation model.

[0092] Furthermore, the first attention and feature fusion path is not exactly the same as the second attention and feature fusion path, but the path structure is consistent. The convolution kernel size, number of channels, and weight initialization method used by the two are different. Therefore, the second feature map and the third feature map obtained by the first attention and feature fusion path and the second attention and feature fusion path are not exactly the same. By combining two different attention and feature fusion paths for image processing and finally fusing them with the first feature map to obtain the output result, the fusion of the semantic feature maps with differences can greatly increase the overall segmentation accuracy and robustness of the real-time semantic segmentation model.

[0093] In the method provided in the embodiment of the present application, the step S200 of “inputting the first, second and third feature maps into the FFM fusion layer for feature fusion to obtain an output result, wherein the output result is a real-time semantic segmentation result of the image to be segmented” includes step S230, and step S230 includes:

[0094] S231: Input the first, second and third feature maps into the FFM fusion layer for feature fusion to obtain a feature map fusion result;

[0095] S232: Upsample and restore the feature map fusion result to obtain the output result.

[0096] Specifically, if Figure 4 As shown, the FFM fusion layer is a feature fusion module. In the work of deep learning such as semantic segmentation, the fusion of features of different scales can effectively improve the accuracy of the final output result and the stability of the model. In the embodiment of the present application, the weight parameter amount of the spatial information extraction path based on deep separable convolution is greatly reduced compared with the traditional convolution, but the accuracy is also lost to a certain extent. Combining it with the output results of the attention and feature fusion path with purposeful semantic segmentation can improve the accuracy of semantic segmentation while reducing the amount of model calculation.

[0097] After the first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain the feature map fusion result, they are upsampled three times by a factor of 2 to restore to the original input size to complete the final output result and semantic segmentation.

[0098] The following describes the application of the method provided in the embodiment of the present application in an actual scenario to enable a clearer understanding of the technical solution of the present application, but it is not intended to limit the embodiment of the present application.

[0099] There is a batch of traffic road map data collected by cameras, in which the main objects include roads, traffic lights, various vehicles, buildings, pedestrians, sky, etc. There are nearly hundreds of categories. It is necessary to train a semantic segmentation model to automatically perform fine object-level segmentation on these images and apply them to autonomous driving scenarios.

[0100] 1. Use a multi-path real-time semantic segmentation method based on attention and feature fusion mechanism to perform this task. First, divide the data into training set, validation set, and test set according to a certain ratio. Then train a multi-path real-time semantic segmentation model based on attention and feature fusion mechanism on the training set, and perform cross-validation on the validation set to complete hyperparameter tuning.

[0101] 2. The segmentation performance of the model was tested on the test set, mainly in four aspects: 1) inference speed (fps); 2) overall segmentation accuracy (Mean Intersection over Union, mIou); 3) object-level segmentation accuracy (Intersection over Union, Iou); 4) model parameter quantity. Repeated test tuning ensures that the inference speed reaches the real-time (60fps) level or above, the overall segmentation accuracy and object-level segmentation accuracy of the model are equal to or slightly higher than other semantic segmentation models currently on the market, and the model parameter quantity is reduced by more than 10 times compared with most semantic segmentation models.

[0102] 3. This model can play a good role in one of the key issues in the field of autonomous driving: road scene understanding. Combining the model's real-time and efficient semantic segmentation capabilities with target detection methods, the contours and position information of all target objects on the road can be obtained in real time. This information can be used to automatically perform semantic analysis and understanding of the scene, and with the coordinated cooperation of autonomous driving technologies such as path planning, autonomous driving tasks can be well performed.

[0103] In summary, the real-time semantic segmentation method provided by the embodiment of the present application includes multiple processing paths, including a spatial information extraction path based on deep separable convolution, a first attention and feature fusion path, and a second attention and feature fusion path. The first feature map can be obtained through the spatial information extraction path based on deep separable convolution, and the second feature map and the third feature map can be obtained through the first attention and feature fusion path and the second attention and feature fusion path, respectively. The first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain real-time semantic segmentation results. The embodiment of the present application produces the following technical effects:

[0104] 1. Using depthwise separable convolution to replace traditional convolution to extract spatial information can greatly reduce the amount of weight parameters involved in the calculation during training and reasoning, and can reduce the number of parameters by up to dozens of times for lightweight models. This makes the model's computational overhead smaller. On the one hand, the memory occupied by the model will be reduced, and on the other hand, it will greatly increase the speed of reasoning. In addition, in order to avoid the problem of weakening the model's learning ability due to a significant decrease in the number of weight parameters, the spatial information extraction path is connected to traditional convolution blocks at both ends of the depthwise separable convolution, which can ensure that the model's learning ability is not affected while the number of parameters is greatly reduced.

[0105] 2. While obtaining the maximum receptive field, the attention and feature fusion mechanism provides a function similar to feature screening, which can purposefully learn the features that are most helpful for segmentation accuracy and ignore or not learn those interfering features. The feature importance vector is automatically calculated based on the existing feature layer, reducing the intervention of manual feature engineering. Since feature fusion is performed on the basis of the attention mechanism, the overall output of the module is a feature without too many biases. The combination of multiple attention and feature fusion paths greatly increases the overall segmentation accuracy and robustness of the model.

[0106] Embodiment 2

[0107] Based on the same inventive concept as the multi-path real-time semantic segmentation method in the aforementioned embodiment, Figure 5 As shown, an embodiment of the present application provides a multi-path real-time semantic segmentation system, wherein the system includes:

[0108] A first obtaining unit 11, wherein the first obtaining unit 11 is used to obtain a processed image data set;

[0109] A first construction unit 12, wherein the first construction unit 12 is used to construct a multi-path model, wherein the multi-path model includes a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path;

[0110] A first processing unit 13, the first processing unit 13 is used to train and test the multi-path model using the image data set to obtain a multi-path real-time semantic segmentation model;

[0111] A second obtaining unit 14, the second obtaining unit 14 is used to obtain a test data set of pictures to be segmented;

[0112] The second processing unit 15 is used to input the test image into the multi-path real-time semantic segmentation model, wherein the test image is processed through the spatial information extraction path to obtain a first feature map, the test image is processed through the first attention and feature fusion path to obtain a second feature map, the test image is processed through the second attention and feature fusion path to obtain a third feature map, the first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain an output result, and the output result is the real-time semantic segmentation result of the test image.

[0113] Furthermore, the system also includes:

[0114] A third processing unit, wherein the third processing unit is used to input the image to be segmented into the front-end convolution block for down-sampling to obtain a first convolution output;

[0115] a fourth processing unit, configured to input the first convolution output into a first depthwise separable convolution layer for downsampling to obtain a second convolution output;

[0116] a fifth processing unit, configured to input the second convolution output into a second depthwise separable convolution layer for downsampling to obtain a third convolution output;

[0117] The sixth processing unit is used to input the third convolution output into the back-end convolution block for cross-channel information fusion to obtain the first feature map.

[0118] Furthermore, the system also includes:

[0119] a seventh processing unit, wherein the seventh processing unit is used to input the image to be segmented into an attention channel, calculate a feature importance vector for the image to be segmented, and obtain a feature importance vector;

[0120] an eighth processing unit, the eighth processing unit being used to perform feature layer division based on the feature importance vector to obtain a trunk input feature layer and a fusion input feature layer;

[0121] A ninth processing unit, the ninth processing unit is used to input the trunk input feature layer into the trunk road for processing, use the feature importance vector as a weight to calculate a new feature layer, and obtain a trunk road output result;

[0122] a tenth processing unit, the tenth processing unit being configured to input the fused input feature layer into a feature fusion channel, fuse the fused input feature layer with the feature layer weighted by the feature importance vector, and obtain a fusion channel output result;

[0123] The eleventh processing unit is used to perform a fusion operation on the main road output result and the fusion channel output result to obtain the second feature map.

[0124] Furthermore, the system also includes:

[0125] A twelfth processing unit, the twelfth processing unit is used to input the image to be segmented into the third convolution block for feature extraction to obtain a fourth convolution output;

[0126] A thirteenth processing unit, the thirteenth processing unit is used to calculate the average importance of the fourth convolution output through a global average pooling layer, obtain the feature importance vector of each feature layer, and numerically map the feature importance vector, and the numerical mapping interval is (0, 1).

[0127] Furthermore, the system also includes:

[0128] A fourteenth processing unit, the fourteenth processing unit is used to input the first, second and third feature maps into the FFM fusion layer for feature fusion to obtain a feature map fusion result;

[0129] The fifteenth processing unit is used to upsample and restore the feature map fusion result to obtain the output result.

[0130] Exemplary Electronic Devices

[0131] Reference below Figure 6 To describe the electronic device of the embodiment of the present application,

[0132] Based on the same inventive concept as a multi-path real-time semantic segmentation method in the aforementioned embodiment, an embodiment of the present application also provides a multi-path real-time semantic segmentation system, including: a processor, the processor is coupled to a memory, the memory is used to store a program, when the program is executed by the processor, the system executes the steps of the method described in embodiment one.

[0133] The electronic device 300 includes: a processor 302, a communication interface 303, and a memory 301. Optionally, the electronic device 300 may also include a bus architecture 304. The communication interface 303, the processor 302, and the memory 301 may be interconnected via the bus architecture 304; the bus architecture 304 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus architecture 304 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0134] The processor 302 may be a CPU, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0135] The communication interface 303 uses any transceiver-like system for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), wired access networks, etc.

[0136] The memory 301 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus architecture 304. The memory may also be integrated with the processor.

[0137] The memory 301 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 302. The processor 302 is used to execute the computer-executable instructions stored in the memory 301, thereby realizing a multi-path real-time semantic segmentation method provided in the above embodiment of the present application.

[0138] Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, which is not specifically limited in the embodiments of the present application.

[0139] The real-time semantic segmentation method provided by the embodiment of the present application includes multiple processing paths, including a spatial information extraction path based on deep separable convolution, a first attention and feature fusion path, and a second attention and feature fusion path. The first feature map can be obtained through the spatial information extraction path based on deep separable convolution, and the second feature map and the third feature map can be obtained through the first attention and feature fusion path and the second attention and feature fusion path, respectively. The first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain real-time semantic segmentation results. The embodiment of the present application produces the following technical effects:

[0140] 1. Using depthwise separable convolution to replace traditional convolution to extract spatial information can greatly reduce the amount of weight parameters involved in the calculation during training and reasoning, and can reduce the number of parameters by up to dozens of times for lightweight models. This makes the model's computational overhead smaller. On the one hand, the memory occupied by the model will be reduced, and on the other hand, it will greatly increase the speed of reasoning. In addition, in order to avoid the problem of weakening the model's learning ability due to a significant decrease in the number of weight parameters, the spatial information extraction path is connected to traditional convolution blocks at both ends of the depthwise separable convolution, which can ensure that the model's learning ability is not affected while the number of parameters is greatly reduced.

[0141] 2. While obtaining the maximum receptive field, the attention and feature fusion mechanism provides a function similar to feature screening, which can purposefully learn the features that are most helpful for segmentation accuracy and ignore or not learn those interfering features. The feature importance vector is automatically calculated based on the existing feature layer, reducing the intervention of manual feature engineering. Since feature fusion is performed on the basis of the attention mechanism, the overall output of the module is a feature without too many biases. The combination of multiple attention and feature fusion paths greatly increases the overall segmentation accuracy and robustness of the model.

[0142] Those of ordinary skill in the art will appreciate that the various digital numbers such as the first and second involved in the present application are only for the convenience of description, and are not used to limit the scope of the embodiments of the present application, nor do they represent the order of precedence. "And / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one" refers to one or more. At least two refers to two or more. "At least one", "any one" or similar expressions refer to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, at least one item (individual, kind) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0143] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0144] The various illustrative logic units and circuits described in the embodiments of the present application can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or the design of any combination of the above functions. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration to implement.

[0145] The steps of the method or algorithm described in the embodiments of the present application can be directly embedded in hardware, a software unit executed by a processor, or a combination of the two. The software unit can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a terminal. Optionally, the processor and the storage medium can also be arranged in different components in the terminal. These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0146] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A multi-path real-time semantic segmentation method, wherein: The method comprises: Obtain an image dataset; Constructing a multi-path model, wherein the multi-path model includes a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path, wherein the convolution kernel size, the number of channels, and the weight initialization method used by the first attention and feature fusion path and the second attention and feature fusion path are different; Using the image data set to train and test the multi-path model, to obtain a multi-path real-time semantic segmentation model; Get the image to be segmented; The picture to be segmented is input into the multi-path real-time semantic segmentation model, wherein the picture to be segmented is processed through the spatial information extraction path to obtain a first feature map, the picture to be segmented is processed through the first attention and feature fusion path to obtain a second feature map, the picture to be segmented is processed through the second attention and feature fusion path to obtain a third feature map, the first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain an output result, and the output result is a real-time semantic segmentation result of the picture to be segmented.

2. The method of claim 1, wherein: The step of processing the image to be segmented through the spatial information extraction path to obtain a first feature map includes: Input the image to be segmented into the front-end convolution block for downsampling to obtain a first convolution output; Inputting the first convolution output into a first depth-wise separable convolution layer for downsampling to obtain a second convolution output; Inputting the second convolution output into a second depth-wise separable convolution layer for downsampling to obtain a third convolution output; The third convolution output is input into the back-end convolution block for cross-channel information fusion to obtain the first feature map.

3. The method of claim 2, wherein: The stride lengths of the front-end convolution block, the first depth-separable convolution layer, and the second depth-separable convolution layer are all 2, and the stride length of the back-end convolution block is 1.

4. The method of claim 2, wherein: The convolution kernel size of the front-end convolution block is 3x3, and the convolution kernel size of the back-end convolution block is 1x1.

5. The method of claim 1, wherein: The step of processing the image to be segmented through the first attention and feature fusion path to obtain a second feature map includes: Inputting the image to be segmented into the attention channel, calculating the feature importance vector of the image to be segmented, and obtaining the feature importance vector; Performing feature layer division based on the feature importance vector to obtain a main input feature layer and a fusion input feature layer; Input the trunk input feature layer into the trunk road for processing, use the feature importance vector as a weight to calculate the new feature layer, and obtain the trunk road output result; Inputting the fused input feature layer into a feature fusion channel, fusing the fused input feature layer with the feature layer weighted by the feature importance vector, and obtaining a fusion channel output result; A fusion operation is performed on the main road output result and the fusion channel output result to obtain the second feature map.

6. The method of claim 5, wherein: The step of inputting the image to be segmented into the attention channel, calculating the feature importance vector of the image to be segmented, and obtaining the feature importance vector includes: Inputting the image to be segmented into the third convolution block for feature extraction to obtain a fourth convolution output; The average importance of the fourth convolution output is calculated through a global average pooling layer to obtain the feature importance vector of each feature layer, and the feature importance vector is numerically mapped, and the numerical mapping interval is (0, 1).

7. The method of claim 1, wherein: The first, second and third feature maps are input into the FFM fusion layer for feature fusion to obtain an output result, wherein the output result is a real-time semantic segmentation result of the image to be segmented, including: Input the first, second and third feature maps into the FFM fusion layer for feature fusion to obtain a feature map fusion result; The feature map fusion result is upsampled and restored to obtain the output result.

8. A multi-path real-time semantic segmentation system, wherein: The system comprises: A first obtaining unit, wherein the first obtaining unit is used to obtain an image data set; A first construction unit, wherein the first construction unit is used to construct a multi-path model, wherein the multi-path model includes a spatial information extraction path, a first attention and feature fusion path, and a second attention and feature fusion path, wherein the convolution kernel size, the number of channels, and the weight initialization method used by the first attention and feature fusion path and the second attention and feature fusion path are different; A first processing unit, the first processing unit is used to train and test the multi-path model using the image data set to obtain a multi-path real-time semantic segmentation model; A second obtaining unit, the second obtaining unit is used to obtain the picture to be segmented; A second processing unit, the second processing unit is used to input the picture to be segmented into the multi-path real-time semantic segmentation model, wherein the picture to be segmented is processed through the spatial information extraction path to obtain a first feature map, the picture to be segmented is processed through the first attention and feature fusion path to obtain a second feature map, the picture to be segmented is processed through the second attention and feature fusion path to obtain a third feature map, the first, second, and third feature maps are input into the FFM fusion layer for feature fusion to obtain an output result, and the output result is a real-time semantic segmentation result of the picture to be segmented.

9. A multi-path real-time semantic segmentation system, comprising: A processor, the processor is coupled to a memory, the memory is used to store a program, when the program is executed by the processor, the system is enabled to perform the steps of the method according to any one of claims 1 to 7.

Images