Image semantic segmentation method and device, and storage medium

Abstract

The application discloses an image semantic segmentation method and device and a storage medium. The semantic segmentation model comprises a backbone network, a feature aggregation network, a segmentation network, a multi-layer perceptron and a post-processing module. The method comprises the following steps: inputting an image to be processed into the backbone network to extract multi-scale features, obtaining multi-scale features corresponding to the image to be processed; inputting the multi-scale features into the feature aggregation network to aggregate high-level semantic information and low-level position information, obtaining aggregated features; inputting the aggregated features into the segmentation network to perform semantic segmentation, obtaining a semantic segmentation graph and target features; inputting the target features into the multi-layer perceptron to predict a region of interest, obtaining a region-of-interest segmentation graph corresponding to the target features; and inputting the region-of-interest segmentation graph and the semantic segmentation graph into the post-processing module to fuse them, obtaining a semantic segmentation result corresponding to the image to be processed. Through the above method, the accuracy of image semantic segmentation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of semantic segmentation, and particularly relates to an image semantic segmentation method and device and a storage medium. BACKGROUND

[0002] Semantic segmentation is to classify each pixel in an image, and pixels belonging to the same class are classified into one class. At present, semantic segmentation is widely used in medical images and unmanned driving, etc. For example, lane line detection, drivable area detection and other tasks in unmanned driving. Due to the limitation of hardware computing power, the related method is to first scale the image to a relatively small size, and then calculate through the model to detect the lane line, the drivable area and other semantic information of interest through pixel-level classification.

[0003] However, the related semantic segmentation method first scales the image to a relatively small size, and then calculates through the model, resulting in low accuracy. SUMMARY

[0004] The present application provides an image semantic segmentation method, device and computer readable storage medium, which can improve the accuracy of image semantic segmentation.

[0005] One technical solution used by the present application is to provide an image semantic segmentation method. The semantic segmentation model includes a backbone network, a feature aggregation network, a segmentation network, a multi-layer perceptron and a post-processing module. The method comprises: inputting a to-be-processed image into the backbone network for multi-scale feature extraction to obtain multi-scale features corresponding to the to-be-processed image; inputting the multi-scale features into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain aggregated features; inputting the aggregated features into the segmentation network for semantic segmentation to obtain a semantic segmentation map and target features; inputting the target features into the multi-layer perceptron for region of interest prediction to obtain a region of interest segmentation map corresponding to the target features; and inputting the region of interest segmentation map and the semantic segmentation map into the post-processing module for fusion to obtain a semantic segmentation result corresponding to the to-be-processed image.

[0006] The backbone network includes a plurality of sequentially connected feature extraction layers, each feature extraction layer has a different feature extraction scale, and the multi-scale feature extraction of the to-be-processed image input into the backbone network includes: inputting the to-be-processed image into the feature extraction layers of the backbone network for multi-scale feature extraction to obtain the backbone extraction features of the corresponding scale output by each feature extraction layer.

[0007] The backbone network comprises a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer and a fifth feature extraction layer connected in sequence; the multi-scale feature is input into the feature extraction layer of the backbone network for multi-scale feature extraction to obtain the backbone extraction feature of a corresponding scale output by each feature extraction layer, including: the multi-scale feature is input into the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, the fourth feature extraction layer and the fifth feature extraction layer for feature extraction to obtain the first backbone extraction feature of a corresponding scale output by the third feature extraction layer, the second backbone extraction feature of a corresponding scale output by the fourth feature extraction layer and the third backbone extraction feature of a corresponding scale output by the fifth feature extraction layer.

[0008] The multi-scale feature is input into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain the aggregated feature, including: the first backbone extraction feature, the second backbone extraction feature and the third backbone extraction feature are input into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain the aggregated feature.

[0009] The multi-scale feature is input into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain the aggregated feature, including:

[0010] The third backbone extraction feature is up-sampled to obtain the first sampling feature;

[0011] The first sampling feature and the second backbone extraction feature are fused to obtain the first fused feature;

[0012] The first fused feature is convoluted to obtain the first convolution feature;

[0013] The first convolution feature is up-sampled to obtain the second sampling feature, and the first convolution feature is down-sampled to obtain the third sampling feature;

[0014] The third backbone extraction feature and the third sampling feature are fused to obtain the second fused feature;

[0015] The first backbone extraction feature and the second sampling feature are fused to obtain the third fused feature;

[0016] The second fused feature is convoluted to obtain the second convolution feature;

[0017] The third fused feature is convoluted to obtain the third convolution feature;

[0018] The third convolution feature is down-sampled to obtain the fourth sampling feature;

[0019] The second main branch feature, the first convolution feature and the fourth sampling feature are fused to obtain a fourth fusion feature;

[0020] The fourth fusion feature is convolved to obtain a fourth convolution feature;

[0021] The second convolution feature is up-sampled to obtain a fifth sampling feature;

[0022] The fourth convolution feature and the fifth sampling feature are fused to obtain a fifth fusion feature;

[0023] The fifth fusion feature is convolved to obtain a fifth convolution feature;

[0024] The fifth convolution feature is up-sampled to obtain a sixth sampling feature;

[0025] The first main branch feature, the third convolution feature and the sixth sampling feature are fused to obtain a sixth fusion feature;

[0026] The sixth fusion feature is convolved to obtain a sixth convolution feature, and the sixth convolution feature is taken as the aggregation feature.

[0027] The target feature is input into the multi-layer perception machine for region of interest prediction to obtain a region of interest segmentation map corresponding to the target feature, including: determining a region of interest in the target feature; and inputting the region of interest into the multi-layer perception machine for region of interest prediction to obtain the region of interest segmentation map corresponding to the target feature.

[0028] The region of interest segmentation map and the semantic segmentation map are input into a post-processing module for fusion to obtain a semantic segmentation result corresponding to the to-be-processed image, and then the real semantic segmentation information and the semantic segmentation result are used to adjust network parameters of the semantic segmentation model.

[0029] The real semantic segmentation information and the semantic segmentation result are used to adjust network parameters of the semantic segmentation model, including: determining a classification loss and an edge loss of semantic segmentation; and using a correlation between the classification loss and the edge loss to adjust network parameters of the semantic segmentation model.

[0030] Another technical solution used in the present application is to provide an image semantic segmentation device, which includes a memory and a processor, the memory is used to store program data, and the processor is used to execute the program data to realize the method provided by the above technical solution.

[0031] Another technical solution used in the present application is to provide a computer readable storage medium, which is used to store a computer program, and the computer program is used to realize the method provided by the above technical solution when executed by a processor.

[0032] The beneficial effects of the present application are: different from the prior art, because the present application inputs multi-scale features into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information, so that the feature aggregation is more sufficient; in addition, the present application inputs target features into a multi-layer perception machine for region of interest prediction, which can recover the lost detail information due to the downsampling involved in the above feature extraction process, more accurately segment the image, and further improve the accuracy of the semantic segmentation result. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0034] Figure 1 is a flowchart of the first embodiment of the image semantic segmentation method provided by the present application;

[0035] Figure 2 is a structural diagram of the backbone network provided by the present application;

[0036] Figure 3 is a structural diagram of Block provided by the present application;

[0037] Figure 4 is a flowchart of step 420 provided by the present application;

[0038] Figure 5 is a flowchart of the third embodiment of the image semantic segmentation method provided by the present application;

[0039] Figure 6 is a schematic diagram of multi-layer perception prediction;

[0040] Figure 7 is an application scenario diagram of an image semantic segmentation method provided by the present application;

[0041] Figure 8 is a segmentation diagram of an image semantic segmentation method provided by the present application;

[0042] Figure 9 is a structural diagram of an embodiment of the image semantic segmentation device provided by the present application;

[0043] Figure 10 is a structural diagram of an embodiment of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limit the present application. In addition, it should be noted that, for the convenience of description, only parts related to the present application are shown in the drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of the present application.

[0045] Reference herein to“an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification is not necessarily all referring to the same embodiment, nor is it necessarily referring to a particular alternative embodiment. It will be explicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] Reference is made to Figure 1 , Figure 1 is a flowchart of a first embodiment of an image semantic segmentation method provided by the present application. The method comprises:

[0047] Step 110: inputting a to-be-processed image into a backbone network to perform multi-scale feature extraction, to obtain multi-scale features corresponding to the to-be-processed image.

[0048] In the embodiment, the backbone network adopts a residual network structure (ResNet), and for a one-million-pixel input, the calculation amount of the present application is limited to 8 GF (10e 9 times of floating-point multiplication and addition operations per GF).

[0049] In some embodiments, the data acquisition of the to-be-processed image can be performed by mounting a vehicle-mounted camera on a vehicle, and collecting image data of various scenes when the vehicle is driving, such as collecting scenes including cities, villages, highways, ramps, etc. The collected images can be preliminarily screened, such as removing repeated images, and removing images with obvious noise and unclear blur.

[0050] Step 120: inputting the multi-scale features into a feature aggregation network to perform feature aggregation of high-level semantic information and low-level position information, to obtain aggregated features.

[0051] In the embodiment, the low-level position information includes contour, edge, color, texture and shape features. Generally, the edge and contour can reflect the content of the image.

[0052] The high-level semantic information refers to the feature information obtained after several times of convolution for feature extraction.

[0053] Exemplarily, low-level position information extraction is performed on a face, and the contour, nose, eyes, etc. of the face can be extracted; and the high-level semantic information of the face is displayed as a face.

[0054] Step 130: input the aggregated features into a segmentation network for semantic segmentation to obtain a semantic segmentation map and target features.

[0055] The segmentation network is composed of two 64-channel 3x3 convolution blocks and one 3x3 convolution layer, and the output channel of the last convolution layer is the classification number, and the original Figure 1 The prediction mask map of the size of 1 / 8 is obtained, and the original image segmentation mask is obtained by upsampling to the original image size.

[0056] The features output by the second convolution block are taken as the target features.

[0057] Step 140: input the target features into a multi-layer perception for region of interest prediction to obtain a region of interest segmentation map corresponding to the target features.

[0058] Step 150: input the region of interest segmentation map and the semantic segmentation map into a post-processing module for fusion to obtain a semantic segmentation result corresponding to the image to be processed.

[0059] There are numerous semantic segmentation techniques based on deep learning, and FCN, UNet, PSPNet and deeplab series have achieved good performance. The main idea of these networks is to perform semantic segmentation on target categories in an encoder-decoder manner. The encoder module is composed of a series of convolution, pooling, normalization and activation layers, and is mainly used to extract semantic features of an image. The downsampling operation gradually extracts higher-level semantic information. The decoder performs upsampling operation according to the semantic information. The conventional upsampling includes nearest neighbor interpolation, bilinear interpolation and deconvolution operation. The upsampling is performed to a certain size (one fourth or one eighth of the original size), the class of each pixel is predicted, and finally the upsampling is performed to the original image size to complete the semantic segmentation of the original image.

[0060] Since the prediction size of the model determines the entire inference time, the prediction size determines the segmentation accuracy of the details. Therefore, the related methods are mostly based on 1 / 8 of the original size, or even lower size for prediction. However, the low size is obtained by continuous convolution and downsampling operation, so the fine position information is lost, and small targets or distant targets will lose features in the convolution and downsampling. That is, the related technology is difficult to balance the time and accuracy.

[0061] The application inputs the target features into the multi-layer perception to perform the region of interest prediction, can restore the lost detail information due to the down-sampling involved in the above feature extraction process, more accurately performs the image segmentation, and further improves the accuracy of the semantic segmentation result.

[0062] In some embodiments, the backbone network can include a plurality of sequentially connected feature extraction layers, each feature extraction layer having a different feature extraction scale, and step 110 in the first embodiment can specifically include the following step 210.

[0063] Step 210: inputting the image to be processed into the feature extraction layer of the backbone network for multi-scale feature extraction to obtain the backbone extraction features of the corresponding scale output by each feature extraction layer.

[0064] For details, please refer to Figure 2 , Figure 2 A structural schematic diagram of the backbone network provided in the application is shown in FIG. 1.

[0065] As shown in Figure 2 , the backbone network is composed of five stages, the first stage includes a 3*3 convolution layer, and a normalization layer and an activation layer. The second stage to the fifth stage adopt the residual network spatial structure.

[0066] In addition, Figure 2 F0 in the formula (1) represents the feature map input into the backbone network, r represents the resolution of the input, and r / 32 represents that the resolution of the feature map finally output is one thirty-second of the input.

[0067] Suppose the network depth is d, then in the residual network spatial structure adopted by the second stage to the fifth stage, the network width w i of the i-th layer has the following linear relationship with the initial network width w0:

[0068] (Formula 1).

[0069] Wherein, w0 is the initial network width, w a is the slope, and w m is an additional parameter.

[0070] In order to ensure that each Block in the second stage to the fifth stage has the same network width, w i， is obtained (Formula 2).

[0071] Wherein, Si is an intermediate parameter and has no actual physical meaning.

[0072] According to formula 1, we can obtain: (Formula 3).

[0073] Substituting Si obtained from Formula 3 into Formula 2, we can obtain the quantized width of each Block.

[0074] Among them, with a network computational cost of 8GF, the parameters w0 and w of the backbone network are... a w m The values ​​are set to 24, 24.48, and 2.54 respectively, with a network depth of 22. The block design is based on residual blocks in ResNet, containing two 1x1 convolutional modules and one 3x3 convolutional module. Each convolutional module consists of a convolutional layer, a normalization layer, and an activation layer. The first block of each stage includes an additional 1x1 convolutional module. Figure 3 As shown in Block-B, the other blocks are as follows: Figure 3 As shown in Blcok-A. In this application, the grouping width of each block is 16, thus the grouping parameter gi for each 3*3 convolutional block can be set according to the block width, where... .

[0075] It should be noted that, Figure 3 In this context, s represents the step size, and ri represents the resolution of the output feature map after the i-th block.

[0076] Based on the above parameters, it can be concluded that the first stage of the backbone network contains a convolutional block with a network width of 32, and the network widths of the second to fifth stages are 32, 64, 160, and 384, respectively. The number of blocks contained in the second to fifth stages are 1, 2, 7, and 12, respectively.

[0077] In some embodiments, the backbone network includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer, and a fifth feature extraction layer connected in sequence, as detailed in the following steps.

[0078] Step 310: Input the image to be processed into the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, the fourth feature extraction layer and the fifth feature extraction layer for feature extraction, and obtain the first backbone extracted features at the corresponding scale output by the third feature extraction layer, the second backbone extracted features at the corresponding scale output by the fourth feature extraction layer and the third backbone extracted features at the corresponding scale output by the fifth feature extraction layer.

[0079] Step 320: Input the multi-scale features into the feature aggregation network to aggregate high-level semantic information and low-level location information to obtain aggregated features.

[0080] Step 330: Input the aggregated features into the segmentation network for semantic segmentation to obtain the semantic segmentation map and target features.

[0081] Step 340: input the target feature into the multi-layer perception machine for region of interest prediction to obtain a region of interest segmentation map corresponding to the target feature.

[0082] Step 350: input the region of interest segmentation map and the semantic segmentation map into a post-processing module for fusion to obtain a semantic segmentation result corresponding to the image to be processed.

[0083] The embodiment can make the feature aggregation more sufficient by inputting the multi-scale features into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information. In addition, the embodiment can restore the lost detail information due to the down-sampling involved in the above feature extraction process, more accurately segment the image, and further improve the accuracy of the semantic segmentation result by inputting the target feature into the multi-layer perception machine for region of interest prediction.

[0084] In some embodiments, step 310 and step 320 can specifically include the following steps:

[0085] Step 410: input the image to be processed into the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, the fourth feature extraction layer and the fifth feature extraction layer for feature extraction to obtain the first backbone extraction feature of the corresponding scale output by the third feature extraction layer, the second backbone extraction feature of the corresponding scale output by the fourth feature extraction layer and the third backbone extraction feature of the corresponding scale output by the fifth feature extraction layer.

[0086] The first backbone extraction feature, the second backbone extraction feature and the third backbone extraction feature are respectively the results obtained by the 1*1 convolution of the third feature extraction layer, the fourth feature extraction layer and the fifth feature extraction layer through the same channel, and the sizes are respectively 1 / 8, 1 / 16 and 1 / 32 of the width and height of the model input.

[0087] Step 420: input the first backbone extraction feature, the second backbone extraction feature and the third backbone extraction feature into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain the aggregated feature.

[0088] In some embodiments, step 420 can include the following flow:

[0089] 1) up-sample the third backbone extraction feature to obtain the first sampling feature.

[0090] The up-sampling can be performed by deconvolution, bilinear interpolation, nearest neighbor interpolation operation. In order to control the amount of calculation, the bilinear interpolation is mainly used for up-sampling operation in the present application.

[0091] 2) fuse the first sampling feature and the second backbone extraction feature to obtain the first fusion feature.

[0092] 3) Convolution is performed on the first fusion feature to obtain a first convolution feature.

[0093] In this application, a 3*3 matrix is used for convolution operation.

[0094] 4) The first convolution feature is up-sampled to obtain a second sampling feature, and the first convolution feature is down-sampled to obtain a third sampling feature.

[0095] 5) The third backbone extraction feature and the third sampling feature are fused to obtain a second fusion feature.

[0096] 6) The first backbone extraction feature and the second sampling feature are fused to obtain a third fusion feature.

[0097] 7) Convolution is performed on the second fusion feature to obtain a second convolution feature.

[0098] 8) Convolution is performed on the third fusion feature to obtain a third convolution feature.

[0099] 9) The third convolution feature is down-sampled to obtain a fourth sampling feature.

[0100] The down-sampling operation can use a convolution layer with a step number of 2, or can be replaced by a pooling layer. In this application, the down-sampling operation is mainly performed by a convolution layer with a step number of 2.

[0101] 10) The second backbone extraction feature, the first convolution feature and the fourth sampling feature are fused to obtain a fourth fusion feature.

[0102] 11) Convolution is performed on the fourth fusion feature to obtain a fourth convolution feature.

[0103] 12) The second convolution feature is up-sampled to obtain a fifth sampling feature.

[0104] 13) The fourth convolution feature and the fifth sampling feature are fused to obtain a fifth fusion feature.

[0105] 14) Convolution is performed on the fifth fusion feature to obtain a fifth convolution feature.

[0106] 15) The fifth convolution feature is up-sampled to obtain a sixth sampling feature.

[0107] 16) The first backbone extraction feature, the third convolution feature and the sixth sampling feature are fused to obtain a sixth fusion feature.

[0108] 17) Convolution is performed on the sixth fusion feature to obtain a sixth convolution feature, and the sixth convolution feature is taken as an aggregation feature.

[0109] For the detailed process of step 420 above, please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart of step 420.

[0110] in, Figure 4 In the diagram, P5 represents the third backbone feature extraction, P4 represents the second backbone feature extraction, P3 represents the first backbone feature extraction, A represents the first convolutional feature, B represents the second convolutional feature, C represents the third convolutional feature, D represents the fourth convolutional feature, E represents the fifth convolutional feature, P3' represents the sixth convolutional feature, 2×upsample represents a 2x upsampling operation, 2×Downsample represents a 2x downsampling operation, ⊕ represents feature addition, and 3*3 conv represents a 3*3 convolution.

[0111] like Figure 4 As shown:

[0112] (1) Extract the feature P5 from the third main branch and perform 2×upsample (i.e., 2 times upsampling) to obtain the first sampled feature.

[0113] (2) The first sampling feature and the second backbone extracted feature P4 are fused to obtain the first fused feature.

[0114] (3) Perform a 3*3 convolution on the first fusion feature to obtain the first convolution feature A.

[0115] (4) Perform 2×upsample (i.e., 2 times upsampling) on ​​the first convolutional feature A to obtain the second sampled feature, and perform 2×downsample (i.e., 2 times downsampling) on ​​the first convolutional feature A to obtain the third sampled feature.

[0116] (5) The third backbone extracted feature P5 and the third sampling feature are fused to obtain the second fused feature.

[0117] (6) The first backbone extracted feature P3 and the second sampling feature are fused to obtain the third fused feature.

[0118] (7) Perform a 3*3 convolution on the second fusion feature to obtain the second convolution feature B.

[0119] (8) Perform a 3*3 convolution on the third fusion feature to obtain the third convolution feature C.

[0120] (9) Perform 2×Downsample on the third convolution feature C to obtain the fourth sampled feature.

[0121] (10) The second backbone extracted feature P4, the first convolution feature A and the fourth sampling feature are fused to obtain the fourth fused feature.

[0122] (11) performing 3*3 convolution on the fourth fused feature to obtain a fourth convolutional feature D.

[0123] (12) performing 2xupsample (i.e., 2 times upsampling) on the second convolutional feature D to obtain a fifth sampling feature.

[0124] (13) fusing the fourth convolutional feature D and the fifth sampling feature to obtain a fifth fused feature.

[0125] (14) performing 3*3 convolution on the fifth fused feature to obtain a fifth convolutional feature E.

[0126] (15) performing 2xupsample (i.e., 2 times upsampling) on the fifth convolutional feature E to obtain a sixth sampling feature.

[0127] (16) fusing the first backbone extracted feature P3, the third convolutional feature C and the sixth sampling feature to obtain a sixth fused feature.

[0128] (17) performing convolution on the sixth fused feature to obtain a sixth convolutional feature P3', and taking the sixth convolutional feature P3' as the aggregated feature.

[0129] The size of the sixth convolutional feature P3' is 1 / 8 of the model input.

[0130] The embodiment performs the feature fusion, convolution, upsampling and downsampling operations on features of different scales, so that the final aggregated feature retains more high-level semantic information and low-level position information, and the accuracy of the subsequently predicted semantic segmentation map is improved.

[0131] Referring to Figure 5 , Figure 5 is a flowchart of a third embodiment of the image semantic segmentation method provided in the present application. The method comprises:

[0132] Step 510: inputting a to-be-processed image into a backbone network to perform multi-scale feature extraction, to obtain multi-scale features corresponding to the to-be-processed image.

[0133] Step 520: inputting the multi-scale features into a feature aggregation network to perform feature aggregation of high-level semantic information and low-level position information, to obtain an aggregated feature.

[0134] Step 530: inputting the aggregated feature into a segmentation network to perform semantic segmentation, to obtain a semantic segmentation map and a target feature.

[0135] Step 540: determining a region of interest in the target feature.

[0136] Specifically, the aggregated features are input into a region of interest pooling layer (i.e., a ROI Pooler layer), and the region of interest is a preset region, such as a region near a vanishing point of an image or a long-distance region that is focused on in the field of autonomous driving. Assuming that the region is a rectangular region with a width of 320 and a height of 128, and the features obtained after the backbone network and the feature aggregation network are 40*16 (8 times of down-sampling are performed), a feature map with a shape of (N, 64, 20, 8) is obtained after the region of interest pooling layer, where N is a batch size of samples selected in one training.

[0137] Step 550: inputting the region of interest into a multilayer perceptron for region of interest prediction to obtain a region of interest segmentation map corresponding to the target features.

[0138] The multilayer perceptron (MLP) includes multiple fully connected layers.

[0139] Specifically, the process of inputting the region of interest into the multilayer perceptron for region of interest prediction to obtain the region of interest segmentation map corresponding to the target features can be referred to in Figure 6 , Figure 6 A schematic diagram of the multilayer perceptron prediction.

[0140] As shown in Figure 6 , the region of interest is input into an input layer fc1 of the multilayer perceptron after the region of interest pooling layer, and an output result of the input layer fc1 is obtained. The output result of the input layer fc1 is input into a hidden layer fc2, and an output result of the hidden layer fc2 is obtained. The output result of the hidden layer fc2 is input into an output layer fc3, and an output feature of the output layer fc3 is obtained. The output feature of the output layer fc3 is reshaped into a tensor of (N, M, 320, 128) first, and then an argmax operation is performed to obtain a segmentation map of the region of interest. M is the number of categories of classification, and N is a batch size of samples selected in one training.

[0141] The reshape operation is an operation of obtaining elements from a matrix in sequence and filling a new matrix according to a specified shape. The input channel number of the input layer fc1 is 64*20*8, the output channel number is 1024, the input channel number of the hidden layer fc2 is 1024, the output channel number is 1024, the input channel number of the output layer fc3 is 1024, and the output channel number is 320*128*M.

[0142] Step 560: inputting the region of interest segmentation map and the semantic segmentation map into a post-processing module for fusion to obtain a semantic segmentation result corresponding to the to-be-processed image.

[0143] The semantic segmentation graph is a coarse prediction graph output by the segmentation network, that is, a prediction graph of a non-interest region, the interest region segmentation graph is an interest region segmentation graph output by the multi-layer perceptron, and the semantic segmentation result obtained by inputting the interest region segmentation graph and the semantic segmentation graph into the post-processing module for fusion is a fine prediction graph as a final output.

[0144] The embodiment inputs multi-scale features into the feature aggregation network to perform feature aggregation of high-level semantic information and low-level position information, so that the feature aggregation is more sufficient. In addition, the embodiment inputs target features into the multi-layer perceptron to perform interest region prediction, can recover lost detail information due to down-sampling involved in the above feature extraction process, more accurately segment the image, and further improve the accuracy of the semantic segmentation result.

[0145] In combination with the above embodiment, the application provides a scene application schematic diagram, which is specifically described with reference to Figure 7 .

[0146] As shown in Figure 7 , H is the height of the to-be-processed image, W is the width of the to-be-processed image, a region with a width of 128 and a height of 320 is selected as the interest region of the to-be-processed image, and after the to-be-processed image passes through the backbone network and the feature aggregation network, a to-be-processed image feature with a height and a width of 1 / 8 of the original and an interest region feature with a width of 16 and a height of 40 are obtained. The to-be-processed image feature is predicted by the segmentation network to segment the whole image, and a rough prediction mask of the original image is obtained. The interest region feature passes through the interest region pooling layer and is output as a 10*10 feature, and then passes through the multi-layer perceptron to obtain a prediction mask of the interest region with a width of 128 and a height of 320. Finally, the rough prediction mask of the original image and the prediction mask of the interest region are input into the post-processing module for fusion to obtain a semantic segmentation result corresponding to the to-be-processed image.

[0147] The fusion process of the post-processing module is to display the prediction mask of the interest region in the corresponding region of the rough prediction mask of the original image.

[0148] In some embodiments, the to-be-processed image is labeled with real semantic segmentation information, and after the interest region segmentation graph and the semantic segmentation graph are input into the post-processing module for fusion to obtain a semantic segmentation result corresponding to the to-be-processed image, the method can further include: adjusting network parameters of the semantic segmentation model by using the real semantic segmentation information and the semantic segmentation result.

[0149] In some embodiments, the classification loss of the semantic segmentation and the edge loss can be determined by

[0150] The correlation between the classification loss and the edge loss is used to adjust the network parameters of the semantic segmentation model to achieve adjusting the network parameters of the semantic segmentation model using the real semantic segmentation information and the semantic segmentation result.

[0151] For example, the classification loss value can be calculated using a classification loss function, such as the following formula:

[0152] .

[0153] where p t is the predicted class probability value, and a t , is a hyperparameter, and the value of a t , is set to 1 and 2 respectively.

[0154] In some embodiments, the edge loss value can be calculated using a cross-entropy loss function, such as the following formula:

[0155] .

[0156] where M is the number of classes, N is the batch size, y ic is the real label 0 and 1, 0 means not this class, and 1 means this class, P ic is the predicted probability value, and c represents the ID of the class, and i represents the index value of the pixel.

[0157] For calculating the edge segmentation in the edge loss, please refer to Figure 8 , Figure 8 is a segmentation schematic diagram in an image semantic segmentation method provided by the present application.

[0158] As shown in Figure 8 , A and B are the predicted image and the real image respectively, and from the figure it can be seen that this segmentation is not very accurate, mainly in the edge part, so before calculating the loss, the edge part of the predicted image and the real image is dynamically obtained, and the width of the edge can be dynamically adjusted, such as setting the width to 2 pixels (that is, the edge of the current class plus the edge of the adjacent class), and the loss value at the edge can be calculated using the cross-entropy loss function described above.

[0159] Referring to Figure 9 , Figure 9 is a structural schematic diagram of an embodiment of an image semantic segmentation device provided by the present application, and the image semantic segmentation device 130 includes a memory 131 and a processor 132, the memory 131 is used to store program data, and the processor 132 is used to execute the program data to realize the following method:

[0160] The image to be processed is input to the backbone network for multi-scale feature extraction to obtain multi-scale features corresponding to the image to be processed; the multi-scale features are input to the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain aggregated features; the aggregated features are input to the segmentation network for semantic segmentation to obtain a semantic segmentation map and target features; the target features are input to the multi-layer perceptron for region of interest prediction to obtain a region of interest segmentation map corresponding to the target features; and the region of interest segmentation map and the semantic segmentation map are input to the post-processing module for fusion to obtain a semantic segmentation result corresponding to the image to be processed.

[0161] It can be understood that the processor 132 is further configured to execute program data to implement the method of any of the above embodiments.

[0162] Referring to Figure 10 , Figure 10 is a structural schematic diagram of an embodiment of the computer readable storage medium provided in the present application, the computer readable storage medium 140 stores program data 141, and the program data 141, when executed by a processor, is configured to implement the following method:

[0163] The image to be processed is input to the backbone network for multi-scale feature extraction to obtain multi-scale features corresponding to the image to be processed; the multi-scale features are input to the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain aggregated features; the aggregated features are input to the segmentation network for semantic segmentation to obtain a semantic segmentation map and target features; the target features are input to the multi-layer perceptron for region of interest prediction to obtain a region of interest segmentation map corresponding to the target features; and the region of interest segmentation map and the semantic segmentation map are input to the post-processing module for fusion to obtain a semantic segmentation result corresponding to the image to be processed.

[0164] It can be understood that the program data 141, when executed by a processor, is further configured to implement the method of any of the above embodiments.

[0165] In several embodiments provided in the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0166] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0167] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0168] The integrated unit in the other embodiments described above, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0169] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent flow transformation based on the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An image semantic segmentation method, characterized in that, The semantic segmentation model comprises a backbone network, a feature aggregation network, a segmentation network, a multi-layer perceptron and a post-processing module, and the method comprises: inputting an image to be processed into the backbone network for multi-scale feature extraction to obtain multi-scale features corresponding to the image to be processed; inputting the multi-scale features into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain aggregated features; inputting the aggregated features into the segmentation network for semantic segmentation to obtain a semantic segmentation map and target features; inputting the target features into the multi-layer perceptron for region of interest prediction to obtain a region of interest segmentation map corresponding to the target features; inputting the region of interest segmentation map and the semantic segmentation map into the post-processing module for fusion to obtain a semantic segmentation result corresponding to the image to be processed; the backbone network comprises a plurality of sequentially connected feature extraction layers, each feature extraction layer having a different feature extraction scale, and the inputting of the image to be processed into the backbone network for multi-scale feature extraction to obtain multi-scale features corresponding to the image to be processed comprises: inputting the image to be processed into the feature extraction layers of the backbone network for multi-scale feature extraction to obtain backbone extraction features of corresponding scales output by each feature extraction layer; the backbone network comprises a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer and a fifth feature extraction layer connected in sequence, and the inputting of the image to be processed into the feature extraction layers of the backbone network for multi-scale feature extraction to obtain backbone extraction features of corresponding scales output by each feature extraction layer comprises: inputting the image to be processed into the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, the fourth feature extraction layer and the fifth feature extraction layer for feature extraction to obtain first backbone extraction features of corresponding scales output by the third feature extraction layer, second backbone extraction features of corresponding scales output by the fourth feature extraction layer and third backbone extraction features of corresponding scales output by the fifth feature extraction layer; the inputting of the multi-scale features into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain aggregated features comprises: inputting the first backbone extraction features, the second backbone extraction features and the third backbone extraction features into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain aggregated features; the inputting of the first backbone extraction features, the second backbone extraction features and the third backbone extraction features into the feature aggregation network for feature aggregation of high-level semantic information and low-level position information to obtain aggregated features comprises: upsampling the third backbone extraction features to obtain first sampling features; fusing the first sampling features and the second backbone extraction features to obtain first fused features; convolving the first fused features to obtain first convolution features; Upsample the first convolutional feature to obtain a second sampling feature, and downsample the first convolutional feature to obtain a third sampling feature; Fuse the third backbone extraction feature and the third sampling feature to obtain a second fusion feature; Fuse the first backbone extraction feature and the second sampling feature to obtain a third fusion feature; Convolve the second fusion feature to obtain a second convolutional feature; Convolve the third fusion feature to obtain a third convolutional feature; Downsample the third convolutional feature to obtain a fourth sampling feature; Fuse the second backbone extraction feature, the first convolutional feature, and the fourth sampling feature to obtain a fourth fusion feature; Convolve the fourth fusion feature to obtain a fourth convolutional feature; Upsample the second convolutional feature to obtain a fifth sampling feature; Fuse the fourth convolutional feature and the fifth sampling feature to obtain a fifth fusion feature; Convolve the fifth fusion feature to obtain a fifth convolutional feature; Upsample the fifth convolutional feature to obtain a sixth sampling feature; Fuse the first backbone extraction feature, the third convolutional feature, and the sixth sampling feature to obtain a sixth fusion feature; Convolve the sixth fusion feature to obtain a sixth convolutional feature, and take the sixth convolutional feature as the aggregation feature.

2. The method of claim 1, wherein, The target feature is input into the multi-layer perceptron for region of interest prediction to obtain a region of interest segmentation map corresponding to the target feature, including: Determining a region of interest in the target feature; Inputting the region of interest into the multi-layer perceptron for region of interest prediction to obtain a region of interest segmentation map corresponding to the target feature.

3. The method of claim 1, wherein, The to-be-processed image is labeled with real semantic segmentation information, and after the region of interest segmentation map and the semantic segmentation map are input into the post-processing module for fusion to obtain a semantic segmentation result corresponding to the to-be-processed image, including: Adjusting network parameters of the semantic segmentation model using the real semantic segmentation information and the semantic segmentation result.

4. The method of claim 3, wherein, The adjusting network parameters of the semantic segmentation model using the real semantic segmentation information and the semantic segmentation result includes: Determining a classification loss and an edge loss of semantic segmentation; Adjusting network parameters of the semantic segmentation model using the correlation of the classification loss and the edge loss.

5. An image semantic segmentation apparatus, characterized by comprising: The image semantic segmentation device includes a memory and a processor, the memory is used to store program data, and the processor is used to execute the program data to realize the image semantic segmentation method in any one of claims 1-4.

6. A computer readable storage medium characterized by The computer readable storage medium stores program data, and the program data is used to realize the image semantic segmentation method in any one of claims 1-4 when executed by a processor.

Images