An image semantic segmentation method and device, an electronic device, and a storage medium

Abstract

The application discloses an image semantic segmentation method applied to an electronic device comprising an image semantic segmentation model, wherein the image semantic segmentation model comprises a down-sampling module and a sparse multi-layer perception module; the image semantic segmentation method comprises the following steps: acquiring an original image, extracting a down-sampled feature map of the original image by using the down-sampling module; performing linear coding on the down-sampled feature map to obtain a linear coding feature map; performing feature extraction on the linear coding feature map by using the sparse multi-layer perception module to obtain an output feature; training the image semantic segmentation model by using the output feature, and performing an image semantic segmentation task by using the trained image semantic segmentation model; and the application can balance the speed and accuracy of image semantic segmentation, so that the image semantic segmentation operation has a faster speed and higher accuracy. The application also discloses an image semantic segmentation device, an electronic device and a storage medium, which have the above beneficial effects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image semantic segmentation method and device, an electronic device and a storage medium. BACKGROUND

[0002] Image semantic segmentation is a basic task in the field of computer vision, which aims to assign a specified semantic label to each pixel in an image. It is essentially a multi-classification problem at the pixel level. With the development of deep learning technology in the field of computer vision, more and more image semantic segmentation algorithms are applied to the fields of autonomous driving, video surveillance, medical image analysis and human-computer interaction. In the related art, image features are usually extracted based on convolutional neural networks or Transformer networks. However, the image features extracted based on convolutional neural networks lack global semantic information, resulting in low accuracy of image semantic segmentation. The computation of image features extracted based on Transformer networks is huge, and the speed of image semantic segmentation is slow.

[0003] Therefore, how to balance the speed and accuracy of image semantic segmentation is a technical problem that needs to be solved by those skilled in the art at present. SUMMARY

[0004] The purpose of the present application is to provide an image semantic segmentation method, an image semantic segmentation device, an electronic device and a storage medium, which can balance the speed and accuracy of image semantic segmentation, so that the image semantic segmentation operation has faster speed and higher accuracy.

[0005] To solve the above technical problems, the present application provides an image semantic segmentation method applied to an electronic device comprising an image semantic segmentation model, wherein the image semantic segmentation model comprises a down-sampling module and a sparse multi-layer perception module. The image semantic segmentation method comprises:

[0006] obtaining an original image, and extracting a down-sampling feature map of the original image by using the down-sampling module;

[0007] linearly encoding the down-sampling feature map to obtain a linearly encoded feature map;

[0008] extracting features of the linearly encoded feature map by using the sparse multi-layer perception module to obtain output features, wherein the output features comprise global semantic information and local semantic information, and the sparse multi-layer perception module comprises a sparse multi-layer perception, a depth separable convolution structure and a residual structure;

[0009] training the image semantic segmentation model by using the output features, and performing an image semantic segmentation task by using the trained image semantic segmentation model.

[0010] Optionally, the sparse multi-layer perception module is used to perform feature extraction on the linearly encoded feature map to obtain output features, including:

[0011] The linearly encoded feature map is subjected to channel splitting at a preset ratio to obtain a first sub-feature map and a second sub-feature map;

[0012] The sparse multi-layer perception is used to extract global semantic information of the first sub-feature map;

[0013] The deep separable convolution structure is used to extract local semantic information of the second sub-feature map;

[0014] The global semantic information and the local semantic information are fused to obtain fused features;

[0015] The fused features are input into the residual structure to obtain the output features.

[0016] Optionally, the sparse multi-layer perception includes a first processing branch and a second processing branch;

[0017] Correspondingly, the sparse multi-layer perception is used to extract global semantic information of the first sub-feature map, including:

[0018] In the first processing branch, full connection layers are used to interact with pixels in the same row in the first sub-feature map to obtain row processing results;

[0019] In the second processing branch, full connection layers are used to interact with pixels in the same column in the first sub-feature map to obtain column processing results;

[0020] The global semantic information of the first sub-feature map is generated according to the row processing results and the column processing results.

[0021] Optionally, the deep separable convolution structure includes a deep separable convolution layer, a batch normalization layer, and an activation function;

[0022] Correspondingly, the deep separable convolution structure is used to extract local semantic information of the second sub-feature map, including:

[0023] The second sub-feature map is input into the deep separable convolution layer to obtain a deep separable convolution result;

[0024] The batch normalization layer and the activation function are used in sequence to process the deep separable convolution result to obtain the local semantic information of the second sub-feature map.

[0025] Optionally, the global semantic information and the local semantic information are fused to obtain fused features, including:

[0026] characteristic fusion is performed on the global semantic information and the local semantic information to obtain a fusion result;

[0027] a channel order of the fusion result is adjusted through a channel shuffle operation to obtain the fusion feature.

[0028] Optionally, the downsampling module comprises a plurality of convolution modules and a channel attention module connected in parallel; the step lengths of all the convolution modules are the same, and the sizes of the convolution kernels of any two of the convolution modules are different.

[0029] Correspondingly, extracting a down-sampling feature map of the original image by using the downsampling module comprises:

[0030] inputting the original image into each of the convolution modules respectively;

[0031] performing feature screening on the output results of all the convolution modules by using the same channel attention module to obtain the down-sampling feature map of the original image.

[0032] Optionally, training the image semantic segmentation model by using the output feature comprises:

[0033] training the image semantic segmentation model by using a loss function of random sampling and the output feature;

[0034] The loss function of random sampling calculates a loss value according to the randomly sampled part of the pixels in the output feature.

[0035] The application further provides an image semantic segmentation device, which comprises a downsampling module, a linear encoding module, a sparse multi-layer perception module and a training module;

[0036] The downsampling module is configured to acquire an original image and extract a down-sampling feature map of the original image by using the downsampling module.

[0037] The linear encoding module is configured to perform linear encoding on the down-sampling feature map to obtain a linearly encoded feature map.

[0038] The sparse multi-layer perception module is configured to perform feature extraction on the linearly encoded feature map by using the sparse multi-layer perception module to obtain an output feature; wherein the output feature comprises global semantic information and local semantic information, and the sparse multi-layer perception module comprises a sparse multi-layer perception, a depth separable convolution structure and a residual structure.

[0039] The training module is configured to train the image semantic segmentation model by using the output feature, so as to perform an image semantic segmentation task by using the trained image semantic segmentation model.

[0040] The application further provides a storage medium, which stores a computer program, and the computer program realizes the steps performed by the image semantic segmentation method when executed.

[0041] The application further provides an electronic device, which comprises a memory and a processor, and the memory stores a computer program, and the processor realizes the steps performed by the image semantic segmentation method when calling the computer program in the memory.

[0042] The application provides an image semantic segmentation method, which is applied to an electronic device comprising an image semantic segmentation model, and the image semantic segmentation model comprises a down-sampling module and a sparse multi-layer perception module. The image semantic segmentation method comprises the following steps: obtaining an original image, extracting a down-sampled feature map of the original image by using the down-sampling module; performing linear coding on the down-sampled feature map to obtain a linear coding feature map; performing feature extraction on the linear coding feature map by using the sparse multi-layer perception module to obtain an output feature; wherein the output feature comprises global semantic information and local semantic information, the sparse multi-layer perception module comprises a sparse multi-layer perception, a depth separable convolution structure and a residual structure; training the image semantic segmentation model by using the output feature, and performing an image semantic segmentation task by using the trained image semantic segmentation model.

[0043] The application performs down-sampling and linear coding on an original image to obtain a linear coding feature map, and performs feature extraction on the linear coding feature map by using a sparse multi-layer perception module to obtain an output feature. The output feature extracted from the linear coding feature map by the sparse multi-layer perception module comprises global semantic information and local semantic information, and the image semantic segmentation model trained based on the output feature can balance the speed and accuracy of image semantic segmentation, so that the image semantic segmentation operation has a faster speed and higher accuracy. The application further provides an image semantic segmentation device, a storage medium and an electronic device, which have the above beneficial effects, and details are not described herein. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the application, the drawings required to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A flowchart of an image semantic segmentation method provided by an embodiment of the application;

[0046] Figure 2 A network structure schematic diagram of a Swin Transformer provided by an embodiment of the application;

[0047] Figure 3 This application provides a multi-scale spatial information aggregation structure diagram.

[0048] Figure 4 This is a structural diagram of a sparse multilayer perceptron module provided in an embodiment of this application;

[0049] Figure 5 A flowchart illustrating how to obtain global context information of a feature, as provided in an embodiment of this application;

[0050] Figure 6 This application provides a flow chart for a depthwise separable convolution module.

[0051] Figure 7 This is a structural diagram of a feature fusion module provided in an embodiment of this application;

[0052] Figure 8 This is a schematic diagram illustrating a random sampling comparison of a loss function provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Please see below. Figure 1 , Figure 1 This is a flowchart of an image semantic segmentation method provided in an embodiment of this application.

[0055] Specific steps may include:

[0056] S101: Obtain the original image and extract the downsampled feature map of the original image using the downsampling module;

[0057] This embodiment can be applied to electronic devices that include an image semantic segmentation model, which performs operations such as image matting and background removal through image semantic segmentation. This embodiment does not limit the type of image semantic segmentation model; for example, it may include Swin Transformer, SegNet, PSPNet, etc. The image semantic segmentation model may include a downsampling module and a sparse multilayer perceptron module.

[0058] Before this step, there can also be an operation of obtaining original images from a sample set, and the number of original images is not limited here. After the original images are input into the downsampling module, the downsampling module can perform feature extraction on the original images to obtain a downsampling feature map. Specifically, the downsampling module can be a single convolutional layer, a single pooling layer, a Focus structure, or multiple convolutional modules connected in parallel.

[0059] S102: linearly encoding the downsampling feature map to obtain a linearly encoded feature map;

[0060] The image semantic segmentation model can further include a linear encoding module (such as an embedding module). After the downsampling feature map is input into the linear encoding module, linear encoding is performed to obtain a linearly encoded feature map.

[0061] S103: performing feature extraction on the linearly encoded feature map by using the sparse multi-layer perception module to obtain an output feature;

[0062] The sparse multi-layer perception module includes a sparse multi-layer perception, a depth separable convolution structure, and a residual structure. The sparse multi-layer perception is used to extract global semantic information of the linearly encoded feature map. The depth separable convolution structure is used to extract local semantic information of the linearly encoded feature map. The residual structure, also known as a residual network, is used to alleviate the problem of gradient vanishing or gradient explosion caused by increasing the depth of a deep neural network. The self-attention, the fully connected layer, the global semantic information, and the local semantic information can be obtained.

[0063] S104: training the image semantic segmentation model by using the output feature, and performing an image semantic segmentation task by using the trained image semantic segmentation model.

[0064] After the output feature containing the global semantic information and the local semantic information is obtained, the image semantic segmentation model is trained by using the output feature. After receiving an image semantic segmentation task, the trained image semantic segmentation model can be used to perform the image semantic segmentation task.

[0065] Specifically, the image semantic segmentation model trained in this embodiment can be applied to a decision device of an autonomous vehicle, and the process of performing an image semantic segmentation task is as follows: a camera of the autonomous vehicle captures a surrounding environment image, generates an image semantic segmentation task according to the surrounding environment image, and processes the surrounding environment image by using the trained image semantic segmentation model to output an image semantic segmentation result. The image semantic segmentation result can include road information such as other vehicles and pedestrians in the surrounding environment image, and a driving route of the vehicle is planned according to the result of semantic segmentation.

[0066] The embodiment performs downsampling and linear coding on the original image to obtain a linear coding feature map, and extracts features from the linear coding feature map by using a sparse multilayer perceptron module to obtain output features. The output features extracted from the linear coding feature map by the sparse multilayer perceptron module include global semantic information and local semantic information, and the image semantic segmentation model trained based on the output features can balance the speed and accuracy of image semantic segmentation, so that the image semantic segmentation operation has a faster speed and higher accuracy.

[0067] As for Figure 1 For further introduction of the corresponding embodiment, the downsampling module can include a plurality of convolution modules connected in parallel, and further include a channel attention module; the step size of each convolution module in the plurality of convolution modules connected in parallel is the same, and the convolution kernel size of any two convolution modules is different. Further, the process of extracting the downsampling feature map of the original image by using the downsampling module in S101 includes: inputting the original image into each convolution module respectively; and filtering the output results of all the convolution modules by using the same channel attention module to obtain the downsampling feature map of the original image.

[0068] In view of the problem of serious information loss when downsampling by the Transformer and the problem of poor small target segmentation effect, the above scheme proposes a multi-scale spatial information aggregation method, which extracts features at different scales by using a plurality of convolution kernels of different sizes, enhances the feature extraction capability of the model, and improves the segmentation effect of the model on small targets. The above embodiment can also use a channel attention mechanism to filter the extracted features, which not only retains important features but also reduces the influence of useless features on deep networks.

[0069] As for Figure 1 For further introduction of the corresponding embodiment, S103 can obtain the output features in the following manner:

[0070] Step 1: performing channel splitting of a preset ratio on the linear coding feature map to obtain a first sub-feature map and a second sub-feature map.

[0071] In this step, the linear coding feature map is split in a preset ratio according to the channel dimension to obtain the first sub-feature map and the second sub-feature map, which have the same number of channels. The preset ratio can be any value, and optionally, the preset ratio can be 1:1, i.e., the channel splitting is performed in equal proportions.

[0072] Step 2: extracting global semantic information of the first sub-feature map by using a sparse multilayer perceptron.

[0073] The sparse multi-layer perceptron includes a first processing branch and a second processing branch. In this embodiment, the first sub-feature map is duplicated twice, and the first sub-feature map is input into the first processing branch and the second processing branch for processing. The specific process is as follows:

[0074] Correspondingly, the sparse multi-layer perceptron is used to extract the global semantic information of the first sub-feature map. In the first processing branch, the pixels in the same row of the first sub-feature map are interacted through a full connection layer to obtain a row processing result. In the second processing branch, the pixels in the same column of the first sub-feature map are interacted through a full connection layer to obtain a column processing result. The operations of the two processing branches can be executed simultaneously or sequentially.

[0075] After the operations of the two processing branches are executed, the global semantic information of the first sub-feature map can be generated according to the row processing result and the column processing result.

[0076] Step 3: The local semantic information of the second sub-feature map is extracted by using the deep separable convolution structure.

[0077] The deep separable convolution structure includes a deep separable convolution layer, a batch normalization layer and an activation function. The process of extracting local semantic information is as follows: the second sub-feature map is input into the deep separable convolution layer to obtain a deep separable convolution result; the batch normalization layer and the activation function are used in sequence to process the deep separable convolution result to obtain the local semantic information of the second sub-feature map.

[0078] Step 4: The global semantic information and the local semantic information are fused to obtain fusion features.

[0079] Specifically, the global semantic information and the local semantic information are fused to obtain a fusion result. The channel order of the fusion result is adjusted through a channel Shuffle operation to obtain the fusion features.

[0080] Step 5: The fusion features are input into the residual structure to obtain the output features.

[0081] Specifically, the fusion features and the features of the skip connection branch in the residual structure are added to obtain the output features. This embodiment proposes a sparse multi-layer perceptron structure to solve the problems of a large amount of matrix calculation and lack of local semantic information in the self-attention of the Transformer network. The sparse connection and parameter sharing are used to reduce the model parameter quantity and the model complexity, thereby reducing the model inference time while maintaining the model accuracy.

[0082] As forFigure 1 For further introduction of the corresponding embodiments, S104 can train the image semantic segmentation model by: training the image semantic segmentation model by using a loss function of random sampling and the output feature; wherein the loss function of random sampling calculates a loss value according to a part of pixel points randomly sampled in the output feature. In view of the difficulty in collecting semantic segmentation data and the problem of alleviating model overfitting, the present scheme proposes a semantic segmentation training scheme based on a loss function calculated by random sampling. When calculating the model gradient, the label is randomly sampled, which not only brings a regularization effect but also speeds up the model convergence.

[0083] The above-described processes are described in the embodiments are illustrated below through examples in practical applications.

[0084] In the field of autonomous driving, the balance between speed and accuracy of image semantic segmentation algorithms, and the fusion of global semantic information and local position information, have gradually become the bottleneck of the development of autonomous driving technology. The semantic segmentation algorithm based on convolutional neural network is limited by the receptive field of the convolution kernel, and the shallow network is difficult to obtain the global semantic information of the image, resulting in poor edge segmentation effect of large targets in real scenes. In order to solve the problem of global semantic information, the present scheme uses a network based on Transformer to extract image features; but the information loss is serious when the Transformer network is down-sampled from the initial image, resulting in poor small target segmentation effect. The present scheme proposes a multi-scale spatial information aggregation method as the down-sampling method of the Transformer network, so that the network model has rich spatial position information at the shallow layer, improving the segmentation effect of small targets; since the self-attention in the Transformer network involves a large amount of matrix calculation, the network model converges slowly during training, and the self-attention lacks local semantic information of the image, the present scheme proposes a sparse multi-layer perceptron structure to reduce the model calculation amount and enhance the local feature expression ability of the model; in order to alleviate the overfitting problem in the model training process, the present scheme calculates the loss by randomly sampling the network model output and the label, and then calculates the gradient by back propagation of the loss value, and then updates the parameters of the model by the gradient, speeding up the model convergence.

[0085] The semantic segmentation method based on sparse multi-layer perceptron of the present scheme includes three links: image down-sampling process based on multi-scale spatial information aggregation layer, feature extraction process of sparse multi-layer perceptron structure, and model training process based on random sampling calculation loss.

[0086] Take Swin Transformer as an example, replace the 4x4 convolution in the image downsampling Patch Partition layer with a multi-scale spatial information aggregation layer, remove the self-attention module, and replace it with a sparse multi-layer perceptron. See Figure 2 , Figure 2 A network structure diagram of a Swin Transformer provided by an embodiment of the present application, wherein N1, N2, N3, and N4 are the number of stacked sparse MLP modules. The four stages of the entire network respectively obtain feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input image images in width and height. Figure 2 In the formula, H represents height, W represents width, C represents the number of channels, PatchPartition represents the image partition layer, Linear Embedding represents the linear encoding module, Sparse MLP represents the sparse multi-layer perceptron, and Patch Merging represents the multi-scale spatial information aggregation layer.

[0087] The image downsampling process based on the multi-scale spatial information aggregation layer is as follows:

[0088] The multi-scale spatial information aggregation module is a downsampling module for feature extraction of an original image. The specific operation is to introduce parallel convolution modules with the same length but different convolution kernel sizes. In this embodiment, 4x4, 8x8, 16x16, and 32x32 scale convolution kernels and a standard convolution module with a step size of 4 are taken as examples. The calculation formula is shown in formula 1. In other scenarios, the number of corresponding convolution modules, the size of the convolution kernel, and the step size can be adjusted according to actual conditions.

[0089] Out = Concat(Conv1(in), Conv2(in), Conv3(in), Conv4(in)); Formula 1

[0090] Out represents the output result of all convolution modules, Concat represents the merging function, Conv1(in), Conv2(in), Conv3(in), and Conv4(in) represent the feature maps output by each convolution module.

[0091] After using the convolution parallel structure, a feature selection operation is performed on the output of each convolution layer: a shared channel attention module Channel attention is used for feature screening to reduce the parameter amount of the model in a parameter sharing manner, and finally, the four feature maps are concatenated to obtain a downsampling feature map, Figure 3 A multi-scale spatial information aggregation structure diagram provided by an embodiment of the present application, as shown in Figure 3As shown, the embodiment reduces the influence of image noise on deep network features while retaining important features.

[0092] The multi-scale spatial information aggregation structure efficiently solves the problem of serious loss of small target position information during initial image downsampling in the Transformer, significantly improves the effect of small target semantic segmentation, and can be applied to other computer vision fields.

[0093] The feature extraction process of the sparse multi-layer perceptron structure is as follows:

[0094] After image downsampling, the feature map is sent to the embedding module for linear coding operation (such as Figure 2 ), to obtain a linear coding feature map, and then the sparse multi-layer perceptron module is used to extract features from the feature map, to extract global semantic information (also known as high-level semantic information) and local semantic information (also known as local position information). The sparse multi-layer perceptron module is composed of a sparse multi-layer perceptron, a depthwise separable convolution, and a residual structure. The structure diagram of the entire module is as shown in Figure 4 , Figure 4 A sparse multi-layer perceptron module structure diagram provided by the embodiment of the present application. Figure 4 In the formula, Channel Split represents channel splitting according to a ratio of (1-r):r, Sparse MLP represents a sparse multi-layer perceptron, 3x3DWConv (Depthwise Convolution) represents a depthwise separable convolution with a convolution kernel size of 3, BN (Batch Normalization) is an algorithm frequently used in deep networks to accelerate neural network training, accelerate convergence speed, and stabilize the network, ReLU represents an activation function, and Shuffle represents a function of randomly sorting all elements of a sequence.

[0095] The specific operations of the sparse multi-layer perceptron module for processing the feature map include the following five parts: (1) feature map channel splitting; (2) MLP module extracting global information; (3) depthwise separable convolution extracting local information; (4) global information and local information fusion; and (5) constructing multiple residual structure blocks.

[0096] (1) Feature map channel splitting: for the input linear coding feature map , the channel dimension is split according to a preset ratio to obtain a first sub-feature map and a second sub-feature map . Taking equal ratio splitting (i.e., r=0.5) as an example, C1=C2, C1+C2=C, C1, C2, and C represent the number of channels of the feature map, and H and W represent the height and width of the feature map.

[0097] (2) MLP module extracts global semantic information: the obtained first sub-feature map f1 is taken as input feature, and a fully connected layer with input and output channel number H is used to process f1 to obtain a feature map with dimension [H, W], which is taken as the global semantic information. Two branches are used to calculate the global semantic information of the pixels in the same row and the same column, respectively, using a fully connected layer. For the row branch, the permute operation is performed on the first sub-feature map f1 to obtain a feature map with dimension [H, W], which is taken as the global semantic information. Then, a fully connected layer with input and output channel number H is used to process the feature map, and the result is taken as the row processing result. That is,

[0098] f H = linear(permute(f1)) Formula 2

[0099] Similarly, for the column branch, a fully connected layer with input and output channel number W is used to process the first sub-feature map f1 to obtain the column processing result. That is,

[0100] f W = linear(f1) Formula 3

[0101] Finally, the row processing result and the column processing result are taken as the matrix point multiplication to obtain a feature map with dimension [C1, H, W], and the matrix point multiplication is performed with the above-mentioned first sub-feature map f1 to obtain the global semantic information. That is,

[0102] f smlp = (f H × f W ) × f1 Formula 4

[0103] In the above process, the flow chart of using parallel branches to obtain the global context information of the feature is shown in Figure 5 .

[0104] (3) Deep separable convolution for local semantic information extraction: In this embodiment, a deep separable convolution is used to enhance the local information of the feature. The second sub-feature map f2 obtained in the process (1) is taken as the input feature, and a deep separable convolution with a convolution kernel size of 3 is used to process f2 to obtain an output with dimension C2 × H × W. A batch normalization layer (Batch Normalization, BN) and a nonlinear activation function ReLU are added after the deep separable convolution layer to process the output, and finally the local semantic information is obtained. The purpose of using normalization and nonlinear activation function is to accelerate the convergence of the model and enhance the generalization ability of the model. The flow chart of the entire deep separable convolution module is shown in Figure 6 , and the calculation formula is:

[0105] f dwReLU(BN(DWConv(f2))) ; Equation 5

[0106] (4) Global information and local information fusion: The output f smlp of the sparse multi-layer perceptron layer and the output f dw of the depth separable convolution module are concatenated and the channel order of the feature map is shuffled through the channel Shuffle operation to obtain the fused feature f out . The Shuffle operation plays a role of channel exchange, which reduces the parameter quantity of the model and reduces the calculation amount. The structural diagram of the feature fusion module is shown in Figure 7 Figure 7 where C1 = C2 = C / 2, and the calculation formula is as follows:

[0107] f out = Shuffle(Concat(f smlp ,f dw )) ; Equation 6

[0108] (5) Constructing multiple residual structure blocks: The residual structure is used to alleviate the gradient vanishing or gradient explosion problem caused by increasing the depth in the deep neural network, and the following calculation is performed to obtain the output f res of the entire sparse multi-layer perceptron module (i.e., the output feature). The entire sparse MLP module is a block structure, and multiple block structures are constructed to complete the final design of the network.

[0109] f res = f + f out ; Equation 7

[0110] The embodiment proposes an efficient sparse multi-layer perceptron structure. The structure only uses two MLP branches to calculate the context information of the same row and the context information of the same column of a pixel position on the feature map, and simultaneously uses a depth separable convolution to extract local features of an image. Compared with the self-attention, the calculation amount of the model is significantly reduced, the local semantic information of the image is obtained, and the feature expression ability of the model is improved; compared with the model of the convolutional neural network architecture, the global semantic information is more easily obtained. In addition, the structure can be embedded into other neural network models as plug and play. The embodiment can simply and efficiently extract global and local semantic information of an image, and improve the feature expression ability of a semantic segmentation model.

[0111] The model training process based on random sampling calculation loss is as follows:

[0112] ​The commonly used semantic segmentation loss function is basically using cross-entropy loss function and Dice loss function. When the depth and width of the model are small, the risk of overfitting of the model is small, but due to the difficulty of data collection and the time-consuming of labeling, the available data is relatively small, and a model with large depth and width is generally used, which is easy to cause model overfitting. To solve this problem, the present scheme proposes a loss function training method of random sampling, which calculates the final gradient by random sampling, which can effectively accelerate the model convergence and bring additional regularization effect. The embodiment fully utilizes the randomness of gradient update, which can efficiently alleviate the overfitting phenomenon and improve the generalization performance of the model.

[0113] The commonly used cross-entropy loss function is calculated by the following formula:

[0114]

[0115] The commonly used cross-entropy loss function is better when the network parameters are less, as shown in the following formula: Figure 8 The general semantic segmentation loss function calculation is to directly calculate the total loss of the output and the corresponding label, and then calculate all the gradients through back propagation and update the parameters. The random sampling loss calculation method proposed in the present scheme is to randomly sample the output image, calculate the loss value of the selected pixels and the corresponding label, and the unselected pixels do not participate in any calculation. Figure 8 The feature map before calculating the loss, the general loss function calculation and the random function calculation of random sampling are shown, and the gray blocks in the figure represent the pixel points participating in the loss calculation. The specific implementation method of the model training method of random sampling loss calculation is as follows:

[0116] Taking the Cityscapes dataset as an example, the input image size is 512x1024, and the feature maps with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image width and height are obtained after the above first and second step processing. The four feature maps are aggregated by FPN (Feature Pyramid Networks) information and processed by UPerNet network to obtain a semantic segmentation result image with a size of 512x1024. After the forward inference of the model, a random sampling loss function based on binary cross-entropy is constructed, and 128x256 pixel points are randomly sampled to calculate the cross-entropy loss value. That is, 32768 pixels are sampled to calculate the loss value, and then the corresponding back propagation and parameter update are performed. The loss calculation formula loss is as follows:

[0117]

[0118] K is the number of selected pixels, p represents the probability of correct prediction, and y i is the label of sample i.

[0119] The loss calculated by the method can accelerate model training and convergence, reduce GPU memory occupancy during training, and because of the random sampling, the model also has randomness during parameter updating. The random parameter updating is more effective than the commonly used regularization method in semantic segmentation. When the model is more complex, random sampling to calculate the loss value can more easily alleviate model overfitting, and the model is more easily converged, greatly relieving the phenomenon of model overfitting.

[0120] The embodiment proposes an efficient semantic segmentation model training method based on random sampling to calculate loss. The loss is calculated by randomly sampling the output of the network model, which brings better regularization effect to the model, promotes the convergence of the model, and improves the generalization performance of the model.

[0121] The image semantic segmentation device provided by the embodiment includes a downsampling module, a linear encoding module, a sparse multi-layer perception module, and a training module.

[0122] The downsampling module is configured to obtain an original image and extract a downsampling feature map of the original image using the downsampling module.

[0123] The linear encoding module is configured to perform linear encoding on the downsampling feature map to obtain a linear encoding feature map.

[0124] The sparse multi-layer perception module is configured to perform feature extraction on the linear encoding feature map using the sparse multi-layer perception module to obtain output features. The output features include global semantic information and local semantic information. The sparse multi-layer perception module includes a sparse multi-layer perception, a depth separable convolution structure, and a residual structure.

[0125] The training module is configured to train the image semantic segmentation model using the output features, so as to perform an image semantic segmentation task using the trained image semantic segmentation model.

[0126] The embodiment performs downsampling and linear encoding on an original image to obtain a linear encoding feature map, and performs feature extraction on the linear encoding feature map using a sparse multi-layer perception module to obtain output features. The output features extracted from the linear encoding feature map by the sparse multi-layer perception module include global semantic information and local semantic information. The image semantic segmentation model trained based on the output features can balance the speed and accuracy of image semantic segmentation, so that the image semantic segmentation operation has a faster speed and higher accuracy.

[0127] Furthermore, the sparse multilayer perceptron module is used to perform channel splitting on the linearly encoded feature map at a preset ratio to obtain a first sub-feature map and a second sub-feature map; it is also used to extract global semantic information from the first sub-feature map using the sparse multilayer perceptron; it is also used to extract local semantic information from the second sub-feature map using the depthwise separable convolutional structure; it is also used to fuse the global semantic information and the local semantic information to obtain fused features; and it is also used to input the fused features into the residual structure to obtain the output features.

[0128] Furthermore, the sparse multilayer perceptron includes a first processing branch and a second processing branch;

[0129] Accordingly, the process by which the sparse multilayer perceptron module extracts global semantic information of the first sub-feature map using the sparse multilayer perceptron includes: interacting with pixels in the same row of the first sub-feature map through a fully connected layer in the first processing branch to obtain a row processing result; interacting with pixels in the same column of the first sub-feature map through a fully connected layer in the second processing branch to obtain a column processing result; and generating global semantic information of the first sub-feature map based on the row processing result and the column processing result.

[0130] Furthermore, the depthwise separable convolutional structure includes depthwise separable convolutional layers, batch normalization layers, and activation functions;

[0131] Accordingly, the process by which the sparse multilayer perceptron module extracts local semantic information of the second sub-feature map using the depthwise separable convolutional structure includes: inputting the second sub-feature map into the depthwise separable convolutional layer to obtain the depthwise separable convolution result; and sequentially processing the depthwise separable convolution result using the batch normalization layer and the activation function to obtain the local semantic information of the second sub-feature map.

[0132] Furthermore, the process by which the sparse multilayer perceptron module fuses the global semantic information and the local semantic information to obtain fused features includes: fusing the global semantic information and the local semantic information to obtain a fusion result; and adjusting the channel order of the fusion result through a channel shuffle operation to obtain the fused features.

[0133] Furthermore, the downsampling module includes multiple convolutional modules and channel attention modules connected in parallel; all the convolutional modules have the same stride, and the kernel sizes of any two convolutional modules are different;

[0134] Correspondingly, the downsampling module is configured to input the original image into each of the convolution modules respectively; and configured to perform feature screening on output results of all the convolution modules by using the same channel attention module to obtain a downsampling feature map of the original image.

[0135] Further, the training module is configured to train the image semantic segmentation model by using a loss function of random sampling and the output feature; and configured to calculate a loss value according to randomly sampled pixel points in the output feature.

[0136] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, and are not described here.

[0137] The application further provides a storage medium having a computer program stored thereon, and the computer program can implement the steps provided by the above embodiments when executed. The storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0138] The application further provides an electronic device, which can include a memory and a processor, the memory has a computer program stored therein, and the processor can implement the steps provided by the above embodiments when calling the computer program in the memory. Of course, the electronic device can also include various network interfaces, power supplies and other components.

[0139] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the application, the application can be improved and modified, and these improvements and modifications also fall within the protection scope of the claims of the application.

[0140] It is further noted that the terminology "first", "second" and the like used in the specification are merely used for distinguishing between similar objects and does not imply or imply any actual relationship or order between the objects. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the element.

Claims

1. An image semantic segmentation method, characterized in that, An image semantic segmentation method is applied to an electronic device that includes an image semantic segmentation model, wherein the image semantic segmentation model includes a downsampling module and a sparse multilayer perceptron module, and the image semantic segmentation method includes: The original image is acquired, and the downsampling module is used to extract the downsampled feature map of the original image. The downsampled feature map is linearly encoded to obtain a linearly encoded feature map; The sparse multilayer perceptron module is used to extract features from the linear encoded feature map to obtain output features; wherein, the output features include global semantic information and local semantic information, and the sparse multilayer perceptron module includes a sparse multilayer perceptron, a depthwise separable convolutional structure and a residual structure; The image semantic segmentation model is trained using the output features, and the trained image semantic segmentation model is used to perform the image semantic segmentation task. The sparse multilayer perceptron module is used to extract features from the linearly encoded feature map to obtain output features, including: The linear encoded feature map is split into channels at a preset ratio to obtain a first sub-feature map and a second sub-feature map; The global semantic information of the first sub-feature map is extracted using the sparse multilayer perceptron. The local semantic information of the second sub-feature map is extracted using the depthwise separable convolutional structure. The global semantic information and the local semantic information are fused to obtain fused features; The fused features are input into the residual structure to obtain the output features.

2. The image semantic segmentation method according to claim 1, characterized in that, The sparse multilayer perceptron includes a first processing branch and a second processing branch. Accordingly, the global semantic information of the first sub-feature map is extracted using the sparse multilayer perceptron, including: In the first processing branch, pixels in the same row of the first sub-feature map are interacted through a fully connected layer to obtain the row processing result; In the second processing branch, pixels in the same column of the first sub-feature map are interacted through a fully connected layer to obtain the column processing result; Global semantic information of the first sub-feature map is generated based on the row processing results and the column processing results.

3. The image semantic segmentation method according to claim 1, characterized in that, The depthwise separable convolutional structure includes a depthwise separable convolutional layer, a batch normalization layer, and an activation function. Accordingly, the local semantic information of the second sub-feature map is extracted using the depthwise separable convolutional structure, including: The second sub-feature map is input into a depthwise separable convolutional layer to obtain a depthwise separable convolution result. The depthwise separable convolution result is processed sequentially using the batch normalization layer and the activation function to obtain the local semantic information of the second sub-feature map.

4. The image semantic segmentation method according to claim 1, characterized in that, The global semantic information and the local semantic information are fused to obtain fused features, including: The global semantic information and the local semantic information are fused to obtain the fusion result; The fusion feature is obtained by adjusting the channel order of the fusion result through a channel shuffle operation.

5. The image semantic segmentation method according to claim 1, characterized in that, The downsampling module includes multiple convolutional modules and channel attention modules connected in parallel; all the convolutional modules have the same stride, and the kernel sizes of any two convolutional modules are different; Accordingly, the downsampling module is used to extract the downsampled feature map of the original image, including: The original image is input into each of the convolutional modules respectively; By using the same channel attention module to perform feature filtering on the output results of all the convolution modules, a downsampled feature map of the original image is obtained.

6. The image semantic segmentation method according to claim 1, characterized in that, Training the image semantic segmentation model using the output features includes: The image semantic segmentation model is trained using a loss function based on random sampling and the output features; The loss function of random sampling calculates the loss value based on a portion of randomly sampled pixels in the output features.

7. An image semantic segmentation device, characterized in that, The image semantic segmentation device includes a downsampling module, a linear coding module, a sparse multilayer perceptron module, and a training module; The downsampling module is used to acquire the original image and extract the downsampling feature map of the original image. The linear encoding module is used to perform linear encoding on the downsampled feature map to obtain a linearly encoded feature map; The sparse multilayer perceptron module is used to extract features from the linear encoded feature map to obtain output features; wherein, the output features include global semantic information and local semantic information, and the sparse multilayer perceptron module includes a sparse multilayer perceptron, a depthwise separable convolutional structure and a residual structure. A training module is used to train an image semantic segmentation model using the output features, so as to perform an image semantic segmentation task using the trained image semantic segmentation model; wherein, the image semantic segmentation model is the image semantic segmentation model according to any one of claims 1 to 6; The sparse multilayer perceptron module is used to split the linear encoded feature map into channels at a preset ratio to obtain a first sub-feature map and a second sub-feature map; it is also used to extract global semantic information from the first sub-feature map using the sparse multilayer perceptron; it is also used to extract local semantic information from the second sub-feature map using the depthwise separable convolutional structure; it is also used to fuse the global semantic information and the local semantic information to obtain fused features; and it is also used to input the fused features into the residual structure to obtain the output features.

8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image semantic segmentation method as described in any one of claims 1 to 6 when it invokes the computer program in the memory.

9. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the image semantic segmentation method as described in any one of claims 1 to 6.

Images