Semantic segmentation method, device, electronic device and storage medium

Abstract

Embodiments of the present application provide a semantic segmentation method, apparatus, electronic device, and storage medium method, the method comprising: extracting original features of each frame in multiple frames of a video, the multiple frames comprising: a current frame and at least one previous frame, the previous frame being located before the current frame; inputting the original features of each frame into a semantic segmentation model to obtain a semantic segmentation result of the current frame, the semantic segmentation model being configured to: determine alignment features of each frame based on the original features of each frame; for each previous frame, perform deformable convolution on the alignment features of the previous frame based on the alignment features of the previous frame and the alignment features of the current frame to obtain alignment spatial features of the previous frame, obtain alignment motion features of the current frame based on the alignment features of the current frame and the alignment spatial features of each previous frame; and predict the semantic segmentation result of the current frame based on the alignment motion features.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and specifically to semantic segmentation methods, devices, electronic devices and storage media. Background Art

[0002] Semantic segmentation is a computer vision technology that determines the semantic category of each pixel in an image, such as vehicle category and pedestrian category. It is widely used in vehicle visual perception and high-precision map construction. When semantic segmentation is performed on the current frame, in order to improve the consistency between the semantic segmentation result of the current frame and the semantic segmentation result of the previous frame before the current frame, it is necessary to align the features between the current frame and the previous frame to obtain the aligned motion features of the current frame. Then, the semantic segmentation result is obtained based on the aligned motion features of the current frame.

[0003] At present, the optical flow between the current frame and each previous frame is usually calculated, and the alignment feature of the current frame is obtained according to the optical flow between the current frame and each previous frame, the features of each previous frame, and the features of the current frame. On the one hand, the calculation of the optical flow between the current frame and the previous frame requires the calculation of the optical flow between any two pixels, one of which is from the current frame and the other is from the previous frame, resulting in the process of obtaining the alignment motion feature of the current frame consuming a lot of computing resources, and semantic segmentation consumes a lot of computing resources. On the other hand, common situations such as the partial occlusion of objects such as vehicles and pedestrians in the current frame and / or the previous frame, and the large change in the viewing angle of the camera that collects the current frame and the previous frame during the collection of the current frame and the previous frame will all lead to the low accuracy of the calculated optical flow between the current frame and the previous frame. The alignment motion feature of the current frame is obtained by using the optical flow between the current frame and the previous frame with low accuracy, resulting in the low accuracy of the alignment motion feature of the current frame. Summary of the invention

[0004] The present application provides a semantic segmentation method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present application, a semantic segmentation method is provided, comprising:

[0006] Extracting original features of each frame in multiple frames of a video, wherein each frame is located continuously in the video, and the multiple frames include: a current frame and at least one previous frame, wherein the previous frame is located before the current frame;

[0007] The original features of each frame are input into the semantic segmentation model to obtain the semantic segmentation result of the current frame, and the semantic segmentation model is configured as follows:

[0008] Based on the original features of each frame, determining features for alignment of each frame;

[0009] For each previous frame, based on the features for alignment of the previous frame and the features for alignment of the current frame, a deformable convolution is performed on the features for alignment of the previous frame to obtain an alignment spatial feature of the previous frame, and based on the features for alignment of the current frame and the alignment spatial features of each previous frame, an alignment motion feature of the current frame is obtained;

[0010] Based on the aligned motion features, a semantic segmentation result of the current frame is predicted, where the semantic segmentation result includes: a semantic category of each current frame pixel in the current frame.

[0011] According to a second aspect of an embodiment of the present application, a semantic segmentation device is provided, including:

[0012] A feature extraction unit is configured to extract original features of each frame in a plurality of frames of a video, wherein each frame is located continuously in the video, and the plurality of frames include: a current frame and at least one previous frame, wherein the previous frame is located before the current frame;

[0013] The semantic segmentation unit is configured to input the original features of each frame into a semantic segmentation model to obtain a semantic segmentation result of the current frame, and the semantic segmentation model is configured to: determine the features for alignment of each frame based on the original features of each frame; for each previous frame, perform deformable convolution on the features for alignment of the previous frame based on the features for alignment of the previous frame and the features for alignment of the current frame to obtain the alignment spatial features of the previous frame, and obtain the alignment motion features of the current frame based on the features for alignment of the current frame and the alignment spatial features of each previous frame; predict the semantic segmentation result of the current frame based on the alignment motion features, and the semantic segmentation result includes: the semantic category of each current frame pixel in the current frame.

[0014] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a semantic segmentation method.

[0015] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, which enables the electronic device to perform a semantic segmentation method when instructions in the storage medium are executed by a processor of an electronic device.

[0016] The semantic segmentation method, device, electronic device and storage medium provided in the embodiments of the present application are

[0017] When aligning features between the current frame and the previous frame during semantic segmentation, the features for alignment of each frame are determined based on the original features of each frame in the multiple frames; for each previous frame, based on the features for alignment of the previous frame and the features for alignment of the current frame, deformable convolution is performed on the features for alignment of the previous frame to obtain the alignment spatial features of the previous frame, and based on the features for alignment of the current frame and the alignment spatial features of each previous frame, the alignment motion features of the current frame are obtained. There is no need to calculate the optical flow between the current frame and each previous frame, saving resources consumed in the process of obtaining the alignment motion features of the current frame, thereby saving computing resources consumed by semantic segmentation. Avoid situations in which the accuracy of the calculated optical flow is low, resulting in low accuracy of the alignment motion features of the current frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] Figure 1 A flowchart of the semantic segmentation method provided by an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of a process of obtaining the alignment motion features of the current frame is shown;

[0021] Figure 3 A schematic diagram of a process of obtaining the semantic category of each pixel in the current frame is shown;

[0022] Figure 4 A structural schematic diagram of a semantic segmentation device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] Figure 1 is a flow chart of the semantic segmentation method provided in the embodiment of the present application. The method comprises the following steps:

[0026] Step 101, extracting original features of each frame in multiple frames of a video.

[0027] In the present application, each frame in the multi-frame is located continuously in the video to which the multi-frame belongs. The multi-frame includes: a current frame, and a previous frame before the current frame. The previous frame is located before the current frame. For example,

[0028] The multiple frames in the video include: the tth frame in the video, the t-1th frame in the video, and the t-2th frame in the video. The tth frame is the current frame, and the t-1th frame and the t-2th frame are both previous frames.

[0029] In the present application, the original features of each frame in the multiple frames are extracted by a feature extraction network. The feature extraction network can be obtained based on a convolutional neural network such as a ResNet network. The feature extraction part in the convolutional neural network can be used as a feature extraction network, and the feature extraction part includes multiple feature extraction units, and each feature extraction unit corresponds to a feature extraction stage. The feature extraction part in the convolutional neural network can be further improved to obtain a feature extraction network. For example, the convolution layer with a step size in the last feature extraction unit in the feature extraction part of ResNet is replaced with a hole convolution layer with a hole rate of 4, and the convolution layer with a step size in the previous feature extraction unit of the last feature extraction unit is replaced with a hole convolution layer with a hole rate of 2 to obtain a feature extraction network.

[0030] For each frame in the multiple frames, the frame can be input into a feature extraction network, and the feature extraction network outputs the original features of the frame.

[0031] Step 102, input the original features of each frame into the semantic segmentation model to obtain the semantic segmentation result of the current frame.

[0032] In the present application, the semantic segmentation model is configured as follows: based on the original features of each frame in multiple frames, determine the features for alignment of each frame; for each previous frame, based on the features for alignment of the previous frame and the features for alignment of the current frame, perform deformable convolution on the features for alignment of the previous frame to obtain the aligned spatial features of the previous frame, and based on the features for alignment of the current frame and the aligned spatial features of each previous frame, obtain the aligned motion features of the current frame; based on the aligned motion features of the current frame, predict the semantic segmentation result of the current frame, and the semantic segmentation result of the current frame includes: the semantic category of each current frame pixel in the current frame.

[0033] In this application, pixels in the current frame are referred to as current frame pixels.

[0034] When the features for alignment of each frame are determined based on the original features of each frame in the multiple frames, for each frame, the original features of the frame may be directly used as the features for alignment of the frame.

[0035] In the present application, the semantic segmentation model may include a deformable convolution (DCN) layer.

[0036] When deformable convolution is performed on the features for alignment of the previous frame based on the features for alignment of the previous frame and the features for alignment of the current frame, the features for alignment of the previous frame and the features for alignment of the current frame can be input into the deformable convolution layer, and the convolution layer for predicting the offset in the deformable convolution layer predicts the offset corresponding to the previous frame based on the features for alignment of the previous frame and the features for alignment of the current frame. The deformable convolution kernel in the deformable convolution layer can perform deformable convolution on the features for alignment of the previous frame based on the offset corresponding to the previous frame to obtain the alignment spatial features of the previous frame.

[0037] In the present application, the semantic segmentation model may include a multi-head attention (Multi-Head Self-Attention module, MHSA for short) module for obtaining aligned motion features. The structure of each multi-head attention module in the present application may adopt the structure of the multi-head attention module in the existing Transformer model.

[0038] When the alignment motion features of the current frame are obtained based on the alignment features of the current frame and the alignment spatial features of each previous frame, the alignment features of the current frame and the alignment spatial features of each previous frame can be input into a multi-head attention module for obtaining the alignment motion features, and the multi-head attention module for obtaining the alignment motion features outputs the alignment motion features of the current frame.

[0039] In the present application, the semantic segmentation result includes: the semantic category of each current frame pixel in the current frame.

[0040] In the present application, the semantic segmentation model may include: a softmax layer for predicting semantic categories of pixels in a frame based on aligned motion features of the frame.

[0041] When predicting the semantic segmentation result of the current frame based on the aligned motion features of the current frame, the aligned motion features of the current frame can be input into a softmax layer used to predict the semantic categories of pixels in the frame based on the aligned motion features of the frame, and probability information of each pixel of the current frame output by the softmax layer used to predict the semantic categories of pixels in the frame based on the aligned motion features of the frame is obtained.

[0042] For each current frame pixel, the probability information of the current frame pixel includes: the probability of the current frame pixel in each preset semantic category.

[0043] For a current frame pixel and a preset semantic category, the probability of the current frame pixel being in the preset semantic category is the probability that the semantic category of the current frame pixel is the preset semantic category.

[0044] For example, preset semantic categories such as vehicle category, pedestrian category...semantic category N are pre-set, and the probability information of a current frame pixel includes: the probability of the current frame pixel in the vehicle category, the probability of the current frame pixel in the pedestrian category...the probability of the current frame pixel in the semantic category N.

[0045] For a current frame pixel and a preset semantic category, the probability of the current frame pixel on the preset semantic category corresponds to the preset semantic category.

[0046] In the present application, after obtaining the probability information of each current frame pixel output by the softmax layer for predicting the semantic category of the pixel in the frame based on the aligned motion feature of the frame, for each current frame pixel, the preset semantic category corresponding to the maximum probability in the probability information of the current frame pixel can be determined, and the preset semantic category corresponding to the maximum probability in the probability information of the current frame pixel is determined as the semantic category of the current frame pixel. Thus, the semantic category of each current frame pixel can be obtained, and the semantic segmentation result can be obtained.

[0047] In the present application, before executing step 101, a semantic segmentation model is pre-trained using a training sample set. Each time the semantic segmentation model is trained, one training sample is used.

[0048] In this application, the last frame in each training sample is uniformly referred to as the current training frame. The current training frame in each training sample is different. The frame before the current training frame in each training sample is uniformly referred to as the previous training frame. The previous training frame in each training sample is different. The training sample includes the current training frame and the previous training frame, and each frame in the training sample is continuous in the position of the video to which the frame in the training sample belongs.

[0049] In a training of a semantic segmentation model, the original features of each training frame in a training sample are extracted. The original features of each training frame are input into the semantic segmentation model to obtain the semantic segmentation result of the current training frame output by the semantic segmentation model. The semantic segmentation result of the current training frame includes: the predicted semantic category of each pixel in the current training frame, and the annotation result of the current training frame includes: the annotated semantic category of each pixel in the current training frame.

[0050] The loss corresponding to the current training frame can be calculated using a cross entropy function, and the loss corresponding to the current training frame is the loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame. According to the loss corresponding to the current training frame, the parameter values ​​of all parameters of the semantic segmentation model related to the loss corresponding to the current training frame can be updated. The parameter values ​​of all parameters of the semantic segmentation model related to the loss corresponding to the current training frame include: at least part of the parameters of the multi-head attention module for obtaining the aligned motion features, and at least part of the parameters of the softmax layer for predicting the semantic category of the pixels in the frame based on the aligned motion features of the frame.

[0051] In some embodiments, based on the original features of each frame, determining the features of each frame for alignment includes: inputting the original features of the current frame into the multi-head attention module corresponding to the current frame, and obtaining the features of the current frame output by the multi-head attention module corresponding to the current frame for alignment; for each previous frame, using the multi-head attention module corresponding to the previous frame based on the original features of the previous frame to obtain the features of the previous frame for alignment.

[0052] In the present application, the number of frames included in the multi-frame is a preset number. The preset number is N, and each time the semantic segmentation of the corresponding current frame is performed, it is for the corresponding N frames. The corresponding current frame in the corresponding multi-frame is the last frame in the corresponding multi-frame, and the frame farthest from the corresponding current frame in the corresponding multi-frame is the first frame in the corresponding multi-frame. The Mth frame in each corresponding multi-frame corresponds to the same multi-head attention module.

[0053] In the present application, when the features for alignment of each frame are determined based on the original features of each frame, the original features of the current frame can be input into the multi-head attention module corresponding to the current frame to obtain the features for alignment of the current frame output by the multi-head attention module corresponding to the current frame.

[0054] For each previous frame, the multi-head attention module corresponding to the previous frame can be used to obtain the features of the previous frame for alignment based on the original features of the previous frame. The original features of the previous frame can be input into the multi-head attention module corresponding to the previous frame to obtain the features of the previous frame for alignment output by the multi-head attention module corresponding to the previous frame.

[0055] In some embodiments, for each previous frame, the multi-head attention module corresponding to the previous frame is used to obtain the features of the previous frame for alignment based on the original features of the previous frame, including: downsampling the original features of the previous frame to obtain the downsampled features of the previous frame; inputting the downsampled features of the previous frame into the multi-head attention module corresponding to the previous frame to obtain the features of the previous frame output by the multi-head attention module corresponding to the previous frame; upsampling the features corresponding to the previous frame to obtain the features of the previous frame for alignment.

[0056] In the present application, the farther the distance between the previous frame and the current frame is, the larger the downsampling factor for the previous frame is, and the downsampling factor is the resolution of the feature map in the downsampled features of the previous frame divided by the resolution of the feature map in the original features of the previous frame. For example, the multiple frames in a video include: the tth frame in the video, the t-1th frame in the video, and the t-2th frame in the video. The current frame is the tth frame, and the t-1th frame and the t-2th frame are both previous frames. The downsampling factor for the t-1th frame is 1 / 2, and the downsampling factor for the t-2th frame is 1 / 4.

[0057] For a previous frame, the feature representation corresponding to the previous frame output by the multi-head attention module corresponding to the previous frame is obtained as:

[0058] X'=Attention(LN(x))+X

[0059] MHSAspa=MLP(ln(X'))+X'

[0060] Among them, MHSAspa represents the feature corresponding to the previous frame, X represents the original feature of the previous frame, Attention() is a function for processing the corresponding features based on the attention mechanism, LN() is a layer regularization function, and MLP() is a multi-layer perceptron function.

[0061] Please refer to Figure 2 , which shows a flow chart of obtaining the alignment motion features of the current frame.

[0062] exist Figure 2In the figure, the process of obtaining the aligned motion features of the current frame for the current frame and two previous frames is exemplarily shown. A structure for obtaining the aligned motion features of the current frame can be called a D-transformer structure, and a structure for obtaining the aligned motion features of the current frame is located in the semantic segmentation model. D-transformer structure: spatial pyramid transformer module and time domain alignment transformer module. The spatial pyramid transformer module includes: an MHSA module corresponding to the current frame and an MHSA module corresponding to each previous frame. The time domain alignment transformer module includes: an MHSA module for obtaining aligned motion features.

[0063] In the spatial pyramid transformer module, the original features of the current frame are input into the MHSA module corresponding to the current frame to obtain the features of the current frame for alignment output by the multi-head attention module corresponding to the current frame. For each previous frame, the original features of the previous frame are downsampled to obtain the downsampled features of the previous frame, and the downsampled features of the previous frame are input into the MHSA module corresponding to the previous frame to obtain the features corresponding to the previous frame output by the MHSA module corresponding to the previous frame, and the features corresponding to the previous frame are upsampled to obtain the features of the previous frame for alignment.

[0064] In the temporal alignment transformer module, for each previous frame, the alignment features of the previous frame are deformed convolved to obtain the alignment spatial features of the previous frame. The alignment features of the current frame and the alignment spatial features of each previous frame are input into the MHSA module for obtaining the alignment motion features to obtain the alignment motion features of the current frame output by the MHSA module for obtaining the alignment motion features.

[0065] The original feature of each frame is a tensor, and n is used to represent the tensor, n = C Fs ×H Fs ×W Fs C Fs Indicates the number of channels of the feature map, H Fs Represents the height of the feature map, W Fs Indicates the width of the feature map. The computational complexity of the spatial pyramid transformer module is C(ATT) = 3×n 2 The computational complexity of the temporal alignment transformer module is C(PST)=2×3n+2×9n=24n. The computational complexity of the D-transformer structure used to obtain the state features of the current frame is: C(D-Transformer)=C(PST)+C(ATT)=3n 2+24n. If the existing transformer model is used to process multiple frames Figure 2 The current frame and the previous frame shown in are processed. Since for any pixel in any frame, the similarity between the pixel and the pixels in every other frame is calculated, the existing transformer model is used to Figure 2 The computational complexity of processing the current frame and the previous frame shown in is C(transgormer)=3n×n×n=3n 3 The computational complexity of the D-transformer structure in the present application for processing multiple frames is much lower than the computational complexity of the existing transformer model for processing multiple frames. Through the D-transformer structure, the aligned motion features of the current frame can be quickly obtained.

[0066] In some embodiments, the original features of each frame are extracted respectively through a feature extraction network, the feature extraction network includes: multiple feature extraction units, the original features are output by the last feature extraction unit in the feature extraction network; and based on the aligned motion features of the current frame, predicting the semantic segmentation result of the current frame includes: inputting the aligned motion features of the current frame and the features corresponding to the current frame output by each feature extraction unit into a multi-head attention module for obtaining state features, and obtaining the state features of the current frame output by the multi-head attention module for obtaining state features, the state features of the current frame include: state sub-features of each pixel of the current frame, wherein the features corresponding to the current frame output by the feature extraction unit are the features output by the feature extraction unit when the current frame is input into the feature extraction network; based on the state features of the current frame, predicting the semantic segmentation result of the current frame.

[0067] In the present application, the original features of each frame can be extracted by a feature extraction network, the feature extraction network includes multiple feature extraction units, and the original features of each frame are output by the last feature extraction unit of the feature extraction network. The output of the Nth feature extraction unit of the feature extraction network is used as the input of the N+1th feature extraction unit of the feature extraction network, where N is less than the number of feature extraction units.

[0068] In the present application, for a current frame, after the current frame is input into a feature extraction network, for each feature extraction unit of the feature extraction network, the feature extraction unit outputs features corresponding to the current frame.

[0069] In the present application, the semantic segmentation model may include a multi-head attention module for obtaining state features.

[0070] The aligned motion features of the current frame and the features corresponding to the current frame output by each feature extraction unit can be input into a multi-head attention module for obtaining state features to obtain the state features of the current frame output by the multi-head attention module for obtaining state features.

[0071] In the present application, the semantic segmentation result of the current frame can be predicted based on the state features of the current frame. When predicting the semantic segmentation result of the current frame based on the state features of the current frame, the state features of the current training frame can be input into a softmax layer for predicting the semantic category of pixels in the frame based on the state features of the frame, and the probability information of the current frame output by the softmax layer for predicting the semantic category of pixels in the frame based on the state features of the frame is obtained.

[0072] For each current frame pixel, the probability information of the current frame pixel includes: the probability of the current frame pixel in each preset semantic category.

[0073] In the present application, after the probability information of the current frame is output by the softmax layer for predicting the semantic category of the pixel in the frame based on the state feature of the frame, for each pixel of the current frame, the preset semantic category corresponding to the maximum probability in the probability information of the pixel of the current frame can be determined, and the preset semantic category corresponding to the maximum probability in the probability information of the pixel of the current frame is determined as the semantic category of the pixel of the current frame. Thus, the semantic category of each pixel of the current frame can be obtained, and the semantic segmentation result can be obtained.

[0074] In the present application, if the semantic segmentation model includes a multi-head attention module for obtaining state features, and predicts the semantic segmentation result of the current frame based on the state features of the current frame, then when the semantic segmentation model is pre-trained, the cross-entropy function is used to calculate the loss corresponding to the current training frame, that is, the loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame, and the parameter values ​​of all parameters of the semantic segmentation model related to the loss corresponding to the current training frame are updated. The parameter values ​​of all parameters of the semantic segmentation model related to the loss corresponding to the current training frame include: at least part of the parameters of the multi-head attention module for obtaining aligned motion features, at least part of the parameters of the multi-head attention module for obtaining state features, and at least part of the parameters of the softmax layer for predicting the semantic category of pixels in the frame based on the state features of the frame.

[0075] In some embodiments, the aligned motion feature of the current frame includes: the aligned motion sub-feature of each current frame pixel; further includes: inputting the aligned motion feature of the current frame into the first logistic regression layer to obtain probability information of each current frame pixel, wherein the probability information of the current frame pixel includes: the probability of the current frame pixel in each preset semantic category; based on the probability information of each current frame pixel, determining each semantic area, wherein the semantic area includes: the area occupied by each relevant current frame pixel of the semantic area, each relevant current frame pixel of the semantic area has the same preset semantic category, and the preset semantic category corresponding to the relevant current frame pixel is the preset semantic category corresponding to the maximum probability in the probability information of the relevant current frame pixel; for each semantic area, based on the aligned motion sub-feature of each relevant current frame pixel of the semantic area and the probability of each relevant current frame pixel in the preset semantic category corresponding to the relevant current frame pixel, determining the characteristics of the semantic area; and based on the state characteristics of the current frame, predicting the semantic segmentation result of the current frame includes: for each current frame pixel, determining the semantic category of the current frame pixel based on the state sub-feature of the current frame pixel and the characteristics of each semantic area.

[0076] In the present application, the aligned motion features of the current frame may be input into the first logistic regression layer to obtain the probability information of each pixel of the current frame.

[0077] In the present application, the first logistic regression layer does not belong to the semantic segmentation model, and the first logistic regression layer can be a softmax layer.

[0078] In the present application, for each current frame pixel, the probability information of the current frame pixel includes: the probability of the current frame pixel in each preset semantic category.

[0079] For a current frame pixel and a preset semantic category, the probability of the current frame pixel being in the preset semantic category is the probability that the semantic category of the current frame pixel is the preset semantic category.

[0080] In the present application, each semantic region may be determined based on probability information of each current frame pixel in the current frame.

[0081] In the present application, for a semantic region, the semantic region includes: an area occupied by each relevant current frame pixel of the semantic region.

[0082] In other words, the semantic region is composed of the area occupied by each relevant current frame pixel of the semantic region.

[0083] In the present application, for a current frame pixel and a preset semantic category, the probability of the current frame pixel in the preset semantic category corresponds to the preset semantic category.

[0084] In the present application, for a semantic area, the preset semantic category corresponding to each relevant current frame pixel of the semantic area is the same, and the preset semantic category corresponding to the relevant current frame pixel of the semantic category is the preset semantic category corresponding to the maximum probability in the probability information of the relevant current frame pixel.

[0085] In the present application, each semantic region has a semantic category. For each semantic region, the semantic category of the semantic region is a preset semantic category corresponding to the relevant current frame pixels of the semantic region.

[0086] For example, for a semantic region, the semantic region includes a plurality of current frame pixels related to the semantic region. For each current frame pixel related to the semantic region, the preset semantic category corresponding to the maximum probability in the probability information of the current frame pixel related to the semantic region is the vehicle category, and the region occupied by each current frame pixel related to the semantic region constitutes the semantic region, and the category of the semantic region is the vehicle category.

[0087] In the present application, the alignment motion feature of the current frame includes: an alignment motion sub-feature of each pixel of the current frame.

[0088] For a semantic region, the feature of the semantic region is determined based on the aligned motion sub-feature of each relevant current frame pixel of the semantic region and the probability of each relevant current frame pixel being in a preset semantic category corresponding to the relevant current frame pixel.

[0089] For each relevant current frame pixel of the semantic area, the probability of the relevant current frame pixel in the preset semantic category corresponding to the relevant current frame pixel can be used as the weight of the aligned motion sub-feature of the relevant current frame pixel, and the weighted sum of the aligned motion sub-features of each relevant current frame pixel is calculated, and the weighted sum is determined as the feature of the semantic area.

[0090] In the present application, for each current frame pixel, when the semantic category of the current frame pixel is determined based on the state sub-features of the current frame pixel and the features of each semantic area, the similarity between the state sub-features of the current frame pixel and the features of each semantic area can be calculated, and the semantic area having the highest similarity with the state sub-features of the current frame pixel is determined, and the semantic category of the determined semantic area is determined as the semantic category of the current frame pixel.

[0091] In some embodiments, for each current frame pixel, determining the semantic category of the current frame pixel based on the state sub-feature of the current frame pixel and the feature of each semantic region includes: calculating the cosine distance between the state sub-feature of the current frame pixel and the feature of each semantic region; determining the target semantic region corresponding to the current frame pixel, wherein the target semantic region corresponding to the current frame pixel is the semantic region having the smallest cosine distance between the feature and the state sub-feature of the current frame pixel; determining the semantic category of the target semantic region corresponding to the current frame pixel as the semantic category of the current frame pixel, wherein the semantic category of the target semantic region corresponding to the current frame pixel is the preset semantic category corresponding to the relevant current frame pixel of the target semantic region.

[0092] The cosine distance between the state sub-feature of the i-th current frame pixel and the feature of the j-th semantic region can be calculated using the following formula:

[0093] d j =1-cos(p i , r j )

[0094]

[0095] Among them, d j represents the cosine distance between the state sub-feature of the i-th current frame pixel and the feature of the j-th semantic region, P i represents the state sub-feature of the i-th pixel in the current frame, r j represents the feature of the jth semantic region, |.| represents the modulus operation of the vector, and P i .r j Represents Pi and r j The dot product of .

[0096] Please refer to Figure 3 , which shows a flowchart of obtaining the semantic category of each pixel in the current frame.

[0097] The aligned motion features of the current frame are obtained through the original features of each frame. The probability information of each pixel of the current frame is obtained through the aligned motion features of the current frame. Each semantic region is determined through the probability information of each pixel of the current frame. The features of each semantic region are determined through the aligned motion features of the current frame and the probability information of each pixel of the current frame.

[0098] By aligning the motion features and the features corresponding to the current frame output by each feature extraction unit, the state features of the current frame are obtained. The state features of the current frame include the state sub-features of each pixel of the current frame.

[0099] For each current frame pixel, the semantic category of the current frame pixel is determined according to the cosine distance between the state sub-feature of the current frame pixel and the feature of each semantic region.

[0100] In some embodiments, before extracting the original features of each frame in the multiple frames of the video, it also includes: training the semantic segmentation model using training samples, including: extracting the original features of each training frame in the training samples, wherein the position of each training frame in the video to which each training frame belongs is continuous, and the training samples include: the current training frame, at least one previous training frame, and the current training frame is located before the previous training frame; inputting the original features of each training frame into the semantic segmentation model to obtain the semantic segmentation result of the current training frame output by the semantic segmentation model; calculating the first loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame, and the annotation result of the current training frame includes: the annotated semantic category of each current training frame pixel in the current training frame; inputting the aligned motion feature of the current training frame into the first logistic regression layer to obtain the first probability information of each current training frame pixel in the current training frame, and determining the first prediction information of the current training frame based on the first probability information of each current training frame pixel, the first probability information of the current training frame pixel The method comprises the following steps: the first prediction information of the current training frame includes: the first probability of the pixel of the current training frame on each preset semantic category, the first prediction information of the current training frame includes: the first predicted semantic category of each pixel of the current training frame; the second loss between the first prediction information of the current training frame and the annotation result of the current training frame is calculated; the state feature of the current training frame is input into the second logistic regression layer to obtain the second probability information of each pixel of the current training frame, and based on the second probability information of each pixel of the current training frame, the second prediction information of the current training frame is determined, the second probability information of the pixel of the current training frame includes: the second probability of the pixel of the current training frame on each preset semantic category, the second prediction information of the current training frame includes: the second predicted semantic category of each pixel of the current training frame; the third loss between the second prediction information of the current training frame and the annotation result of the current training frame is calculated; based on the first loss, the second loss and the third loss, the total loss corresponding to the training sample is determined, and based on the total loss, the parameter values ​​of all parameters of the semantic segmentation model related to the total loss are updated.

[0101] In the present application, before executing step 101, a semantic segmentation model is pre-trained using a training sample set. Each time the semantic segmentation model is trained, one training sample is used.

[0102] In this application, the last frame in each training sample is uniformly referred to as the current training frame. The current training frame in each training sample is different. The frame before the current training frame in each training sample is uniformly referred to as the previous training frame. The previous training frame in each training sample is different.

[0103] In a training of a semantic segmentation model, a feature extraction network is used to extract the original features of each training frame in a training sample. For each training frame in a training sample, the training frame is input into the feature extraction network, and the feature extraction network outputs the original features of the training frame.

[0104] In a training of a semantic segmentation model, the original features of each training frame are input into the semantic segmentation model to obtain the semantic segmentation result of the current training frame output by the semantic segmentation model.

[0105] The semantic segmentation result of the current training frame includes: the predicted semantic category of each pixel of the current training frame.

[0106] In a training of a semantic segmentation model, a cross entropy function is used to calculate the first loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame.

[0107] The annotation result of the current training frame includes: the annotation semantic category of each pixel of the current training frame.

[0108] The first loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame is calculated using the cross entropy function and is expressed as:

[0109] L M =CE(M,G),L M represents the first loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame, CE represents the cross entropy function, M represents the semantic segmentation result of the current training frame, and G represents the annotation result of the current training frame.

[0110] In the present application, the alignment motion feature of the current training frame includes: the alignment motion sub-feature of each current training frame pixel in the current training frame.

[0111] The process of obtaining the alignment motion feature of the current training frame is the same as the process of obtaining the alignment motion feature of the current frame described above. The process of obtaining the alignment motion feature of the current training frame refers to the process of obtaining the alignment motion feature of the current frame described above.

[0112] In a training of a semantic segmentation model, the aligned motion features of the current training frame are input into the first logistic regression layer to obtain the first probability information of each pixel of the current training frame output by the first logistic regression layer.

[0113] The first logistic regression layer is used to predict first probability information of the pixels of the current training frame based on the aligned motion sub-features of the pixels of the current training frame.

[0114] The first logistic regression layer does not belong to the semantic segmentation model, and the first logistic regression layer can be a softmax layer.

[0115] For each current training frame pixel, the first probability information includes: a first probability of the current training frame pixel in each preset semantic category.

[0116] In a training of a semantic segmentation model, first prediction information of the current training frame may be determined based on first probability information of each pixel of the current training frame.

[0117] In the present application, the first prediction information of the current training frame includes: a first predicted semantic category of each current training frame pixel in the current training frame.

[0118] In a training of a semantic segmentation model, for each current training frame pixel in the current training frame, the preset semantic category corresponding to the maximum first probability in the first probability information of the training frame pixel is determined as the first predicted semantic category of the current training frame pixel.

[0119] In a training of a semantic segmentation model, the cross entropy function can be used to calculate the second loss between the first prediction information of the current training frame and the annotation result of the current training frame. The second loss between the first prediction information of the current training frame and the annotation result of the current training frame calculated using the cross entropy function is expressed as:

[0120] L rd =CE(M t’ , G), L rd represents the second loss between the first prediction information of the current training frame and the annotation result of the current training frame, CE represents the cross entropy function, M t’ represents the first prediction information of the current training frame, and G represents the annotation result of the current training frame.

[0121] In the present application, the state feature of the current training frame includes the state sub-feature of each pixel of the current training frame.

[0122] The process of obtaining the state characteristics of the current training frame is the same as the above process of obtaining the state characteristics of the current frame. The process of obtaining the state characteristics of the current training frame refers to the above process of obtaining the state characteristics of the current frame.

[0123] In a training of a semantic segmentation model, the state features of the current training frame may be input into the second logistic regression layer to obtain second probability information of each pixel of the current training frame output by the second logistic regression layer.

[0124] The second logistic regression layer is used to predict the second probability information of the current training frame pixel based on the state sub-feature of the current training frame pixel.

[0125] The second logistic regression layer does not belong to the semantic segmentation model, and the second logistic regression layer can be a softmax layer.

[0126] For each current training frame pixel, the second probability information of the current training frame pixel includes: a second probability of the current training frame pixel in each preset semantic category.

[0127] In a training of a semantic segmentation model, the second prediction information of the current training frame may be determined based on the second probability information of each pixel of the current training frame.

[0128] The second prediction information of the current training frame includes: a second predicted semantic category of each pixel of the current training frame.

[0129] For each current training frame pixel, the preset semantic category corresponding to the maximum second probability in the second probability information of the training frame pixel is determined as the second predicted semantic category of the training frame pixel.

[0130] In a training of a semantic segmentation model, the third loss between the second prediction information of the current training frame and the annotation result of the current training frame can be calculated using the cross entropy function. The third loss between the second prediction information of the current training frame and the annotation result of the current training frame calculated using the cross entropy function is expressed as:

[0131] L pd =CE(S,G),L pd represents the third loss between the second prediction information of the current training frame and the annotation result of the current training frame, CE represents the cross entropy function, S represents the second prediction information of the current training frame, and G represents the annotation result of the current training frame.

[0132] In a training of a semantic segmentation model, the total loss corresponding to the training sample is determined based on the first loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame, the second loss between the first prediction information and the annotation result of the current training frame, and the third loss between the second prediction information and the annotation result of the current training frame. The total loss corresponding to the training sample can be the sum of the first loss, the second loss, and the third loss. The total loss corresponding to the training sample is expressed as: L pd +L rd +L M .

[0133] In a training of a semantic segmentation model, based on the total loss corresponding to the training samples, the parameter values ​​of all parameters of the semantic segmentation model related to the total loss are updated.

[0134] In this application, each time the semantic segmentation model is trained, the total loss corresponding to the training sample is determined based on the first loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame, the second loss between the first prediction information and the annotation result of the current training frame, and the third loss between the second prediction information and the annotation result of the current training frame. Based on the total loss corresponding to the training sample, the parameter values ​​of all parameters of the semantic segmentation model related to the total loss are updated. This allows the semantic segmentation model to simultaneously learn how to more accurately generate aligned motion features, how to more accurately generate state features, and how to more accurately generate semantic segmentation results.

[0135] Please refer to Figure 4 , which shows a structural schematic diagram of a semantic segmentation device provided in an embodiment of the present application. Figure 4 As shown, the semantic segmentation device includes: a feature extraction unit 401 and a semantic segmentation unit 402.

[0136] The feature extraction unit 401 is configured to extract original features of each frame in a plurality of frames of a video, wherein each frame is located continuously in the video, and the plurality of frames include: a current frame and at least one previous frame, wherein the previous frame is located before the current frame;

[0137] The semantic segmentation unit 402 is configured to input the original features of each frame into a semantic segmentation model to obtain a semantic segmentation result of the current frame, and the semantic segmentation model is configured to: determine the features for alignment of each frame based on the original features of each frame; for each previous frame, perform deformable convolution on the features for alignment of the previous frame based on the features for alignment of the previous frame and the features for alignment of the current frame to obtain the alignment spatial features of the previous frame, and obtain the alignment motion features of the current frame based on the features for alignment of the current frame and the alignment spatial features of each previous frame; predict the semantic segmentation result of the current frame based on the alignment motion features, and the semantic segmentation result includes: the semantic category of each current frame pixel in the current frame.

[0138] In some embodiments, the semantic segmentation model is further configured to input the original features of the current frame into the multi-head attention module corresponding to the current frame, and obtain the features of the current frame for alignment output by the multi-head attention module corresponding to the current frame; for each previous frame, the multi-head attention module corresponding to the previous frame is used to obtain the features of the previous frame for alignment based on the original features of the previous frame.

[0139] In some embodiments, the semantic segmentation model is further configured to downsample the original features of the previous frame to obtain the downsampled features of the previous frame; input the downsampled features of the previous frame into the multi-head attention module corresponding to the previous frame to obtain the features corresponding to the previous frame output by the multi-head attention module corresponding to the previous frame; upsample the features corresponding to the previous frame to obtain the features of the previous frame for alignment.

[0140] In some embodiments, the original features of each frame are extracted respectively through a feature extraction network, and the feature extraction network includes: multiple feature extraction units, and the original features are output by the last feature extraction unit in the feature extraction network; the semantic segmentation model is further configured to input the aligned motion features and the features corresponding to the current frame output by each feature extraction unit into a multi-head attention module for obtaining state features, and obtain the state features of the current frame output by the multi-head attention module for obtaining state features, and the state features include: the state sub-features of each pixel of the current frame, wherein the features corresponding to the current frame output by the feature extraction unit are the features output by the feature extraction unit when the current frame is input into the feature extraction network; based on the state features, the semantic segmentation result of the current frame is predicted.

[0141] In some embodiments, the alignment motion feature includes: the alignment motion sub-feature of each current frame pixel; the semantic segmentation device also includes: a semantic region feature determination unit, configured to input the alignment motion feature into the first logistic regression layer to obtain probability information of each current frame pixel, wherein the probability information of the current frame pixel includes: the probability of the current frame pixel on each preset semantic category; based on the probability information of each current frame pixel, each semantic region is determined, wherein all semantic regions include: the region occupied by each relevant current frame pixel of the semantic region, the preset semantic category corresponding to each relevant current frame pixel is the same, and the preset semantic category corresponding to the relevant current frame pixel is the preset semantic category corresponding to the maximum probability in the probability information of the relevant current frame pixel; for each semantic region, based on the alignment motion sub-feature of each relevant current frame pixel of the semantic region and the probability of each relevant current frame pixel on the preset semantic category corresponding to the relevant current frame pixel, the feature of the semantic region is determined; the semantic segmentation model is further configured to determine the semantic category of the current frame pixel for each current frame pixel based on the state sub-feature of the current frame pixel and the feature of each semantic region.

[0142] In some embodiments, the semantic segmentation model is further configured to calculate the cosine distance between the state sub-features of the current frame pixels and the features of each of the semantic regions; determine the target semantic region corresponding to the current frame pixels, wherein the target semantic region is the semantic region having the smallest cosine distance between the features and the state sub-features of the current frame pixels; and determine the semantic category of the target semantic region as the semantic category of the current frame pixels, wherein the semantic category of the target semantic region is a preset semantic category corresponding to the relevant current frame pixels of the target semantic region.

[0143] In some embodiments, the semantic segmentation apparatus further comprises:

[0144] The training unit is configured to train the semantic segmentation model using training samples before extracting the original features of each frame in the multiple frames of the video, including: extracting the original features of each training frame in the training samples, wherein the position of each training frame in the video to which each training frame belongs is continuous, and the training samples include: a current training frame and at least one previous training frame, and the current training frame is located before the previous training frame; inputting the original features of each training frame into the semantic segmentation model to obtain the semantic segmentation result of the current training frame output by the semantic segmentation model; calculating the first loss between the semantic segmentation result of the current training frame and the annotation result of the current training frame, and the annotation result of the current training frame includes: the annotated semantic category of each current training frame pixel in the current training frame; inputting the aligned motion feature of the current training frame into the first logistic regression layer to obtain the first probability information of each current training frame pixel in the current training frame, and determining the first prediction information of the current training frame based on the first probability information of each current training frame pixel, and the first prediction information of the current training frame is determined based on the first probability information of each current training frame pixel. A probability information includes: a first probability of the current training frame pixel on each preset semantic category, the first prediction information includes: the first predicted semantic category of each current training frame pixel; a second loss between the first prediction information and the annotation result of the current training frame is calculated; the state feature of the current training frame is input into the second logistic regression layer to obtain the second probability information of each current training frame pixel, and based on the second probability information of each current training frame pixel, the second prediction information of the current training frame is determined, the second probability information includes: the second probability of the current training frame pixel on each preset semantic category, the second prediction information includes: the second predicted semantic category of each current training frame pixel; a third loss between the second prediction information and the annotation result of the current training frame is calculated; based on the first loss, the second loss, and the third loss, the total loss corresponding to the training sample is determined, and based on the total loss, the parameter values ​​of all parameters of the semantic segmentation model related to the total loss are updated.

[0145] The present application also provides an electronic device, which may be configured with one or more processors; a memory for storing one or more programs, wherein the one or more programs may include instructions for performing the operations described in the above embodiments. When the one or more programs are executed by one or more processors, the one or more processors are caused to perform the instructions for the operations described in the above method embodiments.

[0146] The present application also provides a storage medium, which can be included in an electronic device; or it can exist independently and not be installed in the electronic device. The above storage medium carries one or more programs, and when one or more programs are executed by the electronic device, the electronic device performs the operations described in the above method embodiment.

[0147] It should be noted that the storage medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium that includes or stores a program that can be used by or in combination with a message execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with a message execution system, device or device. The program code included on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0148] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code includes one or more executable messages for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer messages.

[0149] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical embodiments formed by a specific combination of the above technical features, but should also cover other technical embodiments formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) to form a technical embodiment.

Claims

1. A semantic segmentation method, characterized in that: The method comprises: Extracting original features of each frame in multiple frames of a video, wherein each frame is located continuously in the video, and the multiple frames include: a current frame and at least one previous frame, wherein the previous frame is located before the current frame; The original features of each frame are input into the semantic segmentation model to obtain the semantic segmentation result of the current frame, and the semantic segmentation model is configured as follows: Based on the original features of each frame, determining the features for alignment of each frame specifically includes: inputting the original features of the current frame into the multi-head attention module corresponding to the current frame, and obtaining the features for alignment of the current frame output by the multi-head attention module corresponding to the current frame; for each previous frame, using the multi-head attention module corresponding to the previous frame based on the original features of the previous frame to obtain the features for alignment of the previous frame; For each previous frame, based on the features for alignment of the previous frame and the features for alignment of the current frame, the features for alignment of the previous frame are subjected to deformable convolution to obtain the aligned spatial features of the previous frame, and the aligned spatial features of the previous frame and the features for alignment of the current frame are input into a multi-head attention module to obtain the aligned motion features of the current frame; Based on the aligned motion features, a semantic segmentation result of the current frame is predicted, where the semantic segmentation result includes: a semantic category of each current frame pixel in the current frame.

2. The method according to claim 1, characterized in that The features of the previous frame for alignment are obtained by using the multi-head attention module corresponding to the previous frame based on the original features of the previous frame, including: Downsampling the original features of the previous frame to obtain downsampled features of the previous frame; Inputting the downsampled features of the previous frame into the multi-head attention module corresponding to the previous frame, and obtaining the features corresponding to the previous frame output by the multi-head attention module corresponding to the previous frame; The features corresponding to the previous frame are up-sampled to obtain features of the previous frame for alignment.

3. The method according to claim 1, characterized in that The original features of each frame are extracted respectively by a feature extraction network, the feature extraction network comprising: a plurality of feature extraction units, the original features are output by the last feature extraction unit in the feature extraction network; and based on the aligned motion features, predicting the semantic segmentation result of the current frame comprises: Inputting the aligned motion feature and the feature corresponding to the current frame output by each of the feature extraction units into a multi-head attention module for obtaining a state feature, obtaining a state feature of the current frame output by the multi-head attention module for obtaining a state feature, wherein the state feature includes: a state sub-feature of each pixel of the current frame, wherein the feature corresponding to the current frame output by the feature extraction unit is a feature output by the feature extraction unit when the current frame is input into a feature extraction network; Based on the state features, a semantic segmentation result of the current frame is predicted.

4. The method according to claim 3, characterized in that The alignment motion feature includes: the alignment motion sub-feature of each current frame pixel; the method further includes: Inputting the aligned motion features into a first logistic regression layer to obtain probability information of each current frame pixel, wherein the probability information of the current frame pixel includes: the probability of the current frame pixel in each preset semantic category; Based on the probability information of each pixel of the current frame, each semantic region is determined, wherein the semantic region includes: The area occupied by each relevant current frame pixel of the semantic area, the preset semantic category corresponding to each relevant current frame pixel is the same, and the preset semantic category corresponding to the relevant current frame pixel is the preset semantic category corresponding to the maximum probability in the probability information of the relevant current frame pixel; For each semantic region, the characteristics of the semantic region are determined based on the aligned motion sub-features of each relevant current frame pixel of the semantic region and the probability of each relevant current frame pixel in a preset semantic category corresponding to the relevant current frame pixel; and based on the state characteristics, the semantic segmentation result of the current frame is predicted, including: for each current frame pixel, the semantic category of the current frame pixel is determined based on the state sub-features of the current frame pixel and the characteristics of each semantic region.

5. The method according to claim 4, characterized in that Determining the semantic category of the pixel of the current frame based on the state sub-feature of the pixel of the current frame and the feature of each of the semantic regions includes: Calculating the cosine distance between the state sub-feature of the pixel of the current frame and the feature of each of the semantic regions; Determine a target semantic region corresponding to the current frame pixel, wherein the target semantic region is a semantic region having a feature with a minimum cosine distance from a state sub-feature of the current frame pixel; The semantic category of the target semantic area is determined as the semantic category of the current frame pixel, wherein the semantic category of the target semantic area is a preset semantic category corresponding to the current frame pixel related to the target semantic area.

6. The method according to any one of claims 3 to 5, characterized in that: Before extracting the original features of each of the multiple frames of the video, the method further includes: Training the semantic segmentation model using the training samples includes: Extracting original features of each training frame in the training sample, wherein each training frame is located continuously in the video to which each training frame belongs, and the training sample includes: a current training frame and at least one previous training frame, and the current training frame is located before the previous training frame; Inputting the original features of each training frame into the semantic segmentation model to obtain the semantic segmentation result of the current training frame output by the semantic segmentation model; Calculating a first loss between a semantic segmentation result of a current training frame and a labeling result of the current training frame, wherein the labeling result of the current training frame includes: a labeling semantic category of each current training frame pixel in the current training frame; Inputting the aligned motion features of the current training frame into the first logistic regression layer, obtaining first probability information of each current training frame pixel in the current training frame, and determining first prediction information of the current training frame based on the first probability information of each current training frame pixel, wherein the first probability information includes: a first probability of the current training frame pixel on each preset semantic category, and the first prediction information includes: a first predicted semantic category of each current training frame pixel; Calculating a second loss between the first prediction information and the annotation result of the current training frame; Inputting the state feature of the current training frame into the second logistic regression layer to obtain second probability information of each pixel of the current training frame, and determining second prediction information of the current training frame based on the second probability information of each pixel of the current training frame, wherein the second probability information includes: a second probability of the pixel of the current training frame on each preset semantic category, and the second prediction information includes: a second predicted semantic category of each pixel of the current training frame; Calculating a third loss between the second prediction information and the annotation result of the current training frame; Based on the first loss, the second loss, and the third loss, a total loss corresponding to the training sample is determined, and based on the total loss, parameter values ​​of all parameters of the semantic segmentation model related to the total loss are updated.

7. A semantic segmentation device, characterized in that: The device comprises: A feature extraction unit is configured to extract original features of each frame in multiple frames of a video, wherein each frame is continuously positioned in the video, and the multiple frames include: a current frame and at least one previous frame, wherein the previous frame is located before the current frame; a semantic segmentation unit is configured to input the original features of each frame into a semantic segmentation model to obtain a semantic segmentation result of the current frame, wherein the semantic segmentation model is configured to: determine features of each frame for alignment based on the original features of each frame, specifically including: inputting the original features of the current frame into a multi-head attention module corresponding to the current frame, obtaining the features of the current frame output by the multi-head attention module corresponding to the current frame for alignment for each previous frame, using the multi-head attention module corresponding to the previous frame to obtain the features for alignment of the previous frame based on the original features of the previous frame; for each previous frame, based on the features for alignment of the previous frame and the features for alignment of the current frame, performing deformable convolution on the features for alignment of the previous frame to obtain the aligned spatial features of the previous frame, and inputting the aligned spatial features of the previous frame and the features for alignment of the current frame into the multi-head attention module to obtain the aligned motion features of the current frame; based on the aligned motion features, predicting the semantic segmentation result of the current frame, the semantic segmentation result including: the semantic category of each current frame pixel in the current frame.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6. 9 . A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 6.

Images