Video semantic segmentation method and model based on time sequence differential attention mechanism
By introducing the timing differential attention mechanism and adaptive timing feature fusion pyramid in the video semantic segmentation model, the existing model's dependence on the accuracy of optical flow prediction and high computational cost are solved, and a more efficient video semantic segmentation effect is achieved .
Patent Information
- Application Number
- CN202310965812.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-02
- Publication Date
- 2025-02-14
AI Technical Summary
When processing video, existing video semantic segmentation models rely on the accuracy of optical flow prediction and are susceptible to wrong optical flow, resulting in performance degradation, and at the same time, high calculation cost and low efficiency.
The video semantic segmentation method based on the timing differential attention mechanism is adopted, and the timing differential attention module is used to extract the timing characteristics between video frames, and the adaptive timing feature fusion pyramid is used to perform multi-scale feature fusion to reduce the consumption of computing resources.
It improves the prediction accuracy of video semantic segmentation, reduces the computational cost, and enhances the robustness of the model to videos of different motion amplitudes.
Smart Images
Figure CN119445425A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of semantic segmentation technology, and in particular to a video semantic segmentation method and model based on a temporal difference attention mechanism. Background Art
[0002] Semantic segmentation is a fundamental task in computer vision. It attempts to classify each pixel in an image, which is very important for applications such as autonomous driving, video understanding, and intelligent surveillance. Recently, many image semantic segmentation models have achieved good results on single static images. However, a more realistic requirement is to segment dynamic frames of videos containing temporal information. Image-based models usually segment videos frame by frame, which leads to reduced accuracy and increased computational cost. Therefore, research on video semantic segmentation is very important.
[0003] Current video semantic segmentation models mainly focus on extracting and applying temporal features to improve model performance. Currently, researchers generally use two methods to obtain and utilize temporal features. One method is to use temporal features based on optical flow to fuse spatial features from previous frames to segment the target frame. Another method is to use a self-attention mechanism to aggregate spatial features between multiple frames to enhance the target frame. Since the optical flow contains temporal features, optical flow-based models can improve the performance of video semantic segmentation better than image-based segmentation models. However, these models rely on the accuracy of optical flow prediction. Incorrect optical flow will reduce the performance of video semantic segmentation models. In addition, it takes a lot of time to calculate optical flow, which reduces the efficiency of optical flow-based video semantic segmentation models. In order to avoid the optical flow problem, the latter uses a self-attention mechanism to model the global relationship between video frames. This method enhances the features of the target frame by aggregating spatial features from other frames according to the global relationship. However, there are some serious defects in the self-attention method. First, self-attention requires the use of matrix multiplication to model global relationships. This makes these methods require a lot of computing resources, hindering their use in practical applications. In addition, since most frames in videos have similar content, simply modeling all pixels of all video frames will produce a lot of computational redundancy. Summary of the invention
[0004] The purpose of this invention is to provide a video semantic segmentation method and model based on temporal difference attention mechanism to improve prediction accuracy and reduce computational cost.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] A video semantic segmentation method based on temporal difference attention mechanism includes the following steps:
[0007] Step 1) Data annotation: annotate the acquired video image dataset and assign the category to each pixel in the image;
[0008] Step 2) preprocessing the image data set;
[0009] Step 3) extracting multi-scale spatial features of the target frame image in the video based on the encoder;
[0010] Step 4) According to the reference frame and the target frame image, the temporal difference attention module is used to extract the multi-scale temporal features of the image;
[0011] Step 5) fusing temporal features of different scales based on the adaptive temporal feature fusion pyramid;
[0012] Step 6) splicing the spatial features and the fused temporal features to obtain splicing features;
[0013] Step 7) Input the concatenated features into the prediction layer and output the prediction results.
[0014] The step 2) specifically includes: performing numerical standardization on the image data, and rotating and flipping the image with a preset probability.
[0015] The step 4) is specifically as follows: extracting the features of the target frame and the reference frame respectively and performing differentiation to obtain differential features, using the temporal differential attention mechanism to model the pixel relationship between the target frame and the reference frame and using the pixel relationship to fuse the features between the target frame and the reference frame to obtain temporal features.
[0016] The temporal difference attention module includes a fully connected layer, a feature difference calculation layer, a softmax layer, a multiplication layer and a multi-layer addition layer. The input target frame image is mapped into a query feature through the fully connected layer, and the reference frame image is mapped into a key feature and a value feature through the fully connected layer. The feature difference calculation layer performs a difference calculation based on the query feature and the key feature and then inputs the softmax layer. After the output of the sofmax layer and the value feature are jointly input into the multiplication layer, the output and the value feature of the reference frame are input into the multi-layer addition layer for addition to obtain the fused temporal feature.
[0017] The step 5) is specifically as follows: inputting the time series features into the adaptive time series feature fusion pyramid, calculating the weight coefficients of the time series features of different scales according to the global information of the time series features of each scale, and performing weighted fusion of the time series features of each scale using the weight coefficients.
[0018] A video semantic segmentation model based on temporal difference attention mechanism, including sequentially connected:
[0019] The input module is used to obtain a video image dataset with annotation information and perform data preprocessing as model input;
[0020] A spatial feature extraction module, used to extract multi-scale spatial features of a target frame image in a video based on an encoder;
[0021] The temporal difference attention module is used to extract multi-scale temporal features of the image based on the reference frame and the target frame image;
[0022] Adaptive temporal feature fusion pyramid module, used to fuse temporal features of different scales;
[0023] A splicing module is used to splice the spatial features and the fused temporal features to obtain splicing features;
[0024] The output module is used to output semantic segmentation prediction results based on the concatenated features.
[0025] The data preprocessing specifically includes: performing numerical standardization on the image data, and rotating and flipping the image with a preset probability.
[0026] The temporal difference attention module specifically performs the following steps: extracting the features of the target frame and the reference frame respectively and performing differentiation to obtain differential features, using the temporal difference attention mechanism to model the pixel relationship between the target frame and the reference frame and using the pixel relationship to fuse the features between the target frame and the reference frame to obtain temporal features.
[0027] The temporal difference attention module includes a fully connected layer, a feature difference calculation layer, a softmax layer, a multiplication layer and a multi-layer addition layer. The input target frame image is mapped into a query feature through the fully connected layer, and the reference frame image is mapped into a key feature and a value feature through the fully connected layer. The feature difference calculation layer performs a difference calculation based on the query feature and the key feature and then inputs the softmax layer. After the output of the sofmax layer and the value feature are jointly input into the multiplication layer, the output and the value feature of the reference frame are input into the multi-layer addition layer for addition to obtain the fused temporal feature.
[0028] The adaptive temporal feature fusion pyramid module specifically performs the following steps: calculating the weight coefficients of temporal features of different scales according to the global information of temporal features of each scale output by the temporal difference attention module, and weightedly fusing the temporal features of each scale using the weight coefficients.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] (1) The present invention uses a temporal difference attention mechanism to enable the model to efficiently and accurately extract the temporal relationship between frames in the video. The model uses the temporal relationship to enhance the features of the segmentation target frame by the features of the reference frame in the video, thereby improving the segmentation accuracy.
[0031] (2) According to the motion amplitude between frames in the input video, the semantic segmentation model of the present invention uses an adaptive temporal feature fusion pyramid to fuse temporal features of different scales. By adaptively fusing multi-scale temporal features, the model obtains better robustness in the video semantic segmentation prediction task.
[0032] (3) The present invention uses the temporal difference attention mechanism to obtain temporal features to improve the segmentation accuracy. In addition, since this operation only needs to calculate the difference between the target frame and the segmented frame, it avoids the matrix multiplication operation of self-attention and reduces the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 is a flow chart of the method of the present invention;
[0034] Figure 2 It is a schematic diagram of the structure of the semantic segmentation model of the present invention;
[0035] Figure 3 Schematic diagram of the structure of the temporal difference attention module in an embodiment of the present invention;
[0036] Figure 4 Schematic diagram of the structure of the adaptive temporal feature fusion pyramid module in an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0038] This embodiment first provides a video semantic segmentation method based on the temporal difference attention mechanism. Figure 1 As shown, the following steps are included:
[0039] Step 1) Data annotation: annotate the acquired video image dataset and assign the category to each pixel in the image;
[0040] Step 2) Preprocess the image data set: perform numerical standardization on the image data, and rotate and flip the image with a certain probability.
[0041] Step 3) extracting multi-scale spatial features of the target frame image in the video based on the encoder;
[0042] Step 4) According to the reference frame and the target frame image, the temporal difference attention module is used to extract the multi-scale temporal features of the image;
[0043] Step 5) Based on the adaptive temporal feature fusion pyramid, temporal features of different scales are fused to enhance the prediction robustness of the model for videos with different motion amplitudes;
[0044] Step 6) splicing the spatial features and the fused temporal features to obtain splicing features;
[0045] Step 7) Input the concatenated features into the prediction layer and output the prediction results.
[0046] This embodiment also provides a video semantic segmentation model based on the temporal difference attention mechanism, such as Figure 2 As shown, including the following connected in sequence:
[0047] ① Input module, used to obtain video image datasets with annotation information and perform data preprocessing as model input.
[0048] This embodiment makes It is the T frame image of the input video of the network.
[0049] ②Spatial feature extraction module, used to extract multi-scale spatial features of target frame images in the video based on the encoder.
[0050] The structure of the encoder is a common method in this field and will not be described in detail here to avoid blurring the purpose of this application.
[0051] ③ Temporal difference attention module, which is used to extract multi-scale temporal features of images based on reference frame and target frame images.
[0052] The temporal difference attention module specifically performs the following steps: extracting the features of the target frame and the reference frame respectively and performing differentiation to obtain differential features, using the temporal difference attention mechanism to model the pixel relationship between the target frame and the reference frame and using the pixel relationship to fuse the features between the target frame and the reference frame to obtain temporal features.
[0053] In one embodiment, if Figure 3 As shown in the figure, the temporal difference attention module includes a fully connected layer, a feature difference calculation layer, a softmax layer, a multiplication layer, and a multi-layer addition layer, which performs the following steps on the input features:
[0054] 1: Input features i represents the temporal position of the feature in the video. => 1,
[0055] 2: The target frame features at time t After being mapped to query features through the fully connected layer, that is, The reference frame features Mapped to key and value features, i.e. and 1,=>2,MLP
[0056] 3: The feature difference calculation layer calculates the temporal relationship ε based on the query,key obtained in the previous step and normalizes it through softmax. 2,=>3,SoftMax(Sub(query-key));
[0057] 4: Multiply the normalized temporal relationship ε obtained in the previous step by the value feature, and fuse the obtained feature with the value feature of the input target frame.
[0058] ④ Adaptive time series feature fusion pyramid module, used to fuse time series features of different scales.
[0059] The adaptive temporal feature fusion pyramid module specifically performs the following steps: calculates the weight coefficients of temporal features of different scales based on the global information of temporal features of each scale output by the temporal difference attention module, and uses the weight coefficients to perform weighted fusion of temporal features of each scale.
[0060] In one embodiment, Figure 4 As shown in Figure 1, the adaptive temporal feature fusion pyramid module performs the following steps on the input features:
[0061] 1: Input multi-scale temporal semantic features
[0062] 2: Calculate the global information of features at each scale.
[0063] 3: Calculate the weight coefficient α based on the global information X of each scale feature obtained in the previous step. 2,=>3,MLP(X);
[0064] 4: Weight the input multi-scale features according to the weight coefficient α obtained in the previous step.
[0065] 5: Layer-by-layer fusion of temporal features of different scales 4,=>5,f L-1 +Upsample(f L ).
[0066] In this embodiment, the symbol => indicates that the output of the layer represented by the Arabic numeral before the symbol is input into the layer represented by the Arabic numeral after the symbol. MLP represents a multi-layer perceptron network, which is implemented using the SoftMax function in the present invention; Avg represents a global average pooling layer, Max represents a global maximum pooling layer, Sub represents matrix subtraction, and Upsample represents a bilinear interpolation upsampling function.
[0067] ⑤ The splicing module is used to splice the spatial features and the fused temporal features to obtain the splicing features.
[0068] ⑥ Output module, used to output semantic segmentation prediction results based on splicing features.
[0069] The above model is iteratively trained until the model converges to obtain a video speech segmentation model, which is used to achieve accurate video semantic segmentation.
[0070] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A video semantic segmentation method based on temporal difference attention mechanism, characterized in that: The following steps are involved: Step 1) Data annotation: annotate the acquired video image dataset and assign the category to each pixel in the image; Step 2) preprocessing the image data set; Step 3) extracting multi-scale spatial features of the target frame image in the video based on the encoder; Step 4) According to the reference frame and the target frame image, the temporal difference attention module is used to extract the multi-scale temporal features of the image; Step 5) fusing temporal features of different scales based on the adaptive temporal feature fusion pyramid; Step 6) splicing the spatial features and the fused temporal features to obtain splicing features; Step 7) Input the concatenated features into the prediction layer and output the prediction results.
2. According to claim 1, a video semantic segmentation method based on temporal difference attention mechanism is characterized in that: The step 2) specifically includes: performing numerical standardization on the image data, and rotating and flipping the image with a preset probability.
3. According to the video semantic segmentation method based on temporal difference attention mechanism according to claim 1, it is characterized in that: The step 4) is specifically as follows: extracting the features of the target frame and the reference frame respectively and performing differentiation to obtain differential features, using the temporal differential attention mechanism to model the pixel relationship between the target frame and the reference frame and using the pixel relationship to fuse the features between the target frame and the reference frame to obtain temporal features.
4. According to the video semantic segmentation method based on temporal difference attention mechanism according to claim 1, it is characterized in that: The temporal difference attention module includes a fully connected layer, a feature difference calculation layer, a softmax layer, a multiplication layer and a multi-layer addition layer. The input target frame image is mapped into a query feature through the fully connected layer, and the reference frame image is mapped into a key feature and a value feature through the fully connected layer. The feature difference calculation layer performs a difference calculation based on the query feature and the key feature and then inputs the softmax layer. After the output of the sofmax layer and the value feature are jointly input into the multiplication layer, the output and the value feature of the reference frame are input into the multi-layer addition layer for addition to obtain the fused temporal feature.
5. According to claim 1, a video semantic segmentation method based on temporal difference attention mechanism is characterized in that: The step 5) is specifically as follows: inputting the time series features into the adaptive time series feature fusion pyramid, calculating the weight coefficients of the time series features of different scales according to the global information of the time series features of each scale, and performing weighted fusion of the time series features of each scale using the weight coefficients.
6. A video semantic segmentation model based on temporal difference attention mechanism, characterized in that: Includes the following connected in sequence: The input module is used to obtain a video image dataset with annotation information and perform data preprocessing as model input; A spatial feature extraction module, used to extract multi-scale spatial features of a target frame image in a video based on an encoder; The temporal difference attention module is used to extract multi-scale temporal features of the image based on the reference frame and the target frame image; Adaptive temporal feature fusion pyramid module, used to fuse temporal features of different scales; A splicing module is used to splice the spatial features and the fused temporal features to obtain splicing features; The output module is used to output semantic segmentation prediction results based on the concatenated features.
7. The video semantic segmentation model based on temporal difference attention mechanism according to claim 6, characterized in that: The data preprocessing specifically includes: performing numerical standardization on the image data, and rotating and flipping the image with a preset probability.
8. The video semantic segmentation model based on temporal difference attention mechanism according to claim 6, characterized in that: The temporal difference attention module specifically performs the following steps: extracting the features of the target frame and the reference frame respectively and performing differentiation to obtain differential features, using the temporal difference attention mechanism to model the pixel relationship between the target frame and the reference frame and using the pixel relationship to fuse the features between the target frame and the reference frame to obtain temporal features.
9. The video semantic segmentation model based on temporal difference attention mechanism according to claim 6, characterized in that: The temporal difference attention module includes a fully connected layer, a feature difference calculation layer, a softmax layer, a multiplication layer and a multi-layer addition layer. The input target frame image is mapped into a query feature through the fully connected layer, and the reference frame image is mapped into a key feature and a value feature through the fully connected layer. The feature difference calculation layer performs a difference calculation based on the query feature and the key feature and then inputs the softmax layer. After the output of the sofmax layer and the value feature are jointly input into the multiplication layer, the output and the value feature of the reference frame are input into the multi-layer addition layer for addition to obtain the fused temporal feature.
10. The video semantic segmentation model based on temporal difference attention mechanism according to claim 6, characterized in that: The adaptive temporal feature fusion pyramid module specifically performs the following steps: calculating the weight coefficients of temporal features of different scales according to the global information of temporal features of each scale output by the temporal difference attention module, and weightedly fusing the temporal features of each scale using the weight coefficients.