Video semantic segmentation method, device, equipment, storage medium and program product

Abstract

The application provides a video semantic segmentation method, device and equipment, a storage medium and a program product, and relates to the field of computer vision. The video semantic segmentation method comprises: performing static semantic feature aggregation processing on each video frame in a target video frame sequence to obtain the static semantic feature of each video frame in the target video frame sequence; performing dynamic semantic feature aggregation processing on the target video frame sequence according to the static semantic feature of a first video frame, the static semantic feature of a second video frame and the static semantic feature of a third video frame to obtain the dynamic semantic feature of the target video frame sequence; and performing multivariate joint prediction processing on each video frame in the target video frame sequence according to the dynamic semantic feature of the target video frame sequence to obtain a semantic segmentation result of the target video frame sequence. The application can accurately segment the semantics of context content in the target video frame sequence, so that the segmentation result is more consistent, and a large amount of computing resources is not required.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a video semantic segmentation method, apparatus, device, storage medium, and program product. Background Technology

[0002] Video content understanding technology is an interdisciplinary field that includes computer vision, machine learning, and artificial intelligence. It aims to enable computers to understand and interpret video data based on known information, combined with the video's visual characteristics and temporal continuity, and incorporating related technologies such as computer vision and machine learning. This allows them to extract further information from the video, such as scene classification, object recognition, and action recognition.

[0003] The concept of computer vision was proposed to enable computers to simulate the functions of the human visual system, better analyze and understand image and video data, and be widely applied in various life scenarios. With the development of artificial intelligence, a massive amount of image and video data has been generated, creating an ocean of visual information and posing challenges to the application and practice of computer vision. To better analyze and understand this data, computer vision technologies based on deep learning and other machine learning methods are constantly being developed.

[0004] Commonly used computer vision techniques include object detection and recognition, and image segmentation. Object detection and recognition refers to the process of extracting and analyzing features from input images or videos using convolutional neural networks or other deep learning models in different environments, aiming to identify objects in the image and provide necessary information for subsequent image understanding and analysis. Image segmentation, on the other hand, uses pixel-level classification methods to divide an image into multiple regions, ensuring that pixels in the same category have the same color; it is a commonly used image processing and analysis technique.

[0005] In the real world, cameras often capture not just single frames, but continuous video frame sequences. Therefore, segmenting the received video frame sequences is an indispensable key step in understanding video scenes. In traditional image processing, the purpose of image segmentation and detection is to extract key information from the image to better understand and recognize its content, typically using semantic segmentation techniques. Traditional image segmentation techniques are mainly divided into threshold-based segmentation, region-based segmentation, and edge-based segmentation. Threshold-based segmentation divides pixels in an image into target and background by selecting one or more thresholds. Region-based segmentation clusters similar colors or gray levels in the image to form connected regions. Edge-based segmentation identifies abrupt changes in gray levels or colors to determine the boundaries of the target. Video semantic segmentation is a further development of these techniques. Its goal is to perform more refined segmentation on each frame of the video, ensuring that pixels in the same region not only have similar colors and textures but also share the same semantic labels, such as pixels in the same region being labeled as "cat" or "person." Commonly used video semantic segmentation techniques include deep learning methods, such as Fully Convolutional Networks (FCNs), SegNet, and DeepLab. These models are typically trained on large datasets, learn by optimizing pixel-level classification loss functions, and segment and label input videos during testing to achieve the goal of understanding video content.

[0006] Existing semantic segmentation methods for video frame sequences typically utilize optical flow to perform feature distortion or divide the current frame into different regions, and then perform different processing based on the differences between regions to adaptively achieve a balance between performance and efficiency. However, all of these methods are limited by optical flow networks, leading to inconsistent segmentation results. Furthermore, existing semantic segmentation methods for video frame sequences also employ pooling and convolution at different resolutions for frames at different times and implicitly model long-distance relationships between pixels using attention mechanisms. Alternatively, they divide video semantic segmentation into state and motion branches for separate processing and obtain semantic segmentation results by linking pixel-level and region-level descriptors. Or, they establish spatiotemporal relationships between frames by calculating class prototypes of previous frames and then using attention mechanisms. However, the above methods consume a large amount of computational resources. Summary of the Invention

[0007] The purpose of this invention is to provide a video semantic segmentation method, apparatus, device, storage medium, and program product to solve the problems of inconsistent segmentation results and high computational resource consumption in existing semantic segmentation processing methods.

[0008] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0009] This invention provides a video semantic segmentation method, comprising:

[0010] Static semantic feature aggregation is performed on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence.

[0011] Based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame, dynamic semantic feature aggregation processing is performed on the target video frame sequence to obtain the dynamic semantic features of the target video frame sequence. The first video frame is any video frame in the target video frame sequence other than the first two video frames, and the second video frame and the third video frame are the two video frames in the target video frame sequence that are adjacent to the first video frame.

[0012] Based on the dynamic semantic features of the target video frame sequence, multivariate joint prediction processing is performed on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence.

[0013] Optionally, static semantic feature aggregation processing is performed on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence, including:

[0014] For the fourth video frame, a linear attention mechanism is used to obtain the global information of the fourth video frame;

[0015] The global information is aggregated using a deformable convolutional network to obtain the static semantic features of the fourth video frame;

[0016] The fourth video frame is any video frame in the target video frame sequence.

[0017] Optionally, based on the static semantic features of the first video frame, the second video frame, and the third video frame, dynamic semantic feature aggregation processing is performed on the target video frame sequence to obtain the dynamic semantic features of the target video frame sequence, including:

[0018] Based on the perceptual difference between the static semantic features of the third video frame and the static semantic features of the second video frame, the static semantic features of the third video frame are selectively aggregated into the static semantic features of the second video frame to obtain the dynamic semantic features of the second video frame.

[0019] Based on the perceptual difference between the dynamic semantic features of the second video frame and the static semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the static semantic features of the first video frame to obtain the dynamic semantic features of the first video frame.

[0020] Based on the perceptual difference between the dynamic semantic features of the second video frame and the dynamic semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the dynamic semantic features of the first video frame. Based on the perceptual difference between the static semantic features of the third video frame and the dynamic semantic features of the first video frame, the static semantic features of the third video frame are selectively aggregated into the dynamic semantic features of the first video frame, thus obtaining the final dynamic semantic features of the first video frame.

[0021] The third video frame is located before the second video frame.

[0022] Optionally, the selective aggregation process includes:

[0023] The perceptual difference between the first semantic feature of the first target video frame and the second semantic feature of the second target video frame is convolved to obtain the attention offset map.

[0024] For the first pixel in the second target video frame, query the attention offset map for P second pixels of the first target video frame, where P is an integer greater than or equal to 1, and the first pixel is any pixel in the second target video frame.

[0025] The first semantic features of the second pixel are selectively aggregated into the second semantic features of the first pixel;

[0026] Wherein, the first target video frame is the third video frame or the second video frame; the second target video frame is the second video frame or the first video frame; the first semantic feature is a static semantic feature or a dynamic semantic feature; the second semantic feature is a static semantic feature or a dynamic semantic feature;

[0027] When the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the second video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature; or, when the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature.

[0028] Optionally, the method further includes:

[0029] A deformable convolutional network is used to enhance the connection between the second pixel and all other pixels in the first target video frame except the second pixel.

[0030] Optionally, based on the dynamic semantic features of the target video frame sequence, multivariate joint prediction processing is performed on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence, including:

[0031] By using multivariate class prototypes for comparative learning training, a multivariate class prediction model is obtained.

[0032] The dynamic semantic features of the fifth video frame are mapped and processed into multivariate features;

[0033] The multivariate features are input into the multivariate class prediction model to obtain the multivariate prediction result of the fifth video frame output by the multivariate class prediction model.

[0034] Based on the multivariate prediction results of the fifth video frame, the semantic segmentation results of the fifth video frame are obtained;

[0035] The fifth video frame is any video frame in the target video frame sequence.

[0036] Optionally, based on the multivariate prediction result of the fifth video frame, the semantic segmentation result of the fifth video frame is obtained, including:

[0037] The semantic segmentation result of the fifth video frame is obtained by performing joint decision processing and bilinear interpolation on the multivariate prediction results of the fifth video frame.

[0038] This invention also provides a video semantic segmentation apparatus, comprising:

[0039] The first processing module is used to perform static semantic feature aggregation processing on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence.

[0040] The second processing module is used to perform dynamic semantic feature aggregation processing on the target video frame sequence based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame to obtain the dynamic semantic features of the target video frame sequence. The first video frame is any video frame in the target video frame sequence other than the first two video frames, and the second video frame and the third video frame are the two video frames in the target video frame sequence that are adjacent to the first video frame.

[0041] The third processing module is used to perform multivariate joint prediction processing on each video frame in the target video frame sequence based on the dynamic semantic features of the target video frame sequence, so as to obtain the semantic segmentation result of the target video frame sequence.

[0042] This invention also provides a video semantic segmentation device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the video semantic segmentation method as described above.

[0043] This invention also provides a readable storage medium storing a program, which, when executed by a processor, implements the steps of the video semantic segmentation method as described above.

[0044] This invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the video semantic segmentation method as described above.

[0045] At least one of the above-described technical solutions of the present invention has the following beneficial effects:

[0046] The video semantic segmentation method provided by this invention performs static semantic feature aggregation processing on each video frame in the target video frame sequence to obtain the shot semantic features of each video frame in the target video frame sequence. Based on the static semantic features of the first video frame (any video frame in the target video frame sequence other than the first two video frames), the static semantic features of the second video frame, and the static semantic features of the third video frame (the second video frame and the third video frame are the two video frames in the target video frame sequence adjacent to the first video frame), dynamic semantic feature aggregation processing is performed on the target video frame sequence to obtain the dynamic semantic features of the target video frame sequence. Based on the dynamic semantic features of the target video frame sequence, multivariate joint prediction processing is performed on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence. Through the above process, the semantics of the context content in the target video frame sequence can be accurately segmented, making the segmentation results more consistent, and without requiring a large amount of computing resources. Attached Figure Description

[0047] Figure 1 A flowchart of a video semantic segmentation method provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram illustrating the calculation method of the attention diagram provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the structure of the video semantic segmentation device provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of a video semantic segmentation device provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0052] To address the problems of inconsistent segmentation results and high computational resource consumption in existing semantic segmentation methods, this invention provides a video semantic segmentation method, apparatus, device, storage medium, and program product.

[0053] like Figure 1 As shown, this embodiment of the invention provides a video semantic segmentation method, including:

[0054] Step 101: Perform static semantic feature aggregation processing on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence.

[0055] It's important to note that video semantic segmentation actually involves semantic segmentation of a series of images with high frame rates. Under this input, it's necessary to rethink the static and dynamic contexts in video semantics from the perspective of perceptual consistency. This can be understood as follows: the semantic features of a single frame are static, but due to significant repetition between two adjacent video frames, dynamic semantic features are created by constructing connections between static semantic features across multiple frames.

[0056] The target video frame sequence is a continuous sequence of video frames from a certain video, arranged in chronological order of shooting time.

[0057] The video semantic segmentation method provided in this embodiment of the invention is applied to the static-dynamic perceptual consistency (SDPC) framework, which includes a static semantic efficient aggregation module (SSEA) and a dynamic semantic selective aggregation module (DSSA).

[0058] Static semantic features contain more detailed semantic region information, which is a prerequisite for effectively realizing dynamic semantic selective aggregation between frames.

[0059] In this step, static semantic features of each video frame in the target video frame sequence are aggregated using SSEA to obtain the static semantic features of each video frame in the target video frame sequence.

[0060] Step 102: Based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame, perform dynamic semantic feature aggregation processing on the target video frame sequence to obtain the dynamic semantic features of the target video frame sequence. The first video frame is any video frame in the target video frame sequence other than the first two video frames, and the second video frame and the third video frame are the two video frames in the target video frame sequence that are adjacent to the first video frame.

[0061] It should be noted that, due to the strong correlation between frames, capturing the motion relationships between them can reduce perceptual uncertainty and achieve dynamic semantic consistency.

[0062] In this step, for any first video frame, dynamic semantic feature aggregation processing is performed on the first video frame based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame to obtain the dynamic semantic features of the first video frame. Following the above process, dynamic semantic feature aggregation processing is performed on the target video frame sequence to obtain the dynamic semantic features of each first video frame in the target video frame sequence (the dynamic semantic features of the target video frame sequence include the dynamic semantic features of each video frame other than the first two video frames).

[0063] Among them, dynamic semantic features can be understood as the static semantic features of the current video frame that enhance the current video frame by connecting adjacent video frames, since the content of the preceding and following video frames is similar.

[0064] Step 103: Based on the dynamic semantic features of the target video frame sequence, perform multivariate joint prediction processing on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence.

[0065] It should be noted that, inspired by the human cognitive and learning process of different objects, the following characteristics are summarized: (1) Adjacent pixels are more likely to belong to the same category. (2) Pixels within the same entity are interconnected and jointly determine its category. Objects of the same category have small differences, while objects of different categories have large differences. (3) Each category consists of a general feature representation and a personalized feature representation. Therefore, in order to achieve cross-frame perceptual fuzzy matching and enhance the model's representation capability, a class prototype is introduced and extended to a multivariate class prototype (MCP).

[0066] Since humans often use one or more aspects to make joint judgments when learning and recognizing objects, this step involves performing multivariate joint prediction processing on each video frame in the target video frame sequence based on the dynamic semantic features of the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence, which will enhance the robustness and accuracy of the decision.

[0067] In an optional embodiment, static semantic feature aggregation processing is performed on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence, including:

[0068] For the fourth video frame, a linear attention (LA) mechanism is used to obtain the global information of the fourth video frame, where the fourth video frame is any video frame in the target video frame sequence. In other words, introducing LA establishes global dependencies with lower computational complexity, which can solve the problem of high computational cost in static semantic feature extraction caused by the softmax attention mechanism.

[0069] The global information is aggregated using Deformable Convolutional Networks (DCNs) to obtain the static semantic features of the fourth video frame. Specifically, DCNs adaptively explore spatial information and refine global dependencies, aiming to achieve static semantic consistency.

[0070] Specifically, low-level features typically have larger sizes and contain more detailed information, while high-level features typically have smaller sizes and richer semantic information. In this optional embodiment, features from different stages of the backbone network are fused at multiple scales to obtain multi-level representations of the features. Subsequently, a DCN is used instead of traditional convolutions to achieve spatial adaptive aggregation, providing locally deformable receptive fields and long-range dependencies. LA is used instead of the cumbersome multi-head self-attention (MSA) mechanism to provide global pixel dependencies. Furthermore, the global information from LA provides a reliable basis for selecting convolutional regions of the DCN, and the adaptive aggregation of the DCN partially compensates for the poor focusing performance of LA.

[0071] The principle of the linear attention mechanism is explained in detail below:

[0072] Divide a fourth video frame into N (N is an integer greater than or equal to 1) feature maps (tokens). For a given fourth video frame, the input consists of the N tokens of that fourth video frame. H and W represent the height and width of the feature map, respectively (N = H * W), and C represents the number of channels. The general form of attention can be written as:

[0073] Q = xW Q K = xW K V = xW V ,

[0074]

[0075] Where Q, K, and V are the query, key, and value matrices, respectively. Q W K and W V For a learnable mapping matrix, O iThis represents the i-th row of the output, where Sim(·,·) represents the similarity function. When the similarity function is... In this case, the attention mechanism is the original self-attention mechanism in Transformer. To obtain the attention graph, the similarity is calculated for each query key-value pair, a process with a complexity of O(n log n). This leads to quadratic computational complexity. While methods such as smaller attention regions, sliding windows, and sparse global attention patterns have been proposed, these solutions still sacrifice performance in long-range modeling. Therefore, to preserve the function of attention, matrix multiplication is used, and a kernel function φ is introduced to replace the original similarity function, thereby reducing the computational complexity from quadratic to linear:

[0076]

[0077] As can be seen from the above formula, only one calculation is needed. and This reduces computational and memory consumption to

[0078] The following explains the specific principles of deformable convolutional networks:

[0079] Traditional convolution accelerates convergence through static weights and strong inductive biases, but limits the neural network from learning more robust and generalized patterns. To increase the receptive field and establish long-range connections between pixels, DCN is introduced to aggregate spatial information. The definition of DCN is as follows:

[0080]

[0081] Where k is the number of sampling points, x(a) represents the position of feature x in a, Y(a) represents the position of feature Y in a, and a k Let w represent the k-th region centered at a. k Represents the weight, Δa k This represents the learned convolution offset.

[0082] It should be noted that most existing direct methods use additional optical flow networks, which are unreliable and difficult to optimize end-to-end. Existing indirect methods incur prohibitive computational costs due to the need to calculate the correlation matrix of all pixels across frames. Furthermore, the extracted features exhibit significant redundancy due to the high similarity between cross-frame contexts. Therefore, this embodiment achieves dynamic semantic alignment by aggregating only local regions.

[0083] In practice, the greater the time interval between frames, the greater the changes in objects, and the larger the range of pixel changes and associated regions. This means that a larger receptive field is needed to capture global semantic and environmental information. Therefore, this embodiment aggregates dynamic semantic information between frames twice, from coarse to fine, based on the static perceptual differences across frames. Each aggregation selects only a portion of pixels across frames to achieve dynamic semantic alignment with high efficiency and expressiveness.

[0084] Therefore, in the Dynamic Semantic Selective Aggregation (DSSA) module, a cross-frame adaptive offset cross-attention mechanism (CAOCA) is proposed, which can selectively aggregate dynamic semantic information from coarse to fine based on the perceptual differences between frames. In an optional embodiment, the target video frame sequence is subjected to dynamic semantic feature aggregation processing based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame to obtain the dynamic semantic features of the target video frame sequence, including:

[0085] First aggregation (coarse aggregation): Based on the perceptual difference between the static semantic features of the third video frame and the static semantic features of the second video frame, the static semantic features of the third video frame are selectively aggregated into the static semantic features of the second video frame to obtain the dynamic semantic features of the second video frame; and, based on the perceptual difference between the dynamic semantic features of the second video frame and the static semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the static semantic features of the first video frame to obtain the dynamic semantic features of the first video frame, wherein the third video frame is located before the second video frame.

[0086] Second aggregation (fine aggregation): Based on the perceptual difference between the dynamic semantic features of the second video frame and the dynamic semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the dynamic semantic features of the first video frame. Also, based on the perceptual difference between the static semantic features of the third video frame and the dynamic semantic features of the first video frame, the static semantic features of the third video frame are selectively aggregated into the dynamic semantic features of the first video frame, thus obtaining the final dynamic semantic features of the first video frame.

[0087] Based on the above selective aggregation process, it can be understood that the dynamic semantic features of the target video frame sequence include the dynamic semantic features of the first video frame in the target video frame sequence (which is the same as the static semantic features of the first video frame), the dynamic semantic features of the second video frame in the target video frame sequence, and the final dynamic semantic features of each subsequent video frame.

[0088] Specifically, the selective aggregation process includes:

[0089] The perceptual difference between the first semantic feature of the first target video frame and the second semantic feature of the second target video frame is convolved to obtain the attention offset map.

[0090] For the first pixel in the second target video frame, query the attention offset map for P second pixels of the first target video frame, where P is an integer greater than or equal to 1, and the first pixel is any pixel in the second target video frame.

[0091] The first semantic features of the second pixel are selectively aggregated into the second semantic features of the first pixel;

[0092] Wherein, the first target video frame is the third video frame or the second video frame; the second target video frame is the second video frame or the first video frame; the first semantic feature is a static semantic feature or a dynamic semantic feature; the second semantic feature is a static semantic feature or a dynamic semantic feature;

[0093] When the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the second video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature; or, when the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature.

[0094] Specifically, during the first aggregation, the cross frames are arranged according to {F} t-1 F t-2 and {F t F t-1 The order of} is misaligned, where F t Indicates the first video frame, F t-1 Indicates the second video frame, F t-2 Representing the third video frame, the perceptual difference between the static semantic features of the third video frame and the static semantic features of the second video frame is calculated. This static perceptual difference is then passed through multiple convolutional layers to generate an attention shift map. Where H represents the height of the feature map and W represents the width of the feature map, for each pixel of the second video frame (i.e., the first pixel), P second pixels (P is an integer greater than or equal to 1) are selected from the cross frame (i.e., the third video frame) according to the AOM (or it can be understood as querying the first target video frame for P second pixels in the AOM and selecting the P second pixels). That is, in the above formula, P is the number of second pixels. The static semantic features of the P second pixels are aggregated into the static semantic features of the first pixel to obtain the dynamic semantic features of the first pixel. Combining the above steps, the dynamic semantic features of the second video frame are obtained. Next, the perceptual difference between the dynamic semantic features of the second video frame and the static semantic features of the first video frame is calculated. This static perceptual difference is passed through multiple convolutional layers to generate an attention shift map. For each pixel of the first video frame (i.e., the first pixel), P second pixels (P is an integer greater than or equal to 1) are selected from the cross-frame (i.e., the second video frame) according to the attention shift map. The dynamic semantic features of the P second pixels are aggregated into the static semantic features of the first pixel to obtain the dynamic semantic features of the first pixel. Combining the above steps, the dynamic semantic features of the first video frame are obtained.

[0095] In the second aggregation, since the intra-frame and cross-frame spatiotemporal dependencies have been roughly established for each frame pixel, the pixel-wise perceptual difference between the two reference frames (the second video frame and the third video frame) and the target frame (the first video frame) is directly calculated. Then, a second selective aggregation is performed based on the pixel-wise perceptual difference, that is, the perceptual difference between the static semantic features of the third video frame and the dynamic semantic features of the first video frame is calculated. This static perceptual difference generates an attention shift map through multiple convolutional layers. For each pixel of the first video frame (i.e., the first pixel), P second pixels (P is an integer greater than or equal to 1) are selected from the cross-frame (i.e., the third video frame) according to the attention shift map. The static semantic features of these P second pixels are aggregated into the dynamic semantic features of the first pixel, and the dynamic semantic features of the first pixel are obtained again. Combining the above steps, the dynamic semantic features of the first video frame are obtained again. Furthermore, the perceptual difference between the dynamic semantic features of the second video frame and the dynamic semantic features of the first video frame is calculated. This static perceptual difference is passed through multiple convolutional layers to generate an attention shift map. For each pixel of the first video frame (i.e., the first pixel), P second pixels (P is an integer greater than or equal to 1) are selected from the cross-frame (i.e., the second video frame) according to the attention shift map. The dynamic semantic features of the P second pixels are aggregated into the dynamic semantic features of the first pixel to obtain the final dynamic semantic features of the first pixel. Combining the above steps, the final dynamic semantic features of the first video frame are obtained.

[0096] The above aggregation process aims to further refine the spatiotemporal information from the reference frame, capture motion, and model implicit "flow" relationships across frames. Staged message aggregation helps the model obtain a larger receptive field and mitigates the performance loss caused by aggregating only a subset of pixels, thus striking a balance between efficiency and performance.

[0097] Furthermore, after each selective aggregation process, the method further includes:

[0098] Deformable Convolutional Network (DCN) is used to enhance the connectivity between the second pixel and all other pixels in the first target video frame. Specifically, after each aggregation, DCN is used to enhance intra-frame pixel connectivity, indirectly establishing the correlation between the pixel and the "unselected" pixels across frames.

[0099] The formal definition of CAOCA is as follows:

[0100]

[0101] Among them, Q(F) t ) i，j / K(F t-l ) i，j It is F t / F t-l The element at (i, j) of the Q / K mapping matrix. AOM(i, j, p) represents the position of the pixel at (i, j) in frame t corresponding to the p-th region of interest in frame tl. ° represents the Hadamard product. l is the inter-frame distance for cross-frame aggregation. It's worth noting that traditional attention mechanisms compute an N×N attention graph, representing the similarity of N pixels in the current frame to N pixels across frames. However, since only a subset of elements are aggregated, the attention graph contains a large number of zero elements. To utilize this sparsity, the Q matrix (query matrix) and K matrix (key matrix) are expanded and rearranged, and the Hadamard product is used instead of matrix multiplication. Then, aggregation normalization is performed, thereby reducing the computational complexity from... Reduce to Where N >> P. Note the calculation method for the graph as follows: Figure 2 As shown.

[0102] Ignoring the impact of dot product and multi-head on reducing computational cost, this embodiment analyzes the complexity of the proposed scheme. In the prior art, the computational cost of constructing the spatiotemporal relationship between a pixel and other pixels is... If we construct spatiotemporal connections in t (first video frame), t-1 (second video frame), and t-2 (third video frame), then the computational cost is... Therefore, the total computational cost of the vanilla Transformer (existing technology) is In the above scheme of this embodiment, the computational costs of DCN and CAOCA are respectively... and Therefore, the complexity of SSEA is The complexity of DSSA is Therefore, the total complexity of the above model is

[0103] In one optional embodiment, based on the dynamic semantic features of the target video frame sequence, multivariate joint prediction processing is performed on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence, including:

[0104] By using multivariate class prototypes for contrastive learning training, a multivariate class prediction model is obtained. That is, during the training process, contrastive learning and MCP are combined to improve the model's representational ability, and the multivariate class prototypes are only calculated during the training phase, which does not generate a large amount of computational cost.

[0105] Specifically, contrastive learning based on multivariable class prototypes includes:

[0106] The class prototype represents the feature center of the category semantics, and multivariate features are similar to the "basis vectors" that constitute the category feature space. The purpose of this is to maximize the distance between feature centers of different categories, ensuring the separability of category features. Additionally, it increases the similarity between multivariate features belonging to the same category, thereby enriching the expression of category features. Specifically, the multivariate class prototype should be calculated first, and the calculation method can be defined as:

[0107]

[0108] in, Indicates the indicator function, This represents the one-hot encoding of pixel (i, j) belonging to category α. This represents the m-th "basis vector" of category α. For the query pixel... If in S t Belongs to the category Then the set of positive samples is The set of negative samples belonging to other categories is Therefore, the contrastive learning loss can be formally defined as:

[0109]

[0110] Where τ represents the temperature parameter, p + Representative and Other pixels of the same category, p - Representative and Other pixels from different categories. To enhance the model's representational power, this embodiment limits the similarity between multivariate features; formally, this calculation method can be defined as:

[0111]

[0112] in, S represents t The number of categories. Therefore, the overall learning objective can be defined as:

[0113]

[0114] Where λ1 and λ2 are parameters used to balance the loss.

[0115] During testing, joint decisions based on independent predictions of multivariate features are made to enhance the robustness of reasoning.

[0116] The process of performing multivariate joint prediction:

[0117] The dynamic semantic features of the fifth video frame are mapped and processed into multivariate features, where the fifth video frame is any video frame in the target video frame sequence. It should be noted that humans often use one or more aspects to jointly determine objects when learning and recognizing them; such a strategy enhances the robustness and accuracy of the decision. Inspired by this behavior, for any video frame (i.e., the fifth video frame), the output features U of DSSA are... t (Dynamic semantic features) are mapped to multivariate features:

[0118]

[0119] Where H′ and W′ represent the length and width of the feature map, respectively, M represents the number of multivariates, and C represents the number of channels.

[0120] The multivariate features are input into the multivariate class prediction model to obtain the multivariate prediction result of the fifth video frame output by the multivariate class prediction model. Specifically, the multivariate features are independently predicted to obtain the multivariate prediction result.

[0121] Based on the multivariate prediction results of the fifth video frame, the semantic segmentation results of the fifth video frame are obtained.

[0122] Specifically, based on the multivariate prediction result of the fifth video frame, the semantic segmentation result of the fifth video frame is obtained, including:

[0123] The semantic segmentation result of the fifth video frame is obtained by performing joint decision processing and bilinear interpolation on the multivariate prediction results of the fifth video frame.

[0124] That is, by further combining the multivariate prediction results, joint decision processing and bilinear interpolation processing are performed to obtain the semantic segmentation result. Where Ground truth is S t The prediction result will be supervised by minimizing the cross-entropy loss:

[0125]

[0126] This invention provides a Video Sematic Segmentation (VSS) framework that divides video semantics into dynamic and static semantics from the perspective of perceptual consistency, addressing the issues of information inconsistency and high computational cost. Specifically, a static semantics effective aggregation module uses linear and deformable convolutions to establish long-distance pixel-level dependencies to capture global spatial information and adaptively refine local spatial information. A dynamic semantics selective aggregation module selectively aggregates dynamic semantic information from coarse to fine based on perceptual differences between frames. After each selective aggregation, deformable convolutions are used to establish implicit dependencies between "unselected" and "selected" pixels, achieving globally consistent dynamic context with low complexity. A training and inference strategy based on multivariate class prototypes reduces feature differences within and across frames within the same category during training, while also distinguishing feature representations between different categories. During inference, multivariate features are independently predicted and then jointly decided to obtain the final segmentation result. This training and inference strategy enhances the model's category representation ability and robustness.

[0127] In this embodiment of the invention, video semantics is further divided into static and dynamic semantics, and processed separately according to their respective characteristics. During static semantic aggregation, deformable convolution aggregates local spatial information and utilizes a line attention mechanism to establish global pixel associations with low complexity, thereby achieving static semantic consistency and providing reliable intra-frame information as a basis for dynamic semantic aggregation. In dynamic semantic selective aggregation, based on inter-frame perceptual differences, attention mechanisms selectively aggregate parts of reference frames, avoiding redundant inter-frame computation, thus achieving dynamic semantic selective aggregation. A training and inference strategy based on multivariate class prototypes enables the model to acquire personalized inter-class and diverse intra-class category features, and enriches the model's representational ability and enhances its robustness through multivariate joint inference. Class prototypes are only recalculated during each training process, thus avoiding additional computational and memory burdens. Dynamic selective aggregation, under the constraint of multivariate contrastive learning, replaces pixel-level perfect matching in optical flow with inter-frame category-level fuzzy matching, allowing the model to focus more on the task itself rather than pixel details. Furthermore, the proposed scheme eliminates the need for additional labels from the optical flow network, enabling end-to-end model optimization.

[0128] To address the issues of inconsistent segmentation and high computational cost, this invention rethinks static and dynamic contexts in video semantics from the perspective of perceptual consistency. For static contexts, personalized category features are obtained by constraining the perceptual similarity between pixels of the same and different categories within a frame. Then, static semantic consistency can be effectively achieved through intra-frame multi-scale and global dependencies. For dynamic contexts, since cross-frame contexts are highly similar, aggregating partial information across frames is sufficient to achieve dynamic semantic alignment, thereby avoiding redundant computation. Furthermore, during dynamic semantic aggregation, category-level fuzzy matching is established through cross-frame category perceptual consistency constraints and cross-attention mechanisms, replacing pixel-level perfect matching. This allows the model to more easily capture motion information and achieve dynamic semantic alignment.

[0129] This invention proposes a novel Static-Dynamic Perceptual Consistency (SDPC) framework to address the problems of segmentation inconsistency and high computational complexity. Within the SDPC framework, static and dynamic semantic aggregation modules are constructed separately and combined with Multivariable Class Prototype Contrastive Learning (MCP-CL) for training and inference, ensuring perceptual consistency between dynamic and static semantics. In the Static Semantic Efficient Aggregation Module (SSEA), Linear Attention (LA) and Deformable Convolution (DCN) are used to establish long-distance pixel-level dependencies to capture global spatial information and adaptively refine local spatial information, thereby achieving static semantic consistency. In the Dynamic Semantic Selective Aggregation Module (DSSA), a cross-frame adaptive offset cross-attention mechanism (CAOCA) is proposed, enabling selective aggregation of dynamic semantic information from coarse to fine based on perceptual differences between frames. Furthermore, after each selective aggregation, DCN is used to establish implicit dependencies between "unselected" and "selected" pixels, resulting in a more globally consistent dynamic context. During training, MCP-CL mitigates feature differences within the same class both intra-frame and cross-frame, while also distinguishing feature representations between different classes. This approach helps to obtain spatiotemporally coherent and personalized category representations. Furthermore, we enhance category representation capabilities by extending class prototypes to multivariate class prototypes and constraining intra-class feature similarity. In testing, joint decision-making is performed after independent predictions of multivariate features to improve the accuracy and robustness of inference.

[0130] like Figure 3 As shown, embodiments of the present invention also provide a video semantic segmentation apparatus, comprising:

[0131] The first processing module 301 is used to perform static semantic feature aggregation processing on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence.

[0132] The second processing module 302 is used to perform dynamic semantic feature aggregation processing on the target video frame sequence based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame to obtain the dynamic semantic features of the target video frame sequence. The first video frame is any video frame in the target video frame sequence other than the first two video frames, and the second video frame and the third video frame are the two video frames in the target video frame sequence that are adjacent to the first video frame.

[0133] The third processing module 303 is used to perform multivariate joint prediction processing on each video frame in the target video frame sequence based on the dynamic semantic features of the target video frame sequence, so as to obtain the semantic segmentation result of the target video frame sequence.

[0134] Optionally, the first processing module 301 includes:

[0135] The first processing unit is used to obtain global information of the fourth video frame using a linear attention mechanism.

[0136] The second processing unit is used to aggregate the global information using a deformable convolutional network to obtain the static semantic features of the fourth video frame.

[0137] The fourth video frame is any video frame in the target video frame sequence.

[0138] Optionally, the second processing module 302 includes:

[0139] The third processing unit is used to selectively aggregate the static semantic features of the third video frame into the static semantic features of the second video frame based on the perceptual difference between the static semantic features of the third video frame and the static semantic features of the second video frame, so as to obtain the dynamic semantic features of the second video frame.

[0140] The fourth processing unit is configured to selectively aggregate the dynamic semantic features of the second video frame into the static semantic features of the first video frame based on the perceptual difference between the dynamic semantic features of the second video frame and the static semantic features of the first video frame, thereby obtaining the dynamic semantic features of the first video frame.

[0141] The fifth processing unit is configured to selectively aggregate the dynamic semantic features of the second video frame into the dynamic semantic features of the first video frame based on the perceptual difference between the dynamic semantic features of the second video frame and the dynamic semantic features of the first video frame, and to selectively aggregate the static semantic features of the third video frame into the dynamic semantic features of the first video frame based on the perceptual difference between the static semantic features of the third video frame and the dynamic semantic features of the first video frame, thereby obtaining the final dynamic semantic features of the first video frame.

[0142] The third video frame is located before the second video frame.

[0143] Optionally, the first processing module 301 is specifically used for:

[0144] The perceptual difference between the first semantic feature of the first target video frame and the second semantic feature of the second target video frame is convolved to obtain the attention offset map.

[0145] For the first pixel in the second target video frame, query the attention offset map for P second pixels of the first target video frame, where P is an integer greater than or equal to 1, and the first pixel is any pixel in the second target video frame.

[0146] The first semantic features of the second pixel are selectively aggregated into the second semantic features of the first pixel;

[0147] Wherein, the first target video frame is the third video frame or the second video frame; the second target video frame is the second video frame or the first video frame; the first semantic feature is a static semantic feature or a dynamic semantic feature; the second semantic feature is a static semantic feature or a dynamic semantic feature;

[0148] When the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the second video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature; or, when the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature.

[0149] Optionally, the first processing module 301 is further configured to:

[0150] A deformable convolutional network is used to enhance the connection between the second pixel and all other pixels in the first target video frame except the second pixel.

[0151] Optionally, the third processing module 303 includes:

[0152] The sixth processing unit is used to perform comparative learning training using multivariate class prototypes to obtain a multivariate class prediction model;

[0153] The seventh processing unit is used to map and process the dynamic semantic features of the fifth video frame into multivariate features;

[0154] The eighth processing unit is used to input the multivariate features into the multivariate class prediction model to obtain the multivariate prediction result of the fifth video frame output by the multivariate class prediction model.

[0155] The ninth processing unit is used to obtain the semantic segmentation result of the fifth video frame based on the multivariate prediction result of the fifth video frame;

[0156] The fifth video frame is any video frame in the target video frame sequence.

[0157] Optionally, the ninth processing unit is specifically used for:

[0158] The semantic segmentation result of the fifth video frame is obtained by performing joint decision processing and bilinear interpolation on the multivariate prediction results of the fifth video frame.

[0159] It should be noted that the video semantic segmentation apparatus described in the embodiments of the present invention is an apparatus capable of performing the above-described video semantic segmentation method. Therefore, all embodiments of the above-described video semantic segmentation method are applicable to this apparatus and can achieve the same or similar technical effects.

[0160] like Figure 4 As shown, this embodiment of the invention also provides a video semantic segmentation device, including: a processor 401; and a memory 403 connected to the processor 401 via a bus interface 402. The memory 403 is used to store programs and data used by the processor 401 when performing operations, and the processor 401 calls and executes the programs and data stored in the memory 403.

[0161] The transceiver 404 is connected to the bus interface 402 and is used to receive and send data under the control of the processor 401. Specifically, the processor 401 is used to read the program in the memory 403 and executes the following processes:

[0162] Static semantic feature aggregation is performed on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence.

[0163] Based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame, dynamic semantic feature aggregation processing is performed on the target video frame sequence to obtain the dynamic semantic features of the target video frame sequence. The first video frame is any video frame in the target video frame sequence other than the first two video frames, and the second video frame and the third video frame are the two video frames in the target video frame sequence that are adjacent to the first video frame.

[0164] Based on the dynamic semantic features of the target video frame sequence, multivariate joint prediction processing is performed on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence.

[0165] Optionally, the processor 401 is configured to:

[0166] For the fourth video frame, a linear attention mechanism is used to obtain the global information of the fourth video frame;

[0167] The global information is aggregated using a deformable convolutional network to obtain the static semantic features of the fourth video frame;

[0168] The fourth video frame is any video frame in the target video frame sequence.

[0169] Optionally, the processor 401 is configured to:

[0170] Based on the perceptual difference between the static semantic features of the third video frame and the static semantic features of the second video frame, the static semantic features of the third video frame are selectively aggregated into the static semantic features of the second video frame to obtain the dynamic semantic features of the second video frame.

[0171] Based on the perceptual difference between the dynamic semantic features of the second video frame and the static semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the static semantic features of the first video frame to obtain the dynamic semantic features of the first video frame.

[0172] Based on the perceptual difference between the dynamic semantic features of the second video frame and the dynamic semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the dynamic semantic features of the first video frame. Based on the perceptual difference between the static semantic features of the third video frame and the dynamic semantic features of the first video frame, the static semantic features of the third video frame are selectively aggregated into the dynamic semantic features of the first video frame, thus obtaining the final dynamic semantic features of the first video frame.

[0173] The third video frame is located before the second video frame.

[0174] Optionally, the processor 401 is specifically used for:

[0175] The perceptual difference between the first semantic feature of the first target video frame and the second semantic feature of the second target video frame is convolved to obtain the attention offset map.

[0176] For the first pixel in the second target video frame, query the attention offset map for P second pixels of the first target video frame, where P is an integer greater than or equal to 1, and the first pixel is any pixel in the second target video frame.

[0177] The first semantic features of the second pixel are selectively aggregated into the second semantic features of the first pixel;

[0178] Wherein, the first target video frame is the third video frame or the second video frame; the second target video frame is the second video frame or the first video frame; the first semantic feature is a static semantic feature or a dynamic semantic feature; the second semantic feature is a static semantic feature or a dynamic semantic feature;

[0179] When the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the second video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature; or, when the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature.

[0180] Optionally, the processor 401 is further specifically used for:

[0181] A deformable convolutional network is used to enhance the connection between the second pixel and all other pixels in the first target video frame except the second pixel.

[0182] Optionally, the processor 401 is configured to:

[0183] By using multivariate class prototypes for comparative learning training, a multivariate class prediction model is obtained.

[0184] The dynamic semantic features of the fifth video frame are mapped and processed into multivariate features;

[0185] The multivariate features are input into the multivariate class prediction model to obtain the multivariate prediction result of the fifth video frame output by the multivariate class prediction model.

[0186] Based on the multivariate prediction results of the fifth video frame, the semantic segmentation results of the fifth video frame are obtained;

[0187] The fifth video frame is any video frame in the target video frame sequence.

[0188] Optionally, the processor 401 is specifically used for:

[0189] The semantic segmentation result of the fifth video frame is obtained by performing joint decision processing and bilinear interpolation on the multivariate prediction results of the fifth video frame.

[0190] Among them, Figure 4 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 401) and memory (memory 403). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 402 provides user interface 405. Transceiver 404 may be multiple elements, including transmitters and receivers, providing units for communicating with various other devices over a transmission medium. Processor 401 is responsible for managing the bus architecture and general processing, and memory 403 may store data used by processor 401 during operation.

[0191] In addition, specific embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the video semantic segmentation method as described above.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0193] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0194] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] A specific embodiment of the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described functionality. Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0196] The above describes the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A video semantic segmentation method, characterized in that, include: Static semantic feature aggregation is performed on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence. Based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame, dynamic semantic feature aggregation processing is performed on the target video frame sequence to obtain the dynamic semantic features of the target video frame sequence. The first video frame is any video frame in the target video frame sequence other than the first two video frames, and the second video frame and the third video frame are the two video frames in the target video frame sequence that are adjacent to the first video frame. Based on the dynamic semantic features of the target video frame sequence, multivariate joint prediction processing is performed on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence. Specifically, based on the static semantic features of the first video frame, the second video frame, and the third video frame, dynamic semantic feature aggregation processing is performed on the target video frame sequence to obtain the dynamic semantic features of the target video frame sequence, including: Based on the perceptual difference between the static semantic features of the third video frame and the static semantic features of the second video frame, the static semantic features of the third video frame are selectively aggregated into the static semantic features of the second video frame to obtain the dynamic semantic features of the second video frame. Based on the perceptual difference between the dynamic semantic features of the second video frame and the static semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the static semantic features of the first video frame to obtain the dynamic semantic features of the first video frame. Based on the perceptual difference between the dynamic semantic features of the second video frame and the dynamic semantic features of the first video frame, the dynamic semantic features of the second video frame are selectively aggregated into the dynamic semantic features of the first video frame. Based on the perceptual difference between the static semantic features of the third video frame and the dynamic semantic features of the first video frame, the static semantic features of the third video frame are selectively aggregated into the dynamic semantic features of the first video frame, thus obtaining the final dynamic semantic features of the first video frame. The third video frame is located before the second video frame.

2. The method according to claim 1, characterized in that, Static semantic feature aggregation is performed on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence, including: For the fourth video frame, a linear attention mechanism is used to obtain the global information of the fourth video frame; The global information is aggregated using a deformable convolutional network to obtain the static semantic features of the fourth video frame; The fourth video frame is any video frame in the target video frame sequence.

3. The method according to claim 1, characterized in that, The process of selective polymerization includes: The perceptual difference between the first semantic feature of the first target video frame and the second semantic feature of the second target video frame is convolved to obtain the attention offset map. For the first pixel in the second target video frame, query the attention offset map for P second pixels of the first target video frame, where P is an integer greater than or equal to 1, and the first pixel is any pixel in the second target video frame. The first semantic features of the second pixel are selectively aggregated into the second semantic features of the first pixel; Wherein, the first target video frame is the third video frame or the second video frame; the second target video frame is the second video frame or the first video frame; the first semantic feature is a static semantic feature or a dynamic semantic feature; the second semantic feature is a static semantic feature or a dynamic semantic feature; When the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the second video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a static semantic feature; or, when the first target video frame is the second video frame and the first semantic feature is a dynamic semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature; or, when the first target video frame is the third video frame and the first semantic feature is a static semantic feature, the second target video frame is the first video frame and the second semantic feature is a dynamic semantic feature.

4. The method according to claim 3, characterized in that, The method further includes: A deformable convolutional network is used to enhance the connection between the second pixel and all other pixels in the first target video frame except the second pixel.

5. The method according to claim 1, characterized in that, Based on the dynamic semantic features of the target video frame sequence, multivariate joint prediction processing is performed on each video frame in the target video frame sequence to obtain the semantic segmentation result of the target video frame sequence, including: By using multivariate class prototypes for comparative learning training, a multivariate class prediction model is obtained. The dynamic semantic features of the fifth video frame are mapped and processed into multivariate features; The multivariate features are input into the multivariate class prediction model to obtain the multivariate prediction result of the fifth video frame output by the multivariate class prediction model. Based on the multivariate prediction results of the fifth video frame, the semantic segmentation results of the fifth video frame are obtained; The fifth video frame is any video frame in the target video frame sequence.

6. The method according to claim 5, characterized in that, Based on the multivariate prediction results of the fifth video frame, the semantic segmentation results of the fifth video frame are obtained, including: The semantic segmentation result of the fifth video frame is obtained by performing joint decision processing and bilinear interpolation on the multivariate prediction results of the fifth video frame.

7. A video semantic segmentation device, characterized in that, include: The first processing module is used to perform static semantic feature aggregation processing on each video frame in the target video frame sequence to obtain the static semantic features of each video frame in the target video frame sequence. The second processing module is used to perform dynamic semantic feature aggregation processing on the target video frame sequence based on the static semantic features of the first video frame, the static semantic features of the second video frame, and the static semantic features of the third video frame to obtain the dynamic semantic features of the target video frame sequence. The first video frame is any video frame in the target video frame sequence other than the first two video frames, and the second video frame and the third video frame are the two video frames in the target video frame sequence that are adjacent to the first video frame. The third processing module is used to perform multivariate joint prediction processing on each video frame in the target video frame sequence based on the dynamic semantic features of the target video frame sequence, so as to obtain the semantic segmentation result of the target video frame sequence. The second processing module includes: The third processing unit is used to selectively aggregate the static semantic features of the third video frame into the static semantic features of the second video frame based on the perceptual difference between the static semantic features of the third video frame and the static semantic features of the second video frame, so as to obtain the dynamic semantic features of the second video frame. The fourth processing unit is configured to selectively aggregate the dynamic semantic features of the second video frame into the static semantic features of the first video frame based on the perceptual difference between the dynamic semantic features of the second video frame and the static semantic features of the first video frame, thereby obtaining the dynamic semantic features of the first video frame. The fifth processing unit is configured to selectively aggregate the dynamic semantic features of the second video frame into the dynamic semantic features of the first video frame based on the perceptual difference between the dynamic semantic features of the second video frame and the dynamic semantic features of the first video frame, and to selectively aggregate the static semantic features of the third video frame into the dynamic semantic features of the first video frame based on the perceptual difference between the static semantic features of the third video frame and the dynamic semantic features of the first video frame, thereby obtaining the final dynamic semantic features of the first video frame. The third video frame is located before the second video frame.

8. A video semantic segmentation device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the video semantic segmentation method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program that, when executed by a processor, implements the steps of the video semantic segmentation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the video semantic segmentation method as described in any one of claims 1 to 6.

Images