Video understanding method and system for carrying out key frame enhancement based on differential gradient

Abstract

The invention discloses a video understanding method and system for key frame enhancement based on differential gradient. The method comprises the following steps: step 1, constructing a behavior identification data set; 2, preprocessing the behavior generation original video; step 3, space-time gradient attention movement enhancement; and 4, processing the data subjected to the space-time gradient attention movement enhancement processing, and then carrying out space-time average pooling and linear mapping processing to realize understanding of behaviors contained in the video. The difference gradient information of the video directly participates in the calculation of the attention weight, and the attention information conforming to the human intuition can be obtained. According to the method, spatial dimension information is dynamically adjusted according to the generated attention weight, pixel-level affine transformation is achieved, time sequence consistency is kept, differentiation processing can be carried out on different areas, and balance obviously superior to that of a traditional method is achieved between the calculation efficiency and the enhancement effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to a video understanding method and system for key frame enhancement based on differential gradients. Background Art

[0002] FiLM, or Feature-wise Linear Modulation, is a technique with important applications in visual processing. FiLM generates feature-level scaling (γ) and offset (β) parameters based on conditional inputs (such as a language question), applying a channel-by-channel linear transformation to the intermediate features of a target network (such as a CNN). This modulation approach does not rely on complex attention mechanisms or feature concatenation, but instead dynamically adjusts the feature distribution using lightweight parameters, significantly improving both parameter and computational efficiency. FiLM's γ and β parameters are dynamically generated by a separate generative network (such as a GRU or MLP) based on conditional inputs (such as a text question). For example, in visual reasoning tasks, the language question is encoded by a GRU, generating modulation parameters at different levels that are applied layer by layer to the residual blocks of visual features. This design enables deep interaction between language and visual information, rather than fusion at the final stage. Furthermore, FiLM layers can be directly embedded into intermediate layers of existing network architectures (such as ResNet) without changing the underlying network structure. Experiments show that FiLM is robust to network architectures, maintaining stable performance even when batch normalization layers are removed or the number of residual blocks is adjusted. However, FiLM's conditional modulation of static images only generates global γ / β in the channel dimension, failing to reflect differences in spatial position or time steps, nor considering the temporal structure between frames. Furthermore, its conditional sources often rely on external language or classification vectors, making it difficult to adaptively exploit internal spatiotemporal dynamic features.

[0003] The Stand-alone Inter-Frame Attention (SIFA) module is an independent inter-frame attention primitive that accurately exploits temporal correlations and adapts to object deformation by performing self-attention on features from adjacent frames within a local deformable neighborhood. SIFA first constructs a query for each spatial location in the current frame and a key / value pair for the deformable sample points within the corresponding local region in the next frame. It then generates attention weights through dot products and weightedly aggregates the values ​​to achieve cross-frame feature enhancement. It also uses a motion saliency map generated by inter-frame differences to guide the prediction of deformable offsets, enabling the attention computation to focus on high-motion regions and adapt to large deformation scenarios. However, SIFA only computes attention within the local deformable neighborhood of the next frame, making it difficult to capture global temporal dependencies across long time intervals. Its differential recalibration offsets still use unsmoothed pixel differences, making the sample points susceptible to noise. Furthermore, local attention is performed only once per feature map layer, failing to integrate spatiotemporal dynamics at multiple scales and levels.

[0004] The main innovation of the paper "Learning from Temporal Gradient for Semi-supervised Action Recognition" is to introduce temporal gradient (TG) into semi-supervised video action recognition as a second modality that is equally important as RGB, so as to make better use of the motion information of the video. To this end, the authors proposed two multimodal feature constraints: one is a knowledge distillation strategy based on block-wise dense alignment from TG to RGB to extract fine motion features; the other is cross-modal contrastive learning to ensure high-level semantic consistency CVF open access. This method uses TG to enhance the representation ability of the student model in the training phase, but only retains the RGB network in the inference phase, without adding additional computation or parameter overhead. This method directly uses the difference of pixels between adjacent frames (Temporal Gradient, TG) as the "second modality" input, but the gradient signal is not smoothed or denoised, resulting in large gradient noise and susceptibility to interference; in the inference stage, the importance of different time steps is not explicitly weighted or attention is calculated, making it impossible to distinguish key frames from redundant frames; in addition, the TG branch and the RGB branch are only loosely coupled through consistency loss, and multimodal information is not fully integrated at the feature level. Summary of the Invention

[0005] This paper addresses the problems of existing video enhancement methods, which underutilize temporal information; existing spatial modulation methods, which lack global spatial context modeling and thus have limited enhancement effects to local regions; and existing temporal attention methods, which struggle to capture long-range dependencies between frames. The paper proposes a spatiotemporal gradient attention motion enhancement module that directly incorporates the video's differential gradient information into the calculation of attention weights, generating intuitive attention information. The generated attention weights dynamically adjust spatial dimension information to achieve pixel-level affine transformations, maintaining temporal consistency while enabling differentiated processing for different regions. This method achieves a significantly better balance between computational efficiency and enhancement effects than traditional methods.

[0006] To this end, the present invention provides the following technical solutions:

[0007] A video understanding method for key frame enhancement based on differential gradients, comprising:

[0008] Step 1: Build an action recognition dataset;

[0009] The behavior recognition dataset includes the original video of the behavior and the behavior labels;

[0010] The dataset includes the original video and behavior labels. The time when the behavior occurred is not annotated. When creating the dataset, we only need to ensure that the behavior does not appear at the same position in the video to prevent the network from learning incorrect knowledge.

[0011] For example, if the action always occurs in the middle of the video, the network may not learn any temporal and spatial information and simply consider the middle of all videos as the time when the action occurs.

[0012] Step 2: Preprocess the original video of the behavior, and then use the PyTorch Dataset and DataLoader components to read the tensor and corresponding behavior category label in parallel. The preprocessing includes frame extraction and spatial transformation.

[0013] Step 3: Spatiotemporal gradient attention motion enhancement;

[0014] The preprocessed tensor is sequentially subjected to differential feature extraction and Gaussian gradient enhancement processing, and the preprocessed tensor is subjected to spatial information pooling processing. The results of Gaussian gradient enhancement and spatial information pooling processing are then subjected to temporal fusion processing to achieve spatiotemporal gradient attention motion enhancement;

[0015] Step 4: The data after spatiotemporal gradient attention motion enhancement is sequentially processed through 3D batch blocks, four consecutive linear encoding and Video Swin Transformer series blocks, and then spatiotemporal average pooling and linear mapping to understand the behavior contained in the video.

[0016] Furthermore, the frame extraction in the pre-processing is to uniformly sample a frame sequence with a fixed length of 64 frames from each video segment;

[0017] The spatial transformation is to perform center cropping or random cropping on each frame image and scale the cropped image resolution to a uniform size of 224×224.

[0018] Stacked into a tensor of shape (B, C, T, H, W), B is the batch size, C is the 3 RGB channels, T is the number of frames, and H×W is the image resolution.

[0019] Finally, use PyTorch's Dataset and DataLoader components to read the tensor and the corresponding video behavior category labels in parallel;

[0020] Video categories refer to various behaviors of vehicles and pedestrians, including normal behaviors such as vehicle driving and stopping, as well as abnormal behaviors such as vehicle collisions; including normal behaviors such as pedestrian walking, as well as abnormal behaviors such as pedestrian congestion and pedestrian falls.

[0021] Furthermore, the differential feature extraction process is as follows:

[0022] First, the preprocessed tensor Calculate the absolute difference between adjacent frames;

[0023] Then, the preliminary temporal change features are extracted through two layers of small kernels, and the inter-frame difference information is obtained as the differential feature;

[0024] Inter-frame differential information uses express;

[0025] Among them, each layer of small kernels includes kernel 3D convolution, group normalization and GELU processing in sequence. Group normalization refers to GroupNormalization, which divides the channels into several groups and calculates the mean and variance within each group for standardization; GELU is a Gaussian error linear unit of a smooth activation function.

[0026] Furthermore, the Gaussian gradient enhancement process is as follows:

[0027] First, the differential features are smoothed by Gaussian kernel to obtain smooth inter-frame differential information. ;

[0028] Next, the gradient is calculated by finite difference, that is, the gradient of the smoothed inter-frame difference information is calculated using the forward, center, and backward difference operators;

[0029] Finally, the normalization function is used to normalize the gradient of the smooth inter-frame difference information to obtain the smooth inter-frame gradient information .

[0030] Furthermore, the processing process of the time series fusion is as follows:

[0031] First, the original time domain information is processed by key generation convolution to obtain the key vector for attention calculation; and the smoothed inter-frame gradient information is processed by query generation convolution to obtain the query vector for attention calculation;

[0032] Secondly, the original time domain information and the smoothed inter-frame gradient information are concatenated in the channel dimension and then subjected to value generation convolution processing to obtain the value vector used for attention calculation;

[0033] Finally, the query vector is multiplied by the key vector and then processed by the normalization function. The result after normalization is multiplied by the value vector to obtain the temporal fusion attention weight, and the smooth inter-frame gradient information and the temporal fusion attention weight are temporally fused.

[0034] Furthermore, the data after time series fusion processing is subjected to spatiotemporal modulation based on dynamic parameters. The specific process is as follows:

[0035] First, the temporal fusion features are input into the parameter generator, which includes 3D convolution, group normalization, GELU and 3D convolution processing in sequence;

[0036] Next, the data output by the parameter generator Perform an average split on the dimension, and get two dimensions as The original parameters are then normalized to obtain the "adjustment" parameters. With the "Offset" parameter ;

[0037] Finally, use the "Adjustment" parameter With the "Offset" parameter Perform spatiotemporal modulation on the preprocessed tensor, and use the spatiotemporal modulated data as the data processed by spatiotemporal gradient attention motion enhancement;

[0038] The spatiotemporal modulation formula is as follows:

[0039] ,

[0040] in, Represents the data after time and space modulation, Represents a preprocessed tensor.

[0041] Spatiotemporal modulation based on dynamic parameters considers video information as a stack of many frames in the time domain. Before modulation, the parameters in the time domain should be relatively even, that is, each frame has a similar impact on the final output. After spatiotemporal modulation, the parameters of frames near the moment the action occurs will be amplified in the time domain, ultimately having a greater impact on the output.

[0042] The Video Swin Transformer is an existing approach. It's a pure Transformer backbone network designed specifically for video understanding. This model uses a shifted window strategy in both spatiotemporal and temporal dimensions to divide the video into fixed-size 3D patch windows. Self-attention is computed only within these windows, and the window positions are shifted alternately at different levels to balance local fine-grained features with long-range dependencies across windows. After every few attention blocks, the model uses hierarchical patch merging to merge adjacent 2×2×2 patches into a single token, and the number of channels is increased to achieve gradual downsampling and feature upscaling.

[0043] Then enter the Video Swin Transformer network,

[0044] First, the batch encoding layer divides it into several 3D patches of uniform size and maps them into a high-dimensional feature space;

[0045] Then, spatiotemporal downsampling and channel amplification are achieved through multiple layers of batch fusion. Each batch fusion halves the spatial and temporal dimensions of the feature map and doubles the number of channels.

[0046] This is followed by multiple stages of window shifting, in which window self-attention calculation and moving window self-attention calculation are performed alternately to capture fine-grained spatiotemporal dependencies within the local window and long-range dependencies across windows.

[0047] First, the temporal features are smoothed using a one-dimensional Gaussian convolution kernel, and then the gradient is calculated using the central difference operator, which significantly reduces noise and improves signal stability. The normalized gradient amplitude is then used as an attention cue and input into the temporal fusion attention module, which assigns explicit weights to different time steps and highlights key frames. Finally, the smoothed gradient and the projected RGB features are simultaneously fused in the attention calculation, achieving richer spatiotemporal interactions and surpassing the weak fusion strategy of a single consistency loss.

[0048] Furthermore, steps 2 to 4 are repeated, and supervised learning is performed using the cross entropy loss function and the true labels to optimize the classification performance of the video understanding network formed in steps 3 and 4.

[0049] The video understanding network formed in step 3 and step 4 is a Video SwinTransformer network with added spatiotemporal gradient attention motion enhancement.

[0050] In a second aspect, a system using the above-mentioned video understanding method for key frame enhancement based on differential gradients includes:

[0051] Behavior recognition dataset construction module: used to obtain behavior recognition datasets, including original behavior videos, behavior occurrence time, and behavior labels;

[0052] Video preprocessing and tensor reading module: Preprocesses the original video of the behavior, and then uses the PyTorch Dataset and DataLoader components to read the tensor and corresponding behavior category label of the preprocessed video in parallel. The preprocessing includes frame extraction and spatial transformation.

[0053] Spatiotemporal gradient attention motion enhancement module: It is used to perform differential feature extraction and Gaussian gradient enhancement on the preprocessed tensor in sequence, and to perform spatial information pooling on the preprocessed tensor. The results of Gaussian gradient enhancement and spatial information pooling are then temporally fused together to achieve spatiotemporal gradient attention motion enhancement.

[0054] Video Understanding Module: This module is used to sequentially process the data processed by spatiotemporal gradient attention motion enhancement through 3D batch blocks, four consecutive linear encoding and Video Swin Transformer blocks, and then perform spatiotemporal average pooling and linear mapping to understand the behaviors contained in the video.

[0055] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program is called by a processor to implement the steps of the above-mentioned video understanding method for key frame enhancement based on differential gradient.

[0056] According to a fourth aspect, an electronic terminal includes:

[0057] one or more processors;

[0058] a memory storing one or more computer programs;

[0059] The processor calls a computer program to implement:

[0060] The steps of the above-mentioned video understanding method for key frame enhancement based on differential gradient.

[0061] Beneficial effects

[0062] Compared with the existing method, the advantages of the present invention are:

[0063] 1. Explicitly use time-domain differential gradient information for modeling

[0064] In response to the problem that existing video enhancement methods do not fully utilize time domain information and have difficulty in accurately characterizing dynamic details between frames, the technical solution of the present invention innovatively introduces inter-frame differential gradient information into the time domain modeling process, thereby effectively capturing motion changes in the video. Existing methods do not explicitly utilize gradient information, which makes it impossible to distinguish key frames from redundancy, thus losing the advantages of gradient information. At the same time, existing methods that do not explicitly utilize inter-frame differential gradient information do not smooth or denoise the gradient signal, resulting in large gradient noise and susceptibility to interference. The present invention uses a Gaussian gradient enhancement module to suppress noise and accurately characterize dynamic details between frames.

[0065] 2. Perform global pixel-level spatial modulation on the original data based on the attention information.

[0066] Existing methods (FiLM) can only generate channel-level modulation parameters, ignoring differences in spatial and temporal variations, resulting in insufficient enhancement of key details. To address this issue, the present invention proposes a spatiotemporal affine parameter generation module that utilizes a differential gradient-guided attention map to dynamically generate local "adjustment" parameters and global "offset" parameters. This module performs global spatial modulation on the original frame pixels, enabling the model to adaptively adjust feature intensities and baseline deviations, thereby enhancing key areas and details. Furthermore, relying solely on spatial information makes it difficult to locate the moment of motion and fails to focus on specific motion features. This invention incorporates differential gradient information into the generation of modulation parameters, enabling adaptive adjustment of pixel weights based on motion characteristics.

[0067] The spatiotemporal affine parameter generation module dynamically generates two parameters, "adjustment" and "offset," by fusing spatiotemporal attention weights to account for feature importance differences and baseline deviation compensation, respectively. The "adjustment" parameter provides local relative adjustments to highlight important regions, while the "offset" parameter provides global absolute adjustments to maintain the overall adjustment parameter within a reasonable range. In summary, using attention weights to guide the generation of spatial modulation parameters and dynamically adjusting the intensity of motion in different regions aligns with the principles of visual cognition at a physical level and is more suitable for handling spatiotemporal inconsistencies in dynamic scenes than FiLM.

[0068] 3. Perform global time domain dimension attention calculation on the original time domain information and differential gradient information.

[0069] To address the problem that SIFA only calculates attention within a local deformable neighborhood and fails to fully capture long-distance dependencies between frames, the present invention uses a temporal fusion attention module to simultaneously associate the original frame sequence and differential features across the entire temporal scale, thereby achieving global temporal fusion of information.

[0070] The temporal fusion attention module proposed in this technical solution performs attention operations on the complete temporal information to obtain attention information unique to the time domain dimension. Since the spatial dimension and the temporal dimension organize information differently, performing attention calculations separately for the time domain dimension can obtain more meaningful attention information. At the same time, both the original input time domain information and the differential gradient information are Gaussian smoothed to avoid the influence of noise. At the same time, the two are spliced ​​in the channel dimension and used as value vectors for attention calculation, which allows the attention mechanism to focus on "where" there is important content and to capture "when" significant changes occur, thereby obtaining richer feature expressions and significantly improving the model's sensitivity to temporal changes and the fusion of contextual information. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of the spatiotemporal gradient attention motion enhancement module in the technical solution of the present invention;

[0072] Figure 2 Schematic diagram of the temporal fusion attention module in the technical solution of the present invention;

[0073] Figure 3 Schematic diagram of the differential feature extraction module in the technical solution of the present invention;

[0074] Figure 4 Schematic diagram of the Gaussian gradient enhancement module in the technical solution of the present invention;

[0075] Figure 5 Schematic diagram of a spatiotemporal affine parameter generation module in the technical solution of the present invention;

[0076] Figure 6 It is a schematic diagram of the overall network structure in the technical solution of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be further described below with reference to the embodiments.

[0078] Example 1:

[0079] A video understanding method for key frame enhancement based on differential gradients, comprising:

[0080] Step 1: Build an action recognition dataset;

[0081] The behavior recognition dataset includes the original video of the behavior and the behavior labels;

[0082] The behavior recognition dataset does not mark the time when the behavior occurs. When creating the dataset, we only take care not to let the behavior occur at the same position in the video to prevent the network from learning incorrect knowledge.

[0083] For example, if the action always occurs in the middle of the video, the network may not learn any temporal and spatial information and simply consider the middle of all videos as the time when the action occurs.

[0084] Collect a large amount of raw video from public datasets (such as Kinetics-400, UCF-101, and HMDB51) or custom cameras, ensuring coverage of diverse action categories. Collect video data of pedestrians in various environments, keeping the duration of each data point within 3-5 seconds. Be careful to locate the moment of action at a random location within each video point. Label each video segment with predefined action labels, forming (video_path, label) pairs and storing them in a database or CSV file.

[0085] Step 2: Preprocess the original video of the behavior. Then, use PyTorch's Dataset and DataLoader components to read tensors and corresponding behavior category labels in parallel. The preprocessing includes frame extraction and spatial transformation.

[0086] In the pre-processing, frame extraction is to uniformly sample a frame sequence with a fixed length of 64 frames from each video segment;

[0087] The spatial transformation is to perform center cropping or random cropping on each frame image and scale the cropped image resolution to a uniform size of 224×224.

[0088] Stacked into a tensor of shape (B, C, T, H, W), B is the batch size, C is the 3 RGB channels, T is the number of frames, and H×W is the image resolution.

[0089] Finally, use PyTorch's Dataset and DataLoader components to read the tensor and the corresponding video behavior category labels in parallel;

[0090] Video categories refer to various behaviors of vehicles and pedestrians, including normal behaviors such as vehicle driving and stopping, as well as abnormal behaviors such as vehicle collisions; including normal behaviors such as pedestrian walking, as well as abnormal behaviors such as pedestrian congestion and pedestrian falls.

[0091] Step 3: Spatiotemporal gradient attention motion enhancement, such as Figure 1 As shown;

[0092] The preprocessed tensor is sequentially subjected to differential feature extraction and Gaussian gradient enhancement processing, and the preprocessed tensor is subjected to spatial information pooling processing. The results of Gaussian gradient enhancement and spatial information pooling processing are then subjected to temporal fusion processing to achieve spatiotemporal gradient attention motion enhancement;

[0093] like Figure 3 As shown, the differential feature extraction process is as follows:

[0094] First, the preprocessed tensor Calculate the absolute difference between adjacent frames;

[0095] Then, the preliminary temporal change features are extracted through two layers of small kernels, and the inter-frame difference information is obtained as the differential feature;

[0096] Inter-frame differential information uses express;

[0097] Among them, each layer of small kernels includes kernel 3D convolution, group normalization and GELU processing in sequence. Group normalization refers to GroupNormalization, which divides the channels into several groups and calculates the mean and variance within each group for standardization; GELU is a Gaussian error linear unit of a smooth activation function.

[0098] like Figure 4 As shown, the Gaussian gradient enhancement process is as follows:

[0099] First, the differential features are smoothed by Gaussian kernel to obtain smooth inter-frame differential information. ;

[0100] Next, the gradient is calculated by finite difference, that is, the gradient of the smoothed inter-frame difference information is calculated using the forward, center, and backward difference operators;

[0101] Finally, the normalization function is used to normalize the gradient of the smooth inter-frame difference information to obtain the smooth inter-frame gradient information .

[0102] like Figure 2 As shown, the processing process of the time series fusion is as follows:

[0103] First, the original time domain information is processed by key generation convolution to obtain the key vector for attention calculation; and the smoothed inter-frame gradient information is processed by query generation convolution to obtain the query vector for attention calculation;

[0104] Secondly, the original time domain information and the smoothed inter-frame gradient information are concatenated in the channel dimension and then subjected to value generation convolution processing to obtain the value vector used for attention calculation;

[0105] Finally, the query vector is multiplied by the key vector and then processed by the normalization function. The result after normalization is multiplied by the value vector to obtain the temporal fusion attention weight, and the smooth inter-frame gradient information and the temporal fusion attention weight are temporally fused.

[0106] and The dimension is B×C×T×1×1, and Stacked after the batch dimension B, generating parameters of dimension B×2×C×T×1×1 .

[0107] The data after time series fusion processing is then subjected to spatiotemporal modulation based on dynamic parameters, such as Figure 5 As shown, the specific process is as follows:

[0108] First, the temporal fusion features are input into the parameter generator, which includes 3D convolution, group normalization, GELU and 3D convolution processing in sequence;

[0109] Next, the data output by the parameter generator Perform an average split on the dimension, and get two dimensions as The original parameters are then normalized to obtain the "adjustment" parameters. With the "Offset" parameter ;

[0110] Finally, use the "Adjustment" parameter With the "Offset" parameter Perform spatiotemporal modulation on the preprocessed tensor, and use the spatiotemporal modulated data as the data processed by spatiotemporal gradient attention motion enhancement;

[0111] The spatiotemporal modulation formula is as follows:

[0112] ,

[0113] in, Represents the data after time and space modulation, Represents a preprocessed tensor.

[0114] Spatiotemporal modulation based on dynamic parameters considers video information as a stack of many frames in the time domain. Before modulation, the parameters in the time domain should be relatively even, that is, each frame has a similar impact on the final output. After spatiotemporal modulation, the parameters of frames near the moment the action occurs will be amplified in the time domain, ultimately having a greater impact on the output.

[0115] The Video Swin Transformer is an existing approach. It's a pure Transformer backbone network designed specifically for video understanding. This model uses a shifted window strategy in the spatiotemporal dimensions to divide the video into several fixed-size 3D patch windows. Self-attention is calculated only within these windows, and the window positions are shifted alternately at different levels to balance local fine-grained features with long-range dependencies across windows. After every few attention blocks, the model uses hierarchical patch merging to merge adjacent 2×2×2 patches into a single token. The number of channels is then increased to achieve gradual downsampling and feature upscaling.

[0116] Then enter the Video Swin Transformer network:

[0117] First, the batch encoding layer divides it into several 3D patches of uniform size and maps them into a high-dimensional feature space;

[0118] Then, spatiotemporal downsampling and channel amplification are achieved through multiple layers of batch fusion. Each batch fusion halves the spatial and temporal dimensions of the feature map and doubles the number of channels.

[0119] This is followed by multiple stages of window shifting, in which window self-attention calculation and moving window self-attention calculation are performed alternately to capture fine-grained spatiotemporal dependencies within the local window and long-range dependencies across windows.

[0120] Step 4: The data after spatiotemporal gradient attention motion enhancement is sequentially processed through 3D batch blocks, four consecutive linear encoding and Video Swin Transformer series blocks, and then spatiotemporal average pooling and linear mapping to obtain the action category prediction contained in the video and achieve understanding of the behavior contained in the video.

[0121] Repeat steps 2 to 4, and use the cross entropy loss function and the true label for supervised learning to optimize the Figure 6 Classification performance of the video understanding network shown.

[0122] The video understanding network formed in step 3 and step 4 is a Video SwinTransformer network with added spatiotemporal gradient attention motion enhancement.

[0123] Traditional inter-frame difference methods usually only calculate simple differences between pixels or features of adjacent frames, often obtaining binary or noisy motion masks, lacking continuous quantification of motion intensity, making it difficult to distinguish between weak motion and background changes in complex scenes. In contrast, the technical solution of the present invention first performs Gaussian smoothing on the frame difference features and calculates the central differential gradient, and then performs softmax normalization on the absolute gradient value to generate a delicate motion intensity distribution that can more accurately capture subtle changes between different frames. After using this gradient distribution as the query in the attention mechanism, the model can dynamically assign attention weights according to the motion intensity, effectively highlighting sudden motion events and suppressing meaningless background changes, thereby achieving better performance in temporal consistency and spatial target positioning than relying solely on binary differences or traditional optical flow input.

[0124] In traditional temporal self-attention mechanisms, the query, key, and value all derive from the same raw temporal features. This results in an even distribution of attention, making it difficult to actively favor frames with high motion, thus diluting attention to key action information. Our approach uses normalized gradient features as the query to "ask" where motion is occurring; the raw temporal features as the key to provide global contextual localization; and concatenating the two to generate the value through convolution, resulting in an output that combines both static semantics and dynamic cues. This Q / K / V separation makes the attention matrix sharper in the temporal dimension, placing greater weight on frames with motion.

[0125] The technical solution of this invention uses a parameter generation network to output scale and shift parameters that match the feature map resolution, thereby performing a pixel-level affine transformation on the features. This amplifies or offsets the features at each location based on the motion-driven attention weights. This pixel-level modulation based on gradient attention automatically guides the network to focus more computational resources on the locations where motion occurs, effectively suppressing interference from static backgrounds. Compared to motion guidance methods that rely on additional optical flow or region detection, this approach is more lightweight and end-to-end trainable.

[0126] Example 2

[0127] A system using the above-mentioned video understanding method for key frame enhancement based on differential gradients includes:

[0128] Behavior recognition dataset construction module: used to obtain behavior recognition datasets, including original behavior videos, behavior occurrence time, and behavior labels;

[0129] Video preprocessing and tensor reading module: Preprocesses the original video of the behavior, and then uses the PyTorch Dataset and DataLoader components to read the tensor and corresponding behavior category label of the preprocessed video in parallel. The preprocessing includes frame extraction and spatial transformation.

[0130] Spatiotemporal gradient attention motion enhancement module: It is used to perform differential feature extraction and Gaussian gradient enhancement on the preprocessed tensor in sequence. At the same time, it performs spatial information pooling on the preprocessed tensor and performs temporal fusion processing on the results of Gaussian gradient enhancement and spatial information pooling to achieve spatiotemporal gradient attention motion enhancement;

[0131] Video Understanding Module: This module is used to sequentially process the data processed by spatiotemporal gradient attention motion enhancement through 3D batch blocks, four consecutive linear encoding and Video Swin Transformer blocks, and then perform spatiotemporal average pooling and linear mapping to understand the behaviors contained in the video.

[0132] It should also be understood that the specific implementation process of each module please refer to the above method content, the present invention will not go into details here, and the division of the above functional modules is only for example illustration. In some embodiments, some functional modules can be merged, and some functional modules can be split. Each functional module can be implemented in software or hardware or a combination of software and hardware. Among them, software and hardware equipment include but are not limited to general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0133] Example 3

[0134] A computer-readable storage medium stores a computer program, wherein the computer program is called by a processor to implement the steps of the above-mentioned video understanding method for key frame enhancement based on differential gradient.

[0135] For the specific implementation process of each step, please refer to the description of the above method.

[0136] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard disk equipped on the controller, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Furthermore, the readable storage medium can also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0137] Example 4

[0138] An electronic terminal, comprising:

[0139] one or more processors;

[0140] a memory storing one or more computer programs;

[0141] The processor calls a computer program to implement:

[0142] The steps of the above-mentioned video understanding method for key frame enhancement based on differential gradient.

[0143] For the specific implementation process of each step, please refer to the description of the above method.

[0144] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0145] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0146] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is a flow chart according to the method, device (system), and computer program product of the embodiment of the present application and / or the instructions executed by the processor to generate a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product comprising an instruction device, which realizes the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0147] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A video understanding method for key frame enhancement based on differential gradient, characterized in that: include: Step 1: Build an action recognition dataset; The behavior recognition dataset includes the original video of the behavior and the behavior labels; Step 2: Preprocess the original video of the behavior. Then, use PyTorch's Dataset and DataLoader components to read tensors and corresponding behavior category labels in parallel. The preprocessing includes frame extraction and spatial transformation. Step 3: Spatiotemporal gradient attention motion enhancement; The preprocessed tensor is sequentially subjected to differential feature extraction and Gaussian gradient enhancement processing, and the preprocessed tensor is subjected to spatial information pooling processing. The results of Gaussian gradient enhancement and spatial information pooling processing are then subjected to temporal fusion processing to achieve spatiotemporal gradient attention motion enhancement; Step 4: The data after spatiotemporal gradient attention motion enhancement is sequentially processed into 3D batch blocks, four consecutive linear encoding and Video Swin Transformer series blocks, and then spatiotemporal average pooling and linear mapping are performed to understand the behavior contained in the video.

2. The method according to claim 1, characterized in that In the pre-processing, frame extraction is to uniformly sample a frame sequence with a fixed length of 64 frames from each video segment; The spatial transformation is to perform center cropping or random cropping on each frame image and scale the cropped image resolution to a uniform size of 224×224.

3. The method according to claim 1, characterized in that The differential feature extraction process is as follows: First, the preprocessed tensor Calculate the absolute difference between adjacent frames; Then, the preliminary temporal change features are extracted through two layers of small kernels, and the inter-frame difference information is obtained as the differential feature; Inter-frame differential information uses express; Among them, each layer of small kernels includes kernel 3D convolution, group normalization and GELU processing in sequence. Group normalization refers to GroupNormalization, which divides the channels into several groups and calculates the mean and variance within each group for standardization; GELU is a Gaussian error linear unit of a smooth activation function.

4. The method according to claim 3, characterized in that The Gaussian gradient enhancement process is as follows: First, the differential features are smoothed by Gaussian kernel to obtain smooth inter-frame differential information. ; Next, the gradient is calculated by finite difference, that is, the gradient of the smoothed inter-frame difference information is calculated using the forward, center, and backward difference operators; Finally, the normalization function is used to normalize the gradient of the smooth inter-frame difference information to obtain the smooth inter-frame gradient information .

5. The method according to claim 1, characterized in that The processing process of time series fusion is as follows: First, the original time domain information is processed by key generation convolution to obtain the key vector for attention calculation; and the smoothed inter-frame gradient information is processed by query generation convolution to obtain the query vector for attention calculation; Secondly, the original time domain information and the smoothed inter-frame gradient information are concatenated in the channel dimension and then subjected to value generation convolution processing to obtain the value vector used for attention calculation; Finally, the query vector is multiplied by the key vector and then processed by the normalization function. The result after normalization is multiplied by the value vector to obtain the temporal fusion attention weight, and the smooth inter-frame gradient information and the temporal fusion attention weight are temporally fused.

6. The method according to claim 5, characterized in that The data after time series fusion processing is then subjected to spatiotemporal modulation based on dynamic parameters. The specific process is as follows: First, the temporal fusion features are input into the parameter generator, which includes 3D convolution, group normalization, GELU and 3D convolution processing in sequence; Next, the data output by the parameter generator Perform an average split on the dimension, and get two dimensions as The original parameters are then normalized to obtain the "adjustment" parameters. With the "offset" parameter ; Finally, use the "tuning" parameter With the "offset" parameter Perform spatiotemporal modulation on the preprocessed tensor, and use the spatiotemporal modulated data as the data processed by spatiotemporal gradient attention motion enhancement; The spatiotemporal modulation formula is as follows: ， in, Represents the data after time and space modulation, Represents a preprocessed tensor.

7. The method according to any one of claims 1 to 6, characterized in that Repeat steps 2 to 4 and use the cross entropy loss function with the true labels for supervised learning to optimize the classification performance of the video understanding network formed in steps 3 and 4.

8. A system using the video understanding method for key frame enhancement based on differential gradient according to any one of claims 1 to 7, characterized in that: include: Behavior recognition dataset construction module: used to obtain behavior recognition datasets, including original behavior videos, behavior occurrence time, and behavior labels; Video preprocessing and tensor reading module: Preprocesses the original video of the behavior, and then uses the PyTorch Dataset and DataLoader components to read the tensor and corresponding behavior category label of the preprocessed video in parallel. The preprocessing includes frame extraction and spatial transformation. Spatiotemporal gradient attention motion enhancement module: It is used to perform differential feature extraction and Gaussian gradient enhancement on the preprocessed tensor in sequence, and to perform spatial information pooling on the preprocessed tensor. The results of Gaussian gradient enhancement and spatial information pooling are then temporally fused together to achieve spatiotemporal gradient attention motion enhancement. Video Understanding Module: This module is used to sequentially process the data processed by spatiotemporal gradient attention motion enhancement through 3D batch blocks, four consecutive linear encoding and Video Swin Transformer blocks, and then perform spatiotemporal average pooling and linear mapping to understand the behaviors contained in the video.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program is called by a processor to implement the steps of the method according to any one of claims 1 to 7.

10. An electronic terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls a computer program to implement: The steps of the method according to any one of claims 1 to 7.

Images