Video understanding method and system based on key frame enhancement by differential gradient

Abstract

The application discloses a video understanding method and system based on key frame enhancement of differential gradient, and the method comprises the following steps: 1, constructing a behavior recognition dataset; 2, preprocessing the original video of behavior occurrence; 3, spatiotemporal gradient attention motion enhancement; 4, after the data processed by the spatiotemporal gradient attention motion enhancement processing, spatiotemporal average pooling and linear mapping processing are carried out, and the behaviors contained in the video are understood. The differential gradient information of the video is directly involved in the calculation of the attention weight, and the attention information conforming to human intuition can be obtained. The spatial dimension information is dynamically adjusted by the generated attention weight, the pixel-level affine transformation is realized, the time sequence consistency is maintained, and the differentiated processing can be implemented for different regions, and a balance better than the traditional method is achieved between the calculation efficiency and the enhancement effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video processing, and particularly relates to a video understanding method and system for key frame enhancement based on differential gradient. BACKGROUND

[0002] FiLM, Feature-wise Linear Modulation, is a technology with important applications in the field of visual processing. FiLM generates feature-level scaling (γ) and offset (β) parameters through conditional input (such as language problems) to perform linear transformation on the intermediate features of the target network (such as CNN) on a channel-by-channel basis. This modulation method does not rely on complex attention mechanisms or feature concatenation, but dynamically adjusts the feature distribution through lightweight parameters, significantly improving the parameter efficiency and computational efficiency of the model. The γ and β parameters of FiLM are dynamically generated by an independent generation network (such as GRU or MLP) according to the conditional input (such as the text question). For example, in visual reasoning tasks, the language question is encoded by GRU to generate modulation parameters at different levels, which are applied to the residual blocks of visual features layer by layer. This design realizes the deep interaction between language and visual information, rather than just fusing them at the last stage. At the same time, the FiLM layer can be directly embedded into the middle layer of the existing network architecture (such as ResNet), without changing the main structure of the network. Experiments show that FiLM is robust to network architecture, and the performance remains stable even if the batch normalization layer is removed or the number of residual blocks is adjusted. However, the conditional modulation of FiLM for static images only generates global γ / β in the channel dimension, which cannot reflect the differences in spatial position or time step, and does not consider the inter-frame temporal structure; its conditions are mostly dependent on external language or classification vectors, making it difficult to adaptively mine internal spatio-temporal dynamic features.

[0003] Stand-alone Inter-Frame Attention (SIFA) module is a standalone inter-frame attention primitive that precisely mines temporal correlation and adapts object deformation by performing self-attention computation on features of adjacent frames within a locally deformable neighborhood. SIFA first constructs a Query for each spatial position of the current frame, and a Key / Value for the deformable sampling points within the corresponding local region of the next frame, and then generates attention weights by dot product and aggregates the Value weighted, realizing cross-frame feature enhancement; at the same time, the motion saliency map generated by inter-frame difference is used to guide the prediction of deformable offset, so that the attention computation can focus on high motion areas and adapt to large deformation scenarios. However, SIFA only calculates attention within the locally deformable neighborhood of the next frame, making it difficult to capture global temporal dependencies spanning a long time interval; the difference re-labeling offset still uses unsmoothed pixel difference, and the sampling points are easily affected by noise; and the local attention is only performed once for each feature map, failing to integrate spatio-temporal dynamics at multiple scales and levels.

[0004] The main innovation of the paper "Learning from Temporal Gradient for Semi-supervised Action Recognition" is to introduce temporal gradient (TG) into semi-supervised video action recognition as a second modality as important as RGB, so as to make more full use of the motion information of the video. To this end, the authors proposed two kinds of multi-modal feature constraints: one is the block-wise dense alignment knowledge distillation strategy based on TG to RGB, to extract fine motion features; the second is cross-modal contrastive learning, to ensure high-level semantic consistency. The method uses TG to enhance the representation ability of the student model during training, but only retains the RGB network during inference, without increasing additional computation or parameter overhead. This method directly inputs the pixel difference (Temporal Gradient, TG) between adjacent frames as the "second modality", but does not smooth or denoise the gradient signal, resulting in large gradient noise and easily disturbed representation; during the inference stage, the importance of different time steps is not explicitly weighted or attention computed, so it cannot distinguish between key frames and redundant frames; in addition, the TG branch and the RGB branch are only loosely coupled through consistency loss, without fully integrating multi-modal information at the feature level. SUMMARY

[0005] The present application aims at the problems that the existing video enhancement method does not fully utilize the time domain information, the existing spatial modulation method lacks global spatial context modeling and the enhancement effect is limited to local area, and the existing time domain attention method is difficult to capture inter-frame long distance dependency, proposes a spatio-temporal gradient attention motion enhancement module, which directly involves the differential gradient information of the video in the calculation of the attention weight, and can obtain attention information consistent with human intuition. And the spatial dimension information is dynamically adjusted by the generated attention weight, realizing pixel-level affine transformation, which not only maintains the temporal consistency, but also can implement differentiated processing for different areas, achieving a balance between calculation efficiency and enhancement effect which is significantly better than traditional methods.

[0006] To this end, the present application provides the following technical solutions:

[0007] A video understanding method for key frame enhancement based on differential gradient, comprising:

[0008] Step 1: Construct a behavior recognition dataset;

[0009] The behavior recognition dataset includes original videos of behavior occurrence and behavior labels;

[0010] The dataset includes original videos and behavior labels, without any annotation on the behavior occurrence time, only paying attention to not letting the behavior occurrence time all appear in the same position of the video when making the dataset, to avoid the network learning wrong knowledge;

[0011] For example, if the behavior occurrence time all appears in the middle position of the video, the network may not learn any time and space information, and roughly consider the middle position of all videos as the action occurrence time;

[0012] Step 2: Preprocess the original video of behavior occurrence, and then read the tensor and the corresponding behavior category label in parallel through the Dataset and DataLoader components of PyTorch, the preprocessing including frame extraction and spatial transformation;

[0013] Step 3: Spatio-temporal gradient attention motion enhancement;

[0014] The preprocessed tensor is subjected to differential feature extraction and Gaussian gradient enhancement processing in turn, and the preprocessed tensor is subjected to spatial information pooling processing, and the results after Gaussian gradient enhancement and spatial information pooling processing are subjected to time sequence fusion processing together, to realize spatio-temporal gradient attention motion enhancement;

[0015] Step 4: After the data processed by the spatio-temporal gradient attention motion enhancement processing, it is sequentially processed by 3D batch blocking, continuous four linear coding and Video Swin Transformer block processing in series, and then spatio-temporal average pooling and linear mapping processing are performed to realize the understanding of the behaviors contained in the video.

[0016] Further, the frame extraction in the preprocessing is to uniformly sample a frame sequence with a fixed length of 64 frames from each video.

[0017] The spatial transformation is to perform center cropping or random cropping on each frame of image, and the resolution of the cropped image is scaled to a uniform size of 224x224.

[0018] Stacked into a tensor with shape (B, C, T, H, W), where B is the batch size, C is 3 RGB channels, T is the number of frames, and HxW is the image resolution.

[0019] Finally, the PyTorch Dataset and DataLoader components are used to read the tensor and the corresponding behavior category label contained in the video in parallel;

[0020] The categories of the video, i.e. vehicles and pedestrian behaviors, include normal behaviors such as vehicle driving and stopping, and abnormal behaviors such as vehicle collision; include normal behaviors such as pedestrian walking, and abnormal behaviors such as pedestrian congestion and pedestrian falling.

[0021] Further, the differential feature extraction processing process is as follows:

[0022] First, the tensor after preprocessing is Calculate the absolute difference between adjacent frames;

[0023] Then, through two layers of small kernels, the preliminary time sequence change features are extracted, and the inter-frame difference information is obtained as the differential feature;

[0024] The inter-frame difference information is represented as ;

[0025] Wherein, each layer of small kernel includes kernel 3D convolution, group normalization and GELU processing in turn, group normalization refers to GroupNormalization, which divides the channels into several groups and calculates the mean and variance in each group to standardize; GELU is a smooth activation function of Gaussian Error Linear Unit.

[0026] Further, the Gaussian gradient enhancement processing process is as follows:

[0027] First, the differential feature is smoothed by Gaussian kernel to obtain smoothed inter-frame difference information ;

[0028] Next, the gradient is calculated by finite difference, that is, the gradient of the smoothed inter-frame difference information is calculated using the forward, center, and backward difference operators;

[0029] Finally, the normalization function is used to normalize the gradient of the smooth inter-frame difference information to obtain the smooth inter-frame gradient information .

[0030] Furthermore, the processing process of the time series fusion is as follows:

[0031] First, the original time domain information is processed by key generation convolution to obtain the key vector for attention calculation; and the smoothed inter-frame gradient information is processed by query generation convolution to obtain the query vector for attention calculation;

[0032] Secondly, the original time domain information and the smoothed inter-frame gradient information are concatenated in the channel dimension and then subjected to value generation convolution processing to obtain the value vector used for attention calculation;

[0033] Finally, the query vector is multiplied by the key vector and then processed by the normalization function. The result after normalization is multiplied by the value vector to obtain the temporal fusion attention weight, and the smooth inter-frame gradient information and the temporal fusion attention weight are temporally fused.

[0034] Furthermore, the data after time series fusion processing is subjected to spatiotemporal modulation based on dynamic parameters. The specific process is as follows:

[0035] First, the temporal fusion features are input into the parameter generator, which includes 3D convolution, group normalization, GELU and 3D convolution processing in sequence;

[0036] Next, the data output by the parameter generator Perform an average split on the dimension, and get two dimensions as The original parameters are then normalized to obtain the "adjustment" parameters. With the "Offset" parameter ;

[0037] Finally, use the "Adjustment" parameter With the "Offset" parameter Perform spatiotemporal modulation on the preprocessed tensor, and use the spatiotemporal modulated data as the data processed by spatiotemporal gradient attention motion enhancement;

[0038] The spatiotemporal modulation formula is as follows:

[0039] ,

[0040] in, Represents the data after time-space modulation, represents the pre-processed tensor.

[0041] Based on the spatio-temporal modulation of dynamic parameters, the video information is regarded as the stacking of many frames in the time domain direction. Before modulation, the parameters in the time domain direction should be relatively average, that is, each frame has almost the same influence on the final output. After spatio-temporal modulation, in the time domain direction, the parameters of the frames near the action occurrence time are amplified, and the final output is more influential.

[0042] Video Swin Transformer is an existing method. Video Swin Transformer is a pure Transformer backbone network specially designed for video understanding. The model divides the video into several fixed-size 3D patch windows by introducing a shifted window strategy in the space-time dimension, calculates self-attention only within the window, and alternately shifts the window position at different levels to balance local fine-grained features and cross-window long-range dependencies. After every few attention blocks, the model uses a hierarchical patch merging operation to combine adjacent 2x2x2 patches into one token and expand the channel number to achieve stepwise downsampling and feature dimensionality expansion.

[0043] Subsequently, enter the Video Swin Transformer network,

[0044] First, the batch encoding layer divides it into several uniform 3D patches and maps them to a high-dimensional feature space.

[0045] Then, it sequentially passes through multiple batch fusion layers to achieve spatio-temporal downsampling and channel expansion. Each batch fusion halves the spatial and temporal dimensions of the feature map and doubles the channel number.

[0046] Subsequently, after multiple stages of window displacement, in each stage, window self-attention calculation and moving window self-attention calculation are alternately performed to capture fine-grained spatio-temporal dependencies within local windows and cross-window long-range dependencies.

[0047] First, a one-dimensional Gaussian convolution kernel is used to smooth the time series features, and then the central difference operator is used to calculate the gradient, which significantly reduces noise and improves signal stability. Then, the normalized gradient amplitude is used as an attention prompt and input into the time series fusion attention module to assign explicit weights to different time steps and highlight key frames. Finally, the smoothed gradient and projected RGB features are fused in the attention calculation to achieve more rich spatio-temporal interaction, surpassing the weak fusion strategy of single uniformity loss.

[0048] Further, steps 2-4 are repeated to supervise learning with cross-entropy loss function and real labels to optimize the classification performance of the video understanding network formed by steps 3 and 4.

[0049] The video understanding network formed by steps 3-4 is a Video SwinTransformer network with increased spatiotemporal gradient attention motion enhancement.

[0050] In a second aspect, a system employing the above-mentioned video understanding method based on differential gradient for key frame enhancement includes:

[0051] A behavior recognition dataset construction module is configured to obtain a behavior recognition dataset, including original videos of behavior occurrence, behavior occurrence time, and behavior labels.

[0052] A video preprocessing and tensor reading module is configured to preprocess the original videos of behavior occurrence, and then read tensors and corresponding behavior class labels in parallel through PyTorch Dataset and DataLoader components after preprocessing, wherein the preprocessing includes frame extraction and spatial transformation.

[0053] A spatiotemporal gradient attention motion enhancement module is configured to sequentially perform differential feature extraction and Gaussian gradient enhancement on the preprocessed tensors, and perform spatial information pooling on the preprocessed tensors, and then perform temporal fusion on the results of Gaussian gradient enhancement and spatial information pooling to achieve spatiotemporal gradient attention motion enhancement.

[0054] A video understanding module is configured to sequentially perform 3D batch blocking, continuous four linear encoding, and Video SwinTransformer block processing on the data after spatiotemporal gradient attention motion enhancement, and then perform spatiotemporal average pooling and linear mapping to understand the behaviors contained in the video.

[0055] In a third aspect, a computer readable storage medium stores a computer program, which is invoked by a processor to implement the steps of the above-mentioned video understanding method based on differential gradient for key frame enhancement.

[0056] In a fourth aspect, an electronic terminal includes:

[0057] One or more processors;

[0058] A memory storing one or more computer programs;

[0059] The processor invokes the computer program to implement:

[0060] The steps of the above video understanding method based on key frame enhancement by difference gradient.

[0061] Advantages

[0062] Compared with the prior art, the present application has the following advantages:

[0063] 1. Explicitly using temporal difference gradient information for modeling

[0064] In view of the problem that the prior art video enhancement method does not sufficiently utilize temporal information and is difficult to accurately represent inter-frame dynamic details, the technical scheme of the present application innovatively explicitly introduces inter-frame difference gradient information into the temporal modeling process, thereby effectively capturing the motion changes in the video. The prior art non-explicitly utilizes gradient information, which cannot distinguish between key frames and redundancy, and loses the advantage of gradient information. At the same time, the prior art method of non-displaying inter-frame difference gradient information does not perform smoothing or denoising processing on the gradient signal, which has the problem of causing large gradient noise and being susceptible to interference. The present application uses a Gaussian gradient enhancement module to suppress noise and accurately represent inter-frame dynamic details.

[0065] 2. Global pixel-level spatial modulation of the original data according to attention information.

[0066] The prior art (FiLM) can only generate channel-level modulation parameters, ignoring the differences in spatial and temporal changes, resulting in insufficient key detail enhancement. To solve this problem, the present application proposes a spatio-temporal affine parameter generation module that dynamically generates local "regulation" parameters and global "offset" parameters using difference gradient guided attention maps to globally modulate the original frame pixels, so that the model can adaptively adjust the feature intensity and baseline deviation, thereby enhancing the key regions and details. At the same time, relying only on spatial dimension information cannot locate the time of motion occurrence, and cannot specifically focus on motion features. The present application involves difference gradient information in the generation of modulation parameters, so that it can adaptively adjust the pixel weight according to the motion features.

[0067] The spatio-temporal affine parameter generation module dynamically generates "regulation" and "offset" parameters by fusing spatio-temporal attention weights, which respectively realize feature importance difference and baseline deviation compensation. The "regulation" parameter is responsible for local relative adjustment, highlighting important regions, while the "offset" parameter is responsible for global absolute adjustment, controlling the overall regulation parameter within a reasonable range. In general, the spatial modulation parameter is generated by attention weight guidance, and the motion intensity is dynamically adjusted for different regions, which is in line with the visual cognitive principle at the physical level and is more suitable for processing spatio-temporal inconsistency problems in dynamic scenes than FiLM.

[0068] 3. Global temporal dimension attention calculation on original temporal information and difference gradient information.

[0069] In view of the problem that SIFA only calculates attention in a locally deformable neighborhood and fails to comprehensively capture long-distance inter-frame dependencies, the application realizes global time domain fusion of information by a time fusion attention module that simultaneously associates original frame sequences and differential features in the entire time domain scale.

[0070] The time fusion attention module proposed in the technical solution performs attention operation on complete time sequence information to obtain attention information unique to the time domain dimension. Since the spatial dimension and the time domain dimension organize information in different ways, attention calculation for the time domain dimension alone can obtain more meaningful attention information. Meanwhile, the time domain information of the original input and the differential gradient information are both subjected to Gaussian smoothing to avoid noise influence. Meanwhile, the two are spliced in the channel dimension to serve as value vectors for attention calculation, which can enable the attention mechanism to focus on both "where" important content is and "when" significant changes occur, thereby obtaining richer feature expression and significantly improving the model's capture of sensitivity to time sequence changes and fusion of context information. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a schematic diagram of the spatio-temporal gradient attention motion enhancement module in the technical solution of the application;

[0072] Figure 2 is a schematic diagram of the time fusion attention module in the technical solution of the application;

[0073] Figure 3 is a schematic diagram of the differential feature extraction module in the technical solution of the application;

[0074] Figure 4 is a schematic diagram of the Gaussian gradient enhancement module in the technical solution of the application;

[0075] Figure 5 is a schematic diagram of the spatio-temporal affine parameter generation module in the technical solution of the application;

[0076] Figure 6 is a schematic diagram of the overall network structure in the technical solution of the application. DETAILED DESCRIPTION

[0077] The application will be further described below in conjunction with embodiments.

[0078] Embodiment 1

[0079] A video understanding method based on differential gradient for key frame enhancement, comprising:

[0080] Step 1: Construct a behavior recognition dataset;

[0081] The behavior recognition dataset includes original videos in which behaviors occur and behavior labels;

[0082] The behavior recognition dataset does not make any annotations on the time of behavior occurrence, only paying attention to not letting the time of behavior occurrence all appear in the same position of the video when making the dataset, avoiding the network from learning wrong knowledge;

[0083] For example, if the time of behavior occurrence all appears in the middle position of the video, the network may not learn any time and space information, and roughly considers the middle position of all videos as the time of action occurrence;

[0084] Collect a large number of raw videos from public datasets (such as Kinetics-400, UCF-101, HMDB51) or custom cameras, ensuring to cover diverse action categories. Collect video data of pedestrians in various environments, control the duration of each data to be within the length range of 3-5 seconds, and pay attention to positioning the time of action occurrence at a random position in the single video data. Label each video according to the predefined action label to form (video_path, label) pairs and store them in a database or CSV file.

[0085] Step 2: Preprocess the original video of behavior occurrence, and then read the tensor and corresponding behavior category label in parallel through the Dataset and DataLoader components of PyTorch, which includes frame extraction and spatial transformation;

[0086] The frame extraction in the preprocessing uniformly samples a fixed length of 64 frames from each video;

[0087] The spatial transformation is to center crop or randomly crop each frame of image, and scale the resolution of the cropped image to a uniform size of 224x224.

[0088] Stack into a tensor with shape (B, C, T, H, W), where B is the batch size, C is 3 RGB channels, T is the number of frames, and HxW is the image resolution.

[0089] Finally, use the Dataset and DataLoader components of PyTorch to read the tensor and corresponding video behavior category label in parallel;

[0090] The categories of the video, i.e. vehicles and various behaviors of pedestrians, include normal behaviors such as vehicle driving and stopping, and abnormal behaviors such as vehicle collision; include normal behaviors such as pedestrian walking, and abnormal behaviors such as pedestrian congestion and pedestrian falling.

[0091] Step 3: Spatiotemporal gradient attention motion enhancement, as shown in Figure 1 ;

[0092] The preprocessed tensor is sequentially subjected to differential feature extraction and Gaussian gradient enhancement processing, and the preprocessed tensor is subjected to spatial information pooling processing, and the results after Gaussian gradient enhancement and spatial information pooling processing are subjected to time sequence fusion processing together, so as to realize spatio-temporal gradient attention motion enhancement;

[0093] As shown in Figure 3 , the differential feature extraction processing process is as follows:

[0094] First, the preprocessed tensor is subjected to absolute difference calculation between adjacent frames.

[0095] Then, the preliminary time sequence change feature is extracted through two layers of small kernels, and the inter-frame difference information is obtained as the differential feature.

[0096] The inter-frame difference information is represented by .

[0097] Each layer of small kernels sequentially includes kernel 3D convolution, group normalization and GELU processing, the group normalization refers to GroupNormalization, the channel is divided into several groups and the mean and variance are calculated in each group to perform standardization; GELU is a smooth activation function of Gaussian error linear unit.

[0098] As shown in Figure 4 , the Gaussian gradient enhancement processing process is as follows:

[0099] First, the differential feature is subjected to Gaussian kernel smoothing processing to obtain smoothed inter-frame difference information .

[0100] Then, the gradient is calculated by finite difference, that is, the forward, central and backward difference operators are used to calculate the gradient of the smoothed inter-frame difference information.

[0101] Finally, the gradient of the smoothed inter-frame difference information is normalized by using a normalization function to obtain smoothed inter-frame gradient information .

[0102] As shown in Figure 2 , the processing process of the time sequence fusion is as follows:

[0103] First, the original time domain information is subjected to key generation convolution processing to obtain a key vector for attention calculation; and the smoothed inter-frame gradient information is subjected to query generation convolution processing to obtain a query vector for attention calculation.

[0104] Second, after the original time domain information and the smoothed inter-frame gradient information are spliced in the channel dimension, the value generation convolution processing is performed to obtain a value vector for attention calculation.

[0105] Finally, the query vector is multiplied with the key vector, and then normalized function processing is performed, the result after normalized function processing is multiplied with the value vector, and the time sequence fusion attention weight is obtained. The smooth inter-frame gradient information and the time sequence fusion attention weight are time sequence fused.

[0106] and The dimension of is BxCxTx1x1, and and are stacked after the batch dimension B, to generate the parameter with the dimension of Bx2xCxTx1x1.

[0107] The data after time sequence fusion processing is subjected to spatial-temporal modulation based on dynamic parameters, as shown in Figure 5 , and the specific process is as follows:

[0108] First, the time sequence fusion feature is input into a parameter generator, and the parameter generator sequentially includes 3D convolution, group normalization, GELU and 3D convolution processing;

[0109] Then, the data output by the parameter generator is averaged and split in the dimension, to obtain two original parameters with the dimension of , and then the “adjustment” parameter and the “offset” parameter are obtained through normalization functions respectively;

[0110] Finally, the “adjustment” parameter and the “offset” parameter are used to perform spatial-temporal modulation on the preprocessed tensor, and the data after spatial-temporal modulation is taken as the data after spatial-temporal gradient attention motion enhancement processing.

[0111] The spatial-temporal modulation formula is as follows:

[0112] ,

[0113] wherein, represents the data after spatial-temporal modulation, and represents the preprocessed tensor.

[0114] The spatial-temporal modulation based on dynamic parameters regards the video information as a stack of many frames in the time domain direction. Before modulation, the parameters in the time domain direction should be relatively average, that is, the influence of each frame on the final output is similar. After spatial-temporal modulation, in the time domain direction, the parameters of the frames near the action occurrence moment are amplified, and the influence on the output is greater.

[0115] The Video Swin Transformer is an existing approach. It's a pure Transformer backbone network designed specifically for video understanding. This model uses a shifted window strategy in the spatiotemporal dimensions to divide the video into several fixed-size 3D patch windows. Self-attention is calculated only within these windows, and the window positions are shifted alternately at different levels to balance local fine-grained features with long-range dependencies across windows. After every few attention blocks, the model uses hierarchical patch merging to merge adjacent 2×2×2 patches into a single token. The number of channels is then increased to achieve gradual downsampling and feature upscaling.

[0116] Then enter the Video Swin Transformer network:

[0117] First, the batch encoding layer divides it into several 3D patches of uniform size and maps them into a high-dimensional feature space;

[0118] Then, spatiotemporal downsampling and channel amplification are achieved through multiple layers of batch fusion. Each batch fusion halves the spatial and temporal dimensions of the feature map and doubles the number of channels.

[0119] This is followed by multiple stages of window shifting, in which window self-attention calculation and moving window self-attention calculation are performed alternately to capture fine-grained spatiotemporal dependencies within the local window and long-range dependencies across windows.

[0120] Step 4: The data after spatiotemporal gradient attention motion enhancement is sequentially processed through 3D batch blocks, four consecutive linear encoding and Video Swin Transformer series blocks, and then spatiotemporal average pooling and linear mapping to obtain the action category prediction contained in the video and achieve understanding of the behavior contained in the video.

[0121] Repeat steps 2 to 4, and use the cross entropy loss function and the true label for supervised learning to optimize the Figure 6 Classification performance of the video understanding network shown.

[0122] The video understanding network formed in step 3 and step 4 is a Video SwinTransformer network with added spatiotemporal gradient attention motion enhancement.

[0123] Traditional inter-frame difference methods usually only calculate the simple difference of neighboring frame pixels or features, often resulting in binary or noisy motion masks, lacking continuous quantization of motion intensity, making it difficult to distinguish weak motion from background changes in complex scenes. In contrast, the technical solution of the present application first performs Gaussian smoothing on the frame difference features and calculates the central difference gradient, then performs softmax normalization on the absolute gradient value, generating a delicate motion intensity distribution that can more accurately capture the subtle changes between different frames. After using this gradient distribution as the Query in the attention mechanism, the model can dynamically allocate attention weights according to the motion intensity, effectively highlighting sudden motion events and suppressing meaningless background changes, thereby achieving better performance in temporal consistency and spatial target positioning compared to relying only on binary difference or traditional optical flow input

[0124] In traditional temporal self-attention mechanisms, Query, Key and Value all come from the same original temporal features, and the attention distribution tends to be evenly distributed, making it difficult to actively focus on motion-rich frames, thereby diluting the attention to key action information. Our method uses normalized gradient features as Query to "ask" where the motion is; uses original temporal features as Key to provide global context positioning; and concatenates the two to generate Value through convolution, making the output have both static semantics and dynamic cues. This Q / K / V separation design makes the attention matrix more "sharp" in the time dimension, allowing higher weights to be concentrated on motion frames.

[0125] The technical solution of the present application outputs scale and shift parameters with the same resolution as the feature map through the parameter generation network, to perform pixel-level affine transformation on the features, so as to amplify or shift the features at each position according to the motion-driven attention weights. This gradient attention-based pixel-level modulation can automatically guide the network to concentrate more computing resources on the positions where motion occurs, effectively suppressing the interference of static backgrounds, and is more lightweight and end-to-end trainable than motion-guided methods that rely on additional optical flow or region detection.

[0126] Embodiment 2

[0127] A system employing the above-mentioned video understanding method based on differential gradient for key frame enhancement, comprising:

[0128] A behavior recognition dataset construction module is configured to obtain a behavior recognition dataset, including original videos of behavior occurrence, behavior occurrence time, and behavior labels.

[0129] Video preprocessing and tensor reading module: the original video of the behavior occurs is preprocessed, and the preprocessed video is read in parallel with the tensor and the corresponding behavior category label through the Dataset and DataLoader components of PyTorch, and the preprocessing includes frame extraction and spatial transformation;

[0130] Spacetime gradient attention motion enhancement module: for sequentially performing differential feature extraction and Gaussian gradient enhancement processing on the preprocessed tensor, and simultaneously performing spatial information pooling processing on the preprocessed tensor, and performing temporal fusion processing on the results after Gaussian gradient enhancement and spatial information pooling processing, to realize spacetime gradient attention motion enhancement;

[0131] Video understanding module: for sequentially performing 3D batch blocking, continuous four linear coding and Video Swin Transformer block processing on the data after spacetime gradient attention motion enhancement processing, and then performing spacetime average pooling and linear mapping processing, to realize understanding of the behavior contained in the video.

[0132] It should also be understood that the specific implementation process of each module is referred to the above method content, and the present application will not be repeated here, and the division of the above functional modules is only for example, in some embodiments, part of the functional modules can be combined, part of the functional modules can be split, and each functional module can be implemented in software, hardware or combination of software and hardware, wherein the software and hardware devices include but are not limited to general-purpose computer devices, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0133] Embodiment 3

[0134] A computer readable storage medium stores a computer program, the computer program is called by a processor to realize: the steps of the above-mentioned video understanding method based on differential gradient key frame enhancement.

[0135] The specific implementation process of each step is referred to the foregoing method.

[0136] The readable storage medium is a computer readable storage medium, which can be an internal storage unit of the software and hardware device in any of the foregoing embodiments, for example, a hard disk or a memory of the controller. The readable storage medium can also be an external storage device of the controller, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card and the like equipped on the controller. Further, the readable storage medium can also include both the internal storage unit and the external storage device of the controller. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0137] Embodiment 4

[0138] An electronic terminal comprising:

[0139] one or more processors;

[0140] a memory storing one or more computer programs;

[0141] wherein the processor invokes the computer program to implement:

[0142] the steps of the above video understanding method based on key frame enhancement by difference gradient.

[0143] The specific implementation process of each step can refer to the description of the foregoing method.

[0144] It should be understood that, in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The memory can include read-only memory and random access memory, and provide instructions and data for the processor. A part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0145] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0146] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is a flow chart according to the method, device (system), and computer program product of the embodiment of the present application and / or the instructions executed by the processor to generate a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product comprising an instruction device, which realizes the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0147] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A video understanding method for key frame enhancement based on differential gradient, characterized in that: include: Step 1: Build an action recognition dataset; The behavior recognition dataset includes the original video of the behavior and the behavior labels; Step 2: Preprocess the original video of the behavior. Then, use PyTorch's Dataset and DataLoader components to read tensors and corresponding behavior category labels in parallel. The preprocessing includes frame extraction and spatial transformation. Step 3: Spatiotemporal gradient attention motion enhancement; The preprocessed tensor is sequentially subjected to differential feature extraction and Gaussian gradient enhancement processing, and the preprocessed tensor is subjected to spatial information pooling processing. The results of Gaussian gradient enhancement and spatial information pooling processing are then subjected to temporal fusion processing to achieve spatiotemporal gradient attention motion enhancement; Step 4: The data after spatiotemporal gradient attention motion enhancement is sequentially processed into 3D batch blocks, four consecutive linear encoding and Video Swin Transformer series blocks, and then spatiotemporal average pooling and linear mapping are performed to understand the behavior contained in the video.

2. The method according to claim 1, characterized in that In the pre-processing, frame extraction is to uniformly sample a frame sequence with a fixed length of 64 frames from each video segment; The spatial transformation is to perform center cropping or random cropping on each frame image and scale the cropped image resolution to a uniform size of 224×224.

3. The method according to claim 1, characterized in that The differential feature extraction process is as follows: First, the preprocessed tensor Calculate the absolute difference between adjacent frames; Then, the preliminary temporal change features are extracted through two layers of small kernels, and the inter-frame difference information is obtained as the differential feature; Inter-frame differential information uses express; Among them, each layer of small kernels includes kernel 3D convolution, group normalization and GELU processing in sequence. Group normalization refers to GroupNormalization, which divides the channels into several groups and calculates the mean and variance within each group for standardization; GELU is a Gaussian error linear unit of a smooth activation function.

4. The method according to claim 3, characterized in that The Gaussian gradient enhancement process is as follows: First, the differential features are smoothed by Gaussian kernel to obtain smooth inter-frame differential information. ; Next, the gradient is calculated by finite difference, that is, the gradient of the smoothed inter-frame difference information is calculated using the forward, center, and backward difference operators; Finally, the normalization function is used to normalize the gradient of the smooth inter-frame difference information to obtain the smooth inter-frame gradient information .

5. The method according to claim 1, wherein The processing process of the time series fusion is as follows: First, the original time domain information is processed by key generation convolution to obtain the key vector for attention calculation; and the smoothed inter-frame gradient information is processed by query generation convolution to obtain the query vector for attention calculation; Secondly, the original time domain information and the smoothed inter-frame gradient information are concatenated in the channel dimension and then subjected to value generation convolution processing to obtain the value vector used for attention calculation; Finally, the query vector is multiplied by the key vector and then processed by the normalization function. The result after normalization is multiplied by the value vector to obtain the temporal fusion attention weight, and the smooth inter-frame gradient information and the temporal fusion attention weight are temporally fused.

6. The method according to claim 5, characterized in that The data after time series fusion processing is then subjected to spatiotemporal modulation based on dynamic parameters. The specific process is as follows: First, the temporal fusion features are input into the parameter generator, which includes 3D convolution, group normalization, GELU and 3D convolution processing in sequence; Next, the data output by the parameter generator Perform an average split on the dimension, and get two dimensions as The original parameters are then normalized to obtain the "adjustment" parameters With the "offset" parameter ; Finally, use the "tuning" parameter With the "offset" parameter Perform spatiotemporal modulation on the preprocessed tensor, and use the spatiotemporal modulated data as the data processed by spatiotemporal gradient attention motion enhancement; The spatiotemporal modulation formula is as follows: ， in, Represents the data after time-space modulation, Represents a preprocessed tensor.

7. The method according to any one of claims 1 to 6, characterized in that Repeat steps 2 to 4 and use the cross entropy loss function with the true labels for supervised learning to optimize the classification performance of the video understanding network formed in steps 3 and 4.

8. A system using the video understanding method for key frame enhancement based on differential gradient according to any one of claims 1 to 7, characterized in that: include: Behavior recognition dataset construction module: used to obtain behavior recognition datasets, including original videos of behaviors, behavior occurrence time, and behavior labels; Video preprocessing and tensor reading module: Preprocesses the original video of the behavior, and then uses the PyTorch Dataset and DataLoader components to read the tensor and corresponding behavior category label of the preprocessed video in parallel. The preprocessing includes frame extraction and spatial transformation. Spatiotemporal gradient attention motion enhancement module: It is used to perform differential feature extraction and Gaussian gradient enhancement on the preprocessed tensor in sequence, and to perform spatial information pooling on the preprocessed tensor. The results of Gaussian gradient enhancement and spatial information pooling are then temporally fused together to achieve spatiotemporal gradient attention motion enhancement. Video Understanding Module: This module is used to sequentially process the data processed by spatiotemporal gradient attention motion enhancement through 3D batch blocks, four consecutive linear encoding and Video Swin Transformer blocks, and then perform spatiotemporal average pooling and linear mapping to understand the behaviors contained in the video.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program is called by a processor to implement the steps of the method according to any one of claims 1 to 7.

10. An electronic terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls a computer program to implement: The steps of the method according to any one of claims 1 to 7.

Images