A video behavior recognition method, apparatus, device, and storage medium

Abstract

This application discloses a video behavior recognition method, apparatus, device, and storage medium, relating to the field of video recognition technology. The method includes: processing video frames of a training video using a first language model to obtain behavior description information corresponding to each video frame; processing the behavior description information corresponding to a single action category using a second language model to obtain text prompt information corresponding to each action category; training a model based on the text prompt information and visual feature information corresponding to the training video; and using the trained model to recognize the video to be recognized to obtain the corresponding video behavior recognition result. As can be seen, this application can expand concise behavior description information into semantically rich text prompt information and train a model, improving the quality of cross-modal alignment, reducing semantic ambiguity, and obtaining a model with better understanding and recognition performance of video behavior, ultimately improving the video behavior recognition effect.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and in particular to a video behavior recognition method, apparatus, device, and storage medium. Background Technology

[0002] Large-scale contrastive vision-language pre-trained models (such as Contrastive Language-Image Pre-training, CLIP, a multimodal pre-trained model) have demonstrated powerful zero-shot recognition capabilities by aligning images and text in a shared embedding space. However, due to the inherent differences between static images and dynamic video data, directly transferring such image-text models to video action recognition tasks remains challenging. Current adaptation methods often use only short class labels as text supervision, leading to semantic sparsity and limiting the accuracy of cross-modal alignment and the model's generalization ability.

[0003] It is evident that improving the recognition performance of video behavior is a problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a video behavior recognition method, apparatus, device, and storage medium, which can expand concise behavior description information into semantically rich text prompts and perform model training, improving the quality of cross-modal alignment, reducing semantic ambiguity, and obtaining a model with better understanding and recognition performance of video behavior, ultimately improving the recognition effect of video behavior. The specific solution is as follows:

[0005] Firstly, this application provides a video behavior recognition method, including:

[0006] The video frames of the training video are processed using a first language model to obtain behavioral description information corresponding to each video frame;

[0007] The behavior description information corresponding to each individual action category is processed using a second language model to obtain the corresponding text prompt information for each action category.

[0008] The model is trained based on the text prompts and the visual features of the training videos, and the trained model is used to identify the videos to be identified to obtain the corresponding video behavior recognition results.

[0009] Optionally, the step of processing the video frames of the training video using a first language model to obtain behavioral description information corresponding to each video frame includes:

[0010] Each training video is processed according to a preset frame extraction algorithm to obtain a video frame sequence corresponding to each training video; the preset frame extraction algorithm is a frame extraction algorithm based on uniform sampling or based on content change rate.

[0011] The video frames in each video frame sequence are processed using a first language model to obtain behavioral description information for the corresponding video frames.

[0012] Optionally, the step of processing the behavior description information corresponding to a single action category using a second language model to obtain the corresponding text prompt information for each action category includes:

[0013] The behavior description information is grouped and aggregated according to the action category to obtain the aggregated description information corresponding to different action categories;

[0014] The aggregated description information is processed using a second language model to obtain text prompts corresponding to each action category.

[0015] Optionally, the process of extracting the visual feature information includes:

[0016] Local region features of the training video are extracted based on temporal convolution, and dynamic region features of the training video are extracted based on temporal difference and global second-order pooling techniques.

[0017] The local region features, the dynamic region features, and the original features are fused by residual connection to obtain the visual feature information of the training video; the original features are the features obtained by embedding video frames of the training video into image blocks.

[0018] Optionally, the extraction of dynamic region features from the training video based on temporal difference and global second-order pooling techniques includes:

[0019] Initial dynamic region features are obtained by extracting features from the original features based on temporal difference.

[0020] Based on the covariance matrix of the initial dynamic region features, channel attention weights are generated using a preset neural network; the preset neural network is a neural network constructed based on grouped convolution, batch normalization, and nonlinear activation functions.

[0021] The initial dynamic region features are recalibrated using the channel attention weights, and the recalibrated features and the original features are fused using residual connections to obtain the target dynamic region features.

[0022] Optionally, the recalibration of the initial dynamic region features using the channel attention weights includes:

[0023] By using dot product, the channel attention weights are used to enhance or suppress different feature channels of the initial dynamic region features to obtain the corresponding recalibrated features.

[0024] Optionally, the step of training the model based on the text prompts and the visual feature information corresponding to the training videos includes:

[0025] Based on a preset loss function, the model is trained using the text prompts and the visual features of the training videos.

[0026] The preset loss function is a loss function constructed based on the contrastive learning loss function, the similarity classification loss function, the mask loss function, the action classification loss function, and the corresponding weight coefficients.

[0027] Secondly, this application provides a video behavior recognition device, comprising:

[0028] The first information processing module is used to process the video frames of the training video through the first language model to obtain the behavior description information corresponding to each video frame;

[0029] The second information processing module is used to process the behavior description information corresponding to each single action category using a second language model to obtain the text prompt information corresponding to each action category.

[0030] The model training module is used to train the model based on the text prompts and the visual feature information corresponding to the training video.

[0031] The video behavior recognition module is used to identify the video to be recognized using the trained model and obtain the corresponding video behavior recognition results.

[0032] Thirdly, this application provides an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor for executing the computer program to implement the video behavior recognition method as described above.

[0035] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the video behavior recognition method described above.

[0036] Therefore, this application first processes the video frames of the training video using a first language model to obtain behavioral description information corresponding to each video frame; then, it uses a second language model to process the behavioral description information corresponding to each individual action category to obtain text prompt information corresponding to each action category; subsequently, it trains a model based on the text prompt information and the visual feature information corresponding to the training video, and uses the trained model to identify the video to be recognized to obtain the corresponding video behavior recognition result. In this way, this application can expand concise behavioral description information into semantically rich text prompt information and train a model, improving the quality of cross-modal alignment, reducing semantic ambiguity, and obtaining a model with better understanding and recognition performance of video behavior, ultimately improving the video behavior recognition effect. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart of a video behavior recognition method disclosed in this application;

[0039] Figure 2 This is a flowchart of a specific video behavior recognition model training method disclosed in this application;

[0040] Figure 3 This is a schematic diagram of a specific global second-order pooling module structure disclosed in this application;

[0041] Figure 4 This is a schematic diagram of a specific high-order adaptor module structure disclosed in this application;

[0042] Figure 5 Here is a flowchart of a specific semantic prompt information generation method disclosed in this application;

[0043] Figure 6 This is a schematic diagram of the structure of a video behavior recognition device disclosed in this application;

[0044] Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] See Figure 1 As shown, this embodiment of the invention discloses a video behavior recognition method, including:

[0047] Step S11: Process the video frames of the training video using the first language model to obtain the behavioral description information corresponding to each video frame.

[0048] In this application, visual information is first extracted from the video and converted into text descriptions. This process uses short category labels to describe the visual information. Specifically, the video frames of the training video are processed using a first language model. This generates natural language descriptions corresponding to the video frames, i.e., behavioral description information. Further, in a specific embodiment, the process of processing the video frames of the training video using the first language model to obtain behavioral description information corresponding to each video frame may include: processing each training video according to a preset frame extraction algorithm to obtain a sequence of video frames corresponding to each training video; the preset frame extraction algorithm is a frame extraction algorithm based on uniform sampling or content change rate; and processing the video frames in each video frame sequence using the first language model to obtain the behavioral description information of the corresponding video frames. Specifically, for each training video, video frames can be extracted from the training video using a frame extraction algorithm to obtain the corresponding video frame sequence; the frame extraction algorithm here can be a uniform sampling algorithm or an algorithm based on content change rate, extracting one or more representative images from the video. Then, for the extracted video frame sequence, the corresponding natural language description can be generated through the first language model, that is, the behavioral description information of the video frame is obtained; the behavioral description information obtained here includes rich visual semantic information, such as the subject of the action, scene context, object interaction relationship, etc.

[0049] Step S12: Use a second language model to process the behavior description information corresponding to each action category to obtain the text prompt information corresponding to each action category.

[0050] In this embodiment, the behavioral description information of the video frames corresponding to the training video can be obtained through the above steps. To further improve the richness and accuracy of the text description, the behavioral description information is processed according to the action category corresponding to the video frame to generate high-quality, high-coverage description information corresponding to the action category, i.e., text prompt information. It can be understood that the behavioral description information of the same action category can be grouped and aggregated through the above steps. The high-quality, high-coverage text prompt information obtained in this way corresponds to the specific action category and can be used as an information-rich text supervision signal for subsequent cross-modal alignment training.

[0051] In a specific embodiment, processing the behavioral description information corresponding to a single action category using a second language model to obtain text prompt information corresponding to each action category may include: grouping and aggregating the behavioral description information according to the action category to obtain aggregated description information corresponding to different action categories; and processing each aggregated description information separately using the second language model to obtain text prompt information corresponding to each action category. Specifically, to enrich visual semantic information, the behavioral description information can be grouped and aggregated according to the different action categories corresponding to video frames, thus forming a description set corresponding to each action category, i.e., aggregated description information. Further, the aggregated description information can be processed by the second language model to abstract, summarize, and denoise the commonalities and core semantics of all descriptions, ultimately generating high-quality, high-coverage text prompt information corresponding to the action category.

[0052] Step S13: Train the model based on the text prompts and the visual feature information corresponding to the training video, and use the trained model to identify the video to be identified to obtain the corresponding video behavior recognition result.

[0053] In this embodiment, the text prompts corresponding to the training video can be obtained through the above steps. These prompts are generated based on the behavioral descriptions of specific action categories and represent the semantic essence of the entire action category. Then, based on these text prompts and the visual feature information of the training video, the model is trained to obtain a trained model that can be used for video behavior recognition. Furthermore, for the video to be recognized, the trained model can be directly used to perform video behavior recognition to obtain the corresponding video behavior recognition results.

[0054] In one specific embodiment, the process of extracting visual feature information may include: extracting local region features of the training video based on temporal convolution, and extracting dynamic region features of the training video based on temporal differencing and global second-order pooling techniques; fusing the local region features, the dynamic region features, and the original features through residual connections to obtain the visual feature information of the training video; the original features are features obtained by embedding image blocks from the video frames of the training video. Specifically, a dual-branch structure can be used to extract visual features, capturing two key types of information in the video in parallel and complementaryly: local, fine-grained motion patterns and global, structured semantic dependencies. First, to capture local semantic changes within frames and short-term dynamic information between frames, temporal convolution can be performed on the training video. The specific model structure can be two linear projection layers and a temporal convolution between them; this extracts local change trends in the video in a lightweight way, i.e., extracting local region features of the training video. Correspondingly, to capture higher-order, more global semantic structures, temporal differencing and global second-order pooling techniques are needed to extract visual features. Furthermore, temporal difference can extract motion information from the training video, highlighting dynamic regions within the video; global second-order pooling can capture global statistical dependencies between feature channels; thus, global structural information can be used to guide and enhance the original features (the original visual features of the training video) to obtain the corresponding dynamic region features. Going further, the output of the dual-branch structure is finally fused with the original features of the training video through residual addition to obtain the visual feature information of the training video.

[0055] In another specific embodiment, extracting dynamic region features from the training video based on temporal difference and global second-order pooling techniques may include: extracting initial dynamic region features from the original features using temporal difference; generating channel attention weights using a preset neural network based on the covariance matrix of the initial dynamic region features; the preset neural network being a neural network constructed based on grouped convolution, batch normalization, and nonlinear activation functions; recalibrating the initial dynamic region features using the channel attention weights, and fusing the recalibrated features and the original features through residual connections to obtain target dynamic region features. Specifically, in the process of extracting dynamic region features, initial dynamic region features are first extracted using temporal difference to highlight dynamic regions in the video. Then, global second-order pooling is used to model the high-order relationships within the initial dynamic region features. First, the covariance matrix of the initial dynamic region features is calculated to quantify the statistical correlation of different feature channels across the entire spatial dimension. Useful nonlinear relationships can be learned from the covariance matrix. A neural network based on grouped convolution, batch normalization, and nonlinear activation functions can be used to capture the structural patterns within the covariance matrix, ultimately generating channel force weights for the feature channels. The learned channel attention weights can then be used for feature recalibration of the initial dynamic region features. Specifically, this can be achieved through dot products, using the channel attention weights to enhance or suppress different feature channels of the initial dynamic region features, resulting in the corresponding recalibrated features. To ensure the stability of the information flow and training convergence, the recalibrated features and the original features of the training video can be fused through residual connections to obtain the final output target dynamic region features. In a specific embodiment, an additional projection layer can be added before the residual connections to match the dimension.

[0056] In another specific embodiment, model training based on the text prompts and the visual feature information corresponding to the training videos can include: training the model using the text prompts and the visual feature information corresponding to the training videos based on a preset loss function; wherein the preset loss function is a loss function constructed based on a contrastive learning loss function, a similarity classification loss function, a masking loss function, an action classification loss function, and corresponding weight coefficients. Specifically, the model can be optimized using a multi-task learning paradigm, involving multimodal contrastive learning, cross-modal classification, cross-modal masked language modeling, and visual classification, corresponding to four loss functions: contrastive learning loss function, similarity classification loss function, masking loss function, and action classification loss function; and the four loss functions have corresponding weight coefficients, and the weighted sum is the total loss.

[0057] Therefore, this application can expand concise behavioral descriptions into semantically rich textual prompts and train the model accordingly. This enhances textual representation capabilities, enabling the learning of features that are more relevant to the behavior itself and decoupled from the background context. Combining textual prompts and visual features can improve the quality of cross-modal alignment, reduce semantic ambiguity, and result in a model with better understanding and recognition performance of video behaviors, ultimately improving the recognition effect of video behaviors.

[0058] like Figure 2 As shown, this embodiment discloses a training process for a video behavior recognition model. The model framework is based on a frozen backbone network, including a text encoder and a video encoder. A visually driven semantic prompt generation module, a global second-order pooling module, and a higher-order adaptor are used to specifically enhance the two encoders. The outputs of the two encoders are fed into a multi-task decoder for joint training and optimization to obtain the final video behavior recognition model. Specifically, it includes:

[0059] It should be noted that, to effectively address the problem of insufficient ability of traditional visual encoders to capture high-order spatiotemporal structural information when processing video sequences, this application introduces two lightweight and efficient structured modeling modules on the basis of the frozen backbone network. These two modules work together to enhance the expressive power of visual features from different dimensions and levels. They are the Global Second-order Pooling (GSoP) module, which is responsible for capturing intra-frame second-order statistical features, and the High-Order Adapter module, which is responsible for fusing local dynamics and global structure at a higher semantic level.

[0060] like Figure 3 As shown, the core idea of ​​the Global Second-Order Pooling (GSoP) module is to move beyond the first-order statistics (such as mean pooling) relied upon by traditional methods and instead mine the second-order statistics between features, namely covariance. By explicitly modeling the correlation between feature channels, this module can capture more refined and global information about the internal structure of the image. In this application, the GSoP module is strategically placed before each Transformer block in the video encoder as a pre-feature enhancement unit to improve the ability of subsequent self-attention layers to perceive structured information. Specifically, for a single input video frame, the feature map (i.e., the original features) obtained after patch embedding is as follows:

[0061] ;

[0062] Where R refers to the set of real numbers, N is the number of image patches, and C is the number of feature channels.

[0063] Furthermore, feature dimensionality reduction and preparation are performed first. To improve computational efficiency while ensuring model performance, a 1×1 convolutional layer is first used to linearly project the input feature X, achieving channel dimensionality reduction. This operation not only reduces the complexity of subsequent calculations but also provides a certain possibility for feature decoupling.

[0064] ;

[0065] in, It is a temporal convolution that outputs features. ,and .

[0066] Then, the global covariance matrix is ​​calculated; this step is the core of the GSoP module, used to calculate the inter-channel covariance matrix M of the dimensionality-reduced feature Q. This matrix quantifies the statistical correlation of different feature channels across the entire spatial dimension. The calculation formula is as follows:

[0067] ;

[0068] in, It is the identity matrix. It is a vector of all 1s, expressed as follows:

[0069] ;

[0070] Essentially, it's a centered matrix that performs zero-mean normalization on the features before calculating the covariance. The final covariance matrix is... It is a symmetric positive semi-definite matrix, whose diagonal elements represent the variance of each channel feature and the off-diagonal elements represent the covariance between different channel features.

[0071] Then, channel attention generation is performed; to learn useful nonlinear relationships from the covariance matrix M and generate attention weights to guide feature enhancement, a lightweight neural network can be used to process M. This network is typically composed of cascaded grouped convolutions (g-Conv), batch normalization (BN), and nonlinear activation functions (such as sigmoid), designed to efficiently capture the structural patterns within the covariance matrix.

[0072] ;

[0073] in, This represents the Sigmoid activation function, which maps the output to the (0, 1) interval, making it usable as attention weights. The final generated channel attention weight vector... .

[0074] Next, dynamic feature recalibration is performed; specifically, the learned channel attention weights A are applied to the dimensionality-reduced features Q, and different feature channels are dynamically enhanced or suppressed through dot products. This allows the model to adaptively focus on feature channels that are more informative for recognizing the current behavior.

[0075] ;

[0076] in, This represents the characteristics after recalibration.

[0077] Finally, residual connections are performed and output is generated; to ensure the stability of the information flow and the convergence of training, the recalibrated enhanced features are... The enhancement is performed by fusing the original feature X with a residual connection. This operation allows modules to "skip" the enhancement process when needed, directly passing the original information, ensuring the flexibility and robustness of module insertion. Typically, an additional projection layer is required to match the dimensions before addition.

[0078] ;

[0079] Among them, the final output This is an enhanced feature that incorporates second-order statistical information, which will be fed into the subsequent Transformer encoding layer for processing. This represents an additional projection layer used for matching dimensions.

[0080] like Figure 4 As shown, higher-order adaptors are parameter-efficient fine-tuning techniques designed to adapt the knowledge of pre-trained models to downstream video tasks. Higher-order adaptors employ a unique dual-branch structure, inserted after each Transformer coding layer. Their design philosophy is to capture two key pieces of information in the video in parallel and complementary ways: local, fine-grained motion patterns and global, structured semantic dependencies.

[0081] Specifically, one branch of the dual-branch structure is a local modeling branch; this branch focuses on capturing local semantic changes within a frame and short-term dynamic information between frames. Its structure typically consists of two linear projection layers (i.e., fully connected layers) and a temporal convolution between them. This branch extracts local change trends in the video in a lightweight manner.

[0082] ;

[0083] in, It's upsampling. It is downsampling. This represents the information captured by the local modeling branch.

[0084] The other branch of the dual-branch structure is the structural modeling branch; this branch aims to capture higher-order, more global semantic structures. Its internal operations are more complex, specifically including temporal differencing and higher-order relation modeling. Temporal differencing performs a difference operation on the multi-frame features X input to the higher-order adaptor in the temporal dimension to explicitly extract motion information and highlight dynamic regions in the video. Correspondingly, higher-order relation modeling feeds the extracted dynamic features into a built-in GSoP module to perform higher-order relation modeling on the internal structure of the motion features, capturing global statistical dependencies between channels and generating an attention vector. Further, this is multiplicatively applied to the original features X that have not undergone temporal differencing, thereby guiding and enhancing the original features using global structural information.

[0085] ;

[0086] in, The information captured by the structural modeling branch is represented by TemporalDiff, which refers to the temporal-diffference learning algorithm.

[0087] Finally, feature fusion and the final output are performed; the outputs of the two parallel branches are fused with the original feature X from the backbone network by adding the residuals. This fusion method enables the model to synergistically utilize the general features extracted by the backbone network, the fine-grained dynamics captured by the local modeling branches, and the global semantic structure captured by the structural modeling branches.

[0088] ;

[0089] Final output It is a feature representation enhanced with multi-dimensional information, which serves as the final output of the entire visual encoder.

[0090] like Figure 5 As shown, the visual-driven semantic cue generation strategy aims to address the semantic sparsity problem caused by using only short category labels in traditional methods. It enhances the supervision signal by automatically generating rich and accurate text descriptions from visual content. The overall process is a semantic refinement from concrete to abstract: first, visual information is extracted from a single video instance and transformed into a text description; then, semantic aggregation and enhancement are performed across multiple instances of the same category, ultimately forming a text cue that represents the semantic essence of the entire category. The advantage of this method is that the generated text cue not only contains more discriminative action information but also incorporates rich scene context, providing the model with stronger semantic constraints and better cross-modal alignment anchors. The specific steps are as follows: First, extract the keyframes of the video. For each video in the training set (training videos)... By using uniform sampling or content-rate-of-change-based keyframe extraction algorithms, one or more of the most representative keyframe images can be extracted from a video sequence. To ensure a balance between computational efficiency and representativeness, a frame is typically extracted from the middle of the video. A description is then generated using a visual language model; specifically, this involves using the extracted keyframes... The input is fed into a pre-trained vision-language model (i.e., a first language model) to generate a natural language description. (i.e., behavioral description information). This process can be formally represented as:

[0091] ;

[0092] in, This refers to the Visual Language Model (VLM), where N is the total number of training videos. The generated descriptions... It contains rich visual semantic information, such as the subject of the action, the scene context, and the interaction relationships between objects.

[0093] Furthermore, regarding the generated description Grouping and aggregating by action category. Based on the video. Action category tag Describe all videos Group the data to form a description set corresponding to each action category c.

[0094] Then, a text prompt is generated using a large language model (i.e., a second language model). Specifically, the description set for each category c is input into a large language model. By designing a specific prompt, the large language model is instructed to abstract, summarize, and denoise the commonalities and core semantics of all descriptions for that category, generating a high-quality, high-coverage category-level semantic prompt (i.e., text prompt information). This process can be represented as:

[0095] ;

[0096] Where C represents the total number of categories, and LLM stands for Large Language Model. The generated T will serve as a new, information-rich text supervision signal for that category, used for subsequent cross-modal alignment training.

[0097] Finally, multi-task training and optimization are performed. Specifically, the model can be optimized using a multi-task learning paradigm, updating only the adaptor parameters. The decoder contains four task branches, and the four loss functions are jointly optimized using a multi-task decoder. Specifically, this includes: multimodal contrastive learning: symmetric InfoNCE loss (Information Noise-Contrastive Estimation Loss, a loss function based on noise contrast estimation) can be used to bring matching video-text pairs closer in the embedding space and push away mismatched video-text pairs.

[0098] Cross-modal classification: Similarity is calculated between video features and text prompt features of all categories in the dataset, transforming it into a classification problem, and optimized using cross-entropy loss; Cross-modal masked language modeling: Some words are randomly masked in the text prompts, and video features are used as context to guide the model to recover the masked words; Visual classification: A linear classifier is added on top of the video features to directly classify actions. The corresponding total model training loss is the weighted sum of the above four losses.

[0099] As can be seen, by introducing a global second-order pooling module, this embodiment can capture the covariance of the feature map, i.e., second-order statistical information, thereby better modeling the spatial structure and channel dependencies in the video and improving the model's ability to understand complex scenes. Furthermore, the vision-driven semantic cueing strategy expands sparse category labels into rich descriptions containing action details, scene and object interactions, providing the model with more accurate semantic anchors, effectively improving the quality of cross-modal alignment and reducing semantic ambiguity during the learning process. Further, this embodiment only requires training lightweight adaptor modules (GSoP and higher-order adaptors) without fine-tuning the massive backbone network, greatly reducing computational and storage overhead and making it suitable for resource-constrained scenarios. Overall, this application, by synergistically enhancing the representation capabilities of both visual and textual modalities, can learn features that are more relevant to the behavior itself and decoupled from the background context, achieving higher behavior recognition performance on video behavior recognition datasets and demonstrating superior generalization ability.

[0100] like Figure 6 As shown, this embodiment discloses a video behavior recognition device, including:

[0101] The first information processing module 11 is used to process the video frames of the training video through the first language model to obtain the behavior description information corresponding to each video frame;

[0102] The second information processing module 12 is used to process the behavior description information corresponding to a single action category using a second language model to obtain the text prompt information corresponding to each action category.

[0103] Model training module 13 is used to train the model based on the text prompt information and the visual feature information corresponding to the training video;

[0104] The video behavior recognition module 14 is used to recognize the video to be recognized using the trained model to obtain the corresponding video behavior recognition results.

[0105] Therefore, this application can expand concise behavioral description information into semantically rich text prompts and train the model, thereby improving the quality of cross-modal alignment, reducing semantic ambiguity, and obtaining a model with better understanding and recognition performance of video behavior, ultimately improving the video behavior recognition effect.

[0106] In one specific embodiment, the first information processing module 11 may include:

[0107] A frame extraction unit is used to process each training video according to a preset frame extraction algorithm to obtain a video frame sequence corresponding to each training video; the preset frame extraction algorithm is a frame extraction algorithm based on uniform sampling or based on content change rate.

[0108] The first information processing unit is used to process the video frames in each video frame sequence using a first language model to obtain behavioral description information of the corresponding video frames.

[0109] In one specific embodiment, the second information processing module 12 may include:

[0110] The grouping and aggregation unit is used to group and aggregate the behavior description information according to the action category to obtain the aggregated description information corresponding to different action categories;

[0111] The second information processing unit is used to process each of the aggregated description information using a second language model to obtain text prompt information corresponding to each of the action categories.

[0112] In one specific embodiment, the device may include:

[0113] The feature extraction module is used to extract local region features of the training video based on temporal convolution, and to extract dynamic region features of the training video based on temporal difference and global second-order pooling techniques.

[0114] The feature fusion module is used to fuse the local region features, the dynamic region features, and the original features through residual connections to obtain the visual feature information of the training video; the original features are the features obtained by embedding video frames of the training video into image blocks.

[0115] In another specific embodiment, the feature extraction module may include:

[0116] The feature extraction unit is used to extract initial dynamic region features from the original features based on temporal difference.

[0117] The weight generation unit is used to generate channel attention weights based on the covariance matrix of the initial dynamic region features using a preset neural network; the preset neural network is a neural network constructed based on grouped convolution, batch normalization, and nonlinear activation functions.

[0118] The feature calibration submodule is used to recalibrate the initial dynamic region features using the channel attention weights, and to fuse the recalibrated features and the original features through residual connections to obtain the target dynamic region features.

[0119] In yet another specific embodiment, the feature calibration submodule may include:

[0120] The feature calibration unit is used to enhance or suppress different feature channels of the initial dynamic region features by means of the channel attention weights in the dot product method, so as to obtain the corresponding recalibrated features.

[0121] In one specific embodiment, the model training module 13 may include:

[0122] The model training unit is used to train the model based on a preset loss function, utilizing the text prompts and visual feature information corresponding to the training videos; wherein, the preset loss function is a loss function constructed based on a contrastive learning loss function, a similarity classification loss function, a mask loss function, an action classification loss function, and corresponding weight coefficients.

[0123] Furthermore, embodiments of this application also disclose an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0124] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the video behavior recognition method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0125] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0126] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0127] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the video behavior recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0128] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed video behavior recognition method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0130] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0132] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0133] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video behavior recognition method, characterized in that, include: The video frames of the training video are processed using a first language model to obtain behavioral description information corresponding to each video frame; The behavior description information corresponding to each individual action category is processed using a second language model to obtain the corresponding text prompt information for each action category. The model is trained based on the text prompts and the visual feature information corresponding to the training video, and the trained model is used to identify the video to be identified to obtain the corresponding video behavior recognition results. The step of processing the behavior description information corresponding to a single action category using a second language model to obtain the corresponding text prompt information for each action category includes: The behavior description information is grouped and aggregated according to the action category to obtain the aggregated description information corresponding to different action categories; The aggregated description information is processed using a second language model to obtain text prompt information corresponding to each action category. The process of extracting the visual feature information includes: Local region features of the training video are extracted based on temporal convolution, and dynamic region features of the training video are extracted based on temporal difference and global second-order pooling techniques. The local region features, the dynamic region features, and the original features are fused using residual connections to obtain the visual feature information of the training video; the original features are the features obtained by embedding image blocks from the video frames of the training video. The step of extracting dynamic region features from the training video based on temporal difference and global second-order pooling techniques includes: Initial dynamic region features are obtained by extracting features from the original features based on temporal difference. Based on the covariance matrix of the initial dynamic region features, channel attention weights are generated using a preset neural network; the preset neural network is a neural network constructed based on grouped convolution, batch normalization, and nonlinear activation functions. The initial dynamic region features are recalibrated using the channel attention weights, and the recalibrated features and the original features are fused using residual connections to obtain the target dynamic region features.

2. The video behavior recognition method according to claim 1, characterized in that, The step of processing video frames of the training video using a first language model to obtain behavioral description information corresponding to each video frame includes: Each training video is processed according to a preset frame extraction algorithm to obtain a video frame sequence corresponding to each training video; the preset frame extraction algorithm is a frame extraction algorithm based on uniform sampling or based on content change rate. The video frames in each video frame sequence are processed using a first language model to obtain behavioral description information for the corresponding video frames.

3. The video behavior recognition method according to claim 1, characterized in that, The recalibration of the initial dynamic region features using the channel attention weights includes: By using dot product, the channel attention weights are used to enhance or suppress different feature channels of the initial dynamic region features to obtain the corresponding recalibrated features.

4. The video behavior recognition method according to any one of claims 1 to 3, characterized in that, The model training based on the text prompts and the visual feature information corresponding to the training videos includes: Based on a preset loss function, the model is trained using the text prompts and the visual features of the training videos. The preset loss function is a loss function constructed based on the contrastive learning loss function, the similarity classification loss function, the mask loss function, the action classification loss function, and the corresponding weight coefficients.

5. A video behavior recognition device, characterized in that, include: The first information processing module is used to process the video frames of the training video through the first language model to obtain the behavior description information corresponding to each video frame; The second information processing module is used to process the behavior description information corresponding to a single action category using a second language model to obtain the text prompt information corresponding to each action category. The model training module is used to train the model based on the text prompts and the visual feature information corresponding to the training video. The video behavior recognition module is used to identify the video to be recognized using the trained model and obtain the corresponding video behavior recognition results. The second information processing module includes: The grouping and aggregation unit is used to group and aggregate the behavior description information according to the action category to obtain the aggregated description information corresponding to different action categories; The second information processing unit is used to process each of the aggregated description information using a second language model to obtain text prompt information corresponding to each of the action categories. The device includes: The feature extraction module is used to extract local region features of the training video based on temporal convolution, and to extract dynamic region features of the training video based on temporal difference and global second-order pooling techniques. The feature fusion module is used to fuse the local region features, the dynamic region features, and the original features through residual connections to obtain the visual feature information of the training video; the original features are the features obtained by embedding video frames of the training video into image blocks. The feature extraction module includes: The feature extraction unit is used to extract initial dynamic region features from the original features based on temporal difference. The weight generation unit is used to generate channel attention weights based on the covariance matrix of the initial dynamic region features using a preset neural network; the preset neural network is a neural network constructed based on grouped convolution, batch normalization, and nonlinear activation functions. The feature calibration submodule is used to recalibrate the initial dynamic region features using the channel attention weights, and to fuse the recalibrated features and the original features through residual connections to obtain the target dynamic region features.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the video behavior recognition method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the video behavior recognition method as described in any one of claims 1 to 4.

Images