Open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration

Abstract

The invention discloses an open vocabulary behavior recognition method and device based on cross-batch optimization and Gaussian self-integration, a medium and equipment. The method comprises the steps of obtaining a to-be-processed video frame sequence; processing the to-be-processed video frame sequence in a pre-trained open vocabulary behavior recognition model to obtain a recognized behavior; and outputting the identified behavior. According to the method, a cross-batch optimization method is introduced in the video fine tuning process of the video learning model, and Gaussian self-integration is performed on the model parameters, so that the universality improvement of the video model from'known 'to'open' and the depolarization enhancement of the video model from'image 'to'video' information are efficiently promoted, and the more robust universality is realized.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, medium, and device for open vocabulary behavior recognition based on cross-batch optimization and Gaussian self-integration. Background Art

[0002] Open vocabulary (zero-shot) action recognition aims to identify new actions not seen during model training. By understanding video content and establishing semantic relationships between action categories, the model can recognize categories not seen during training. Zero-shot action recognition can alleviate the problems of scarce labeled action data or severely imbalanced data distribution, providing important support for the development of intelligent video understanding and artificial intelligence technology.

[0003] In recent years, visual language models (VLMs) have been trained on large-scale image-text pairs, efficiently aligning visual and semantic spaces through contrastive learning, resulting in powerful understanding of visual content and semantic knowledge. One of the most representative works, CLIP, is an image-based VLM consisting of two separate image and text encoders, jointly optimized via the infoNCE loss. It exhibits high flexibility and versatility, as well as strong zero-shot reasoning capabilities. Fine-tuning CLIP with a specialized temporal modeling component captures temporal information, enhancing its expertise in video recognition. However, because the video dataset used for fine-tuning is much smaller than the image-text dataset used for CLIP pre-training (0.2M vs. 400M), the temporal learning module often suffers from overfitting, significantly reducing the model's generalization and failing to generalize well to unseen categories in the test set. This weakens CLIP's original zero-shot capabilities during fine-tuning, inevitably leading to the challenge of catastrophic forgetting. Therefore, there is a trade-off between expertise and generality when adapting pre-trained CLIP to the video domain, especially for novel zero-shot tasks. Open-VCLIP proposes a continuous learning-based model interpolation weight optimization strategy. During training, a novel interpolation weight regularization is introduced. By sampling the balancing coefficient during training to optimize the model weights, CLIP is extended to become a powerful zero-shot visual learner that generalizes well to unknown video behaviors, rather than completely retaining knowledge of image tasks. To maintain the strong generalization ability of the pre-trained CLIP model and effectively adapt it to video tasks, FROSTER proposes a residual feature distillation method that enables the model to adapt to action recognition tasks while maintaining the generalization ability of CLIP. By using the frozen CLIP model as a teacher model, supervised feature learning is used to extract video features, thus bridging the gap between images and videos. By adjusting the balancing coefficient in the residual feature distillation, it allows for flexible adaptive features from images to videos without sacrificing the generalization ability of the pre-trained CLIP, achieving a balance between general features and video features.

[0004] The success of existing methods in open-vocabulary action recognition often benefits from the efficient visual-text alignment capabilities of large-scale image-language pre-trained models (e.g., CLIP) and their generality in the static domain of images. However, on the one hand, most full-scale fine-tuning and parameter-efficient fine-tuning methods suffer from overfitting on the video fine-tuning data, which to some extent reduces the generalization and versatility of the original large-scale pre-trained models and fails to achieve good action recognition performance in open-vocabulary situations. On the other hand, some methods propose to introduce CLIP regularization during the video fine-tuning process to retain its strong static generality, effectively improving the performance of open-vocabulary action recognition. However, due to CLIP's severe static bias, such methods rely too much on static cues in the video, hindering the learning of video-related dynamic features, resulting in poor performance on challenging out-of-distribution data and a lack of robust generality. Summary of the Invention

[0005] The main purpose of this application is to provide an open vocabulary behavior recognition method, device, medium and equipment based on cross-batch optimization and Gaussian self-integration, aiming to solve the technical problem of severe static bias in CLIP.

[0006] To achieve the above-mentioned objectives, the present application provides an open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration, comprising: obtaining a sequence of video frames to be processed; processing the sequence of video frames to be processed using a pre-trained open vocabulary behavior recognition model to obtain recognized behaviors; and outputting the recognized behaviors; wherein the training of the pre-built open vocabulary behavior recognition model includes cross-batch optimization training and Gaussian self-integration training.

[0007] Optionally, the cross-batch meta-optimization training of the visual learner of the CLIP model includes: obtaining an original visual learner based on the CLIP model; obtaining training set data, obtaining support batch data and query batch data based on sampling of the training set data, and constructing multiple generalization tasks from known to open based on the support batch data and the query batch data, wherein the support batch data and the query batch data are obtained based on two adjacent sampling batches; for each task, calculating the first loss gradient and temporary fast weight of the original visual learner on the support batch data in the meta-training stage; calculating the second loss gradient of the original visual learner on the query batch data under the temporary fast weight in the meta-testing stage, and using the second loss gradient as the generalization evaluation feedback of the visual learner; determining the latest parameters of the visual learner based on the generalization evaluation feedback and the first loss gradient in the meta-optimization stage, and updating the parameters of the original visual learner based on the latest parameters, completing a round of cross-batch meta-optimization training of the original visual learner; wherein the meta-training, meta-testing and meta-optimization are all determined based on model-independent meta-learning.

[0008] Optionally, the calculating of the first loss gradient and temporary fast weight of the original visual learner on the support batch data in the meta-training stage includes: obtaining the original parameters of the original visual learner; for each task, meta-training the original visual learner using the support batch data to obtain the first loss gradient of the original visual learner; and obtaining the temporary fast weight based on the difference between the original parameters and the gradient of the first loss gradient calculated by the back-propagation algorithm.

[0009] Optionally, the step of calculating the second loss gradient of the original visual learner under the temporary fast weight on the query batch data in the meta-test phase, and using the second loss gradient as generalization evaluation feedback of the visual learner, includes: updating the original parameters of the original visual learner based on the temporary fast weight to obtain a visual learner with updated parameters; performing a meta-test on the visual learner with updated parameters using the query batch data to obtain the category loss of the visual learner on the query batch data; obtaining the second loss gradient based on the category loss, and using the second loss gradient as the generalization evaluation feedback.

[0010] Optionally, the latest parameters of the visual learner are determined based on the generalization evaluation feedback and the first loss gradient in the meta-optimization stage, including: determining an adaptation cost function based on the generalization performance feedback and the first loss gradient function, and determining a meta-optimization objective function based on the adaptation cost function; performing meta-optimization on each of the tasks according to the meta-optimization objective function to obtain a meta-optimized adaptation cost function for each task; determining a parameter optimization objective function for all tasks based on the original parameters and the weighted adaptation cost functions of all the tasks; and solving the parameter optimization objective function using a back-propagation algorithm to obtain the latest parameters of the original visual learner.

[0011] Optionally, the Gaussian self-integration training of the visual learner of the CLIP model includes: obtaining the latest parameters of each round, as well as the iterative weight parameters of each round and the iterative weight of the Rth round based on the current R rounds of cross-batch optimization training; calculating global parameters based on the latest parameters of each round, the weighted sum of the iterative parameters of each round and the iterative weight of the Rth round of cross-batch optimization training, and updating the original parameters of the visual learner based on the global parameters.

[0012] Optionally, the performing Gaussian self-ensemble training on the visual learner of the CLIP model further includes:

[0013] Update the latest parameters of the tth round of cross-batch optimization training according to the sum of the iterative weights of the t-1th round of cross-batch optimization training, the sum of the iterative weights of the tth round of cross-batch optimization training, and the global parameters of the tth round of cross-batch optimization training.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides an open vocabulary behavior recognition device based on cross-batch optimization and Gaussian self-integration, including: an acquisition module for acquiring a video frame sequence to be processed; a processing module for processing the video frame sequence to be processed using an open vocabulary behavior recognition model to obtain the recognized behavior, wherein the open vocabulary behavior recognition model is constructed based on a CLIP visual learner after video fine-tuning and a CLIP text encoder, and the CLIP visual learner after video fine-tuning is obtained by sequentially performing cross-batch optimization training and Gaussian self-integration training on the visual learner of the CLIP model; and an output module for outputting the recognized behavior.

[0015] To achieve the above objectives, the present application also provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration provided in any of the above embodiments.

[0016] To achieve the above-mentioned objectives, the present application also provides an electronic device, comprising: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration provided in the aforementioned embodiment.

[0017] The embodiments of the present application propose an open vocabulary behavior recognition method, apparatus, medium and device based on cross-batch optimization and Gaussian self-integration, which obtains a sequence of video frames to be processed; processes the sequence of video frames to be processed using a pre-trained open vocabulary behavior recognition model to obtain recognized behaviors; and outputs the recognized behaviors. The present application introduces a cross-batch optimization method and Gaussian self-integration of model parameters in the process of fine-tuning the video learning model, which efficiently promotes the improvement of the universality of the video model from "known" to "open" and the debiasing enhancement of information from "image" to "video" to achieve more robust universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart illustrating an embodiment of an open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-ensembling is provided in this application;

[0019] FIG2 is a schematic diagram of an embodiment of an open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-ensembling according to the present application;

[0020] Figure 3 This is a functional module block diagram provided for an embodiment of an open vocabulary behavior recognition device based on cross-batch optimization and Gaussian self-integration.

[0021] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0023] In the existing technology, the standard fine-tuning paradigm is generally adopted. This standard fine-tuning paradigm is generally based on the CLIP model. The CLIP model is based on contrastive learning. By pre-training on large-scale image and text data, two separate unimodal encoders are learned to process images and texts respectively, and embed them into a joint feature space. Taking the CLIP-based visual learner as an example, its visual encoder adopts the general ViT architecture and introduces temporal modeling technology for video understanding. The standard fine-tuning paradigm of vision only converts such visual learners into (with parameters θ) with frozen text encoder Applied to action recognition. Specifically, given a training set D tr Video clip of V i and candidate behavior labels T j ∈Z tr , the tag is described using a predefined text template (e.g., “a video of {action}”). The similarity of visual semantics is calculated as follows:

[0024]

[0025] Among them, the training goal is to maximize the matching video V i and label T j The similarity s i,j , otherwise minimize s i,j The loss function is cross entropy loss:

[0026]

[0027] Among them, B and K represent the batch size and the number of known behavior types respectively. If the i-th video belongs to the k-th category, y i,k is equal to 1; otherwise, y i,k Equal to 0.

[0028] In open vocabulary action recognition, the trained visual learner needs to be tested on the test data T i ∈Z te It has good generalization properties, where Zte ∩Z tr =□.

[0029] The above process of standardized video fine-tuning relies too much on CLIP regularization, which is not conducive to improving the versatility of videos in open vocabulary action recognition.

[0030] This application provides a solution. Inspired by the "learning to learn" paradigm of meta-learning, the present invention proposes a cross-batch meta-optimization method for "learning to generalize", which naturally treats the training data of adjacent batches as support data and query data for a generalization task from a known to an open one. In the meta-training phase, the model optimizes parameters by learning behavior-related knowledge on the support data, obtaining temporary weights adapted to the support batch. Next, in the meta-testing phase, the model needs to perform a virtual evaluation on the query batch data based on the temporarily obtained adaptation weights, and the gradient of the evaluation loss is used as feedback information in a simulated open scenario. In the meta-optimization phase, the feedback information and behavior-type-specific knowledge are used together as conditions for model optimization, encouraging the model to optimize weights in a more general direction and learn more general and transferable video features from various tasks, rather than relying on the original static generality of CLIP for regularization. Furthermore, the present invention adopts a new self-integration stabilization strategy during the model training process, using Gaussian distribution to self-integrate the model optimization trajectory, so that the model achieves a better balance between specialization and generality.

[0031] Reference Figure 1 The first embodiment of the present application provides an open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration. The method can be executed by a processor of a terminal or server. The method may include:

[0032] S10, obtaining a video frame sequence to be processed;

[0033] S20, processing the sequence of video frames to be processed using an open vocabulary action recognition model to obtain recognized actions, wherein the open vocabulary action recognition model is constructed based on a CLIP visual learner fine-tuned for the video and a CLIP text encoder, and the CLIP visual learner fine-tuned for the video is obtained by sequentially performing cross-batch optimization training and Gaussian self-ensemble training on the visual learner of the CLIP model;

[0034] S30: Output the recognized behavior.

[0035] The following further illustrates the technical solution of the open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration with reference to the accompanying drawings.

[0036] Referring to Figure 2, Figure 2(a) shows that the cross-batch meta-optimization method simulates a known-to-open generalization task by performing gradient descent updates on the support batch data (i.e., meta-training) and virtual evaluation on the query batch data (i.e., meta-testing). Afterwards, the visual learner is optimized with a category-specific loss from the support batch data and task feedback from the query batch data to learn more generalizable video behavior knowledge that is resistant to inherent known and static biases. Figure 2(b) shows that during the episodic training process, the Gaussian self-ensemble strategy utilizes the optimization trajectory of the visual learner to perform Gaussian weighted averaging (GWA) to obtain a universal optimal solution for robust generalization.

[0037] In an embodiment of the present application, performing cross-batch optimization training on the visual learner of the CLIP model includes:

[0038] S201, obtain the original visual learner based on the CLIP model;

[0039] S202: Acquire training set data, obtain support batch data and query batch data based on sampling of the training set data, and construct multiple generalization tasks from known to open based on the support batch data and query batch data, wherein the support batch data and query batch data are obtained based on two adjacent sampling batches;

[0040] S203. For each task, calculate the first loss gradient and temporary fast weight of the original visual learner on the support batch data in the meta-training phase;

[0041] S204. Calculate the second loss gradient of the original visual learner under the temporary fast weight on the query batch data in the meta-test phase, and use the second loss gradient as generalization evaluation feedback of the visual learner;

[0042] S205. In the meta-optimization phase, the latest parameters of the visual learner are determined based on the generalization evaluation feedback and the first loss gradient, and the parameters of the original visual learner are updated based on the latest parameters, completing a round of cross-batch meta-optimization training of the original visual learner.

[0043] Among them, meta-training, meta-testing and meta-optimization are all determined based on model-independent meta-learning.

[0044] The processor implements a round of cross-batch meta-optimization training of the original visual learner by executing steps S201 to S205. It should be noted that meta-optimization training includes three processes: meta-training, meta-testing and meta-optimization. All three processes are determined based on model-independent meta-learning. Model-independent meta-learning (MAML) is a gradient-based meta-learning method that aims to learn an initialization parameter so that the model can quickly adapt to new tasks after a small amount of gradient updates. The core idea is to obtain a good initial parameter through meta-learning, rather than directly learning parameters for a specific task, so that the model can better adapt to unseen data. Unlike the regularization method that uses the static universality of the CLIP pre-trained model, the cross-batch meta-optimization method introduces MAML into open vocabulary behavior recognition, and promotes the video model to learn more generalizable video-specific knowledge through meta-training and meta-testing, rather than over-exploiting static biases.

[0045] This application performs cross-batch meta-optimization training on the visual learner of the CLIP model, and directly uses the default training data sampler to take the data of adjacent batches as the support data and query data in a "known" to "open" meta-task. In fact, due to the different label distributions between different batches of data, the known deviations in the support data do not hold in the query data. Therefore, the arbitrariness of sampling different batches of data can naturally provide convenience for constructing a "known" to "open" generalization task. Similar to the round-based training of MAML, the cross-batch meta-optimization training method expands the original gradient step size into two consecutive small batches of samples, where the current batch is used as support data and the subsequent batch is used as query data, thereby creating a known to open generalization task.

[0046] The implementation of each stage is described below. First, in the meta-training phase, the visual learner is trained. By calculating the first loss gradient of the visual learner on each batch of support data, the visual learner is updated to obtain temporary fast weights. Then, in the meta-testing phase, the second loss gradient of the visual learner under the fast weights on each batch of query data is used as feedback to evaluate the learner's generalization, thereby guiding the learner to achieve more general parameter optimization.

[0047] The implementation of each stage is described in detail below:

[0048] In an embodiment of the present application, step S203 may include the following execution process:

[0049] S2031. Obtain original parameters of the original visual learner;

[0050] S2032. For each task, meta-train the original visual learner using the support batch data to obtain the first loss gradient of the original visual learner;

[0051] S2033. Obtain a temporary fast weight according to the difference between the original parameter and the gradient of the first loss gradient calculated by the back propagation algorithm.

[0052] The processor completes the meta-training of the visual learner by executing steps S2031 to S2033, and obtains the temporary fast weight and the first loss gradient after the meta-training.

[0053] For each iteration of the training phase, in the meta-training phase, we first use i Each support batch S={V i ,T i} B To train the visual learner f θ (with parameter θ), a standard gradient step optimization is performed on the batch data loss of each support batch to obtain the first loss gradient, where the batch data loss expression is as follows:

[0054]

[0055] Where L(·) is the loss function. Then, by temporarily copying the original parameters of the learner, the gradient of the first loss gradient is calculated according to the backpropagation algorithm, and the gradient of the first loss gradient is used to update the intermediate parameters:

[0056]

[0057] where α represents the learning rate for meta-training. Intuitively, this step simulates a direct update to the parameters of the visual learner to train the visual learner to acquire category-specific knowledge that supports the batch data.

[0058] In an embodiment of the present application, step S204 may include the following execution process:

[0059] S2041. Update original parameters of the original visual learner based on the temporary fast weights to obtain a visual learner with updated parameters;

[0060] S2042. Perform a meta-test on the visual learner after parameter update using the query batch data to obtain the category loss of the visual learner on the query batch data.

[0061] S2043. Obtain a second loss gradient based on the category loss, and use the second loss gradient as generalization evaluation feedback.

[0062] The processor completes the meta-test of the visual learner by executing steps S2041 to S2043, and obtains generalization evaluation feedback of the visual learner after the meta-test.

[0063] Specifically, after the meta-training on the supporting batch data, a virtual meta-testing process is performed, using the query batch data Q = {V i ,T i} B (in, ) to evaluate visual learners Formally, the generalization performance of task T is measured by calculating the category loss on the query batch data. i Known to open performance on:

[0064]

[0065] The above formulation is closely related to the standard fine-tuning procedure, which aims to obtain good category-specific performance for all training batches. The difference is that since the original parameters θ are preserved and not affected by the task-specific updates, this step only evaluates the intermediate learner's known-to-open generalization ability on each task and can therefore be used to provide feedback so that the learner can learn more general video-specific knowledge from the known and static deviations to achieve robust generalization in the subsequent meta-optimization.

[0066] In an embodiment of the present application, step S205 may include the following execution process:

[0067] S2051, determining an adaptation cost function based on the generalization performance feedback and the first loss gradient function, and determining a meta-optimization objective function according to the adaptation cost function;

[0068] S2052. Perform meta-optimization on each task according to the meta-optimization objective function, and obtain a meta-optimized adaptive cost function for each task.

[0069] S2053. Determine the parameter optimization objective function for all tasks based on the original parameters and the weighted adaptation cost functions of all tasks;

[0070] S2054. Use the back propagation algorithm to solve the parameter optimization objective function and obtain the latest parameters of the original visual learner.

[0071] The processor completes the meta-optimization of the visual learner by executing steps S2051 to S2054. Through the meta-optimization stage, the latest parameters of the visual learner for all tasks in one round of iteration are obtained.

[0072] Referring to Figure 2(a), in the meta-optimization phase, as above, the virtual evaluation during meta-testing in this application can provide useful generalization performance feedback to guide and encourage the learner to meta-train on the support data (i.e., θ i ′←θ) and then learn a more robust representation for fast known-to-open generalization. It is worth noting that the original MAML method focuses on optimizing parameters to obtain a strong initialization in order to quickly adapt to new tasks with minimal gradient updates. In this application, open vocabulary recognition requires zero-shot capabilities, which cannot be further adapted to new tasks. Therefore, category-specific knowledge should be strengthened from a global optimization perspective. To this end, in the inner loop, the parameters of the learner are optimized to simultaneously minimize the category-specific loss of the support data and the adaptation cost of the query data. Therefore, the meta-optimization objective can be defined as:

[0073]

[0074] Here, the first term refers to the class-specific knowledge learned on the support batch, while the second term is based on θ i ′ Provides a task for T i The known-to-open generalization feedback of θ is used to achieve robust generalization for robust representation learning. The optimization process of the learner parameter θ can be expressed as:

[0075]

[0076] Where N is the batch size of the meta-optimization task. Since the meta-gradient update of MAML needs to be differentiated through the optimization process (i.e., the gradient of the gradient), this is not an ideal solution when a large number of tasks need to be optimized during the training phase. Therefore, we choose to omit θ←θ i ′ Back propagation is used to perform the first-order approximate update, and the above formula can be replaced by:

[0077]

[0078] Where β and δ are the meta-optimization learning rates. Since the actual update of the learner does not introduce any CLIP regularization, the actual training process can implement a multi-task parallel version of cross-batch meta-optimization, that is, evaluate on N known-to-open tasks with different category distributions (i.e., category-specific knowledge), thereby encouraging the learning of more generalizable features to combat known and static biases.

[0079] In an embodiment of the present application, Gaussian self-ensemble training is performed on the visual learner of the CLIP model, including:

[0080] S206. Obtain the latest parameters of each round, as well as the iterative weight parameters of each round and the iterative weight of the Rth round according to the current R rounds of cross-batch optimization training;

[0081] S207, calculating global parameters based on the latest parameters of each round, the weighted sum of the iterative parameters of each round, and the iterative weights of the R-th round of cross-batch optimization training, and updating the original parameters of the visual learner based on the global parameters;

[0082] Wherein, R is a positive integer.

[0083] The processor implements a global update of the visual learner parameters by executing steps S206 to S207, thereby obtaining the latest parameters of the visual learner. It should be noted that the global update of the parameters is obtained after R rounds of cross-batch optimization training. Each round of cross-batch optimization training includes multiple tasks, and each task has its own video fine-tuning parameters. After calculating the parameters of each round, this application uses the prior distribution to reduce the contribution of parameters near the initial and final iterations, thereby obtaining robust generalization and versatility.

[0084] Referring to Figure 2(b), since the video category-specific knowledge in the training phase is limited, while the open vocabulary task is unrestricted, the static generalization from large-scale image-text pre-training should be exploited to enhance the adaptability of the visual learner while minimizing the impact of static bias. Therefore, the present invention further proposes a self-integration stabilization strategy on the optimization trajectory of the visual learner, which utilizes the knowledge of previous training iterations to obtain a more generalized visual learner. Specifically, during the fine-tuning process of R rounds (each round is l steps long), the optimization trajectory of the learner is expressed as Where θ0 is the pre-training weight. The weight of the self-ensemble learner is:

[0085]

[0086] Among them, w t is the weight of the parameter at the tth iteration. Intuitively, in the early stages of the fine-tuning process (i.e., smaller iteration t), the visual learner lacks the maturity to effectively capture video-specific knowledge, while still retaining a large amount of static information from large-scale pre-training, which introduces divergent information for video temporal understanding. In contrast, the parameters of the last few rounds (i.e., larger iteration t) integrate more video-specific knowledge and highly reflect the distribution of downstream supervision tasks, while failing to guarantee zero-shot capability plasticity. Since both aspects reduce the final open vocabulary generalization ability, the present invention aims to achieve robust generalization and versatility by using a prior distribution to reduce the contribution of parameters near the initial and final iterations.

[0087] Continuing with FIG2( b ), in an embodiment of the present application, Gaussian self-ensemble training is performed on the visual learner of the CLIP model, further comprising:

[0088] S208, updating the latest parameters of the tth round of cross-batch optimization training according to the sum of the iterative weights of the t-1th round of cross-batch optimization training, the sum of the iterative weights of the tth round of cross-batch optimization training, and the global parameters of the tth round of cross-batch optimization training;

[0089] Wherein, t is a positive integer less than or equal to R.

[0090] For example, to further supplement the self-integration stabilization technique, the present invention proposes Gaussian weighted averaging (GWA), which assigns lower weights to the parameters of the initial iteration, higher weights to the mid-term iterations, and relatively lower weights to the final iterations. Given a Gaussian distribution w defined on the number of iterations t ~N(μ,σ 2 ), the parameter θ t The weight value of is its corresponding probability in the distribution:

[0091]

[0092] In practice, it is necessary to determine a suitable μ value based on the number of iterations. Then, the weights are normalized to ensure At the same time, GWA can also be used as a moving average by dynamically updating the current learner θ t The average of ℓ to avoid the storage cost of saving multiple parameter snapshots:

[0093]

[0094] Among them, θ GWA Indicates the latest parameters obtained in the previous iteration.

[0095] The following examples illustrate the technical solution of this application:

[0096] The model parameters were fine-tuned using Kinectis-400 training videos, pre-trained weights using CLIP ViT-B / 16, and paired with the VCLIP visual learner. During meta-optimization, each batch contained four tasks, each containing eight support samples and eight query samples sampled from the training set. The initial learning rate α for the inner loop (support batch) and the learning rate β for the outer loop (query batch) were 3.33×10 -6 , and decayed to 3.33×10 using the AdamW optimizer and the cosine decay learning rate scheduler -8 The hyperparameter δ of the query batch is set to 1.67×10 -3For cross-dataset evaluation, we first pre-train for 2 rounds on the Kinectis-400 dataset, and then further fine-tune the visual learner for an additional 20 rounds. For base-to-novel evaluation, we train the visual learner for 12 rounds, of which the first two rounds are pre-trained. The zero-shot inference stage uses 3 temporal views and 1 spatial view of each video for inference, linearly aggregating the recognition results of different views.

[0097] Given a training set D tr Video clip of V i and candidate behavior labels T j ∈Z tr , the tag is described using a predefined text template (e.g., "a video of {action}"). Calculate the similarity of visual semantics:

[0098]

[0099] And use cross entropy loss as the loss function:

[0100]

[0101] Among them, B and K represent the batch size and the number of known behavior types respectively. If the i-th video belongs to the k-th category, y i,k is equal to 1; otherwise, y i,k Equal to 0. The cross-batch optimization method updates and fine-tunes the parameters of the visual learner through the sampled support data and query data:

[0102]

[0103] In addition, a Gaussian self-ensembling strategy is applied to the learner parameters after each epoch:

[0104]

[0105] Finally, we obtain more general and robust visual learner weights that can be widely adapted to different open vocabulary scenarios.

[0106] Based on the above method embodiments, the present application also provides an open vocabulary behavior recognition device based on cross-batch optimization and Gaussian self-integration, the device including an acquisition module, a processing module and an output module, wherein the acquisition module is used to acquire the video frame sequence to be processed; the processing module is used to process the video frame sequence to be processed using the open vocabulary behavior recognition model to obtain the recognized behavior, wherein the open vocabulary behavior recognition model is constructed based on the CLIP visual learner after video fine-tuning and the CLIP text encoder, and the CLIP visual learner after video fine-tuning is obtained by sequentially performing cross-batch optimization training and Gaussian self-integration training on the visual learner of the CLIP model; the output module is used to output the recognized behavior.

[0107] It should be understood that the open vocabulary behavior recognition device based on cross-batch optimization and Gaussian self-integration described in this embodiment has the same beneficial effects as those described in any of the above-mentioned open vocabulary behavior recognition method embodiments based on cross-batch optimization and Gaussian self-integration, and will not be repeated here.

[0108] Based on the above embodiments, the present application also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes the open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration provided in any of the above embodiments.

[0109] Based on the above embodiments, the present application also provides an electronic device, comprising: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration provided in the above embodiments.

[0110] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for open vocabulary behavior recognition based on cross-batch optimization and Gaussian self-integration, characterized in that: include: Get the video frame sequence to be processed; Processing the to-be-processed video frame sequence using an open vocabulary behavior recognition model to obtain recognized behaviors, wherein the open vocabulary behavior recognition model is constructed based on a CLIP visual learner fine-tuned by the video and a CLIP text encoder, and the CLIP visual learner fine-tuned by the video is obtained by sequentially performing cross-batch optimization training and Gaussian self-ensemble training on the visual learner of the CLIP model; Output the identified behavior.

2. The open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration as claimed in claim 1, characterized in that: The cross-batch optimization training of the visual learner of the CLIP model includes: Obtain the original visual learner based on the CLIP model; Acquire training set data, obtain support batch data and query batch data based on the training set data sampling, and construct multiple generalization tasks from known to open based on the support batch data and the query batch data, wherein the support batch data and the query batch data are obtained based on two adjacent sampling batches; For each task, calculating the first loss gradient and temporary fast weight of the original visual learner on the support batch data in the meta-training phase; Calculate the second loss gradient of the original visual learner under the temporary fast weight on the query batch data in the meta-test phase, and use the second loss gradient as the generalization evaluation feedback of the visual learner; In the meta-optimization stage, the latest parameters of the visual learner are determined based on the generalization evaluation feedback and the first loss gradient, and the parameters of the original visual learner are updated based on the latest parameters to complete a round of cross-batch meta-optimization training of the original visual learner; Wherein, the meta-training, meta-testing and meta-optimization are all determined based on model-independent meta-learning.

3. The open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration as claimed in claim 2, characterized in that: The step of calculating the first loss gradient and temporary fast weight of the original visual learner on the supporting batch data in the meta-training stage includes: Obtaining original parameters of the original visual learner; For each task, meta-training the original visual learner using the support batch data to obtain a first loss gradient of the original visual learner; The temporary fast weight is obtained according to the difference between the original parameter and the gradient of the first loss gradient calculated by the back propagation algorithm.

4. The open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration as claimed in claim 3, characterized in that: The step of calculating the second loss gradient of the original visual learner under the temporary fast weight on the query batch data in the meta-test phase, and using the second loss gradient as generalization evaluation feedback of the visual learner, comprises: Based on the temporary fast weights, original parameters of the original visual learner are updated to obtain a visual learner with updated parameters; Using the query batch data, meta-testing the visual learner after the parameter update to obtain the category loss of the visual learner on the query batch data; The second loss gradient is obtained based on the category loss, and the second loss gradient is used as the generalization evaluation feedback.

5. The open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration as claimed in claim 4, characterized in that: The step of determining the latest parameters of the visual learner based on the generalization evaluation feedback and the first loss gradient in the meta-optimization stage includes: Determine an adaptation cost function based on the generalization performance feedback and the first loss gradient function, and determine a meta-optimization objective function according to the adaptation cost function; Performing meta-optimization on each of the tasks according to the meta-optimization objective function, and obtaining a meta-optimized adaptive cost function for each of the tasks; Determine the parameter optimization objective function of all tasks according to the original parameters and the weighted adaptation cost functions of all the tasks; The parameter optimization objective function is solved by using a back propagation algorithm to obtain the latest parameters of the original visual learner.

6. The open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration as claimed in claim 3, characterized in that: The Gaussian self-integration training of the visual learner of the CLIP model includes: According to the current R rounds of cross-batch optimization training, the latest parameters of each round, as well as the iterative weight parameters of each round and the iterative weight of the R round are obtained; Calculate global parameters based on the latest parameters of each round, the weighted sum of iterative parameters of each round, and the iterative weights of the R-th round of cross-batch optimization training, and update the original parameters of the visual learner based on the global parameters; Wherein, R is a positive integer.

7. The open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration as claimed in claim 6, characterized in that: The Gaussian self-integration training of the visual learner of the CLIP model further includes: According to the sum of the iterative weights of the t-1th round of cross-batch optimization training, the sum of the iterative weights of the tth round of cross-batch optimization training, and the global parameters of the tth round of cross-batch optimization training, the latest parameters of the tth round of cross-batch optimization training are updated; Wherein, t is a positive integer less than or equal to R.

8. An open vocabulary behavior recognition device based on cross-batch optimization and Gaussian self-integration, characterized in that: include: An acquisition module, used for acquiring a video frame sequence to be processed; a processing module, configured to process the to-be-processed video frame sequence using an open vocabulary behavior recognition model to obtain recognized behaviors, wherein the open vocabulary behavior recognition model is constructed based on a CLIP visual learner fine-tuned by the video and a CLIP text encoder, wherein the CLIP visual learner fine-tuned by the video is obtained by sequentially performing cross-batch optimization training and Gaussian self-ensemble training on the visual learner of the CLIP model; The output module is used to output the recognized behavior.

9. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the open vocabulary behavior recognition method based on cross-batch optimization and Gaussian self-integration according to any one of claims 1 to 7.

Images