Semi-supervised fine-grained action recognition method based on time sequence disturbance and learning stabilization

Abstract

The invention provides a semi-supervised fine-grained action recognition method based on time sequence disturbance and learning stabilization, and provides a semi-supervised learning fine-grained action recognition algorithm framework SeFAR based on a FixMatch semi-supervised learning system for a highly challenging task of semi-supervised fine-grained action recognition. According to the method, a two-stage time sequence element sampling method is provided, fine-granularity time sequence elements and time sequence contexts are combined, and multi-granularity time sequence information can be effectively captured, so that the capability of distinguishing subtle actions in a video is enhanced; the method comprises the following steps of: obtaining fine-grained time sequence elements, considering the importance of time sequence dynamics and designing a new'strong enhancement 'strategy, introducing proper time sequence disturbance into the obtained fine-grained time sequence elements, and keeping the time sequence of context elements, so that a weak-strong enhancement comparison data pair more suitable for a fine-grained action recognition task is constructed; besides, a self-adaptive learning adjustment strategy is further designed in the method, and back propagation loss in training is adjusted by calculating a self-adaptive coefficient so as to stabilize the training process.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition and deep learning, and specifically to a semi-supervised fine-grained action recognition method based on timing perturbation and learning stabilization. Background Art

[0002] With the popularization of 5G Internet and the rapid development of artificial intelligence technology, a huge amount of video data has been generated. Video understanding has attracted more and more attention because it contains vivid visual information and rich temporal dynamics, attracting more and more researchers to devote themselves to the research of video understanding. As one of the core issues in video understanding, action recognition has shown broad application prospects in monitoring systems, human-computer interaction, and video retrieval. In recent years, thanks to the development of deep learning technology and the construction of large-scale video datasets, supervised action recognition research has made significant progress. Several powerful multimodal large language models (MLLMs) have been proposed, such as GPT-4v (OpenAI 2024); VideoChat (Li et al. 2024b); Video-LLaVA (Lin et al. 2023); InternLM (Zhang et al. 2024), with the goal of obtaining more general and comprehensive capabilities. These large multimodal models can achieve very perfect recognition results in classic action recognition, but classic action recognition only requires the model to provide relatively coarse-grained categories, such as "gymnastics". For fine-grained action recognition that focuses on more specific task scenarios and aims to provide more detailed, specific and semantically accurate descriptions, such as "pushing on the bar with legs together and looping back - turning 180 degrees in a handstand", the current mainstream large multimodal models perform poorly and fail to correctly recognize the fine-grained semantics of a given action. For example, Figure 1 shown. Summary of the invention

[0003] Technical issues to be solved:

[0004] Currently, there is very limited research on fine-grained action recognition. The main reasons are:

[0005] (1) The construction of fine-grained action datasets is extremely complex. For example, providing annotations for actions such as “switch legs, split, jump, and turn 360 degrees” requires sufficient expert knowledge, extremely long annotation time, and a lot of inspection work to ensure quality. This leads to a scarcity of datasets with fine-grained labels, making it very difficult to directly retrain or fine-tune large models using supervised learning.

[0006] (2) To address the problem of scarce labeled datasets, the traditional solution is to use semi-supervised learning to improve model performance by providing stable and reasonable supervision (pseudo-labels) for unlabeled data. However, fine-grained action recognition requires sufficient visual details, effective information aggregation, and a comprehensive understanding of temporal dynamics. Therefore, this task itself is very challenging. Human annotators find it difficult to distinguish fine-grained actions, which makes the pseudo-labels generated in traditional solutions very likely to be unreliable, making the entire semi-supervised learning process very easy to fail.

[0007] The technical solution of the present invention:

[0008] In order to solve the problems existing in the prior art and realize accurate recognition of fine-grained actions, the present invention proposes a semi-supervised fine-grained action recognition method based on time series perturbation and learning stabilization, which specifically includes the following steps:

[0009] Step 1: The video to be identified is sampled using a two-level temporal element sampling method;

[0010] Step 2: Input the frame sequence sampled in step 1 into the trained model to obtain the recognition result;

[0011] The model is trained through the following process:

[0012] Step a: obtaining training samples, wherein the training samples include a small number of labeled videos and a large number of unlabeled videos;

[0013] Step b: The labeled video is sampled using a two-level temporal element sampling method, and the labeled frame sequence samples obtained by sampling are used to obtain an initial model through supervised training;

[0014] The teacher model and the student model are initialized using the initial model; the iterative training is performed through the following steps, where the kth iteration process is:

[0015] Step c: sampling both labeled and unlabeled videos using a two-level temporal element modeling method, weakly enhancing the labeled frame sequence samples obtained by sampling, and weakly enhancing and strongly enhancing the unlabeled frame sequence samples obtained by sampling, to obtain a batch of frame sequence samples; the batch of frame sequence samples includes the labeled frame sequence samples after weak enhancement and the unlabeled frame sequence samples after weak enhancement and strong enhancement;

[0016] Step d: Input the weakly enhanced labeled frame sequence samples into the teacher model for recognition, and obtain the recognition result and the loss L between the recognition result and the label sup ; The recognition result is a probability distribution of various possible categories;

[0017] Step e: Input the weakly enhanced unlabeled frame sequence samples into the teacher model for recognition to obtain the recognition results;

[0018] Step f: Repeat step e N times to obtain N recognition results, and calculate the average probability μ of each category in the N recognition results. i As a pseudo label, i represents the i-th category;

[0019] Step g: Calculate the maximum value μ of the probability average of various categories * =max(μ i ), and μ * The probability standard deviation σ of the corresponding category in N recognition results * ;

[0020] According to μ * and σ * , calculate the dynamic coefficients τ1 and τ2:

[0021]

[0022] Among them, e is a natural constant, β and ε are set constants; the adaptive coefficient η=τ1τ2 is calculated according to the dynamic coefficients τ1 and τ2;

[0023] Step h: Input the strongly enhanced unlabeled frame sequence samples into the student model for recognition to obtain the recognition results;

[0024] Step i: Use the recognition results and pseudo labels of the student model to obtain the loss L of the unsupervised learning process un ;

[0025] Step j: Combine the strongly enhanced unlabeled frame sequence samples in the batch in pairs to obtain several pairs of unlabeled frame sequence sample pairs, process each of the unlabeled frame sequence sample pairs by the Tube TokenMix (TTMix) method to obtain a new unlabeled frame sequence sample, input the new unlabeled frame sequence sample into the student model for recognition, and obtain the recognition result;

[0026] Step k: One of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η1 and the corresponding pseudo label ρ1; the other unlabeled frame sequence sample in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η2 and the corresponding pseudo label ρ2; according to the formula

[0027] η ′ =λη1+(1-λ)η2

[0028] ρ ′ =λρ1+(1-λ)ρ2

[0029] Get the mixed loss weight η ′ and the mixed pseudo-label ρ ′ , where λ is the set mixing ratio; using the recognition result of step j and the mixed pseudo label ρ ′ , we get the mixed loss L mix ;

[0030] Step 1: Use the formula

[0031] L=L sup +ξ(ηL un +η′L mix )

[0032] Get the loss L of this iteration process, where ξ is the warmup coefficient;

[0033] Step m: Use the loss L to adjust the student model and get the student model parameters after the kth iteration Reuse the adjusted student model parameters According to the formula Get the teacher model parameters after the kth iteration Then return to step c until the set total number of iterations is reached and the model training is completed.

[0034] In a further preferred embodiment, the two-stage sequential element sampling is:

[0035] Sampling the video in segments to obtain K frames of images;

[0036] Densely sampling the K frame images with a relatively small sampling step to obtain a set of context frame sequences that reflect global temporal information;

[0037] The K frame images are sparsely sampled with a relatively large sampling step to obtain multiple groups of fine-grained frame sequences that reflect detail information.

[0038] In a further preferred embodiment, in step a, the small amount of labeled videos and the large amount of unlabeled videos means that the labeled videos account for no more than 10% and no less than 1%.

[0039] In a further preferred embodiment, the strong enhancement is a moderate timing disturbance.

[0040] In a further preferred embodiment, the moderate timing perturbation is to arrange each group of fine-grained frame sequences in reverse order, while the order of the context frame sequences remains unchanged.

[0041] In a further preferred embodiment, in step g, β is related to the dropout of the model, and ε is a stable parameter.

[0042] In a further preferred embodiment, in step k, λ=Beta(10,10), where Beta() represents Beta distribution.

[0043] In a further preferred embodiment, in step 1, the warmup coefficient k is the current iteration number, M n is the total number of iterations set.

[0044] Beneficial Effects

[0045] The present invention has the following beneficial effects:

[0046] 1. This paper proposes for the first time a semi-supervised learning fine-grained action recognition algorithm framework, SeFAR;

[0047] 2. The present invention proposes a two-level temporal element sampling method, which combines fine-grained temporal elements with temporal context, and can effectively capture multi-granularity temporal information to enhance the ability to distinguish subtle movements in the video;

[0048] 3. The present invention considers the importance of temporal dynamics and designs a new "strong enhancement" strategy, which introduces moderate temporal perturbations into the obtained fine-grained temporal elements while maintaining the temporal order of contextual elements, thereby constructing weak-strong enhancement contrast data pairs that are more suitable for fine-grained action recognition tasks;

[0049] 4. In order to provide reliable pseudo-labels for unlabeled data when the teacher model prediction is unstable, the present invention designs an adaptive learning adjustment strategy, which adjusts the backpropagation loss in training by calculating the adaptive coefficient to stabilize the training process.

[0050] The examples show that the semi-supervised learning fine-grained action recognition algorithm framework SeFAR proposed in this paper has achieved state-of-the-art performance on both fine-grained (FineGym, FineDiving) and coarse-grained action recognition datasets (UCF101, HMDB51), proving its effectiveness. And SeFAR can be used as a powerful visual encoder, embedded in the current powerful large-scale multimodal basic model, to improve the performance of the current MLLM in specific domain scenarios.

[0051] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0053] Figure 1 : Existing multi-modal large model fine-grained action recognition examples;

[0054] Figure 2 : Flowchart of the SeFAR algorithm framework proposed in the present invention;

[0055] Figure 3 : Schematic diagram of visualization of various ablation experiments;

[0056] (a) Comparison of the effects of different sampling combinations on SeFAR-B performance in the Gym-99 dataset with a 5% annotation rate; as the number of sampled fine-grained frame sequences increases, the model accuracy continues to rise; as the number of context frame sequences increases, the model accuracy also continues to rise; this shows that the two-level temporal element sampling proposed in the present invention can improve model performance by enriching fine-grained information and global information;

[0057] (b) The performance of the fixed threshold method and the adaptive adjustment strategy proposed in the present invention on the FineDiving dataset at a 5% annotation rate is compared; compared with different fixed thresholds, the adaptive adjustment method proposed in the present invention has better stability and higher accuracy;

[0058] (c) shows the fluctuation of the prediction results of the teacher model on different data sets. It can be seen that compared with the coarse-grained action recognition data set, the prediction fluctuation of the teacher model on the fine-grained action recognition data set FineGym (blue line) is significantly more dramatic, which illustrates the necessity of adaptively adjusting the weights according to the teacher model. DETAILED DESCRIPTION

[0059] Embodiments of the present invention are described in detail below. The embodiments are exemplary and intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0060] Figure 1The results of the four current mainstream powerful multimodal large models MLLMs: GPT-4v (OpenAI 2024); VideoChat (Li et al. 2024b); Video-LLaVA (Lin et al. 2023); InternLM (Zhang et al. 2024) on coarse-grained and fine-grained action recognition are shown. The upper part of the figure shows two fine-grained video action samples selected from the FineGym dataset (Shao et al., 2020) to show the subtle differences in fine-grained action recognition. Among them, the top video sample example is "legs and legs push the bar backward loop-handstand 180 degrees", and the bottom sample is "legs and legs push the bar backward loop-handstand 360 degrees". We use the video action examples at the bottom as test samples, and the evaluation shows the performance of the four multimodal large models on coarse-grained and fine-grained tasks. The first question focuses on coarse-grained semantics, and the second question focuses on fine-grained semantics. It can be seen that for the first question, the four multimodal large models all answered correctly, while for the second question focusing on fine-grained semantics, the four multimodal large models failed to correctly identify the fine-grained semantics of the given action.

[0061] In order to explore the highly challenging task of semi-supervised fine-grained action recognition, this embodiment proposes a semi-supervised learning fine-grained action recognition algorithm framework SeFAR based on the FixMatch semi-supervised learning system, designs a new enhancement strategy, and optimizes the original teacher-student learning model according to the new task. The specific algorithm framework is as follows Figure 2 As shown, the following steps are included:

[0062] Step 1: The video to be identified is sampled using a two-level temporal element sampling method.

[0063] Two-level temporal element sampling is one of the innovations of this invention, which is used to distinguish the subtle differences of fine-grained actions and mine rich and detailed visual information. The specific process is:

[0064] The fine-grained action video is divided into K segments and a frame is randomly sampled in each segment to obtain a frame sequence consisting of K frames {f1,…,f K}.

[0065] Since most visual contents (such as scenes and objects) in fine-grained action recognition are highly similar, the model needs to perceive subtle changes and rich details to achieve accurate distinction. Therefore, this embodiment proposes to use a frame sequence {f1,…,f K} is sparsely sampled with a relatively large sampling step size to construct multiple groups of small time series elements p that reflect detailed information i (i.e., multiple groups of fine-grained frame sequences), and each time element p iThe length L (i.e., the number of frames) is moderate to help the model focus on fast and subtle changes. After M sparse samplings, M groups of time series elements with the same time series length are obtained, expressed as:

[0066] where |p i |=L

[0067] In addition to the above-mentioned fine-grained temporal elements, this embodiment also proposes a frame sequence {f1,…,f K} is densely sampled with a relatively small sampling step size to construct a context element p that reflects the global timing information. context (i.e., context frame sequence) to encode long-term information and macroscopic temporal dynamics. Since dense sampling is performed with a relatively small sampling step size, p context It consists of more frames, usually fine-grained temporal elements p i More than twice.

[0068] This two-level temporal element sampling ensures the preservation of multi-level and multi-granular information, thereby obtaining an effective representation of the input fine-grained action video, denoted as {p1,…,p M ,p context}.

[0069] like Figure 2 As shown, in this embodiment, the fine-grained action video is divided into 8 segments and a frame is randomly sampled in each segment to obtain a frame sequence {f1, ..., f8} consisting of 8 frames of images.

[0070] After performing sparse sampling twice with a sampling step of 4 on the frame sequence {f1,…,f8}, two groups of time series elements with a length of 2 are obtained, namely {f2,f6} and {f4,f8}.

[0071] The context elements {f1, f3, f5, f7} are obtained by densely sampling the frame sequence {f1, ..., f8} with a sampling step of 2.

[0072] Step 2: Input the frame sequence sampled in step 1 into the trained model to obtain the recognition result;

[0073] The model is trained through the following process:

[0074] Step a: Obtain training samples, which include a small number of labeled videos and a large number of unlabeled videos. In this embodiment, the labeled videos account for 5% and the unlabeled videos account for 95%.

[0075] Step b: The labeled video is sampled using a two-level temporal element sampling method, and the labeled frame sequence samples obtained by sampling are used to obtain an initial model through supervised training.

[0076] The teacher model and the student model are initialized using the initial model; the iterative training is performed through the following steps, where the kth iteration process is:

[0077] Step c: Both labeled and unlabeled videos are sampled using a two-level temporal element modeling method, the labeled frame sequence samples obtained by sampling are weakly enhanced, and the unlabeled frame sequence samples obtained by sampling are weakly enhanced and strongly enhanced to obtain a batch of frame sequence samples; the batch of frame sequence samples includes the labeled frame sequence samples after weak enhancement and the unlabeled frame sequence samples after weak enhancement and strong enhancement.

[0078] Weak enhancement and strong enhancement are typical data enhancement methods in semi-supervised learning. In existing methods, both weak enhancement and strong enhancement are designed for images, but “strong enhancement” designed for images is not enough for video tasks because it completely ignores the development of temporal dynamics in the video. For fine-grained video tasks, temporal changes are more critical, requiring the model to pay extreme attention to temporal information, and traditional spatial enhancement often destroys key information presented in action. For this reason, in this embodiment, weak enhancement still uses conventional random horizontal flipping or random scaling, while for strong enhancement, moderate temporal perturbation is used to perturb the most critical part of the data that we want the model to pay attention to without affecting the semantic distinction of the action category. For this reason, we combine strong enhancement with two-level temporal element sampling, and arrange each group of fine-grained frame sequences in reverse order, while keeping the order of the context frame sequence unchanged to ensure the temporal directionality in the action, such as Figure 2 As shown, two groups of time series elements {f2, f6} and {f4, f8} with a length of 2 are strongly enhanced to {f6, f2} and {f8, f4}; while the context elements {f1, f3, f5, f7} remain unchanged.

[0079] In addition, a large number of timing perturbation methods are compared in this embodiment, and the results are shown in the subsequent tables. The results show that the strong enhancement strategy of this embodiment is better than the completely disrupted timing perturbation, and the moderate perturbation strategy we designed is better than other previous strategies, such as timing distortion.

[0080] Step d: Input the weakly enhanced labeled frame sequence samples into the teacher model for recognition, and obtain the recognition result and the loss L between the recognition result and the label sup ; The recognition result is the probability distribution of each possible category.

[0081] Step e: Input the weakly enhanced unlabeled frame sequence samples into the teacher model for recognition to obtain the recognition results.

[0082] Step f: Repeat step e N times to obtain N recognition results, and calculate the average probability μ of each category in the N recognition results. i As a pseudo label, i represents the i-th category.

[0083] Step g: Calculate the maximum value μ of the probability average of various categories * =max(μ i ), and μ * The probability standard deviation σ of the corresponding category in N recognition results * , for example, μ * corresponds to the fifth category, then take the standard deviation σ of the probability of the fifth category in the N recognition results * .

[0084] According to μ * and σ * , calculate the dynamic coefficients τ1 and τ2 to obtain the adaptive coefficient η, which is used to adjust the loss from the unlabeled frame sequence samples:

[0085]

[0086] Wherein, e is a natural constant, β and ε are set constants, β is related to the dropout of the model and is taken as 20 in this embodiment, ε is a stable parameter and is taken as 1e-9 in this embodiment; the adaptive coefficient η=τ1τ2 is calculated according to the dynamic coefficients τ1 and τ2.

[0087] Step h: Input the strongly enhanced unlabeled frame sequence samples into the student model for recognition to obtain the recognition results.

[0088] Step i: Use the recognition results and pseudo labels of the student model to obtain the loss L of the unsupervised learning process un .

[0089] Step j: Combine the strongly enhanced unlabeled frame sequence samples in the batch in pairs to obtain several pairs of unlabeled frame sequence samples, process each of the unlabeled frame sequence sample pairs by the tubemix method to obtain a new unlabeled frame sequence sample, input the new unlabeled frame sequence sample into the student model for recognition, and obtain the recognition result.

[0090] Step k: one of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η1 and the corresponding pseudo label ρ1; that is, one of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair replaces the weakly enhanced unlabeled frame sequence sample in step e and inputs it into the teacher model for recognition, and proceeds to step g to obtain the adaptive coefficient η1 and the corresponding pseudo label ρ1; and the other unlabeled frame sequence sample in the unlabeled frame sequence sample pair is also subjected to the process of step e-step g to obtain the adaptive coefficient η2 and the corresponding pseudo label ρ2; then according to the formula

[0091] η ′ =λη1+(1-λ)η2

[0092] ρ ′ =λρ1+(1-λ)ρ2

[0093] Get the mixed loss weight η ′ and the mixed pseudo-label ρ ′ , where λ is the set mixing ratio. In this embodiment, λ=Beta(10,10), where Beta() represents Beta distribution; using the recognition result of step j and the mixed pseudo label ρ ′ , we get the mixed loss L mix .

[0094] Step 1: Use the formula

[0095] L=L sup +ξ(ηL un +η′L mix )

[0096] The loss L of this iteration process is obtained, where ξ is the warmup coefficient, and k is the current iteration number, M n is the total number of iterations set.

[0097] Since the dropout mechanism is enabled in the model, the parameters used in each prediction in the teacher model are different (a random subset of all model parameters), and due to the high complexity of the fine-grained action recognition task, the model usually oscillates between categories with subtle differences, and the N recognition results in step f vary greatly. The greater the uncertainty of the teacher model's prediction, the less reliable its prediction. This unstable prediction of the teacher model will cause the student model to use ambiguous and invalid pseudo-labels, which will eventually cause serious damage to the entire learning and optimization process. To this end, in this embodiment, the learning optimization process is stabilized by adaptive adjustment. First, step f is used to allow the teacher model to generate N recognition results for the weakly enhanced unlabeled frame sequence samples. These recognition results may have large inconsistencies. Then, based on these recognition results, the probability average of each category in the N recognition results is calculated as the pseudo-label, as well as the maximum value of the probability average of various categories and the probability standard deviation of the category corresponding to the maximum value in the N recognition results to calculate the dynamic coefficients τ1 and τ2. Specifically, τ1 increases with the maximum value μ * On the other hand, τ2 suppresses unstable predictions, i.e., predictions with high standard deviation σ * The adaptive coefficient η = τ1τ2 obtained in this way is more flexible and beneficial than the predefined hyperparameters.

[0098] Step m: Use the loss L to adjust the student model and get the student model parameters after the kth iteration Reuse the adjusted student model parameters According to the formula Get the teacher model parameters after the kth iteration Then return to step c until the set number of iterations is reached and the model training is completed.

[0099] In summary, this embodiment targets the semi-supervised fine-grained action recognition task. When most of the input samples are unlabeled data, in the unsupervised learning stage, two-level temporal element sampling is adopted, and the data is enhanced by two methods, “weak enhancement” and “strong enhancement”. Specifically, the “strong enhancement” distorted samples with moderate temporal perturbations will be used as input to the student model, while the teacher model generates pseudo labels based on the “weak enhancement” samples. By minimizing the unsupervised loss (L un ) to enforce the consistency of model predictions. An adaptive adjustment mechanism is used to further adjust the unsupervised loss to stabilize the learning process. The entire framework is based on the supervised loss (L sup ) and the unsupervised loss (L un ), and mixed loss (L mix ) is used for optimization training.

[0100] Case analysis:

[0101] Next, the advantages of the method proposed in this invention are illustrated by comparing relevant examples:

[0102] Table 1 Comparison with the most advanced semi-supervised action recognition methods on fine-grained datasets

[0103]

[0104] As shown in Table 1, quantitative experiments are conducted on the fine-grained action recognition datasets Gym99, Gym288 and FineDiving for the SeFAR-B model (larger version model), SeFAR-S model (lightweight version) and three advanced semi-supervised action recognition models MemDPC, LTG and SVFormer proposed in the present invention.

[0105] The SeFAR model with the sampling combination {2-2-4} is used for the experiment. The main evaluation indicator is the Top-1 accuracy. In Table 1, "V" in the "Input" column represents RGB video and "G" represents temporal gradient. "ImgNet" means using a model pre-trained on ImageNet, and "#F" represents the number of input frames. The annotation ratios of the data annotations are "5%" and "10%" respectively. The best results are shown in bold and the second best results are underlined.

[0106] Experimental results show that the SeFAR framework we proposed significantly outperforms existing semi-supervised action recognition methods, whether on the Gym99, Gym288, or FineDiving datasets, especially in tasks of all semantic granularity. It is worth noting that the lightweight version of SeFAR-S has been able to surpass previous open source methods with a moderate number of parameters. When compared with larger models such as SVFormer, our larger version model SeFAR-B shows better performance. These results fully demonstrate the effectiveness and adaptability of the SeFAR framework in dealing with the complex task of fine-grained action recognition.

[0107] To further verify the robustness and generalization ability of SeFAR, we also conducted experiments on two classic coarse-grained action recognition datasets, UCF-101 and HMDB-51. As shown in Table 2, the average accuracy of SeFAR-B on the UCF-101 dataset is improved by about 3.3% compared with the baseline, and the average accuracy on the HMDB-51 dataset is improved by about 1.7%, both reaching the current state-of-the-art level. These results further prove that the SeFAR framework not only performs well on fine-grained tasks, but is also effective on traditional coarse-grained video semantic understanding tasks, once again proving the powerful performance and wide applicability of our proposed algorithm.

[0108] Table 2 Comparison with the most advanced semi-supervised action recognition methods on coarse-grained datasets

[0109]

[0110] In order to gain a deeper understanding of the proposed SeFAR framework, we conducted an ablation study on its core components, including dual-level temporal element sampling (Dual-Ele), moderate temporal perturbation (Mod-Perturb), and adaptive regulation (Ada-Reg). As shown in Table 3, these modules, as the basic components of SeFAR, have made significant contributions to the model performance.

[0111] Table 3 Ablation experiments of SeFAR core components

[0112]

[0113] In Table 3, “√” indicates that the component is included. In order to follow the consistency regularization principle in semi-supervised learning (SSL), we adopt a strong enhancement strategy consistent with that in SVFormer, namely, time series distortion, after removing Mod-Perturb.

[0114] In addition, we conducted a comprehensive analysis of the design and selection of each strategy or module, as follows:

[0115] Two-level sequential element sampling analysis

[0116] We first conducted comparative experiments on different temporal element sampling combinations, testing different temporal lengths of each context element (e.g., 4, 6, and 8 frames). For ease of comparison, the length of the fine-grained temporal element is fixed to 2, consistent with the default setting {2-2-4}. The experimental results are shown in Figure 3(a). It is worth noting that even with only 6 frames (i.e., {2-4}) as input, SeFAR is already able to surpass the performance of the baseline model SVFormer with 8 frames of input. This result shows that two-level temporal element sampling can effectively capture rich detail information in video data, thereby helping the model to better distinguish subtle differences between fine-grained actions. Further experiments show that increasing the number of fine-grained elements (e.g., {2-2-2-4}) or extending the temporal length of context elements (e.g., {2-2-6} and {2-2-8}) can improve model performance. This is because more frames provide the model with richer action information, thereby further enhancing the ability of fine-grained action recognition.

[0117] Moderate Timing Perturbation Analysis

[0118] To verify the effectiveness of moderate timing perturbation (Mod-Perturb), we selected 40 pairs of action categories that are reversed in time from the FineGym dataset (e.g., "backward big loop" and "forward big loop") and constructed a subset called Gym-New to evaluate the performance of the model on fine-grained actions. As shown in Table 4, SeFAR also maintains superior performance on these actions. Subsequently, we compared Mod-Perturb with other timing perturbation strategies (e.g., slow rate perturbation (Singh et al., 2021), timing distortion (Xing et al., 2023), T-Drop and T-Half (Zou et al., 2023), and the results are also listed in Table 4. Experimental results show that: 1) Mod-Perturb shows significant advantages in retaining temporal information and enhancing stability compared with other temporal enhancement strategies and strategies that only perform spatial enhancement; 2) The strategy that only performs spatial enhancement has poor results on Gym99, but outperforms most temporal enhancement strategies on Gym-New. This shows that for more complex datasets, maintaining accurate temporal information is crucial, while in larger and more diverse datasets, reasonable temporal perturbations can enhance the stability of the model. The Mod-Perturb strategy we proposed takes into account both the retention of temporal information and the rationality of perturbations, thus showing good performance on different datasets.

[0119] Table 4 Comparative experiments of different timing enhancement strategies

[0120]

[0121] In Table 4, “Spe.” indicates the focusing speed of the strategy, and “Ord.” indicates the focusing order.

[0122] Adaptive Regulation Analysis

[0123] We further verified the effectiveness of the stability coefficient design in the adaptive regulation module (Ada-Reg) by analyzing the following two experiments:

[0124] 1. Comparison with fixed threshold strategy: We compare adaptive adjustment with the fixed threshold strategy commonly used in classical semi-supervised learning (SSL) methods. Figure 3 As shown in (b), adaptive regulation outperforms the fixed threshold strategy in terms of stability and performance.

[0125] 2. Analysis of teacher model prediction distribution: In order to further explore the design motivation of Ada-Reg, we randomly selected 30 data samples from multiple datasets (UCF101, HMDB51, and FineGym), let the teacher model make multiple predictions, and analyze the stability of the prediction distribution. Figure 3 As shown in (c), the predictions on coarse-grained data (such as UCF101 and HMDB51) vary little, while the predictions on fine-grained data (such as FineGym) fluctuate greatly. This result shows that the predictions of the teacher model in fine-grained action recognition tasks are often unstable, and the adaptive adjustment module can effectively alleviate this problem, thereby improving the performance of the model on fine-grained tasks.

[0126] SeFAR feature analysis

[0127] To further verify the potential of SeFAR in enhancing the capabilities of multimodal large language models (MLLMs), we constructed a dataset called Gym-QA based on FineGym converted into multiple-choice questions (such as Figure 1 As shown in Table 5, the SeFAR-enhanced model achieves significant performance improvement on the Gym-QA task compared with the commonly used MLLM visual encoder. This shows that SeFAR can not only assist the multimodal large model (MLLM) in coping with the challenges of fine-tuning in low-annotation scenarios, but also demonstrates its potential in enhancing the visual perception and modeling capabilities of multimodal large models.

[0128] Table 5 Comparative experiments on large pre-trained multimodal model visual encoders

[0129]

[0130] The above examples demonstrate the superior performance of the proposed SeFAR framework in enhancing the fine-grained visual understanding capabilities of the multimodal large model (MLLM). SeFAR not only significantly outperforms all baseline methods on representative fine-grained video action datasets Gym99, Gym288, and FineDiving, but also achieves new state-of-the-art results on classic benchmark human action recognition datasets (i.e., UCF101 and HMDB51).

[0131] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization, characterized by: The following steps are involved: Step 1: The video to be identified is sampled using a two-level temporal element sampling method; Step 2: Input the frame sequence sampled in step 1 into the trained model to obtain the recognition result; The model is trained through the following process: Step a: obtaining training samples, wherein the training samples include a small number of labeled videos and a large number of unlabeled videos; Step b: The labeled video is sampled using a two-level temporal element sampling method, and the labeled frame sequence samples obtained by sampling are used to obtain an initial model through supervised training; Initialize the teacher model and the student model using the initial model; Iterative training is performed through the following steps, where the kth iteration process is: Step c: sampling both labeled and unlabeled videos using a two-level temporal element modeling method, weakly enhancing the labeled frame sequence samples obtained by sampling, and weakly enhancing and strongly enhancing the unlabeled frame sequence samples obtained by sampling, to obtain a batch of frame sequence samples; the batch of frame sequence samples includes the labeled frame sequence samples after weak enhancement and the unlabeled frame sequence samples after weak enhancement and strong enhancement; Step d: Input the weakly enhanced labeled frame sequence samples into the teacher model for recognition, and obtain the recognition result and the loss L between the recognition result and the label sup ; The recognition result is a probability distribution of various possible categories; Step e: Input the weakly enhanced unlabeled frame sequence samples into the teacher model for recognition to obtain the recognition results; Step f: Repeat step e N times to obtain N recognition results, and calculate the average probability μ of each category in the N recognition results. i As a pseudo-label, i represents the i-th category; Step g: Calculate the maximum value μ of the probability average of various categories * =max(μ i ), and μ * The probability standard deviation σ of the corresponding category in N recognition results * ; According to μ * and σ * , calculate the dynamic coefficients τ1 and τ2: τ1=sigmoid(e μ* -e) Among them, e is a natural constant, β and ε are set constants; the adaptive coefficient η=τ1τ2 is calculated according to the dynamic coefficients τ1 and τ2; Step h: Input the strongly enhanced unlabeled frame sequence samples into the student model for recognition to obtain the recognition results; Step i: Use the recognition results and pseudo labels of the student model to obtain the loss L of the unsupervised learning process un ; Step j: Combine the strongly enhanced unlabeled frame sequence samples in the batch in pairs to obtain several pairs of unlabeled frame sequence sample pairs, process each of the unlabeled frame sequence sample pairs by the Tube TokenMix method to obtain a new unlabeled frame sequence sample, input the new unlabeled frame sequence sample into the student model for recognition, and obtain a recognition result; Step k: One of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η1 and the corresponding pseudo label ρ1; the other unlabeled frame sequence sample in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η2 and the corresponding pseudo label ρ2; according to the formula or ′ =λη1+(1-λ)η2 r ′ =λρ1+(1-λ)ρ2 Get the mixed loss weight η ′ and the mixed pseudo-label ρ ′ , where λ is the set mixing ratio; using the recognition result of step j and the mixed pseudo label ρ ′ , we get the mixing loss L mix ; Step 1: Use the formula L=L sup +ξ(ηL un +η′L mix ) Get the loss L of this iteration process, where ξ is the warmup coefficient; Step m: Use the loss L to adjust the student model and get the student model parameters after the kth iteration Reuse the adjusted student model parameters According to the formula Get the teacher model parameters after the kth iteration Then return to step c until the set total number of iterations is reached and the model training is completed.

2. According to claim 1, a semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization is characterized by: The two-stage sequential element sampling is: Sampling the video in segments to obtain K frames of images; Densely sampling the K frame images with a relatively small sampling step to obtain a set of context frame sequences that reflect global temporal information; The K frame images are sparsely sampled with a relatively large sampling step to obtain multiple groups of fine-grained frame sequences that reflect detail information.

3. A semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1 or 2, characterized in that: In step a, the small amount of labeled videos and the large amount of unlabeled videos means that the labeled videos account for no more than 10% and no less than 1%.

4. According to claim 2, a semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization is characterized by: The strong enhancement is a moderate timing disturbance.

5. A semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 4, characterized in that: The moderate timing perturbation is to reverse the order of each group of fine-grained frame sequences, while the order of the context frame sequences remains unchanged.

6. The semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1, characterized in that: In step g, β is related to the dropout of the model, and ε is a stable parameter.

7. The semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1, characterized in that: In step k, λ=Beta(10,10), where Beta() represents Beta distribution.

8. The semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1, characterized in that: In step 1, the warmup coefficient k is the current iteration number, M n is the total number of iterations set.

Images