Semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization

Abstract

The application provides a semi-supervised fine-grained action recognition method based on time sequence disturbance and learning stabilization, and aims at the highly challenging semi-supervised fine-grained action recognition task, and the method is based on the FixMatch semi-supervised learning system, and a semi-supervised learning fine-grained action recognition algorithm framework SeFAR is provided. The method provides a two-stage time sequence element sampling method, combines fine-grained time sequence elements and time sequence contexts, can effectively capture multi-granularity time sequence information, and can enhance the ability to distinguish fine actions in the video; and the importance of time sequence dynamics is considered, and a new "strong enhancement" strategy is designed, moderate time sequence disturbance is introduced in the obtained fine-grained time sequence elements, and the time sequence order of the context elements is maintained, so that weak-strong enhancement contrast data pairs more suitable for the fine-grained action recognition task are constructed; in addition, the method also designs an adaptive learning adjustment strategy, and an adaptive coefficient is calculated to adjust the loss of back propagation in training, so that the training process is stabilized.

Description

Technical Field

[0001] The present invention relates to the field of action recognition and deep learning technology, and specifically to a semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization. Background Art

[0002] With the widespread adoption of 5G internet and the rapid development of artificial intelligence (AI), massive amounts of video data are being generated. Video understanding, owing to its vivid visual information and rich temporal dynamics, has attracted increasing attention and attracted a growing number of researchers to this field. Action recognition, as one of the core issues in video understanding, holds broad application prospects in surveillance systems, human-computer interaction, and video retrieval. In recent years, thanks to advances in deep learning techniques and the construction of large-scale video datasets, supervised action recognition research has made significant progress. Several powerful multimodal large language models (MLLMs) have been proposed, such as GPT-4v (OpenAI 2024); VideoChat (Li et al. 2024b); Video-LLaVA (Lin et al. 2023); and InternLM (Zhang et al. 2024), aiming to achieve more general and comprehensive capabilities. These large multimodal models can achieve very good recognition results in classic action recognition. However, classic action recognition only requires the model to provide relatively coarse-grained categories, such as "gymnastics". For fine-grained action recognition that focuses on more specific task scenarios and aims to provide more detailed, specific and semantically accurate descriptions, such as "legs together and push the bar backward loop - handstand 180 degrees", the current mainstream large multimodal models perform poorly and fail to correctly recognize the fine-grained semantics of a given action. Figure 1 shown. Summary of the Invention

[0003] Technical issues to be solved:

[0004] Currently, research on fine-grained action recognition is very limited. The main reasons are:

[0005] (1) The construction of fine-grained action datasets is extremely complex. For example, providing annotations for actions such as "exchange legs, splits, jump, and turn 360 degrees" requires sufficient expert knowledge, extremely long annotation time, and a lot of inspection work to ensure quality. This leads to the scarcity of datasets with fine-grained labels, making it very difficult to directly retrain or fine-tune large models using supervised learning.

[0006] (2) To address the scarcity of labeled datasets, the traditional approach is to use semi-supervised learning to improve model performance by providing stable and reasonable supervision (pseudo-labels) for unlabeled data. However, fine-grained action recognition requires sufficient visual details, effective information aggregation, and a comprehensive understanding of temporal dynamics. Therefore, this task itself is very challenging. Human annotators find it difficult to distinguish fine-grained actions, resulting in the pseudo-labels generated by traditional approaches being highly unreliable, which in turn makes the entire semi-supervised learning process very prone to failure.

[0007] The technical solution of the present invention:

[0008] To solve the problems existing in the prior art and achieve accurate recognition of fine-grained actions, the present invention proposes a semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization, which specifically includes the following steps:

[0009] Step 1: The video to be identified is sampled using a two-stage temporal element sampling method;

[0010] Step 2: Input the frame sequence sampled in step 1 into the trained model to obtain the recognition result;

[0011] The model is trained through the following process:

[0012] Step a: Obtain training samples, which include a small number of labeled videos and a large number of unlabeled videos;

[0013] Step b: Sample the labeled video using a two-level temporal element sampling method, and use the labeled frame sequence samples obtained through supervised training to obtain an initial model;

[0014] Initialize the teacher model and student model using the initial model; perform iterative training through the following steps, where the kth iteration process is:

[0015] Step c: sampling both labeled and unlabeled videos using a two-level temporal element modeling method, performing weak enhancement on the labeled frame sequence samples obtained by sampling, and performing weak enhancement and strong enhancement on the unlabeled frame sequence samples obtained by sampling, to obtain a batch of frame sequence samples; the batch of frame sequence samples includes the labeled frame sequence samples after weak enhancement and the unlabeled frame sequence samples after weak enhancement and strong enhancement;

[0016] Step d: Input the weakly enhanced labeled frame sequence samples into the teacher model for recognition, and obtain the recognition results and the loss L between the recognition results and the labels. sup ; The recognition result is a probability distribution of various possible categories;

[0017] Step e: Input the weakly enhanced unlabeled frame sequence samples into the teacher model for recognition to obtain the recognition results;

[0018] Step f: Repeat step e N times to obtain N recognition results, and calculate the average probability μ of each category in the N recognition results. i As a pseudo label, i represents the i-th category;

[0019] Step g: Calculate the maximum value μ of the probability average of various categories * =max(μ i ), and μ * The probability standard deviation σ of the corresponding category in N recognition results * ;

[0020] According to μ * and σ * , calculate the dynamic coefficients τ1 and τ2:

[0021]

[0022] Where e is a natural constant, β and ε are set constants; the adaptive coefficient η = τ1τ2 is calculated based on the dynamic coefficients τ1 and τ2;

[0023] Step h: Input the strongly enhanced unlabeled frame sequence samples into the student model for recognition to obtain the recognition results;

[0024] Step i: Using the recognition results and pseudo labels of the student model, we can get the loss L of the unsupervised learning process. un ;

[0025] Step j: Combine the strongly enhanced unlabeled frame sequence samples in the batch in pairs to obtain several pairs of unlabeled frame sequence samples, process each of the unlabeled frame sequence sample pairs using the Tube TokenMix (TTMix) method to obtain a new unlabeled frame sequence sample, input the new unlabeled frame sequence sample into the student model for recognition, and obtain the recognition result;

[0026] Step k: One of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η1 and the corresponding pseudo label ρ1; the other unlabeled frame sequence sample in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η2 and the corresponding pseudo label ρ2; according to the formula

[0027] η ′ =λη1+(1-λ)η2

[0028] ρ ′ =λρ1+(1-λ)ρ2

[0029] Get the mixed loss weight η ′ and the mixed pseudo-label ρ ′ , where λ is the set mixing ratio; using the recognition result of step j and the mixed pseudo label ρ ′ , we get the mixing loss L mix ;

[0030] Step 1: Use the formula

[0031] L=L sup +ξ(ηL un +η′L mix )

[0032] Get the loss L of this iterative process, where ξ is the warmup coefficient;

[0033] Step m: Use the loss L to adjust the student model and obtain the student model parameters after the kth iteration Reusing the adjusted student model parameters According to the formula Get the teacher model parameters after the kth iteration Then return to step c until the set total number of iterations is reached and the model training is completed.

[0034] In a further preferred embodiment, the two-stage sequential element sampling is as follows:

[0035] Sampling the video in segments to obtain K frames of images;

[0036] Densely sampling the K frames of image data with a relatively small sampling step size to obtain a set of context frame sequences reflecting global temporal information;

[0037] The K frames of images are sparsely sampled with a relatively large sampling step size to obtain multiple groups of fine-grained frame sequences that reflect detail information.

[0038] In a further preferred embodiment, in step a, the small number of labeled videos and the large number of unlabeled videos mean that the labeled videos account for no more than 10% and no less than 1%.

[0039] In a further preferred embodiment, the strong enhancement is a moderate timing disturbance.

[0040] In a further preferred solution, the moderate temporal perturbation is to reverse the order of each group of fine-grained frame sequences, while the order of the context frame sequences remains unchanged.

[0041] In a further preferred embodiment, in step g, β is related to the dropout of the model, and ε is a stable parameter.

[0042] Further preferably, in step k, lambda = Beta(10, 10) is taken, where Beta() represents a Beta distribution.

[0043] Further preferably, in step l, the warmup coefficient k is the current iteration number, M n is the total iteration number set.

[0044] Advantages

[0045] The present application has the following advantages:

[0046] 1. The present application first proposes a semi-supervised learning fine-grained action recognition algorithm framework SeFAR.

[0047] 2. The present application proposes a two-stage time sequence element sampling method, which combines fine-grained time sequence elements and time sequence context, and can effectively capture multi-granularity time sequence information to enhance the ability to distinguish subtle actions in videos.

[0048] 3. The present application considers the importance of time sequence dynamics and designs a new "strong enhancement" strategy, which introduces moderate time sequence disturbance in the obtained fine-grained time sequence elements while maintaining the time sequence order of the context elements, thereby constructing weak-strong enhancement contrast data pairs more suitable for fine-grained action recognition tasks.

[0049] 4. The present application designs an adaptive learning adjustment strategy to provide reliable pseudo labels for unlabeled data when the teacher model prediction is unstable, and adjusts the loss in the training by calculating an adaptive coefficient to stabilize the training process.

[0050] The present application proposes a semi-supervised learning fine-grained action recognition algorithm framework SeFAR, which achieves the most advanced performance on fine-grained (FineGym, FineDiving) and coarse-grained action recognition datasets (UCF101, HMDB51), proving its effectiveness. And SeFAR can be embedded as a powerful visual encoder into the current powerful large multi-modal base model to improve the performance of the current MLLM in specific domain scenarios.

[0051] Additional aspects and advantages of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings, in which:

[0053] Figure 1 : existing multi-modal large model fine-grained action recognition instance;

[0054] Figure 2 SeFAR algorithm framework flowchart proposed by the present application;

[0055] Figure 3 Multiple ablation experiment visualization schematic diagram;

[0056] (a) Comparison of the influence of different sampling combinations on the performance of SeFAR-B under the condition of 5% annotation rate in Gym-99 dataset; With the increase of the number of fine-grained frame sequences, the model accuracy is constantly rising; With the increase of the number of context frame sequences, the model accuracy is constantly rising; It shows that the two-level time sequence element sampling proposed by the present application can improve the model performance by enriching fine-grained information and global information;

[0057] (b) Comparison of the performance of fixed threshold method and the adaptive adjustment strategy proposed by the present application under 5% annotation rate in FineDiving dataset; Compared with different fixed thresholds, the adaptive adjustment method proposed by the present application has better stability and higher accuracy;

[0058] (c) The prediction result fluctuation of the teacher model on different datasets is shown, and it can be seen that compared with the coarse-grained action recognition dataset, the prediction fluctuation of the teacher model on the fine-grained action recognition dataset FineGym(blue line) is significantly violent, thus illustrating the necessity of the adaptive adjustment weight according to the teacher model. DETAILED DESCRIPTION

[0059] The embodiments of the present application are described in detail below, which are exemplary and intended to explain the present application, and cannot be understood as a limitation of the present application.

[0060] Figure 1The current mainstream four powerful multimodal large models MLMs: GPT-4v (OpenAI 2024); VideoChat (Li et al. 2024b); Video-LLaVA (Lin et al. 2023); InternLM (Zhang et al. 2024) are shown on the results of coarse-grained and fine-grained action recognition. The upper part of the figure shows two fine-grained video action samples selected from the FineGym dataset (Shao et al., 2020) to present the subtle differences of fine-grained action recognition. Among them, the top video sample example is "leg press back to the ring - hand handstand 180 degrees", and the bottom sample is "leg press back to the ring - hand handstand 360 degrees". We use the bottom video action instance as a test sample to evaluate the performance of the four multimodal large models on coarse-grained and fine-grained tasks, where the first question focuses on coarse-grained semantics, and the second question focuses on fine-grained semantics. It can be seen that the first question, the four multimodal large models all answer correctly, and for the second question focusing on fine-grained semantics, the four multimodal large models all fail to correctly identify the fine-grained semantics of the given action.

[0061] In order to explore the highly challenging task of semi-supervised fine-grained action recognition, this embodiment proposes a semi-supervised learning fine-grained action recognition algorithm framework SeFAR based on the FixMatch semi-supervised learning system, designs a new enhancement strategy, and optimizes the original teacher-student learning mode according to the new task. The specific algorithm framework is shown in Figure 2 , which includes the following steps:

[0062] Step 1: Use a two-level time sequence element sampling method to sample the video to be recognized.

[0063] Two-level time sequence element sampling is one of the innovations of the present application, which is used to distinguish the subtle differences of fine-grained actions and to mine rich and detailed visual information. The specific process is as follows:

[0064] The fine-grained action video is divided into K segments and a frame is randomly sampled in each segment to obtain a frame sequence {f1,…,f K}.

[0065] Since most of the visual content (such as scenes and objects) in fine-grained action recognition has high similarity, the model needs to perceive subtle changes and rich details to achieve accurate distinction. Therefore, in this embodiment, sparse sampling is performed on the frame sequence {f1,…,f K} composed of K frames of images with a relatively large sampling step to construct multiple groups of time sequence elements p i (i.e. multiple groups of fine-grained frame sequences), and each time sequence element p iThe length L (i.e., the number of frames) is moderate to help the model focus on rapid and subtle changes. After M times of sparse sampling, M groups of time series elements with the same time series length are obtained, which can be expressed as:

[0066] where |p i |=L

[0067] In addition to the above-mentioned fine-grained temporal elements, this embodiment also proposes a frame sequence {f1,…,f K} is densely sampled with a relatively small sampling step size to construct the context element p that reflects the global temporal information context (i.e., context frame sequence) to encode long-term information and macro-temporal dynamics. Since dense sampling is performed with a relatively small sampling step, p context Consists of more frames, usually fine-grained temporal elements p i more than twice.

[0068] This two-level temporal element sampling ensures the preservation of multi-level and multi-granular information, thereby obtaining an effective representation of the input fine-grained action video, denoted as {p1,…,p M ,p context}.

[0069] like Figure 2 As shown, in this embodiment, the fine-grained action video is divided into 8 segments and a frame is randomly sampled in each segment to obtain a frame sequence {f1, ..., f8} consisting of 8 frames.

[0070] After performing sparse sampling twice with a sampling step of 4 on the frame sequence {f1,…,f8}, two groups of time series elements with a length of 2 are obtained, namely {f2,f6} and {f4,f8}.

[0071] The context elements {f1, f3, f5, f7} are obtained by densely sampling the frame sequence {f1, ..., f8} with a sampling step of 2.

[0072] Step 2: Input the frame sequence sampled in step 1 into the trained model to obtain the recognition result;

[0073] The model is trained through the following process:

[0074] Step a: Obtain training samples, which include a small number of labeled videos and a large number of unlabeled videos. In this embodiment, the labeled videos account for 5% and the unlabeled videos account for 95%.

[0075] Step b: The labeled video is sampled using a two-level temporal element sampling method, and the labeled frame sequence samples obtained by sampling are used to obtain an initial model through supervised training.

[0076] Initialize the teacher model and student model using the initial model; perform iterative training through the following steps, where the kth iteration process is:

[0077] Step c: Both labeled and unlabeled videos are sampled using a two-level temporal element modeling method, the labeled frame sequence samples obtained by sampling are weakly enhanced, and the unlabeled frame sequence samples obtained by sampling are weakly enhanced and strongly enhanced to obtain a batch of frame sequence samples; the batch of frame sequence samples includes the labeled frame sequence samples after weak enhancement and the unlabeled frame sequence samples after weak enhancement and strong enhancement.

[0078] Weak enhancement and strong enhancement are typical data enhancement methods in semi-supervised learning. In existing methods, both weak enhancement and strong enhancement are designed for images, but “strong enhancement” designed for images is not enough for video tasks because it completely ignores the development of temporal dynamics in the video. For fine-grained video tasks, temporal changes are more critical, requiring the model to pay extreme attention to temporal information, and traditional spatial enhancement often destroys key information presented in action. For this reason, in this embodiment, weak enhancement still uses conventional random horizontal flipping or random scaling, while for strong enhancement, moderate temporal perturbation is used, which not only perturbs the most critical part of the data that we want the model to pay attention to, but also does not affect the semantic distinguishability of the action category. For this reason, we combine strong enhancement with two-level temporal element sampling, and arrange each group of fine-grained frame sequences in reverse order, while keeping the order of the context frame sequences unchanged to ensure the temporal directionality in the action, such as Figure 2 As shown in Figure 2, two groups of time series elements {f2, f6} and {f4, f8} with a length of 2 are strongly enhanced to {f6, f2} and {f8, f4}; while the context elements {f1, f3, f5, f7} remain unchanged.

[0079] In addition, this embodiment compares a large number of timing perturbation methods, and the results are shown in the subsequent tables. The results show that the strong enhancement strategy of this embodiment is more effective than the completely disrupted timing perturbation, and the moderate perturbation strategy we designed is better than other previous strategies, such as timing distortion.

[0080] Step d: Input the weakly enhanced labeled frame sequence samples into the teacher model for recognition, and obtain the recognition results and the loss L between the recognition results and the labels. sup ; The recognition result is the probability distribution of each possible category.

[0081] Step e: Input the weakly enhanced unlabeled frame sequence samples into the teacher model for recognition to obtain the recognition results.

[0082] Step f: Repeat step e N times to obtain N recognition results, and calculate the average probability μ of each category in the N recognition results. i As a pseudo label, i represents the i-th category.

[0083] Step g: Calculate the maximum value μ of the probability average of various categories * =max(μ i ), and μ * The probability standard deviation σ of the corresponding category in N recognition results * , for example, μ * It corresponds to the fifth category, then take the standard deviation σ of the probability of the fifth category in N recognition results * .

[0084] According to μ * and σ * , calculate the dynamic coefficients τ1 and τ2 to obtain the adaptive coefficient η, which is used to adjust the loss from the unlabeled frame sequence samples:

[0085]

[0086] Wherein, e is a natural constant, β and ε are set constants, β is related to the dropout of the model and is taken as 20 in this embodiment, and ε is a stable parameter and is taken as 1e-9 in this embodiment; the adaptive coefficient η = τ1τ2 is calculated based on the dynamic coefficients τ1 and τ2.

[0087] Step h: Input the strongly enhanced unlabeled frame sequence samples into the student model for recognition to obtain the recognition results.

[0088] Step i: Using the recognition results and pseudo labels of the student model, we can get the loss L of the unsupervised learning process. un .

[0089] Step j: Combine the strongly enhanced unlabeled frame sequence samples in the batch in pairs to obtain several pairs of unlabeled frame sequence samples, process each of the unlabeled frame sequence sample pairs using the tubemix method to obtain a new unlabeled frame sequence sample, input the new unlabeled frame sequence sample into the student model for recognition, and obtain the recognition result.

[0090] Step k: one of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair is subjected to the process of step e-step g to obtain η1 and the corresponding pseudo label ρ1; that is, one of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair is replaced with the weakly enhanced unlabeled frame sequence sample in step e and input into the teacher model for recognition, and proceed to step g to obtain the adaptive coefficient η1 and the corresponding pseudo label ρ1; and the other unlabeled frame sequence sample in the unlabeled frame sequence sample pair is also subjected to the process of step e-step g to obtain the adaptive coefficient η2 and the corresponding pseudo label ρ2; then according to the formula

[0091] η ′ =λη1+(1-λ)η2

[0092] ρ ′ =λρ1+(1-λ)ρ2

[0093] Get the mixed loss weight η ′ and the mixed pseudo-label ρ ′ , where λ is the set mixing ratio. In this embodiment, λ=Beta(10,10), where Beta() represents Beta distribution. The recognition result of step j is combined with the mixed pseudo label ρ ′ , we get the mixing loss L mix .

[0094] Step 1: Use the formula

[0095] L=L sup +ξ(ηL un +η′L mix )

[0096] Get the loss L of this iteration process, where ξ is the warmup coefficient, take k is the current iteration number, M n is the total number of iterations set.

[0097] Because of the dropout mechanism in the model, the parameters used in the teacher model are different each time the prediction is made (a random subset of all model parameters), and because of the high complexity of the fine-grained action recognition task, the model often wavers between classes with subtle differences, and the N recognition results in step f have large differences. The greater the uncertainty of the teacher model prediction, the more unreliable the prediction. Such unstable prediction of the teacher model will lead the student model to use ambiguous and invalid pseudo-labels, ultimately causing the entire learning and optimization process to be severely damaged. Therefore, in this embodiment, the stable learning optimization process is adjusted adaptively. First, the teacher model generates N recognition results for the weakly enhanced unlabeled frame sequence sample in step f. These recognition results may have large inconsistencies. Then, according to these recognition results, the probability average of each class in the N recognition results is calculated as a pseudo-label, and the maximum value of the probability average of each class and the probability standard deviation of the class corresponding to the maximum value in the N recognition results are calculated to calculate dynamic coefficients τ1 and τ2. Specifically, τ1 rises rapidly with the increase of the maximum value μ * , which enhances the high-confidence prediction, and on the other hand, τ2 suppresses unstable predictions, that is, predictions with high standard deviation σ * . The adaptive coefficient η = τ1τ2 obtained in this way is more flexible and beneficial than the predefined hyperparameter.

[0098] Step m: adjust the student model using the loss L to obtain the student model parameters after the kth iteration Then use the adjusted student model parameters According to the formula the teacher model parameters after the kth iteration are obtained Then return to step c until the set number of iterations is reached, and the model training is complete.

[0099] In summary, for the semi-supervised fine-grained action recognition task, in the case of most input samples being unlabeled data, in the unsupervised learning phase, two-level time sequence element sampling is used, and the data is enhanced in two ways, "weak enhancement" and "strong enhancement". Specifically, the "strong enhancement" distorted sample with moderate time sequence disturbance will be used as the input to the student model, and the teacher model will generate pseudo-labels based on the "weak enhancement" sample. The consistency of the model prediction is forced to be maintained by minimizing the unsupervised loss (L un ). And an adaptive adjustment mechanism is used to further adjust the unsupervised loss to stabilize the learning process. The entire framework is optimized and trained through the weighted combination of the supervised loss (L sup ), the unsupervised loss (L un ), and the hybrid loss (L mix ).

[0100] Example analysis:

[0101] Next, the superiority of the method proposed in the application is illustrated by relevant examples:

[0102] Table 1 Comparison with the most advanced semi-supervised action recognition methods on fine-grained datasets

[0103]

[0104] As shown in Table 1, quantitative experiments were conducted on the fine-grained action recognition datasets Gym99, Gym288 and FineDiving for the SeFAR-B model (a larger version model), the SeFAR-S model (a lightweight version) proposed in the application and three advanced semi-supervised action recognition models MemDPC, LTG and SVFormer.

[0105] Here, the SeFAR model using the sampling combination {2-2-4} was used for experiments. The main evaluation index was Top-1 accuracy. In Table 1, “V” in the “input” column represents RGB video, and “G” represents temporal gradient. “ImgNet” indicates that a model pre-trained on ImageNet is used, and “#F” indicates the number of input frames. The annotation ratio of data annotation is “5%” and “10%” respectively. The best result is in bold, and the second best result is underlined.

[0106] The experimental results show that, whether on the Gym99, Gym288 or FineDiving dataset, the SeFAR framework proposed in the application is significantly superior to existing semi-supervised action recognition methods, especially achieving a substantial performance improvement on all semantic granularity tasks. It is worth noting that the lightweight version SeFAR-S has already surpassed previous open-source methods with a moderate parameter amount. When compared with larger models such as SVFormer, the larger version model SeFAR-B of the application exhibits better performance. These results fully demonstrate the effectiveness and adaptability of the SeFAR framework in dealing with the complex task of fine-grained action recognition.

[0107] To further verify the robustness and generalization ability of SeFAR, we also conduct experiments on two classical coarse-grained action recognition datasets, UCF-101 and HMDB-51. As shown in Table 2, SeFAR-B improves the average accuracy by about 3.3% on the UCF-101 dataset and by about 1.7% on the HMDB-51 dataset, reaching the current state-of-the-art level. These results further demonstrate that the SeFAR framework not only performs well on fine-grained tasks but also has effectiveness on traditional coarse-grained video semantic understanding tasks, again proving the strong performance and broad applicability of the proposed algorithm.

[0108] Table 2 Comparison with state-of-the-art semi-supervised action recognition methods on coarse-grained datasets

[0109]

[0110] To further understand the proposed SeFAR framework, we conduct ablation studies on its core components, including Dual-Ele, Mod-Perturb, and Ada-Reg. As shown in Table 3, these modules, as the basic components of SeFAR, have significant contributions to the model performance.

[0111] Table 3 Ablation experiments on core components of SeFAR

[0112]

[0113] In Table 3, "√" indicates that the component is included. To follow the consistency regularization principle in semi-supervised learning (SSL), we use the same strong augmentation strategy as in SVFormer, i.e., temporal distortion, after removing Mod-Perturb.

[0114] In addition, we conduct a comprehensive analysis of the design and selection of each strategy or module, as follows:

[0115] Analysis of Dual-Ele

[0116] We first conduct comparative experiments on different combinations of temporal element sampling, testing different temporal lengths of each contextual element (e.g., 4, 6, and 8 frames). For ease of comparison, the length of fine-grained temporal elements is fixed at 2, consistent with the default setting {2-2-4}. The experimental results are as follows: Figure 3(a) as shown. Notably, even with only 6 frames (i.e., {2-4}) input, SeFAR has already outperformed the baseline model SVFormer with 8 frames input. This result indicates that the dual-level temporal element sampling can effectively capture the rich detailed information in the video data, thus helping the model to better distinguish the subtle differences between fine-grained actions. Further experiments show that increasing the number of fine-grained elements (e.g., {2-2-2-4}) or lengthening the temporal length of the contextual elements (e.g., {2-2-6} and {2-2-8}) can all improve the model performance. This is because more frames provide the model with richer action information, thus further enhancing the ability of fine-grained action recognition.

[0117] Moderate temporal perturbation analysis

[0118] To verify the effectiveness of the moderate temporal perturbation (Mod-Perturb), we selected 40 pairs of action categories that are inversely ordered in time from the FineGym dataset (e.g., “Big Backward Circle” and “Big Forward Circle”), and constructed a subset named Gym-New to evaluate the model’s performance on fine-grained actions. As shown in Table 4, SeFAR also maintains superior performance on these actions. Subsequently, we compared Mod-Perturb with other temporal perturbation strategies (e.g., slow rate perturbation (Singh et al., 2021), temporal twist (Xing et al., 2023), T-Drop and T-Half (Zou et al., 2023)), and the results are also listed in Table 4. Experimental results show that: 1) Compared with other temporal enhancement strategies and strategies that only perform spatial enhancement, Mod-Perturb performs significantly better in terms of temporal information preservation and enhancement stability; 2) Strategies that only perform spatial enhancement perform poorly on Gym99, but outperform most temporal enhancement strategies on Gym-New. This indicates that for more complex datasets, it is crucial to maintain accurate temporal information, while in larger and more diverse datasets, reasonable temporal perturbation can enhance the stability of the model. The Mod-Perturb strategy we propose balances the preservation of temporal information and the reasonableness of perturbation, thus performing well on different datasets.

[0119] Table 4 Comparison of different temporal enhancement strategies

[0120]

[0121] In Table 4, “Spe.” represents the strategy focusing on speed, and “Ord.” represents focusing on order.

[0122] Adaptive adjustment analysis

[0123] We further validate the effectiveness of the stability coefficient design in the adaptive regulation module (Ada-Reg) through the following two experiments:

[0124] 1. Comparison with fixed threshold strategy: We compare the adaptive regulation with the fixed threshold strategy commonly used in classical semi-supervised learning (SSL) methods. As shown in Fig. Figure 3 (b), the adaptive regulation outperforms the fixed threshold strategy in both stability and performance.

[0125] 2. Analysis of teacher model prediction distribution: To further explore the design motivation of Ada-Reg, we randomly select 30 data samples from multiple datasets (UCF101, HMDB51, and FineGym) and let the teacher model make multiple predictions, and analyze the stability of the prediction distribution. As shown in Fig. Figure 3 (c), the prediction changes less on coarse-grained data (such as UCF101 and HMDB51), while the prediction fluctuates more on fine-grained data (such as FineGym). This result shows that the teacher model's prediction in fine-grained action recognition tasks is often unstable, and the adaptive regulation module can effectively alleviate this problem, thereby improving the model's performance on fine-grained tasks.

[0126] SeFAR feature analysis

[0127] To further verify the potential of SeFAR in enhancing the capabilities of multi-modal large language models (MLLMs), we construct a dataset called Gym-QA, which is based on FineGym and converted into a multiple-choice question format (as shown in Fig. Figure 1 In the experiment, we select three widely used visual encoders as comparative baselines: CLIP-ViT-L / 16, EVA-CLIP ViT-G / 14, and ViT-L / 14. To ensure fair comparison, we perform semi-supervised training on these backbone networks with 5% labeled data, then freeze the encoder weights and the training weights of SeFAR, and fine-tune Q-Former on the Gym-QA dataset with 5% labeled data. As shown in Table 5, compared with the commonly used visual encoders of MLLM, the SeFAR-enhanced model achieves significant performance improvement on the Gym-QA task. This indicates that SeFAR not only helps multi-modal large models (MLLMs) cope with the challenges of fine-tuning in low-labeled scenarios, but also exhibits its potential in enhancing the visual perception and modeling capabilities of multi-modal large models.

[0128] Table 5 Comparison experiments for visual encoders of large pre-trained multi-modal models

[0129]

[0130] The above examples show that the SeFAR framework proposed by the present application has superior performance in enhancing the fine-grained visual understanding capability of the multi-modal large model (MLLM). SeFAR not only significantly outperforms all baseline methods on the representative fine-grained video action data sets Gym99, Gym288 and FineDiving, but also achieves new state-of-the-art results on the classic benchmark human action recognition data sets (i.e., UCF101 and HMDB51).

[0131] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application without departing from the principles and spirit of the present application.

Claims

1. A semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization, characterized by: The following steps are involved: Step 1: The video to be identified is sampled using a two-stage temporal element sampling method; Step 2: Input the frame sequence sampled in step 1 into the trained model to obtain the recognition result; The model is trained through the following process: Step a: Obtain training samples, wherein the training samples include a small number of labeled videos and a large number of unlabeled videos; the small number of labeled videos means that the proportion of labeled videos does not exceed 10% and is not less than 1%; Step b: Sample the labeled video using a two-level temporal element sampling method, and use the labeled frame sequence samples obtained through supervised training to obtain an initial model; Initialize the teacher model and student model using the initial model; Iterate training through the following steps; Step c: Both labeled and unlabeled videos are sampled using a two-level temporal element modeling method. The labeled frame sequence samples obtained by sampling are weakly enhanced, and the unlabeled frame sequence samples obtained by sampling are weakly enhanced and strongly enhanced to obtain a batch of frame sequence samples. The weak enhancement is random horizontal flipping or random scaling, and the strong enhancement is moderate temporal perturbation, which is to reverse the order of each group of fine-grained frame sequences while keeping the order of the context frame sequences unchanged; The batch of frame sequence samples includes weakly enhanced labeled frame sequence samples and weakly and strongly enhanced unlabeled frame sequence samples; Step d: Input the weakly enhanced labeled frame sequence samples into the teacher model for recognition, and obtain the recognition results and the loss of the recognition results and labels. ; The recognition result is a probability distribution of various possible categories; Step e: Input the weakly enhanced unlabeled frame sequence samples into the teacher model for recognition to obtain the recognition results; Step f: Repeat step e N times to obtain N recognition results, and calculate the average probability of each category in the N recognition results. As a pseudo label, i represents the i-th category; Step g: Calculate the maximum value of the probability average of various categories ,as well as The probability standard deviation of the corresponding category in N recognition results ; according to and , calculate the dynamic coefficient and : in, is a natural constant, and To set the constant; according to the dynamic coefficient and Calculate the adaptive coefficient ; Step h: Input the strongly enhanced unlabeled frame sequence samples into the student model for recognition to obtain the recognition results; Step i: Use the recognition results and pseudo labels of the student model to obtain the loss of the unsupervised learning process ; Step j: Combine the strongly enhanced unlabeled frame sequence samples in the batch in pairs to obtain several pairs of unlabeled frame sequence samples, process each of the unlabeled frame sequence sample pairs using the Tube TokenMix method to obtain a new unlabeled frame sequence sample, input the new unlabeled frame sequence sample into the student model for recognition, and obtain the recognition result; Step k: Take one of the unlabeled frame sequence samples in the unlabeled frame sequence sample pair through the process of step e-step g to obtain And the corresponding pseudo labels ; The other unlabeled frame sequence sample in the unlabeled frame sequence sample pair is obtained through the process of step e-step g. And the corresponding pseudo labels According to the formula Get the mixed loss weight and mixed pseudo labels ,in is the set mixing ratio; using the recognition result of step j and the mixed pseudo label , we get the mixing loss ; Step 1: Use the formula Get the loss of this iteration process ,in is the warmup coefficient; Step m: Exploiting losses Adjust the student model to get the student model parameters after the kth iteration , and then use the adjusted student model parameters , according to the formula , get the teacher model parameters after the kth iteration ; Then return to step c until the set total number of iterations is reached and the model training is completed.

2. The semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1, characterized in that: The two-stage sequential element sampling is: Sampling the video in segments to obtain K frames of images; Densely sampling the K frames of image data with a relatively small sampling step size to obtain a set of context frame sequences reflecting global temporal information; The K frames of images are sparsely sampled with a relatively large sampling step size to obtain multiple groups of fine-grained frame sequences that reflect detail information.

3. The semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1, characterized in that: In step g, Related to the dropout of the model, is a stable parameter.

4. The semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1, characterized in that: In step k, take ,in represents a Beta distribution.

5. The semi-supervised fine-grained action recognition method based on temporal perturbation and learning stabilization according to claim 1, characterized in that: In step 1, the warmup coefficient , k is the current iteration number, is the total number of iterations set.

Images