Behavior recognition method based on adaptation during compact testing, medium and equipment

Abstract

The invention discloses a behavior recognition method based on adaptation during compact testing, a medium and equipment. The method comprises the following steps: extracting cross-modal representation of a video and a category text by adopting a pre-trained vision-language model and realizing alignment; obtaining a pseudo tag and confidence of the test video through cross-modal matching; core image blocks are screened based on cross-modal attention, and compact video representation is constructed through space and time extension; category caches are constructed and maintained according to the pseudo labels, and weighted fusion or sample replacement of the caches is realized through visual similarity and confidence; and after the steps are repeated on a new test video to generate a compact representation, performing same-mode matching on the compact representation and the visual prototype in the category cache to complete category judgment. According to the method, pre-training model parameters and extra training do not need to be changed, redundant information is reduced through compact representation construction, the test adaptation efficiency is improved, meanwhile, the accuracy of behavior recognition is guaranteed by means of a cache mechanism, and the method is suitable for various behavior recognition scenes needing adaptation during efficient testing.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and video understanding, and in particular to a behavior recognition method, medium, and device based on compact testing adaptation. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, vision-language models (VLMs) have been widely used in image and video understanding tasks. Particularly in action recognition tasks, deep neural network models trained on large-scale labeled data have achieved remarkable results. However, the cost of labeling video data is extremely high; data acquisition and manual labeling are often time-consuming and labor-intensive, making it difficult to meet the demands for rapid deployment and generalization capabilities in practical applications.

[0003] In recent years, Test-Time Adaptation (TTA) has been widely studied as an effective transfer paradigm in behavior recognition. Existing TTA methods are mainly divided into two categories: training-required methods and training-free methods. A typical training-required method is Test-Time Prompt Tuning (TPT), which fine-tunes the model's prompts by introducing unsupervised semantic consistency loss to improve the model's ability to recognize samples from the target domain. However, this type of method has high computational overhead and is not suitable for resource-constrained deployment scenarios because it still requires backpropagation and gradient updates during inference. In contrast, training-free TTA methods alleviate the data offset problem between the source and target domains by constructing a category cache and post-processing the output results without changing the model parameters. However, in this type of method, the alignment between cached video and behavior text is semantically guided, while the alignment between test video and cached video is appearance-driven, leading to a break in the mapping chain. If two videos are similar due to semantically unrelated factors, such as background, they may be incorrectly classified into the same category. For example, playing golf on a grassy field and riding a bicycle on a grassy field have significant semantic differences, but because their backgrounds are similar, misclassification is likely. Specifically, existing technologies have the following problems.

[0004] First, existing test-time adaptation methods that do not require training suffer from semantic inconsistency during visual-visual alignment. Current methods typically build category caches based on visual-text alignment and complete the final matching during the inference phase using visual-visual similarity. However, since visual-text alignment is semantically driven, while visual-visual alignment is strongly influenced by appearance factors, there is a lack of structural transitive consistency between the two, leading to broken mapping chains. When the test video and the cached video are similar in background, scene, or object appearance but semantically different (e.g., swinging a golf club on grass versus cycling on grass), the model is prone to making incorrect matches, significantly reducing generalization performance.

[0005] Secondly, existing technologies generally rely on global video representations for cache construction, which cannot effectively remove semantically irrelevant redundant information. Videos typically contain a large number of background blocks, static scenes, or visual noise, while action semantics are often concentrated only in local image patches and local temporal dynamics. Directly using global representations not only dilutes semantic information but also amplifies appearance bias during cache alignment, causing the model to focus more on background consistency than the action itself, resulting in insufficient cache discriminativeness.

[0006] Furthermore, existing methods lack effective mechanisms for image patch selection and temporal modeling. Current attention-based image patch selection methods often rely on a single high-response region, resulting in overly concentrated image patches that fail to cover complete action semantics. Meanwhile, simple spatial or temporal expansion strategies cannot capture real temporal changes, leading to fragmented video representation structures, dynamic discontinuities, and reduced classification reliability.

[0007] Furthermore, existing training-free TTA methods lack semantically guided video representation compression mechanisms. Although the visual-text alignment structure contains rich semantic activation information, this signal is not fully utilized. The cache construction process is still dominated by visual appearance and lacks a semantically driven filtering strategy, thus failing to guarantee the consistency between the cache and the text semantics. Summary of the Invention

[0008] Purpose of the invention: To address the aforementioned existing technologies, this invention proposes a behavior recognition method based on compact test-time adaptation, which allows the matching between test videos and cached videos to be guided by semantics rather than dominated by appearance, thereby improving the generalization performance and practical value in behavior recognition tasks.

[0009] Technical Solution: A behavior recognition method based on compact test-time adaptation (CTTA) enables a pre-trained visual-language model to adapt to new target domains without additional training by semantically refining and updating video representations during the testing phase. The core idea of ​​this method is to construct a more compact, semantically guided video representation, ensuring that the matching between the video and the category cache is no longer interfered with by background or appearance factors. This addresses issues such as insufficient cross-modal consistency, severe visual-visual alignment bias, and excessive redundancy in video representations found in existing CTTA methods.

[0010] In the specific process, a pre-trained vision-language model is first used to perform cross-modal encoding on the test video and category text to obtain initial prediction results. This includes selecting pseudo-labels for the test video based on the matching score and calculating confidence based on the entropy of the prediction distribution. Subsequently, a semantic activation-guided image patch selection mechanism is used to compact the video frames. To this end, each frame of the video is divided into multiple image patches. Cross-modal attention is used to select the image patches with the highest semantic relevance to the action as core image patches, thereby filtering out irrelevant information such as background and static scenes, making the video's expressive structure more focused on the action itself. To enhance the spatial integrity and temporal continuity of the selection results, the selected image patches are further spatially and temporally expanded, so that the representation is not limited to a single local region, but can cover continuous action regions and dynamic changes across frames, resulting in a compact and semantically dense video representation.

[0011] After completing the compact representation, a caching mechanism is used to construct a visual prototype, with one cache corresponding to each category. As test videos are input sequentially, the model writes the compact video representations into the corresponding category cache based on pseudo-labels. Since cache quality directly affects the accuracy of subsequent same-modal matching, a video reliability assessment mechanism is introduced, using prediction entropy to measure sample confidence, and only reliable video representations are written into the cache. When cache capacity is limited, momentum fusion or sample replacement is performed based on visual similarity and confidence when writing new samples, ensuring that the cache has both intra-class representativeness and sufficient diversity and generalization ability. Through the above cache update strategy, this method effectively alleviates the problems of visual cache being dominated by background, severe sample homogenization, and semantic offset accumulation in traditional testing adaptation.

[0012] In the subsequent adaptation phase, the compact video representation of the test video is compared with the visual prototype in the category cache to calculate visual-visual similarity, and finally, the category is determined based on the same modality similarity. Because the video representation generated by this method is more compact, semantically focused, and has a continuous temporal structure, the visual-visual matching process is no longer affected by non-semantic factors, and can more accurately reflect the semantic consistency between the test video and the target category, thereby achieving efficient test-time adaptation of the pre-trained visual-language model.

[0013] Through the above technical solutions, this method achieves semantic compression of video representations, reliable optimization of cache structures, and efficient modeling of video action semantics without changing model parameters or requiring training. It effectively solves problems such as weak semantic dependence, heavy representation redundancy, large appearance interference, and unstable adaptation effect in existing technologies, and significantly enhances the accuracy, discriminativeness, and practical application value of behavior recognition.

[0014] Beneficial Effects: This invention addresses the problems of semantic inconsistency, severe representation redundancy, and susceptibility to background interference in behavior recognition by existing adaptation methods during testing. By introducing a semantically guided compact video representation and caching construction mechanism, it significantly improves the accuracy and stability of video semantic alignment and adaptation. Compared with existing technologies, this invention mainly has the following beneficial effects:

[0015] 1. Significantly improves the semantic concentration and discriminative ability of video representation.

[0016] This invention utilizes semantic activation of a visual-language model to filter image patches in each frame that best reflect the meaning of the action, effectively eliminating background, static areas, and other irrelevant visual content. This allows the video representation to focus more intently on the action-related subject area. Furthermore, spatial expansion fully covers the action region structure, and temporal expansion captures cross-frame action changes, resulting in a compact video representation that combines high semantic density with a continuous temporal structure. Compared to traditional methods using global representations, the video representation generated by this invention better reflects the essence of the action and improves the accuracy of cross-modal alignment and visual matching.

[0017] 2. Improve the semantic consistency and reliability of prototypes through compact visual caching.

[0018] This invention writes a compact video representation, after semantic filtering and spatial-temporal expansion, into a category cache. This eliminates the impact of video background or irrelevant image blocks on the cache, significantly improving the semantic quality of the prototype. Simultaneously, because the cache stores only semantically dense video representations, the prototype is more stable and can more accurately reflect the semantic center of the action category. Compared to existing technologies that use global visual representations to build caches, the cache of this invention has greater semantic consistency and interference resistance, thereby improving the reliability of visual-to-visual matching.

[0019] 3. Reduce the computational burden caused by redundant representations and achieve efficient test-time adaptation.

[0020] By employing semantic filtering and compact video representation construction, this invention significantly compresses the representation dimensionality of videos, reducing unnecessary background and noise information, and fundamentally lowering the complexity of cache construction and similarity calculation. Compact video representation is not only more efficient but also enables faster and more accurate identification of semantic associations with the prototype during the matching process. This allows the entire testing-time adaptation process to be completed efficiently without training, with adaptation overhead far lower than traditional methods that rely on global representations.

[0021] 4. Enhance the model's adaptability to different video scenarios without requiring training.

[0022] The entire adaptation process of this invention does not rely on model parameter updates; adaptation is completed solely by constructing a cache using compact video representations and performing matching. Because the cache has high semantic concentration and temporal continuity, the model can maintain stable recognition of action semantics in new video scenes, and will not easily make misjudgments due to background changes or appearance shifts. This characteristic enables this invention to significantly enhance the model's generalization ability without training, thereby solving the problem of unstable performance of existing methods in cross-scene testing.

[0023] In summary, this invention achieves a balance between high semantic concentration, high anti-interference capability, and high adaptation efficiency in behavior recognition through a semantically driven compact video representation and a semantically consistent caching construction mechanism, significantly improving the accuracy, stability, and practical value of adaptation during testing. Attached Figure Description

[0024] Figure 1 This is an architecture diagram of the behavior recognition method based on compact testing adaptation of the present invention;

[0025] Figure 2 This is a flowchart illustrating the construction of a compact video representation in the method of this invention;

[0026] Figure 3 This is a schematic diagram of the behavior recognition results of the present invention. Detailed Implementation

[0027] The invention will now be further explained with reference to the accompanying drawings.

[0028] like Figure 1 As shown, a behavior recognition method based on compact test-time adaptation is presented. The overall process includes cross-modal representation extraction, semantically guided compact video representation construction, category cache construction and maintenance, and cache-based same-modal matching. The specific implementation process is as follows.

[0029] Step 1: Cross-modal characterization extraction.

[0030] A pre-trained vision-language model is used as the base model. First, a visual encoder encodes the test video, consisting of multiple frames of images, into a video representation. Simultaneously, a text encoder encodes the categorical text into a text representation, serving as a semantic prototype. This step aligns the video and text in the same vector space and lays the foundation for cross-modal matching.

[0031] Step 2: Cross-modal matching and initial confidence acquisition.

[0032] The similarity between the video representation obtained in step 1 and the semantic prototype is calculated to obtain the cross-modal matching result. A pseudo-label for the test video is selected based on the matching score, and the confidence level is calculated based on the entropy of the prediction distribution. This confidence level is used in step 4 to determine whether the compact video representation corresponding to the test video is suitable for being written into the cache as a visual prototype.

[0033] Step 3: Semantically guided construction of compact video representations.

[0034] To reduce interference from non-semantic factors such as background and static areas, video compression is performed based on semantic activation. For example... Figure 2 As shown, firstly, each frame of image is divided into multiple image patches. Cross-modal attention is used to select the image patch with the highest semantic relevance to the action, which is then used as the core image patch. Specifically, the semantic attention signal is obtained based on the gradient G and attention map R in the visual-language model as follows:

[0035]

[0036] In the formula Indicates the number of layers in the visual encoder. This indicates that the average is taken across the dimensions of the attention head. Indicates the first The gradient corresponding to the layer visual encoder, Indicates the first Attention map corresponding to the layer visual encoder This represents the identity matrix. Based on semantic attention signals, image regions most relevant to action semantics can be sampled, thereby filtering out irrelevant information such as background and static scenes, making the video's expressive structure more focused on the action itself.

[0037] Subsequently, these core image patches are expanded spatially to cover the complete action region; and semantic association expansion is performed on adjacent frames in the temporal dimension to capture the dynamic structure of action across frames. After the above spatial-temporal expansion, a compact video representation that combines compactness and temporal structure can be obtained.

[0038] Step 4: Category cache building and maintenance.

[0039] The compact video representations obtained in step 3 are written into the corresponding category cache based on the pseudo-labels from step 2. To ensure the representativeness and diversity of the cache, this invention limits the cache capacity; in this embodiment, the cache capacity is set to 10 compact video representations. A similarity judgment mechanism is introduced in the cache update: when a new compact video representation is highly similar to a certain compact video representation in the cache, in this embodiment, the cosine similarity between the two is greater than 0.9, and the compact video representation is updated through a weighted fusion method, with each weight being 0.5; when it differs significantly from all compact video representations, the compact video representation with the lowest confidence in the cache is replaced. This mechanism ensures that the category cache can continuously accumulate high-quality, semantically consistent visual prototypes.

[0040] Step 5: Cache-based same-modal matching.

[0041] After the cache is built, for a new test video, its compact video representation is first generated following steps 1 to 3. Then, the compact video representation is compared with the visual prototypes in each cache to calculate the same-modal similarity, resulting in more stable, semantically driven matching results. This step relies only on the compact video representation and the visual prototypes in the cache, no longer depending on the background or appearance, effectively correcting biases caused by cross-modal predictions during the testing phase.

[0042] Through the above steps, the present invention can construct semantically dense, temporally structured, compact video representations without training and achieve efficient test-time adaptation through category caching, significantly improving the accuracy and robustness of behavior recognition.

[0043] like Figure 3 As shown, this embodiment can identify the category that best matches the test video from three candidate categories, namely "singing," "dancing," and "playing the piano," by following these steps:

[0044] Step 1: Encode the three categories of text, namely “singing”, “dancing” and “playing piano”, into three text representations using the CLIP model’s text encoder, and encode the test video into a video representation using the CLIP model’s visual encoder.

[0045] Step 2: Calculate the similarity between the video representation and the text representation for each category to obtain the cross-modal matching result. At this stage, the model may misclassify the test video as the "dancing" category. Cosine similarity is used for the similarity calculation.

[0046] Step 3: Construct a compact video representation by selecting the most relevant image patch based on the semantic activation value of each image patch.

[0047] Step 4: During the testing process, continuously maintain the cache for the three categories of "singing", "dancing" and "playing the piano".

[0048] Step 5: Perform sequence matching between the visual representation of the test video and the visual prototypes in the three category caches of "singing", "dancing" and "playing the piano" to obtain the same modal matching result. At this time, the test video is highly similar to the visual prototype corresponding to "playing the piano". Therefore, the same modal matching result can correct the wrong cross-modal matching result and correctly predict the test video in this embodiment as the "playing the piano" category.

[0049] Through the detailed steps described above, this invention provides an efficient and accurate behavior recognition scheme, which significantly improves the quality and application feasibility of behavior recognition.

[0050] It is obvious to those skilled in the art that the steps of the behavior recognition method based on compact test adaptation in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using device-executable program code, thereby storing them in a storage device for execution by a computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0051] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A behavior recognition method based on compact test-time adaptation, characterized in that, include: Step 1: Using a pre-trained vision-language model, the test video is encoded into a video representation through a visual encoder, and the category text is encoded into a text representation as a semantic prototype through a text encoder, so as to achieve the alignment of video and text in the same vector space; Step 2: Calculate the similarity between the video representation obtained in Step 1 and the semantic prototype, select pseudo-labels for the test video based on the matching score, and calculate the confidence level based on the entropy of the prediction distribution. Step 3: Divide the video frame into multiple image blocks, select the core image blocks with the highest relevance to action semantics through cross-modal attention, spatially expand the core image blocks to cover the complete action region, and temporally expand them to capture the dynamic structure of cross-frame action, thus obtaining a compact video representation. Step 4: Based on the pseudo-labels in Step 2, write the compact video representation obtained in Step 3 into the corresponding category cache, limit the cache capacity, and when writing the new compact video representation, perform weighted fusion or sample replacement based on visual similarity and confidence. Step 5: Repeat steps 1-3 for the new test video to generate a compact video representation. Calculate the modal similarity between the compact video representation and the visual prototypes in each category cache. Determine the category based on the similarity results.

2. The behavior recognition method based on compact test adaptation according to claim 1, characterized in that, In step 3, when filtering core image patches through cross-modal attention, a semantic attention signal is calculated based on the gradient G and attention map R in the visual-language model. The semantic attention signal is represented as follows: ； In the formula Indicates the number of layers in the visual encoder. This indicates that the average is taken across the dimensions of the attention head. Indicates the first The gradient corresponding to the layer visual encoder, Indicates the first Attention map corresponding to the layer visual encoder The identity matrix is ​​represented; based on the semantic attention signal, the image block with the highest semantic relevance to the action is selected as the core image block.

3. The behavior recognition method based on compact test time adaptation according to claim 1, characterized in that, In step 3, the temporal expansion is to perform semantic association expansion on adjacent frames in the temporal dimension to capture the dynamic structure of actions across frames.

4. The behavior recognition method based on compact test adaptation according to claim 1, characterized in that, In step 4, visual similarity is determined using cosine similarity. When the cosine similarity between the new compact video representation and a certain compact video representation in the cache is greater than 0.9, weighted fusion is performed, with each fusion weight being 0.

5. When the cosine similarity with all compact video representations is not greater than 0.9, the compact video representation with the lowest confidence in the cache is replaced.

5. The behavior recognition method based on compact test adaptation according to claim 1, characterized in that, The pre-trained visual-language model is the CLIP model.

6. The behavior recognition method based on compact test time adaptation according to claim 1, characterized in that, The test video is a video sequence consisting of multiple frames.

7. The behavior recognition method based on compact test time adaptation according to claim 1, characterized in that, The parameters of the pre-trained visual-language model are not changed throughout the entire behavior recognition process, and no additional training is required.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the behavior recognition method based on compact test adaptation as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that performs the behavior recognition method based on compact test adaptation as described in any one of claims 1-7.

Images