Zero-shot video classification method based on frame-level cache and video-level cache update

Abstract

The application discloses a zero sample video classification method based on frame-level cache and video-level cache updating. First, visual features and text features of a test video are extracted. Then, cosine similarity calculation is performed on the test video and text prototypes of each category to obtain cross-modal zero sample matching results, which contain predicted pseudo labels and prediction result entropy. According to the prediction result entropy and time sequence probability difference criterion, the credibility of each frame is measured, and a two-stage Top-K strategy is adopted. First, high-confidence frames satisfying a preset value are selected according to the prediction entropy, and then frames most discriminative in time sequence are further selected according to the time sequence probability difference for caching. A video-level cache updating module is used to maintain the cache. Visual features of the test video are matched with visual prototypes in the cache of each category to obtain same-modal matching results. The cross-modal zero sample matching results and the same-modal matching results are fused to obtain final inference results.

Description

Technical Field

[0001] This invention relates to a zero-shot video classification method based on frame-level caching and video-level cache updates. Specifically, it is an implementation method for zero-shot video classification based on the joint adaptation of frame-level cache refinement and video-level cache updates using a hierarchical adapter, belonging to the fields of artificial intelligence and video understanding technology. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, visual-language models have been widely applied in image and video understanding tasks. Particularly in video classification tasks, deep neural network models trained on large-scale labeled data have achieved significant results. However, the cost of labeling video data is extremely high; data collection and manual labeling are often time-consuming and labor-intensive, making it difficult to meet the demands for rapid deployment and generalization capabilities in practical applications. Therefore, zero-shot learning techniques have gradually gained attention. Their core objective is to complete classification tasks using the knowledge transfer capabilities of pre-trained models, even when there are no labeled samples for the target category.

[0003] In recent years, test-time adaptation, as an effective transfer paradigm in zero-shot video classification, has received extensive research. Existing test-time adaptation methods are mainly divided into two categories: training-required methods and training-free methods. Typical training-required methods include test-time cue tuning, which fine-tunes the model's cues by introducing unsupervised semantic consistency loss to improve the model's ability to recognize samples in the target domain. However, these methods have high computational costs and are not suitable for resource-constrained deployment scenarios because they still require backpropagation and gradient updates during inference. In contrast, training-free test-time adaptation methods alleviate the data offset problem between the source and target domains by constructing class caches or support sets and post-processing the output results without changing the model parameters. These methods have made some progress in the image domain, such as constructing caches using historical test samples or expanding cached samples with generative models. However, directly applying these methods to the video domain still presents significant problems, mainly including the following two aspects: First, how to represent multi-frame video information with high quality is a key problem that urgently needs to be solved. Existing methods typically construct video representations using simple feature averaging or keyframe selection. The former easily introduces redundancy and noise, reducing semantic discriminability, while the latter may lose crucial temporal action information, affecting the ability to model complex actions. Secondly, effectively constructing discriminative category caches is also a challenge. While existing entropy-based cache update strategies can select high-confidence samples, they are prone to sample homogenization, meaning the selected samples have similar distributions in the visual feature space, making it difficult to cover the diverse features within action categories, resulting in a lack of representativeness and generalization ability in the cache.

[0004] Therefore, there is an urgent need for an efficient, reliable, and versatile adaptation scheme to improve the generalization performance and practical value of zero-shot video classification tasks. Summary of the Invention

[0005] Purpose of the invention: This invention aims to solve the following key technical problems existing in current zero-shot video classification technologies:

[0006] First, existing test-time adaptation methods that do not require training have shortcomings in temporal modeling of video data. Since videos consist of multiple frames, there is rich temporal information between frames. However, existing methods typically employ simple averaging strategies or select a single keyframe to construct a video representation. These strategies fail to effectively model the temporal dynamics of the video, easily introducing redundant noise or ignoring key action phases. This leads to a decrease in the discriminative power of the cached video representation, affecting the final classification performance.

[0007] Secondly, existing methods lack fine-grained control over sample diversity and representation reliability when constructing category caches. Current mainstream cache update strategies mostly select high-confidence samples based on confidence metrics such as prediction entropy. However, samples with low entropy values ​​tend to cluster in the visual feature space, lacking representativeness and diversity, leading to a "homogenization" problem in cache construction. This fails to cover complex action variations within a category, thereby reducing the model's generalization ability in the target domain.

[0008] Furthermore, existing technologies lack a hierarchical adaptation mechanism. Current cache update and filtering operations are typically performed only at the video level, failing to incorporate frame-level information for fine-grained filtering and selection. This results in unreliable or redundant frames being included in the final representation, thereby affecting the overall quality and discriminability of the video cache.

[0009] Therefore, this invention proposes a reliable and diverse hierarchical adapter to address the aforementioned key issues. Specifically, this invention optimizes frame-level caching by introducing frame-level reliability metrics (such as prediction entropy and temporal probability difference) and introduces a diversity metric mechanism at the video level to improve cache coverage and generalization ability. Simultaneously, by designing a hierarchical "frame-video" adaptation framework, high-quality test-time adaptation is achieved without training, effectively improving the performance and practicality of pre-trained vision-language models in zero-shot video classification tasks.

[0010] Technical Solution: A zero-shot video classification method based on frame-level caching and video-level cache updates aims to achieve test-time adaptation of pre-trained visual language models without relying on additional training. This solution improves the model's ability to model complex video action semantics by having frame-level cache refinement and video-level cache updates work collaboratively, thus addressing the problems of high adaptation overhead, insufficient robustness, and weak semantic generalization ability in existing methods.

[0011] To achieve the above objectives, the present invention provides the following solution:

[0012] Given C category labels and a sequence V = v1, v2, ..., v3 consisting of N test videos. N The goal of zero-shot video classification is to classify each test video v when the set of categories is not visible to the pre-trained model. i Predict its category, i=1……N.

[0013] Given any pre-trained visual language model (VLM) consisting of a visual encoder and a text encoder, the test video is first encoded into visual features f. v Simultaneously, the category text is encoded into text features, namely the text prototype f for each category. c Then, the cosine similarity between the test video and the text prototype of each category is calculated to obtain the cross-modal zero-sample matching result, which includes both the predicted pseudo-label and the entropy of the prediction result.

[0014] To further enrich the semantic information of the text prototype, a visual prototype is constructed through a caching mechanism:

[0015] First, encode the current (i-th) test video into visual features f. v Then, it is placed into the cache along with the visual features of the previous i-1 test videos. There are a total of C caches, one for each category. When storing the visual features of a test video into the cache, it is necessary to store them into the corresponding cache based on the cross-modal zero-sample matching results.

[0016] After the cache is built, the visual features of the test video are compared with the visual prototypes in each category cache to calculate the cosine similarity, and the same modality zero-sample matching results are obtained.

[0017] To enhance the representation of the test video, a frame-level buffer refinement module was designed:

[0018] This module performs frame-level screening by integrating two metrics: (1) prediction entropy, which measures the uncertainty of the model in single-frame prediction; and (2) temporal probability difference, which measures the sensitivity of the model to time dynamic cues.

[0019] The prediction entropy is given in the cross-modal zero-sample matching results, and the formula is: Where e represents the prediction entropy, This is the classification result of the test video, where v is the test video, and there is no limit to the base of the logarithm, but it is generally set to base e.

[0020] To calculate the temporal probability difference, each frame of the current test video is first divided into 7x7 image blocks. Then, each frame is shuffled at the image block level with its neighboring frames to obtain new video frames. A pre-trained visual language model is then used to perform a cross-modal zero-shot matching. The temporal probability difference for each frame is obtained by subtracting the matching result from the original test video. The formula is: Where v is the original test video, v t The test video is composed of shuffled frames. * These are the pseudo-labels for the original test video. The larger the value of the temporal probability difference, the more the cross-modal zero-shot matching relies on temporal cues, indicating that the pseudo-labels are more reliable; conversely, the smaller the value of the temporal probability difference, the more the cross-modal zero-shot matching relies on static cues such as video background, indicating that the pseudo-labels are less reliable, thus affecting the representational ability of the visual prototype.

[0021] A two-stage Top-K strategy is adopted. First, high-confidence frames are selected based on prediction entropy. Then, the frames with the most time-series discriminative power, such as the frames containing key action nodes, are further filtered based on the temporal probability difference to construct a more representative cached input.

[0022] To limit cache size while improving cache diversity and generalization performance, a video-level cache update module is designed:

[0023] To limit the memory and video memory usage of the visual prototype cache to a reasonable range, each cache can store a maximum of 10 visual features from test videos.

[0024] When cache capacity is limited, if a new sample is too similar to an existing sample in the cache, it is fused using momentum update, with the formula: q←μq+(1-μ) f v The left arrow (←) sign indicates an assignment operation. v The visual features of the test video are represented by q, where q represents the closest to f. v The visual prototype, μ, represents the momentum coefficient, which measures the relative importance of the visual prototype to the new visual feature. Its value ranges from 0 to 1. The smaller the value of μ, the stronger the new visual feature f. v The higher the proportion, the larger the value of μ, indicating that the existing visual prototype q has a higher proportion.

[0025] If a new sample has low similarity to all existing samples in the cache, the new sample is replaced. The confidence criterion for the sample is consistent with the confidence criterion in the frame-level cache refining module, which jointly considers prediction entropy and temporal probability difference.

[0026] To more accurately assess the similarity between two videos, a dynamic time warping algorithm is designed:

[0027] Given two video features f to be matched v and f w First, a similarity matrix S is constructed based on cosine similarity, where the element S(i,j) in the i-th row and j-th column of the matrix is ​​f. v Features of the i-th frame and f w The cosine similarity of the features in the j-th frame. Let S be the similarity of the features in the j-th frame. seq (i,j) represents f v The first i-frame and f w The sequence similarity between the first j frames can be used to establish the following state transition equation:

[0028] S seq (i,j)=S(i,j)+min{ S seq (i-1,j), S seq (i,j-1), S seq (i-1,j-1)},

[0029] Ultimately, f v and f w The temporal similarity between them is determined by S seq (n,m) is given, where n is the video feature f. v The number of video frames, where m is the video feature f. w The number of video frames.

[0030] The present invention effectively solves the shortcomings of existing technologies in video feature representation, caching diversity and generalization, and video feature similarity calculation through the above technical solution, and significantly improves the quality and applicability of zero-shot video classification.

[0031] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the zero-sample video classification method based on frame-level caching and video-level caching updates as described above.

[0032] A computer-readable storage medium storing a computer program that performs the zero-shot video classification method based on frame-level buffer and video-level buffer updates as described above.

[0033] Compared with existing technologies, this invention addresses the shortcomings of existing zero-shot video classification techniques by introducing a hierarchical caching management mechanism, significantly improving the performance and efficiency of video classification. This invention has the following significant advantages:

[0034] (1) Improve the model’s sensitivity and expressive ability to key action timing information.

[0035] The frame-level cache refinement module can effectively remove low-confidence and temporally redundant frames, retaining only the most discriminative frames for classification, such as frames containing key action nodes, thereby constructing more semantically representative cache features and improving the ability to discriminate complex video actions.

[0036] (2) Improve the diversity and generalization performance of caching

[0037] This invention introduces a cache update strategy based on temporal similarity to ensure that the cache contains diverse representations of action categories. This overcomes the problems of excessively high visual similarity and insufficient semantic coverage between high-confidence samples in traditional methods, thereby improving the cache's coverage of the test video distribution and the overall robustness of the model.

[0038] (3) No training required, low adaptation cost, suitable for resource-constrained environments

[0039] This invention does not rely on gradient updates or model fine-tuning at all. The adaptation process is based solely on cache building and query operations, resulting in high inference efficiency and low computational resource consumption. It is suitable for computationally constrained scenarios such as embedded devices and real-time inference.

[0040] (4) It has strong scalability and is suitable for various pre-trained visual language models.

[0041] The present invention has a highly versatile structure, supports deployment on various mainstream VLMs (such as CLIP, ViFi-CLIP, etc.), and has verified its wide applicability and superior performance on multiple datasets (such as HMDB-51, UCF-101, Kinetics-600, ActivityNet-200).

[0042] In summary, this invention achieves joint modeling of video action diversity and reliability through a hierarchical adaptation mechanism that requires no training during testing. This breaks the limitation of zero-shot classification where it is difficult to achieve both high accuracy and low computational cost, and has significant practical value and industrial transformation prospects. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the method of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating how the temporal probability difference is calculated.

[0045] Figure 3 This is a schematic diagram of the video classification results of the present invention. Detailed Implementation

[0046] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0047] like Figure 1 As shown, a zero-shot video classification method based on frame-level caching and video-level caching updates aims to achieve accurate and efficient zero-shot video classification through cross-modal reasoning and intra-modal reasoning collaborative reasoning. The implementation process of the technical solution includes the following main steps:

[0048] Step 1: Feature Extraction

[0049] (11) The input to the visual language model contains C category labels and a sequence V = v1, v2, ..., v consisting of N test videos. N Each category label is in text form, and each test video consists of T frames of RGB images.

[0050] (12) Given any pre-trained visual language model, such as CLIP or ViFiCLIP, use a visual encoder to encode the test video into visual features f. v This process extracts high-level features of the actions, reducing the computational complexity of the model.

[0051] (13) The text encoder of the pre-trained visual language model encodes the text corresponding to the C category labels into text features, i.e., text prototypes f. c .

[0052] Step 2: Cross-modal zero-sample matching

[0053] (21) The visual features f of the test video v Cosine similarity is calculated with the text prototype of each category to obtain cross-modal zero-sample matching, i.e. .

[0054] (22) Based on the logits of the cross-modal zero-sample matching results, calculate the pseudo-label of the current test video, i.e., the video classification result: , where v is the current test video.

[0055] (23) Based on the results of cross-modal zero-sample matching (logits), calculate the entropy e of the model prediction in this study, which will be used to evaluate the confidence level of the test samples in the future.

[0056] , where v is the current test video.

[0057] Step 3: Build a cache based on the frame-level cache refinement module

[0058] (31) First, the current test video is refined at the frame level. The frame-level cache refinement module measures the credibility of each frame based on the prediction entropy e and the temporal probability difference criterion in step (23). The prediction entropy is used to measure the uncertainty of the model in single-frame prediction, and the temporal probability difference is used to measure the sensitivity of the model to temporal dynamic cues. In order to calculate the temporal probability difference, each frame image is first divided into 7x7 image blocks, and then each frame and its adjacent frames are shuffled at the image block level to obtain new video frames. Then, a cross-modal zero-sample matching is performed on them using the pre-trained visual language model. The temporal probability difference of each frame image is obtained by subtracting the matching result of the original test video from the new matching result. The formula is:

[0059] Where v is the original test video, v t The test video is composed of shuffled frames. * These are pseudo tags from the original test video.

[0060] (32) A two-stage Top-K strategy is adopted. First, high-confidence frames that meet the preset value are selected based on the prediction entropy. Then, the frames with the most time-series discriminative power are further filtered based on the temporal probability difference to construct a more representative cached input. The frames with the most time-series discriminative power are the frames containing key action nodes. For example, for the "long jump" action, the take-off and landing can be regarded as the most time-series discriminative. Here, to avoid ambiguity, "the frames with the most time-series discriminative power" can be modified to "the frames containing key action nodes".

[0061] (33) Put the refined test video representation into the cache. There are C caches in total, with one cache for each category. When storing the visual features of the test video into the cache, it is necessary to store them into the corresponding cache according to the cross-modal zero-sample matching results.

[0062] Step 4: Maintain the cache based on the video-level cache update module

[0063] (41) To limit the memory and video memory occupied by the visual prototype cache to a reasonable range, each cache can store a maximum of 10 visual features from test videos. If the visual features of the current sample are too similar to existing samples in the cache, they are fused using momentum update, with the formula: q←μq+(1-μ) f v The left arrow (←) sign indicates an assignment operation. v The visual features of the test video are represented by q, where q represents the closest to f. v The visual prototype, μ represents the momentum coefficient, which is generally set to 0.5. If the cosine similarity between two features is higher than 0.95, the two features are considered to be too similar.

[0064] (42) If the new sample has low similarity to all existing samples in the cache (the cosine similarity between the two features is less than 0.95), then replace the new sample. The confidence criterion for the sample is consistent with the confidence criterion in (31), that is, jointly considering the prediction entropy and the temporal probability difference.

[0065] Step 5: Cache-based same-modal matching

[0066] (51) After the cache is built and maintained, the visual features of the test video are matched sequentially with the visual prototypes in each category cache to obtain the same modality matching result logits'. Given two video features f to be matched v and f w First, a similarity matrix S is constructed based on cosine similarity, where the element in the i-th row and j-th column of the similarity matrix is ​​f. v Features of the i-th frame and f w The cosine similarity of the features in the j-th frame. Let S be the similarity of the features in the j-th frame. seq (i,j) represents f v The first i-frame and f w The sequence similarity between the first j frames can be used to establish the following state transition equation: S seq (i,j)=S(i,j)+min{ S seq (i-1,j), S seq (i,j-1), S seq (i-1,j-1)}, ultimately, f v and f w The temporal similarity between them is determined by S seq (n,m) is given, where n is the video feature f. v The number of video frames, m is another video feature f. w The number of video frames.

[0067] (52) By fusing the cross-modal zero-sample matching result logits in (21) and the same-modal matching result logits' in (51), the final inference result is obtained: logits'' = logits + logits', where logits'' is a one-dimensional vector of length C, representing the probability of the test video belonging to each category. The category with the highest probability is the final zero-sample video classification result.

[0068] Application Examples

[0069] like Figure 3 As shown, the method proposed in this invention can identify the category that best matches the test video from three candidate categories, namely "singing," "dancing," and "playing the piano," by following these steps:

[0070] Step 1: Encode the three categories of text, namely “singing”, “dancing” and “playing piano”, into three text features using the CLIP model’s text encoder, and then encode the test video into a three-frame video representation using the CLIP model’s visual encoder.

[0071] Step 2: Calculate the similarity between the video representation and the text features of each category to obtain the cross-modal zero-sample matching results. At this point, the model may misclassify the test video as the "dancing" category.

[0072] Step 3: Refine the test video at the frame level and place it in the corresponding category's cache. For example, the three-frame representation in this case can be refined into a two-frame representation or a single-frame keyframe representation, and then placed in the cache corresponding to the "Dancing" category.

[0073] Step 4: During the testing process, continuously update and maintain the cache for the three categories of "singing", "dancing" and "playing the piano".

[0074] Step 5: Perform sequence matching between the visual features of the test video and the visual prototypes in the three category caches of "singing", "dancing" and "playing the piano" to obtain the same modal matching result. At this time, the test video is highly similar to the visual prototype corresponding to "playing the piano". Therefore, the same modal matching result can correct the wrong cross-modal zero-sample matching result and correctly predict the test video in this example as the "playing the piano" class.

[0075] Through the detailed steps described above, this invention provides an efficient and accurate zero-shot video classification scheme, which significantly improves the quality and feasibility of video classification.

[0076] In summary, this invention provides a reasoning scheme based on cross-modal reasoning and intra-modal reasoning collaborative reasoning, and designs a reliable and diverse hierarchical adapter, significantly improving the accuracy and efficiency of zero-shot video classification. This technology can progressively correct erroneous predictions during the reasoning process, improving video classification quality while maintaining reasoning efficiency, and has significant technical value.

[0077] Obviously, those skilled in the art should understand that the steps of the zero-shot video classification method based on frame-level caching and video-level caching updates described in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using device-executable program code, thereby storing them in a storage device for execution by a computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A zero-shot video classification method based on frame-level caching and video-level caching updates, characterized in that, First, feature extraction is performed to extract visual features and textual features of categories from the test video. Then, cosine similarity is calculated between the test video and the text prototype of each category to obtain cross-modal zero-sample matching results. These results include both predicted pseudo-labels and the entropy of the prediction results. Based on the entropy of the prediction results and the temporal probability difference criterion, the credibility of each frame is measured. A two-stage Top-K strategy is adopted: first, high-confidence frames that meet the preset value are selected based on the prediction entropy; then, frames containing key action nodes are further filtered based on the temporal probability difference and cached. The cache is maintained based on the video-level cache update module. The visual features of the test video are matched sequentially with the visual prototypes in each category cache to obtain the same-modal matching results. The cross-modal zero-sample matching results and the same-modal matching results are then fused to obtain the final inference result. The cache is built based on the frame-level cache refinement module, including: (31) First, the current test video is refined at the frame level; the frame-level buffer refinement module measures the credibility of each frame based on the prediction entropy e and the temporal probability difference criterion; in order to calculate the temporal probability difference, each frame image is first divided into 7x7 image blocks, and then each frame and its adjacent frames are shuffled at the image block level to obtain new video frames. Then, a pre-trained visual language model is used to perform a cross-modal zero-sample matching; the new matching result is subtracted from the matching result of the original test video to obtain the temporal probability difference of each frame image, and the formula is: Where v is the original test video, v t The test video is composed of shuffled frames. * These are pseudo-tags from the original test video. These are the classification results of the test videos; (32) A two-stage Top-K strategy is adopted. First, high-confidence frames that meet the preset values ​​are selected based on the prediction entropy. Then, the frames where the key action nodes are located are further screened based on the temporal probability difference, so as to construct a more representative cache input. (33) Put the refined test video representation into the cache; there are a total of C caches, one cache for each category. When storing the visual features of the test video into the cache, it is necessary to store them into the corresponding cache according to the cross-modal zero-sample matching results.

2. The zero-shot video classification method based on frame-level caching and video-level caching updates according to claim 1, characterized in that, Given C category labels and a sequence V = v1, v2, ..., v consisting of N test videos. N The goal of zero-shot video classification is to classify each test video v when the set of categories is not visible to the pre-trained model. i Predict its category, i=1……N.

3. The zero-shot video classification method based on frame-level caching and video-level caching updates according to claim 1, characterized in that, The feature extraction includes: (11) The input to the visual language model contains C category labels and a sequence V = v1, v2, ..., v consisting of N test videos. N； Each category label is in text form, and each test video consists of T frames of RGB images. (12) Given any pre-trained visual language model, use a visual encoder to encode the test video into visual features f. v ; (13) The text encoder of the pre-trained visual language model encodes the text corresponding to the C category labels into text features, i.e., text prototypes f. c .

4. The zero-shot video classification method based on frame-level caching and video-level caching updates according to claim 1, characterized in that, Cross-modal zero-shot matching includes: (21) The visual features f of the test video v With the text prototype f of each category c Cosine similarity is calculated to obtain cross-modal zero-sample matching, i.e. ; (22) Based on the logits of the cross-modal zero-sample matching results, calculate the pseudo-label of the current test video, i.e., the video classification result: Where v is the current test video. These are the classification results of the test videos; (23) Based on the results of cross-modal zero-sample matching (logits), calculate the entropy e of the model prediction in this study, which will be used to evaluate the confidence level of the test samples in the future. , where v is the current test video and C represents the number of categories.

5. The zero-shot video classification method based on frame-level caching and video-level caching updates according to claim 1, characterized in that, The cache is maintained based on a video-level cache update module. The specific implementation steps are as follows: (41) Each cache can store visual features of up to 10 test videos; If the visual features of the current sample are too similar to those of existing samples in the cache, they are fused using momentum updates, with the formula: q←μq+(1-μ) f v The left arrow (←) sign indicates an assignment operation. v The visual features of the test video are represented by q, where q represents the closest to f. v The visual prototype, μ represents the momentum coefficient; if the cosine similarity between two features is higher than a set value, then the two features are considered to be too similar. (42) If the new sample has a low similarity to all existing samples in the cache, then replace the new sample; low similarity means that the cosine similarity between the two features is lower than the set value.

6. The zero-shot video classification method based on frame-level caching and video-level caching updates according to claim 1, characterized in that, The implementation process of cache-based same-modal matching is as follows: (51) After the cache is built and maintained, the visual features of the test video are matched with the visual prototypes in each category cache to obtain the same modality matching result logits'; given two video features f to be matched v and f w First, a similarity matrix S is constructed based on cosine similarity, where the element S(i,j) in the i-th row and j-th column of the similarity matrix is ​​f. v Features of the i-th frame and f w The cosine similarity of the features in the j-th frame; let S seq (i,j) represents f v The first i-frame and f w Based on the sequence similarity between the first j frames, the following state transition equation is established: S seq (i,j)=S(i,j)+min{ S seq (i-1,j), S seq (i,j-1), S seq (i-1,j-1)}, ultimately, f v and f w The temporal similarity between them is determined by S seq (n,m) is given, where n is the video feature f. v The number of video frames, m is another video feature f. w The number of video frames; (52) The cross-modal zero-sample matching result logits and the same-modal matching result logits' are fused to obtain the final inference result: logits''=logits+logits', where logits'' is a one-dimensional vector of length C, representing the probability of the test video belonging to each category. The category with the highest probability is the final zero-sample video classification result.

7. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the zero-sample video classification method based on frame-level caching and video-level caching updates as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that performs a zero-sample video classification method based on frame-level buffering and video-level buffering updates as described in any one of claims 1-6.

Images