A small sample action recognition method based on multimodal multi-level matching

Abstract

This invention discloses a few-shot action recognition method based on multimodal, multi-level matching, relating to the field of computer technology. The method includes: training a few-shot action recognition model; during training, determining the global matching distance based on the few-shot action recognition model by determining the cosine similarity between the global feature representations of the query video and the supporting video set; obtaining the frame-level matching distance between the query video and the supporting videos through the matching relationship between video frames; obtaining the image patch-level matching distance between the query video and the supporting video set through the matching relationship between image patch features; calculating the loss based on the global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the query video, and adjusting the parameters of the few-shot action recognition model; and identifying the category of the video to be recognized using the few-shot action recognition model. This method can improve the accuracy of video category recognition.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a small sample action recognition method based on multimodal and multi-level matching. Background Technology

[0002] In recent years, research in the field of Few-Shot Action Recognition (FSAR) has developed rapidly. The core goal of FSAR technology is to achieve accurate action classification with limited labeled data. In FSAR, metric-based methods dominate, originating from prototype networks, and have diverged into two distinct branches. One branch focuses on various matching strategies for video sequences, emphasizing the temporal relationships between sequences. The other branch focuses on prototype representation, tending to use global representations from all video frames to construct a representative prototype.

[0003] However, due to the inherent complexity of videos, even videos of the same category may exhibit low clustering performance due to differences in visual background and temporal distribution, resulting in high intra-class scatter. In the field of few-shot action recognition, since sampling for each category is randomized during each training iteration, high intra-class scatter leads to unstable prototype representations of each video action type based on few-shot video data. Furthermore, for some fine-grained video data, the similarity between different action categories results in low inter-class scatter, meaning that video prototypes of different categories are often quite similar, leading to low discriminative power between action categories. Relying solely on a small number of videos often makes it difficult to learn patterns that can distinguish subtle action details, resulting in low accuracy in few-shot action recognition. Summary of the Invention

[0004] Therefore, it is necessary to provide a few-sample action recognition method based on multimodal, multi-level matching to address the aforementioned technical problems. This method can improve the accuracy of video category recognition.

[0005] The present invention adopts the following technical solution:

[0006] This invention provides a few-sample action recognition method based on multimodal, multi-level matching, including:

[0007] Obtain the supporting video set and the query video set as the training set; the supporting video set includes supporting videos of multiple categories; the query video set includes query videos of multiple categories.

[0008] The few-shot action recognition model is trained using a training set. During training, based on the few-shot action recognition model, the global matching distance between the query video and the supporting video set is obtained by determining the cosine similarity between the global feature representations of the query video and the supporting video set. The frame-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the video frames of the query video and the supporting video set. The image patch-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the image patch features of the query video and the supporting video set.

[0009] Loss is calculated based on global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the video to be queried. The parameters of the few-shot action recognition model are then adjusted based on the calculated loss results to obtain a trained few-shot action recognition model.

[0010] The trained few-sample action recognition model identifies the category of the video to be recognized.

[0011] Preferably, the process of determining the global matching distance specifically includes:

[0012] Perform video averaging on videos of the same category in the supporting video set to obtain... N Original supporting video prototypes for each category;

[0013] The text encoder encodes all categories in the supporting video set to obtain the label prompts corresponding to each category in the supporting video set;

[0014] The original supporting video prototype and its corresponding tag prompts are merged separately to obtain... N A supporting video prototype;

[0015] Calculate the relationship between each query video in the query video set and... N Cosine similarity between supporting video prototypes;

[0016] according to N Each cosine similarity is used to determine the query video. N Dimensional class probability distribution, and N The probability distribution of each category is used to determine the global matching distance.

[0017] Preferably, the label prompt is as follows:

[0018] ;

[0019] in, Tokenizer For text encoders, prompt_template For text templates, For category i Real video action tags, For category i The label prompts, i ={1,2,..., N}, N The number of categories.

[0020] Preferably, the process of determining the frame-level matching distance specifically includes:

[0021] For each video to be queried Calculate the video to be queried separately The KL divergence between the target video and each supporting video prototype is calculated, and the supporting video prototype with the smallest KL divergence is determined as the target supporting video corresponding to the query video.

[0022] Video to be queried With supporting video prototype Subtract each frame sequentially, sum the results, and take the absolute value to obtain the video to be queried. The frame-level matching distance between the target supporting video prototype and the target.

[0023] Preferably, the formula for calculating the KL divergence is:

[0024] ;

[0025] in, T For an ideal matching distribution, M The number of frames in the video to be queried. , , D ( i , j ) is the first i Frame and the j The actual matching distribution between frames, D( i , k ) is the first i Frame and the k The actual matching distribution between frames, T ( i , j ) is the first i Frame and the j Ideal matching distribution between frames T ( i , k ) is the first i Frame and the k The actual matching distribution between frames, and These represent the row-normalized versions of D and T, respectively.

[0026] Preferably, the process of determining the image patch-level matching distance specifically includes:

[0027] For each supporting video prototype, the similarity between each frame of the query video and each frame of the supporting video prototype is calculated, and then analyzed using TOP. K Select the K frames that have the highest similarity to each frame in the video to be queried;

[0028] Calculate the similarity between the tag prompts corresponding to the supporting video prototype and the image patch features of the video to be queried, and generate weighted probabilities based on the similarity.

[0029] Based on weighted probabilities, the similarity between the image patch features of the query video and the image patch features of each supporting video prototype is weighted and summed.

[0030] The weighted summation result is determined as the image patch-level matching distance.

[0031] Preferably, loss calculation is performed based on global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the query video, specifically including:

[0032] Based on the global matching distance and the image patch-level matching distance respectively, the cross-entropy loss function is used in conjunction with the true category of the query video in the query video set to obtain the global level loss and the image patch level loss.

[0033] The loss at the frame sequence level is determined based on the frame-level matching distance;

[0034] The weighted sum of the global level loss, the frame sequence level loss, and the image patch level loss is determined as the loss of the few-sample action recognition model.

[0035] Preferably, the loss at the frame sequence level is determined based on the frame-level matching distance, specifically including:

[0036] Will N The frame-level matching distance of the supporting video prototype is converted into a frame-level distance vector;

[0037] The frame-level distance vector is determined as the loss at the frame sequence level.

[0038] Preferably, the formula corresponding to the loss of the few-sample action recognition model is:

[0039] ;

[0040] in, The loss of the few-sample action recognition model, For global level loss, For the loss at the frame sequence level, For the loss at the image patch block level, This represents the coefficient of the loss at the global level. These are the coefficients of the loss at the frame sequence level. The coefficients represent the loss at the image patch block level.

[0041] This invention provides a few-sample action recognition device based on multimodal and multi-level matching, comprising:

[0042] The acquisition module is used to acquire the supporting video set and the query video set as the training set; the supporting video set includes supporting videos of multiple categories; the query video set includes query videos of multiple categories.

[0043] The training module is used to train the few-shot action recognition model using the training set. During training, based on the few-shot action recognition model, the global matching distance between the query video and the supporting video set is obtained by determining the cosine similarity between the global feature representations of the query video and the supporting video set; the frame-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the video frames of the query video and the supporting video set; and the image patch-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the image patch features of the query video and the supporting video set.

[0044] The adjustment module is used to calculate the loss based on the global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the video to be queried, and adjust the parameters of the few-shot action recognition model based on the calculated loss results to obtain the trained few-shot action recognition model.

[0045] The recognition module is used to identify the category of the video to be recognized using a trained few-sample action recognition model.

[0046] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described small sample action recognition method based on multimodal and multi-level matching.

[0047] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described small sample action recognition method based on multimodal multi-level matching.

[0048] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects:

[0049] The few-shot action recognition model is trained using a support video set and a query video set. During training, based on the few-shot action recognition model, the global matching distance between the query video and the support video set is obtained by determining the cosine similarity between their global feature representations; the frame-level matching distance is obtained by analyzing the matching relationships between video frames in the query video and the support video set; and the image patch-level matching distance is obtained by analyzing the matching relationships between image patch features in the query video and the support video set. The global matching distance can capture the overall motion pattern of the video, solving the problem of macroscopic representation of long-term actions, while the frame-level matching distance can optimize short-term action clips through temporal alignment. Segment matching enhances temporal modeling capabilities, while image patch-level matching distance extracts local spatial details and improves fine-grained action discrimination. Loss calculations are performed based on global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the query video. The parameters of the few-shot action recognition model are adjusted based on the calculated loss results to obtain a trained few-shot action recognition model. The loss is calculated at the global level and the image patch level, respectively, using the cross-entropy loss function combined with the true category of the query video in the query video set. The frame sequence level loss is determined based on the frame-level matching distance. Finally, the trained few-shot action recognition model identifies the category of the video to be recognized. This method improves the accuracy of video category recognition. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0051] Figure 1 This is a schematic diagram illustrating video action classification errors provided by the present invention.

[0052] Figure 2 A schematic diagram of a small sample action recognition method based on multimodal and multi-level matching provided by the present invention;

[0053] Figure 3 This is a schematic diagram of frame-level matching provided by the present invention;

[0054] Figure 4 A schematic diagram of the small sample action recognition model provided by the present invention;

[0055] Figure 5 A schematic diagram of a small sample action recognition device based on multimodal and multi-level matching provided by the present invention;

[0056] Figure 6This is a schematic diagram of a computer device for implementing a small-sample action recognition method based on multimodal and multi-level matching, as provided by the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0058] Devices such as desktop computers, servers, and laptops are capable of executing the solutions of this invention. For ease of explanation, the following description will focus on servers as the executing entity.

[0059] Few-shot video action recognition technology emerged in the development of computer vision to address the challenges of data scarcity and model generalization in practical applications. In security monitoring, real-time detection and early warning of specific behaviors such as falls and running are required. Few-shot video action recognition can efficiently classify and detect actions, promptly identifying anomalies. In smart homes, recognizing and classifying the daily behaviors of family members, such as walking and watching television, helps to achieve personalized services and improve the intelligence level of the home. Video action recognition involves automatically analyzing and classifying the actions of people or objects in videos. Its technical implementation includes preprocessing, feature extraction, classifier design, and post-processing. Traditional methods typically capture temporal information by exploring the relationships between all frames, but this can be ineffective when dealing with changes in temporal order. Therefore, a model that can effectively integrate spatial information and temporal dynamics is needed, and few-shot video action recognition technology was developed to address this need.

[0060] The limitations of traditional video action recognition methods include: high data acquisition costs, as they rely on large-scale and strongly labeled video datasets, requiring significant human and material resources to acquire. For example, in specific fields such as medical action recognition, obtaining large amounts of labeled data is very difficult. Furthermore, there are challenges in model generalization ability. In reality, labeled data is often insufficient in many scenarios, and models are prone to overfitting on small sample sizes, resulting in poor generalization. For instance, in new surveillance scenarios, only a small number of labeled videos may be available for training the action recognition model.

[0061] Few-shot learning aims to identify new categories using a small number of samples. Its research can be broadly categorized into several groups, including data augmentation, model optimization, and metric-based methods. These studies have provided the theoretical and methodological foundation for few-shot video action recognition, driving the development of this technology.

[0062] Figure 1 This is a schematic diagram of video action classification errors provided by the present invention, such as... Figure 1 The left side of the image is a classification error caused by high intra-class dispersion. Figure 1 The right side of the diagram represents classification errors caused by low intra-class dispersion.

[0063] The method provided by this invention utilizes a Contrastive Language-Image Pre-training (CLIP) model and fully leverages the visual encoding output by the visual encoder at different levels and the text encoding output by the text encoder. This invention performs global-level matching, frame-sequence-level matching, and image patch-level matching between videos, from coarse-grained to fine-grained.

[0064] Since temporal matching is crucial for video analysis, the model in this invention needs to maintain sequence alignment to ensure its basic performance. This invention first employs a temporal Transformer mechanism to process the frame embeddings of the supporting video and the query video, which have already passed through the backbone network. Subsequently, this invention uses KL divergence to align the probability distributions of the supporting video and the query video between actual and ideal matches.

[0065] To address the problem of high intra-class dispersion, where the patterns of videos of the same category may differ significantly in different training rounds, this invention alleviates this problem by injecting textual information into visual information, since the labels for videos of the same category are fixed.

[0066] Regarding the issue of low inter-class dispersion, sequence matching based on average prototypes is not discriminative enough for fine-grained action classification.

[0067] However, in determining fine-grained video matching, the spatial fine-grainedness depends on the temporal fine-grainedness. Therefore, this invention first performs frame matching in the temporal dimension, and then performs local image patch matching in the spatial dimension based on the temporally matched frames, enabling the model to focus on visual details. To reduce noise, this invention reweights the contribution of each supporting image patch to the final matching score based on the similarity between the cue embedding and the image patch. For image patch-level matching, this invention specifically proposes a weighted Hausdorff distance for matching supporting videos and query videos.

[0068] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0069] Figure 2This is a flowchart illustrating a few-sample action recognition method based on multimodal and multi-level matching in this invention, specifically including the following steps:

[0070] S201: Obtain the support video set and query video set as training sets; the support video set includes support videos of multiple categories; the query video set includes query videos of multiple categories.

[0071] S202: Train the few-shot action recognition model using the training set; during training, based on the few-shot action recognition model, obtain the global matching distance between the query video and the supporting video set by determining the cosine similarity between the global feature representations of the query video and the supporting video set; obtain the frame-level matching distance between the query video and the supporting video set by the matching relationship between the video frames of the query video and the supporting video set; obtain the image patch-level matching distance between the query video and the supporting video set by the matching relationship between the image patch features of the query video and the supporting video set.

[0072] This invention uses CLIP's visual encoder as the backbone to extract features from video frames, while simultaneously using CLIP's text encoder to encode the cues for each category label. This invention modifies the return value of the CLIP visual encoder to provide not only frame features but also image patch features for each frame.

[0073] A modified CLIP pre-trained model is used to extract visual and textual features. The frame-level output of the CLIP model is... and ,in, T For each frame of the video, the image patch-level feature output is: ,in, , E It is the number of image patch blocks per frame, each and One-to-one correspondence, the feature output of CLIP's text feature encoder is .

[0074] In one exemplary embodiment, the process of determining the global matching distance specifically includes: performing a video averaging operation on videos of the same category in the supporting video set to obtain... N The original supporting video prototypes for each category are generated; all categories in the supporting video set are encoded using a text encoder to obtain the tag hints corresponding to each category in the supporting video set; the original supporting video prototypes and their corresponding tag hints are then merged to obtain... N A supporting video prototype; calculate the relationship between each query video in the query video set and... N Cosine similarity between supporting video prototypes; based onN Each cosine similarity is used to determine the query video. N Dimensional class probability distribution, and N The probability distribution of each category is used to determine the global matching distance.

[0075] Specifically, the global matching distance is dist g In the global-level matcher, this invention employs Video Global Representation Calibration (VGRC) to process each video, thereby generating a global representation for the corresponding video. This invention fuses the global representations of the supporting videos in the supporting video set with the embeddings of the supporting video category labels, and calculates the cosine distance between the fused features and the global representations of the query videos in the query video set, using this as the global-level matching score. The cross-entropy loss of the cosine distance and the ground truth labels serves as the global-level loss. By quantifying the global-level matching degree through cosine distance and guiding the model to learn semantic alignment through cross-entropy loss, this approach is suitable for tasks requiring refined similarity modeling, such as video retrieval and classification.

[0076] Given video frames ,in, Typically, mean aggregation is used to represent the global features of a video. However, since these frames are obtained through sampling, randomly sampled frames are not highly correlated with the main content of the video. Therefore, simply using average pooling cannot obtain the optimal global representation of the video and introduces inductive bias. To address this issue, this invention proposes calibrating the mean by considering the cosine similarity between different frames. By incorporating similarity into the aggregation process, this invention aims to optimize the representation of global features, ensuring that global features can more effectively capture subtle details.

[0077] The cosine similarity is calculated as shown in formula (1):

[0078] (1);

[0079] in, i and j The sequence number of the frame. For the first i frame, For the first j In a frame, <> represents the inner product operation, and |||| represents the modulo operation.

[0080] No. i The average similarity between a frame and other frames is calculated as shown in formula (2):

[0081] (2);

[0082] in, T For the number of frames, For other frames and the first i The average similarity of frames.

[0083] The calculation of the global video representation calibration is shown in formula (3):

[0084] (3);

[0085] in, Calibrate the global representation of the video.

[0086] In each training round, this invention uses the CLIP text encoder to obtain the tag hint embedding. The tag hint is shown in formula (4):

[0087] (4);

[0088] in, Tokenizer For text encoders, prompt_template For text templates, For category i Real video action tags, For category i The label prompts, , N The number of categories.

[0089] For each supporting video prototype in the supporting set The average operation of the enhanced prototype in this invention is shown in the following formula (5):

[0090] (5);

[0091] in, To enhance the supporting video prototype, For category i The label prompts, Indicates the first i One supporting sample, For the first i A prototype of a supporting sample.

[0092] Text embedding essentially captures the overall meaning of a video. The method provided by this invention generates a comprehensive visual representation with the aid of global video representation calibration. This invention achieves global alignment between the supporting video and the query video by comparing the global representation of the query video with the global representation of the supporting video prototype that has been enhanced with text embedding. The similarity, or registration probability, between a query sample and the global representation of each supporting video prototype is shown in Equation (6):

[0093] (6);

[0094] in, For similarity calculation, For temperature coefficient, For exponential operations, B To support the number of video prototypes, q To query samples, Indicates the first i One supporting sample.

[0095] In an exemplary embodiment, the process of determining the frame-level matching distance specifically includes: for each video to be queried Calculate the video to be queried separately Calculate the KL divergence between the query video and each supporting video prototype, and determine the supporting video prototype with the smallest KL divergence as the target supporting video corresponding to the query video; then, select the query video... With supporting video prototype Subtract each frame sequentially, sum the results, and take the absolute value to obtain the video to be queried. The frame-level matching distance between the target supporting video prototype and the target.

[0096] Specifically, the frame-level matching distance is dist f This invention continues to use a frame level matcher to process video frame features. For both the supporting video and the query video, this invention uses a Transformer to enable each frame to focus on the features of other frames in the sequence, thereby endowing the frame sequence with contextual features. Ideally, each frame of the supporting video should match each frame of the query video one-to-one. The matching probability is based on a two-dimensional Gaussian distribution along the diagonal of the frame matching. Therefore, this invention calculates the KL divergence between the actual matching distribution and the ideal matching distribution, using it as the frame level loss. The comparison of frame temporal order serves as the frame-level matching distance for the frame level.

[0097] Specifically, use This represents the mean feature value of multiple samples in a certain class of supporting samples. Therefore... and The comparison is the comparison between the query sample and the center point of a certain type of supporting sample.

[0098] Video frame matching is described as an optimal transmission problem. After processing by the Transformer, it is assumed that each frame can obtain information from other frames. Specifically, assume a query video... Actions belonging to a certain class, the supporting sample prototype of that class is... This method promotes The starting frame and Alignment of the starting frame in the middle, while prompting Subsequent frames in Alignment with subsequent frames. First, a text modality is introduced for each frame supporting the video prototype. ,in, and These are all weighting coefficients. Each frame is compared with... The matching score for each frame is converted into a probability distribution, and then expressed as D∈R T×T This represents the probability distribution obtained through matching actual samples.

[0099] If the two are strictly matched in the order of frames, then the ideal matching probability distribution is assumed to be: where the edge distribution along any straight line perpendicular to the diagonal is a Gaussian distribution centered at the intersection of the diagonals, as shown in formula (7):

[0100] (7);

[0101] in, l(i,j) Indicates position ( i , j The distance to the diagonal. The standard deviation is a Gaussian distribution. T The peak value is located on the diagonal and gradually decreases along a direction perpendicular to the diagonal. i , j That is, the first i Frame and the j frame.

[0102] The aforementioned prior conditions that preserve the temporal order are applied to few-shot video classification. Let... and They represent D and T The row-normalized version is shown in the following formula (8):

[0103] (8);

[0104] in, M The number of frames in the video to be queried. For position ( i , j Gaussian distribution, For position ( i , k Gaussian distribution, D ( i , j ) is the location ( i , j The actual probability distribution of ) D ( i , k ) is the location ( i , k The actual probability distribution of ) and its location ( i , j ) is the first i Frame and the j Frame, position ( i , k ) is the first i Frame and the k frame.

[0105] like Figure 3 The diagram shown is a frame-level matching schematic provided by the present invention. Figure 3 The left side of the graph represents the true probability distribution. D , Figure 3 The right side of the figure represents the ideal probability distribution. T The goal is to make the actual probability distribution approximate the ideal probability distribution. and The formula for calculating the KL divergence is shown in formula (9):

[0106] (9);

[0107] Where KL is the divergence calculation, D For the true probability distribution, for and The KL divergence.

[0108] The smaller this distance, the more similar the actual distribution is to the ideal distribution.

[0109] In the context of fine-grained alignment, the objective of this invention is to align all image patch blocks in each support sample and query sample. However, since the number of image patch blocks in each video is E×T, where, E It is the number of image patch blocks in each frame. T This approach presents two challenges regarding the number of frames per video: firstly, it involves a large amount of computation; secondly, excessive fragmentation patterns may lead to high ambiguity in the matching process.

[0110] Frame-level features processed by the CLIP visual encoder and temporal Transformer have already achieved good alignment performance. Therefore, image patches from the CLIP visual encoder are filtered based on matching scores between support and query sample features processed by the Transformer. Only image patches corresponding to highly matched frame features are selected. This approach allows us to reduce computational burden and alleviate matching ambiguity by focusing on a subset of image patches that are more likely to be relevant to alignment.

[0111] In an exemplary embodiment, the process of determining the image patch-level matching distance specifically includes: for each supporting video prototype, calculating the similarity between each frame of the query video and each frame of the supporting video prototype, and then using TOP... K Select the K frames with the highest similarity to each frame in the query video; calculate the similarity between the tag prompts corresponding to the supporting video prototypes and the image patch features of the query video, and generate weighted probabilities based on the similarity; based on the weighted probabilities, perform a weighted summation of the similarities between the image patch features of the query video and the image patch features of each supporting video prototype; determine the weighted summation result as the image patch-level matching distance.

[0112] Specifically, the image patch-level matching distance is dist p This invention uses a patches-level matcher to process fine-grained image patch-level features of the supporting video and the query video. The invention inputs the image patch features of the supporting video, along with the frame-level representations of the supporting video and the query video (processed by a Transformer), into an image patches filter. Based on the matching similarity of the supporting video and the query video at the frame level, the invention selects the K most similar supporting video frames for each query video frame. Then, the image patch-level features of each query video frame are matched with the patch features corresponding to the selected supporting video frames. Simultaneously, the similarity between the label features corresponding to the supporting video and the features of each selected image patch is calculated, generating a weighted probability. The similarity between each query image patch and each supporting image patch is weighted and summed based on this weighted probability. The weighted sum is used as the image patch-level distance at the patches level. The cross-entropy between the image patch-level distance and the ground truth label is used as the loss at the image patch level.

[0113] In the context of fine-grained alignment, the goal of this invention is to align all image patch blocks in each support sample and query sample. However, since the number of image patch blocks in each video is E×T, where E is the number of image patch blocks per frame and T is the number of frames in each video, this method presents two challenges: first, it is computationally intensive; second, excessive fragmentation patterns may lead to high ambiguity in the matching process.

[0114] Building upon previous research, frame-level features processed by the CLIP visual encoder and temporal Transformer have achieved good alignment performance. Therefore, image patches from the CLIP visual encoder are filtered based on matching scores between support and query sample features processed by the Transformer. Only image patches corresponding to highly matched frame features are selected. This approach allows us to reduce computational burden and alleviate matching ambiguity by focusing on a subset of image patches more likely to be relevant to alignment.

[0115] Assuming the given frame-level features are and Meanwhile, given the characteristics of the image patch block at the patch level, ,in, .

[0116] Each and One-to-one correspondence, and similarly, each and One-to-one correspondence, meaning one frame corresponds to E patches. For each frame feature of the query video, cosine similarity is used to calculate the similarity between the frame and the features of all frames in the supporting video. Then, the image patch features corresponding to the top K frames of the supporting video with the highest cosine similarity are selected. K The similarity is calculated as shown in formula (10):

[0117] (10);

[0118] in, For each query i The former K Large similarity, For each query i The former K The index of the supporting frame corresponding to the large similarity. K As a dimension, T The number of frames per video, query represents the videos in the query video set, and support represents the videos in the supporting video set.

[0119] Based on the selected supporting video The frame index, derived from the image patch block features supporting the video. The corresponding image patch features are selected from the data. The selected image patch features are shown in formula (11):

[0120] (11);

[0121] in, For the features of the selected image patch blocks, To extract values ​​from an input tensor based on a specified index, Image patch block features to support video.

[0122] Will Expand and display the supporting video. s The index of the filtered image patch features is redefined as ,in, E represents the number of patches in a frame pair.

[0123] according to time Come to Perform a flattening operation to obtain ,in, .

[0124] Based on the image patches selected by the image patch filter for both support and query samples, we next need to match these image patches. It's clear that within a single frame, some parts have almost no relation to the action. Similarly, when comparing two frames, there may be two similar image patches that are barely related to the action. Such alignment can mislead the final score. To suppress noise from semantically irrelevant image patch pairs, this invention proposes reweighting each image patch pair based on its semantic relevance to the action label cues.

[0125] The standard Hausdorff distance is shown in Equations (12)-(14):

[0126] (12);

[0127] This involves an operation that first finds the minimum value (min) and then the maximum value (max) within a set. To support the image patch block features of the video, D For Hausdorf distance.

[0128] (13);

[0129] The existing improved Hausdorff distance is shown in Equation (13):

[0130] (14);

[0131] in, For the existing improved Hausdorf distance, To support the number of sample prototypes. To query the number of samples.

[0132] This invention proposes a weighted front-K Hausdorff distance for image patch matching between support samples and query samples. As shown in equations (15) to (16):

[0133] (15);

[0134] in, To support the number of sample prototypes. To query the number of samples. As shown in formula (16).

[0135] (16);

[0136] in, To support the sample prototype s The similarity between a certain patch and its corresponding label semantic vector. This indicates that for a specific query sample q The number of image patches in the frames of the selected supporting samples for comparison. d The dimension of each vector.

[0137] The expression represents the probability distribution of similarity between the patch and label semantics of each selected supporting sample, indicating that for a specific query sample... q The number of image patches in the frame of the selected supporting sample for comparison.

[0138] These two parameters represent the semantic importance of the image patch block in the frame for each supporting sample.

[0139] S203: Calculate the loss based on the global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the video to be queried. Adjust the parameters of the few-shot action recognition model based on the calculated loss results to obtain the trained few-shot action recognition model.

[0140] In an exemplary embodiment, loss calculation is performed based on global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the query video. Specifically, this includes: processing the global-level loss and image patch-level loss respectively based on the global matching distance and image patch-level matching distance using the cross-entropy loss function in conjunction with the true category of the query video in the query video set; determining the frame sequence-level loss based on the frame-level matching distance; and determining the loss of the few-sample action recognition model by weighting the global-level loss, the frame sequence-level loss, and the image patch-level loss.

[0141] In an exemplary embodiment, determining the loss at the frame sequence level based on the frame-level matching distance specifically includes: N The frame-level matching distance of the supporting video prototype is converted into a frame-level distance vector; the frame-level distance vector is then used as the loss at the frame sequence level.

[0142] During the training phase, after three levels of matching, the global matching distance can be obtained. dist g ), frame-level matching distance ( dist f ) and image patch-level matching distance ( dist p For the global matching distance and the image patch-level matching distance, the cross-entropy loss function is used in conjunction with the true values ​​of the video tags to be queried, thereby obtaining the loss for different levels of matching.

[0143] In one exemplary embodiment, the loss of the few-sample action recognition model is shown in Equation (17):

[0144] (17);

[0145] in, The loss of the few-sample action recognition model, For global level loss, For the loss at the frame sequence level, For the loss at the image patch block level, This represents the coefficient of the loss at the global level. These are the coefficients of the loss at the frame sequence level. The coefficients represent the loss at the image patch block level.

[0146] S204: Identify the category of the video to be identified using a trained few-sample action recognition model.

[0147] In one exemplary embodiment, the present invention provides as follows Figure 4 The small sample action recognition model shown is an example. Figure 4As shown, the few-sample action recognition model includes a backbone network, a global hierarchical matcher, a frame sequence hierarchical matcher, and an image patch hierarchical matcher. C1, C2, and C3 represent the matching methods of different hierarchical matchers. Figure 4 The supporting sample set mentioned earlier in this invention is the supporting video set. Figure 4 The query sample set is the same as the query video set mentioned earlier. First, CLIP's visual encoder is used as the backbone to extract features from video frames, while CLIP's text encoder encodes the prompts for each category label. This invention modifies the return value of the CLIP visual encoder so that it not only provides frame features but also returns the image patch features for each frame. The support sample set, a known sample set, provides a small amount of labeled data for the model to learn how to quickly adapt to new categories or tasks. It contains three categories (cat, dog, bird), with two labeled images for each category. The model learns the feature representation of "cat / dog / bird" based on these six images, providing prior knowledge and constraining the model's learning direction (e.g., generating class prototypes). The query sample set is an unknown sample set used to evaluate whether the model can deduce unseen data of the same category based on the knowledge learned from the support sample set. It contains 15 images (5 images per category) of the same three categories, with labels hidden from the model. The model needs to classify the query images into the three categories and calculate the accuracy, providing prior knowledge and constraining the model's learning direction (e.g., generating class prototypes). Figure 4 The dark green squares in the image represent samples from the support sample set, and the modified CLIP output frame sequence. Figure 4 In the image, four squares represent a frame. The light green square represents the global sequence of the original output of the clip, and one square is one frame. The dark brown square represents the sample in the query sample set, and the light brown square represents the global sequence of the original output of the clip corresponding to the query sample set.

[0148] Secondly, a Global Level Matcher is used to process video frame features. In the Global Level Matcher, Video Global Representation Calibration (VGRC) is employed to process each support video and query video, generating a global representation for each video. The global representation of the support video is fused with the embedding of the support video category label hints, and the cosine distance between the fused features and the global representation of the query video is calculated, serving as the global level matching score. The cross-entropy of distance and ground truth labels is used as the loss. Quantifying the global level matching degree through cosine distance and using cross-entropy loss to guide the model in learning semantic alignment is suitable for tasks requiring refined similarity modeling, such as video retrieval and classification.

[0149] Next, the Frame Level Matcher is used to process video frame features. For both the supporting and query videos, a Transformer is used to enable each frame to focus on features of other frames in the sequence, thus giving them sequence context features. Ideally, each frame of the supporting video should match each frame of the query video one-to-one. The matching probability is based on a two-dimensional Gaussian distribution along the diagonal of the frame matching. Therefore, this invention calculates the KL divergence between the actual matching distribution and the ideal matching distribution, using it as the loss for the Frame Level. The comparison of frame temporal sequences is used as the distance between the Frame Levels.

[0150] Then, this invention uses a patches-level matcher to process the fine-grained image patch-level features of the supporting video and the query video. This invention inputs the image patch features of the supporting video, along with the frame-level representations of the supporting video and the query video (which have already been processed by the Transformer), into an image patches filter. Based on the matching similarity of the supporting video and the query video at the frame level features, this invention selects the K most similar supporting video frames for each query video frame. Then, the image patch-level features of each query video frame are matched with the patch features corresponding to the selected supporting video frames. Simultaneously, the similarity between the label features corresponding to the supporting video and the features of each selected image patch is calculated, generating a weighted probability. Based on this probability, the similarity between each query image patch and each supporting image patch is weighted and summed. This sum is used as the patches-level matching score. Distance and the cross-entropy of the ground truth labels are used as the loss.

[0151] Finally, the present invention fuses matching scores from the global level, frame sequence level, and image patch block level, and compares them with the true labels to calculate the loss.

[0152] When applying the small-sample action recognition method based on multimodal and multi-level matching provided by this invention, it is not necessary to rely on... Figure 2 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this invention does not impose any restrictions on it.

[0153] The above describes a few-shot action recognition method based on multimodal, multi-level matching, provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding few-shot action recognition device based on multimodal, multi-level matching, such as… Figure 5 As shown.

[0154] Figure 5A schematic diagram of a small-sample action recognition device based on multimodal and multi-level matching provided by the present invention includes:

[0155] The acquisition module 501 is used to acquire a support video set and a query video set as training sets; the support video set includes support videos of multiple categories; the query video set includes query videos of multiple categories.

[0156] Training module 502 is used to train the few-shot action recognition model using the training set. During training, based on the few-shot action recognition model, the global matching distance between the query video and the supporting video set is obtained by determining the cosine similarity between the global feature representations of the query video and the supporting video set. The frame-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the video frames of the query video and the supporting video set. The image patch-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the image patch features of the query video and the supporting video set.

[0157] The adjustment module 503 is used to calculate the loss based on the global matching distance, frame-level matching distance, image patch-level matching distance, and the true category of the video to be queried, and to adjust the parameters of the few-shot action recognition model based on the calculated loss results to obtain the trained few-shot action recognition model.

[0158] The recognition module 504 is used to identify the category of the video to be recognized using a trained few-sample action recognition model.

[0159] Specific limitations regarding the device for the few-sample action recognition method based on multimodal multi-level matching can be found in the limitations of the method described above, and will not be repeated here. Each module in the aforementioned device for the few-sample action recognition method based on multimodal multi-level matching can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0160] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 A few-sample action recognition method based on multimodal and multi-level matching is provided.

[0161] The present invention also provides Figure 6 The schematic diagram of the computer device shown is as follows: Figure 6As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 2 A few-sample action recognition method based on multimodal and multi-level matching is provided.

[0162] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A few-sample action recognition method based on multimodal, multi-level matching, characterized in that, include: Obtain the supporting video set and the query video set as the training set; The supporting video set includes supporting videos of multiple categories; The query video set includes videos from multiple categories. The few-sample action recognition model is trained using the training set. During training, based on the few-sample action recognition model, the global matching distance between the query video and the supporting video set is obtained by determining the cosine similarity between the global feature representations of the query video and the supporting video set; the frame-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the video frames of the query video and the supporting video set; and the image patch-level matching distance between the query video and the supporting video set is obtained by the matching relationship between the image patch features of the query video and the supporting video set. The loss is calculated based on the global matching distance, the frame-level matching distance, the image patch-level matching distance, and the true category of the video to be queried. The parameters of the few-shot action recognition model are then adjusted based on the calculated loss results to obtain the trained few-shot action recognition model. The trained few-sample action recognition model identifies the category of the video to be recognized; The process of determining the image patch-level matching distance specifically includes: for each supporting video prototype, calculating the similarity between each frame of the query video and each frame of the supporting video prototype, and then using TOP... K Select the K frames with the highest similarity to each frame in the query video; calculate the similarity between the tag prompts corresponding to the supporting video prototypes and the image patch features of the query video, and generate weighted probabilities based on the similarity; based on the weighted probabilities, perform a weighted summation of the similarities between the image patch features of the query video and the image patch features of each supporting video prototype; determine the weighted summation result as the image patch-level matching distance; the formula for calculating the image patch-level matching distance is: ； in, To support the number of sample prototypes. To query the number of samples; ； in, To support the sample prototype s The similarity between patches and their corresponding label semantic vectors. This indicates that for a specific query sample q The number of image patches in the frames of the selected supporting samples for comparison. d The dimension of the semantic vector for each label.

2. The method as described in claim 1, characterized in that, The process of determining the global matching distance specifically includes: Perform video averaging on videos of the same category in the supporting video set to obtain... N Original supporting video prototypes for each category; The text encoder encodes all categories in the supporting video set to obtain the label prompts corresponding to each category in the supporting video set; The original supporting video prototype and its corresponding tag prompts are merged separately to obtain... N A supporting video prototype; Calculate the relationship between each query video in the query video set and... N Cosine similarity between supporting video prototypes; according to N Each cosine similarity is used to determine the query video. N Dimensional category probability distribution, and the N The probability distribution of each category is determined as the global matching distance.

3. The method as described in claim 2, characterized in that, The label indicates: ； in, Tokenizer For text encoders, prompt_template For text templates, For category i Real video action tags, For category i The label prompts, i ={1,2,..., N }, N The number of categories.

4. The method as described in claim 2, characterized in that, The process of determining the frame-level matching distance specifically includes: For each video to be queried Calculate the video to be queried separately The KL divergence between each supporting video prototype and the target supporting video corresponding to the query video is determined. Video to be queried With supporting video prototype Subtract each frame sequentially, sum the results, and take the absolute value to obtain the video to be queried. The frame-level matching distance between the target supporting video prototype and the target.

5. The method as described in claim 4, characterized in that, The formula for calculating the KL divergence is: ； in, T For an ideal matching distribution, M The number of frames in the video to be queried. , , D ( i , j ) is the first i Frame and the j The actual matching distribution between frames, D( i , k ) is the first i Frame and the k The actual matching distribution between frames, T ( i , j ) is the first i Frame and the j Ideal matching distribution between frames T ( i , k ) is the first i Frame and the k The actual matching distribution between frames, and These represent the row-normalized versions of D and T, respectively.

6. The method as described in claim 1, characterized in that, The loss calculation based on the global matching distance, the frame-level matching distance, the image patch-level matching distance, and the true category of the video to be queried specifically includes: Based on the global matching distance and the image patch-level matching distance, respectively, the cross-entropy loss function is used in conjunction with the true category of the query video in the query video set to obtain the global-level loss and the image patch-level loss. Based on the frame-level matching distance, determine the loss at the frame sequence level; The weighted sum of the global level loss, the frame sequence level loss, and the image patch level loss is determined as the loss of the few-sample action recognition model.

7. The method as described in claim 6, characterized in that, The step of determining the loss at the frame sequence level based on the frame-level matching distance specifically includes: Will N The frame-level matching distance of the supporting video prototype is converted into a frame-level distance vector; The frame-level distance vector is determined as the loss at the frame sequence level.

8. The method as described in claim 7, characterized in that, The formula for the loss of the few-sample action recognition model is: ； in, The loss of the few-sample action recognition model, For global level loss, For the loss at the frame sequence level, For the loss at the image patch block level, This represents the coefficient of the loss at the global level. These are the coefficients of the loss at the frame sequence level. The coefficients represent the loss at the image patch block level.

Images