Small sample action recognition method based on semantic-time sequence adaptive representation learning

Abstract

The invention relates to a small sample action recognition method based on semantic-time sequence adaptive representation learning, which comprises the following steps of: randomly sampling action categories, and constructing a task sample pair containing a support set and a query set; obtaining temporal and spatial features of category semantic embedding and support samples and query samples; based on an explicit semantic guidance focusing mechanism, the spatial-temporal features of the support samples and corresponding category semantic embedding are aligned to obtain semantic enhanced spatial-temporal features, and multi-frequency sub-sequence construction, one-way modeling and two-way modeling are sequentially carried out on the semantic enhanced spatial-temporal features and the spatial-temporal features of the query samples to obtain the spatial-temporal features of the query samples. Focusing time sequence prototype representation is obtained, and prototype matching is carried out; performing fine-grained video-text alignment on the spatial-temporal characteristics and category semantic embedding of the support sample by using a frame-level cross-modal attention mechanism; prototype matching is carried out on the to-be-recognized sample, and recognition of the query set action category under the condition of few samples is achieved. Compared with the prior art, the method has the advantages of effectively relieving insufficient characterization capability under the condition of few samples and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and video understanding, and in particular to a small sample action recognition method based on semantic-temporal adaptive representation learning. BACKGROUND

[0002] Few-Shot Action Recognition (FSAR) is an important task in video understanding, aiming to achieve accurate classification of new class actions under the condition of only a few labeled samples. This task is of great significance to solve the problem of high labeling cost in large-scale video data. However, due to the limited number of data and class distribution in the small sample scene, it still faces many challenges to improve the generalization ability of the model on new classes.

[0003] CN118747916A discloses a small sample action recognition method based on semantic guidance multi-modal fusion, which generates rich and comprehensive text knowledge covering various action categories by using a large language model, ensures the comprehensiveness of the extracted semantic information of the small sample action recognition task, and performs preliminary classification in the text branch by matching and measuring the extracted discriminative semantic information and visual information of unknown class samples. In addition, a semantic-guided visual interaction module is designed in the visual branch to promote the effective integration of semantic and visual information, improve the quality of feature representation in the sample, and enable more timely understanding of new classes with only a small number of samples. However, this method has two defects: 1) This method relies on a prototype construction mechanism based on metric learning, trying to complete the recognition of new classes through global representation matching. However, this kind of method usually ignores the key dynamic changes in action sequences when constructing video prototypes, only using compression operations such as global average pooling, which makes it difficult to capture the subtle correspondence between action dynamics and semantic description, thereby limiting the classification performance. 2) Although this method considers using action category information to provide guidance for temporal modeling, helping the model focus on key action segments and improving the discriminability between classes. However, this method lacks explicit modeling of the association between semantics and time, and cannot fully utilize semantic information for effective guidance. In complex or semantically ambiguous action scenarios, there may be deviations or even conflicts between text semantics and visual features, which further affects the model's understanding and discrimination ability of action categories.

[0004] Therefore, the existing FSAR methods still have significant deficiencies in fine-grained temporal modeling, semantic guidance mechanism, and cross-modal feature alignment, and there is an urgent need for a new representation learning framework that can realize semantic-temporal collaborative modeling to improve the recognition ability and generalization performance of the model in the small sample scene. SUMMARY

[0005] The application aims to provide a small sample action recognition method based on semantic-temporal adaptive representation learning, solve the problems of insufficient semantic guidance, inaccurate temporal modeling and insufficient cross-modal alignment in the prior art, and enhance the generalization ability of the model.

[0006] The application aims to provide a small sample action recognition method based on semantic-temporal adaptive representation learning, solve the problems of insufficient semantic guidance, inaccurate temporal modeling and insufficient cross-modal alignment in the prior art, and enhance the generalization ability of the model.

[0007] A small sample action recognition method based on semantic-temporal adaptive representation learning comprises the following steps:

[0008] In each round of scenario learning task, action categories are randomly sampled from the training video data set to construct a task sample pair containing a support set and a query set;

[0009] Temporal dependent semantic category descriptions generated by an offline language model are obtained, category semantic embeddings are obtained by using a text encoder, and spatiotemporal features of the support sample and the query sample are respectively extracted by using a visual encoder;

[0010] The spatiotemporal features of the support sample are aligned with the corresponding category semantic embedding based on an explicit semantic guidance focusing mechanism to obtain semantic enhanced spatiotemporal features, and the semantic enhanced spatiotemporal features and the spatiotemporal features of the query sample are sequentially subjected to multi-frequency subsequence construction, one-way modeling based on an action-specific dynamic temporal modeling module, first feature fusion, two-way modeling based on an action center unified temporal modeling module and second feature fusion to obtain focused temporal prototype representations of the support sample and the query sample, the focused temporal prototype representations of all support samples in the support set are clustered to obtain category prototypes, a matching distance between the focused temporal prototype representation of the query sample and the category prototypes is calculated to generate a matching probability distribution, a few sample matching loss is calculated based on the matching probability distribution to optimize the prototype matching process;

[0011] The spatiotemporal features of the support sample and the category semantic embedding are subjected to fine-grained video-text alignment by using a frame-level cross-modal attention mechanism to obtain category enhanced representations, a video-text alignment loss is calculated based on the similarity between the category enhanced representations and the corresponding category semantic embedding to optimize the video-text alignment target and adjust the spatiotemporal feature representation of the support set sample;

[0012] The spatiotemporal features of the support sample and the category semantic embedding are subjected to fine-grained video-text alignment by using a frame-level cross-modal attention mechanism to obtain category enhanced representations, a video-text alignment loss is calculated based on the similarity between the category enhanced representations and the corresponding category semantic embedding to optimize the video-text alignment target and adjust the spatiotemporal feature representation of the support set sample;

[0013] The spatiotemporal features of the support sample and the category semantic embedding are subjected to fine-grained video-text alignment by using a frame-level cross-modal attention mechanism to obtain category enhanced representations, a video-text alignment loss is calculated based on the similarity between the category enhanced representations and the corresponding category semantic embedding to optimize the video-text alignment target and adjust the spatiotemporal feature representation of the support set sample;

[0014] Let the spatiotemporal feature of the kth support sample be The corresponding category semantic embedding is T k ∈R D Where F represents the number of video frames and D represents the feature embedding dimension;

[0015] After normalizing the category semantic embedding, a standardized text representation is obtained:

[0016] T norm =LayerNorm(T k )

[0017] Where LayerNorm represents normalization, T norm A standardized text representation representing category semantic embedding;

[0018] Temporal semantic weights are calculated on the standardized text representations of the spatiotemporal features and category semantic embeddings of the supporting samples to obtain the frame-level semantic relevance distribution:

[0019]

[0020] Among them, W video-text For temporal semantic weights, it represents the degree of association between each frame and the current semantics;

[0021] The temporal semantic weights are broadcast along the channel dimension and then residually fused with the element-wise weighted spatiotemporal features of the supporting samples to generate semantically enhanced spatiotemporal features:

[0022]

[0023] Here, ⊙ represents the multiplication operation. This represents semantically enhanced spatiotemporal features.

[0024] The construction of the multi-frequency subsequence specifically involves: to simulate action change patterns at different time granularities, constructing multi-frequency subsequences from semantically enhanced spatiotemporal features and query sample spatiotemporal features to capture the dynamic features of actions at various time scales.

[0025] Define a set of time steps W = {w1, w2, ..., w n}, where n is the number of time steps. For each time step w∈W, a set of subsequences with different starting offsets is constructed from the semantically enhanced spatiotemporal features and the spatiotemporal features of the query samples:

[0026]

[0027] Where 'o' represents the starting offset. Enhancing Spatiotemporal Features with Semantic Meaning The subsequence corresponding to step size w and starting offset o. To query the spatiotemporal features v of the sample qcorresponding sub-sequences under the step length w and the starting offset o, respectively, are semantic enhanced spatio-temporal features and sub-sequence sets of spatio-temporal features of query samples.

[0028] The one-way modeling of the action-specific dynamic time series modeling module is specifically: to extract local dynamic change features in each sub-sequence, the action-specific dynamic time series modeling module is used to perform one-way modeling on each sub-sequence to obtain representation results under a time causal structure, wherein the action-specific dynamic time series modeling module takes the sub-sequence corresponding to the semantic enhanced spatio-temporal feature and the sub-sequence corresponding to the spatio-temporal feature of the query sample as inputs, adopts a discretized one-way state space modeling operation TSSM to establish an expression relationship of system state evolution over time, and obtains corresponding local modeling outputs:

[0029]

[0030] wherein TSSM fwd (·) represents a state space modeling operation under a forward time flow, respectively, are local modeling results of the sub-sequence corresponding to the semantic enhanced spatio-temporal feature and the sub-sequence corresponding to the spatio-temporal feature of the query sample.

[0031] The first feature fusion is specifically: to unify the sub-sequence local modeling results obtained under different step lengths to a consistent time scale, and to realize cross-offset and cross-frequency feature fusion, the sub-sequence local modeling results are up-sampled and integrated in the time dimension to form an intermediate representation with a global time series structure, wherein

[0032] For the local modeling result of the semantic enhanced spatio-temporal feature of the support video under each step length w is up-sampled to a length F consistent with the original video frame number through an interpolation operation, and is integrated in the time dimension, and then

[0033] is superimposed with the semantic enhanced spatio-temporal feature to obtain a corresponding first multi-frequency fusion feature

[0034]

[0035] For the local modeling result of the spatio-temporal feature of the query video under each step length w only up-sampling and splicing operations are performed to obtain a corresponding second multi-frequency fusion feature

[0036]

[0037] wherein Upsample(·) represents time interpolation on each sub-sequence to align the frame number to F; and Concat(·) represents splicing and fusing multiple offset sub-sequences along the time dimension.

[0038] The bidirectional modeling of the action center unified timing modeling module is specifically: to enhance the perception of global dependency structure of timing modeling, a bidirectional state space modeling mechanism is introduced, the forward state space modeling module and the reverse state space modeling module in the action center unified timing modeling module are used for forward and reverse timing encoding fusion, and unified modeling of semantic continuity before and after the action is realized: the first multi-frequency fusion feature obtained through the first feature fusion and the second multi-frequency fusion feature respectively construct the time-reversed sequence thereof:

[0039]

[0040] In the time forward direction, the forward state space modeling module is input; in the time reverse direction, the reverse state space modeling module is input, and the output result is reversed to restore the order and fused with the forward result to obtain the final bidirectional modeling result:

[0041]

[0042] wherein TSSM fwd (·) and TSSM bwd (·) respectively represent state space modeling operations in the time forward and reverse directions; Reverse(·) represents a reverse operation in the time dimension, respectively represent the bidirectional modeling results corresponding to the semantic enhanced spatio-temporal feature and the spatio-temporal feature of the query sample.

[0043] The second feature fusion is specifically: to unify the modeling results at different time steps, the bidirectional modeling results at multiple frequencies are weighted and fused, and a global context guided channel adjustment mechanism is combined to generate the final focused timing prototype representation:

[0044] At all steps w e W, the bidirectional modeling results corresponding to the semantic enhanced spatio-temporal feature and the spatio-temporal feature of the query sample are respectively aggregated and fused to obtain a multi-scale fusion output:

[0045]

[0046] wherein a w is the aggregation weight of the step w, and Refine(·) represents a channel adaptive adjustment function, which adjusts the weight distribution of each channel according to global feature statistical information, respectively represent the focused timing prototype representations of the support sample and the query sample.

[0047] The focused time sequence prototype representation of the support set is clustered to obtain a class prototype, and the matching distance of the focused time sequence prototype representation of the query sample and the class prototype is calculated to generate a matching probability distribution, specifically:

[0048] The focused time sequence prototype representation of the K support samples for each class c in the support set is averaged and aggregated within the class to obtain the class prototype U of the class. c :

[0049]

[0050] The OTAM metric strategy based on optimal transport is used to calculate the matching distance dist(q, c) between the focused time sequence prototype representation of the query sample and the class prototype U. c The specific steps are as follows:

[0051]

[0052] Where OTAM(·,·) represents the OTAM metric strategy, and the cost is updated by forward dynamic accumulation, and <·,·> represents the inner product of the feature vector.

[0053] Based on the calculated matching distance dist(q, c), the relative matching strength of each class is generated to generate the matching probability distribution of the query sample under all support classes:

[0054]

[0055] Where p few-shot (y=i∣q) represents the matching probability distribution, dist(q,c i ) represents the matching distance between the focused time sequence prototype representation of the query sample q and the class prototype U ci corresponding to class i, and N is the number of classes.

[0056] The few-shot matching loss is defined in the form of cross-entropy:

[0057] L few-shot =-logp few-shot (y=y q ∣∣q)

[0058] Where L few-shot is the few-shot matching loss, y q is the true class label of the query sample, and p few-shot is the matching probability distribution.

[0059] The process of optimizing the video-text alignment target specifically includes the following steps:​​

[0060] The normalized text representation T obtained after normalization processing based on the category semantic embedding norm The query matrix is constructed based on the frame-level spatio-temporal features V of the support set and the query set videos s And V q Form a key-value pair V ∈ {V s ,V q} to perform multi-head attention calculation to realize fine-grained video-text alignment, and output the category enhanced representation V enhanced ∈ R N×D N is the number of categories, and D represents the feature embedding dimension;

[0061] The similarity between the extracted category enhanced representation V enhanced and its corresponding category semantic embedding is calculated to obtain a video-text alignment loss, which optimizes the video-text alignment target and adjusts the spatio-temporal feature representation of the support set sample, and the video-text alignment loss is represented as:

[0062]

[0063] Wherein, is the video-text alignment loss, represents the category enhanced representation most relevant to the i-th category in the video, c (j) ∈ R D represents the category semantic embedding of the j-th category, sim(·,·) represents the normalized cosine similarity calculation, τ ∈ R + is a temperature parameter that can be adjusted, which is used to control the flexibility of the similarity distribution.

[0064] Compared with the prior art, the present application has the following beneficial effects:

[0065] (1) The present application proposes a novel semantic-temporal adaptive representation learning method, which realizes the collaborative optimization of explicit semantic guidance and temporal modeling in the small sample action recognition (FSAR) task by introducing semantic prior enhancement and multi-scale dynamic temporal modeling. The present application establishes a semantic-temporal collaborative path at the frame level, so that the model has stronger discriminative ability in both cross-modal alignment and temporal modeling dimensions, effectively alleviating the problem of insufficient representation ability under the condition of small sample.

[0066] (2) To solve the problem of insufficient existing semantic guidance ability, the application designs a frame-level cross-modal attention mechanism to adaptively align the key information in the video frame and the text, and enhance the fine-grained association expression between vision and semantics. At the same time, unlike the static category embedding constructed by manual construction in the prior art, the application proposes to combine the time-dependent semantic description generated by the offline large language model to provide long-range and structured semantic guidance at the category level, and inject explicit semantic prior into subsequent time series modeling, thereby improving the understanding and differentiation ability of the model for complex action categories;

[0067] (3) To further enhance the dynamic modeling ability of the time series features, the application develops a key action-oriented focus prototype synthesis module, which first explicitly realizes the collaborative modeling of semantics and time series in the FSAR task. The module guides the model to focus on the foreground action area by fusing semantic and time series information. Unlike the static "long-short term modeling" time series modeling method of the prior art, the mechanism further introduces a multi-frequency sampling mechanism and a state space recursive model to dynamically extract and update hierarchical time series information; at the same time, it introduces bidirectional time series modeling to strengthen context dependence and improve the model's perception ability of the relationship between actions. Finally, under the guidance of semantics, a discriminative time series prototype is generated, effectively improving the recognition accuracy and thus improving the recognition accuracy under small sample conditions. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The method flowchart of the application;

[0069] Figure 2 The small sample action recognition process diagram of the application;

[0070] Figure 3 The focus prototype synthesis module corresponding to the support video of the application. DETAILED DESCRIPTION

[0071] The application will be described in detail below in combination with the drawings and specific embodiments. The embodiments are implemented on the premise of the technical solution of the application, and detailed implementation methods and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.

[0072] The embodiment provides a small sample action recognition method based on semantic-time series adaptive representation learning, as shown in Figure 1 The method comprises the following steps:

[0073] S1, in each round of scenario learning task, randomly sample action categories from the training video data set to construct a task sample pair containing a support set and a query set.

[0074] Specifically, in each round of scenario learning task, N action categories are randomly sampled from the training dataset to construct a task sample pair containing a support set and a query set to provide supervision information in the current task; meanwhile, P samples are randomly selected from the remaining samples under the same category to construct the query set to verify the recognition ability of the model in this task. All samples in the support set and the query set are accompanied by explicit action category labels, and their categories are limited to the N categories sampled in the current round, i.e., there is no category crossing. To avoid information leakage, the sample instances in the support set and the query set are mutually exclusive in data, i.e., they satisfy

[0075] S2, obtain the time-dependent semantic category description generated by the offline language model, obtain the category semantic embedding using the text encoder, and extract the spatio-temporal features of the support samples and the query samples using the visual encoder.

[0076] Specifically, the embodiment uses the visual encoder of the contrastive learning language-image pre-training model (CLIP) to extract the spatio-temporal features V s and V q of the support set and the query set, respectively, where the feature dimension of each video is FxD, F represents the number of video frames, and D represents the feature embedding dimension. At the same time, the time-dependent semantic category description C′ train generated by the offline language model is obtained by the text encoder of CLIP to obtain the category embedding T c ∈R N×D .

[0077] S3, as shown in Figure 2 , in the prototype construction stage, based on the obtained spatio-temporal features and category semantic embedding, the focus prototype synthesis module and the explicit semantic guided focus mechanism are used to realize the collaborative modeling of semantic information and time sequence features, combined with multi-scale frequency sampling and bidirectional state space modeling strategy, to generate the prototype of the focused key action. Based on the constructed category prototype representation and the features of the query video, the frame-level time sequence matching relationship is calculated to realize the matching of the support set and the query set.

[0078] The model of the focus prototype synthesis module is as shown in Figure 3 , which corresponds to the processing process of the following steps S31-S36.

[0079] Specifically, S3 includes the following steps:

[0080] S31. To construct an explicit semantic-temporal association mechanism, the spatiotemporal features of supporting samples are aligned with their corresponding category semantic embeddings based on an explicit semantic guidance focusing mechanism, thereby obtaining semantically enhanced spatiotemporal features and enhancing the model's semantic perception ability of key action frames.

[0081] Let the spatiotemporal features of the k-th support sample be . The corresponding category semantic embedding is T k ∈R D Where F represents the number of video frames and D represents the feature embedding dimension;

[0082] After normalizing the category semantic embedding, a standardized text representation is obtained:

[0083] T norm =LayerNorm(T k )

[0084] Where LayerNorm represents normalization, T norm A standardized text representation representing category semantic embedding.

[0085] Temporal semantic weights are calculated on the standardized text representations of the spatiotemporal features and category semantic embeddings of the supporting samples to obtain the frame-level semantic relevance distribution:

[0086]

[0087] Among them, W video-text The temporal semantic weight represents the degree of association between each frame and the current semantics.

[0088] The temporal semantic weights are broadcast along the channel dimension and then residually fused with the element-wise weighted spatiotemporal features of the supporting samples to generate semantically enhanced spatiotemporal features:

[0089]

[0090] Here, ⊙ represents a multiplication operation, which enhances the ability to express semantically significant regions of actions and improves the model's ability to identify key moments. This represents semantically enhanced spatiotemporal features.

[0091] S32, in order to simulate the action change patterns at different time granularities, constructs multi-frequency subsequences for the semantically enhanced spatiotemporal features and the spatiotemporal features of the query samples respectively, to capture the dynamic features of actions at each time scale.

[0092] Define a set of time steps W = {w1, w2, ..., w n}, where n is the number of time steps. For each time step w∈W, a set of subsequences with different starting offsets is constructed from the semantically enhanced spatiotemporal features and the spatiotemporal features of the query samples:

[0093]

[0094] where o denotes the starting offset, semantic-enhanced spatio-temporal features corresponding sub-sequence under the step size w and the starting offset o, spatio-temporal features of the query sample v q corresponding sub-sequence under the step size w and the starting offset o, semantic-enhanced spatio-temporal features and sub-sequence set of spatio-temporal features of the query sample, respectively.

[0095] and each sub-sequence represents a time series segment obtained by sampling under the step size w and the offset o, which covers the action evolution state under different time densities, and is helpful to extract rich dynamic clues in subsequent modeling.

[0096] S33, one-way modeling based on the action-specific dynamic time series modeling module.

[0097] To extract the local dynamic change features in each sub-sequence, the action-specific dynamic time series modeling module is used to model each sub-sequence in one direction, and the representation result under the time causal structure is obtained. The action-specific dynamic time series modeling module takes the sub-sequence corresponding to the semantic-enhanced spatio-temporal features and the sub-sequence corresponding to the spatio-temporal features of the query sample v as inputs, respectively, uses the discretized one-way state space modeling operation (Temporal State Space Model, TSSM) to establish the expression relationship of the system state evolution over time, and obtains the corresponding local modeling output:

[0098]

[0099] where TSSM fwd (·) represents the state space modeling operation under the forward time flow, which can capture the action evolution process in the local area and improve the time sensitivity, semantic-enhanced spatio-temporal features and sub-sequence of the spatio-temporal features of the query sample, respectively.

[0100] wherein the process of the one-way state space modeling operation is as follows:

[0101] The model under continuous time can be represented as:

[0102] h'(t) = Ah(t) + Bx(t), y(t) = Ch(t)

[0103] where h(t) ∈ RH x(t) e R is the hidden state D y(t) e R is the input feature D A e R is the output feature H×H B, C e R H×D are state transition matrix, input projection matrix and output mapping matrix respectively.

[0104] Considering the practical requirements of discrete sequence modeling, a learnable time scale parameter L is introduced

[0105]

[0106] At discrete time step t, state update and output calculation are expressed as:

[0107]

[0108] Then, one-way state space modeling operation (TSSM) is performed on each subsequence to obtain local dynamic representation. The above formula represents that the state at the current time is determined by the previous state and the current input, and is mapped to the result space through the output matrix C, which has explicit time causality.

[0109] S34, first feature fusion

[0110] To unify the local modeling results of sub-sequences obtained under different step lengths to a consistent time scale, and realize cross-offset and cross-frequency feature fusion, the local modeling results of sub-sequences are up-sampled and integrated in the time dimension to form an intermediate representation with global time sequence structure, wherein

[0111] For the local modeling results of the semantic enhanced spatio-temporal features of the video at each step w Up-sampling to the length F consistent with the original video frame number through interpolation operation, and then splicing and integrating in the time dimension, followed by residual superposition with the semantic enhanced spatio-temporal features to obtain the corresponding first multi-frequency fusion features

[0112]

[0113] For the local modeling results of the spatio-temporal features of the query video at each step w Only up-sampling and splicing operations are performed to obtain the corresponding second multi-frequency fusion features

[0114]

[0115] wherein Upsample(·) denotes time interpolation on each sub-sequence to align its frame number to F; Concat(·) denotes concatenation of multiple offset sub-sequences along the time dimension. Through the residual connection, the features of the support set fuse the static representation based on semantic attention and the dynamic structure based on state modeling.

[0116] This process realizes the unified representation of modeling results of different time granularities, which helps the further modeling of global temporal dependency.

[0117] S35, bidirectional modeling based on the action center unified temporal modeling module

[0118] To enhance the perception ability of temporal modeling on global dependency structure, a bidirectional state space modeling mechanism is introduced. The forward and reverse temporal encoding fusion is realized by the forward state space modeling module and the reverse state space modeling module in the action center unified temporal modeling module, realizing the unified modeling of semantic continuity before and after the action:

[0119] The first multi-frequency fusion feature obtained by the first feature fusion and the second multi-frequency fusion feature respectively construct its time-reversed sequence:

[0120]

[0121] In the time forward direction, the forward state space modeling module is input; in the time reverse direction, the reverse state space modeling module is input, and the output result is reversed to restore the order and fused with the forward result to obtain the final bidirectional modeling result:

[0122]

[0123] wherein TSSM fwd (·) and TSSM bwd (·) denote state space modeling operations in the time forward and reverse directions respectively; Reverse(·) denotes the reverse operation in the time dimension, respectively denote the bidirectional modeling results corresponding to the semantic enhanced spatio-temporal features and the spatio-temporal features of the query sample.

[0124] Through bidirectional modeling and result fusion, the model can capture the pre-position clues and subsequent evolution of the action at the same time, effectively improving the perception ability of the complete action semantic structure.

[0125] S36, second feature fusion

[0126] To unify the modeling results under different time steps and improve the overall consistency and generalization ability of features, the bidirectional modeling results under multiple frequencies are weighted and fused, and combined with the channel adjustment mechanism guided by the global context, the final focused temporal prototype representation is generated.

[0127] For all step sizes w∈W, the bidirectional modeling results corresponding to the semantically enhanced spatiotemporal features and the spatiotemporal features of the query samples are aggregated and fused to obtain a multi-scale fusion output:

[0128]

[0129] Where, α w The aggregate weights are defined by step size w. Refine(·) represents the channel adaptive adjustment function, which adjusts the weight distribution of each channel based on global feature statistics to improve the discriminativeness and semantic consistency of the final temporal features. These represent the focused temporal prototype representations of the supporting samples and the query samples, respectively.

[0130] S37, cluster the focused temporal prototype representations of all supporting samples in the support set to obtain the category prototypes.

[0131] Focused temporal prototype representation of K support samples for each class c in the support set Perform average pooling and aggregate within the class to obtain the class prototype U for that class. c :

[0132]

[0133] S38, calculate the matching distance between the focused temporal prototype representation of the query sample and the category prototype to generate the matching probability distribution.

[0134] To measure the temporal matching degree between query videos and prototypes of each category, an Optimal Transport Alignment Module (OTAM) metric strategy based on optimal transport is adopted to calculate the focused temporal prototype representation of the query samples. With category prototype U c The matching distance dist(q,c) is calculated using the following steps:

[0135]

[0136] Where OTAM(·,·) represents the OTAM metric strategy, which updates the cost through forward dynamic accumulation, and <·,·> represents the inner product of feature vectors.

[0137] Specifically, the distance metric method for the OTAM metric strategy is as follows:

[0138]

[0139] C ij =D i,j +min{C i-1,j ,C i,j-1 ,C i-1,j-1}

[0140] By calculating the pair-wise distance D i,j between each frame, the inter-frame alignment matrix is constructed, and the minimum cumulative distance on the total alignment path is output as the matching distance between the two sequences. Where |·|2 is the L2 norm.

[0141] Based on the calculated matching distance dist(q,c), according to the relative matching strength of each category, the matching probability distribution of the query sample under all support categories is generated:

[0142]

[0143] Where p few-shot (y=i∣q) represents the matching probability distribution, dist(q,c i ) represents the matching distance between the focused time sequence prototype representation of the query sample q and the category prototype U ci corresponding to category i, and N is the number of categories.

[0144] S39, based on the matching probability distribution, the few-shot matching loss is calculated to optimize the prototype matching.

[0145] The few-shot matching loss is defined in the form of cross entropy:

[0146] L few-shot =-logp few-shot (y=y q ∣∝q)

[0147] Where L few-shot is the few-shot matching loss, and y q is the true category label of the query sample.

[0148] S4, through the frame-level cross-modal attention mechanism, the spatio-temporal features and category semantic embeddings of the support samples are fine-grained video-text alignment to obtain the category enhanced representation. Based on the similarity between the category enhanced representation and its corresponding category semantic embedding, the video-text alignment loss is calculated to optimize the video-text alignment target and adjust the spatio-temporal feature representation of the support set samples to improve the cross-modal alignment accuracy and representation consistency.

[0149] In the FSAR task, existing methods rely on global features for semantic alignment, making it difficult to capture fine-grained correspondence between video frames and text descriptions. Therefore, as Figure 2As shown, step S4 introduces a frame-level cross-modal attention mechanism (i.e., temporal semantic attention mechanism) to improve the alignment accuracy between video features and semantic information.

[0150] S4 specifically includes the following steps:

[0151] S41, the normalized text representation T obtained after normalization based on the category semantic embedding norm A query matrix is constructed based on the frame-level spatio-temporal features V of the support set and the query set videos s and V q Form a key-value pair V∈{V s ,V q}, perform multi-head attention calculation to achieve fine-grained video-text alignment, and output the category enhanced representation V enhanced ∈R N×D , N is the number of categories, and D represents the feature embedding dimension. The attention calculation process is as follows:

[0152] Q=T norm W Q ,K=VW K ,V′=VW V

[0153]

[0154] V attn =AV′

[0155]

[0156] Based on the multi-head attention mechanism, linear transformation is performed on the query, key, and value respectively, wherein W Q ,W K ,W V ∈R D×d is a learnable parameter matrix, and in each attention head, the attention score A is calculated, and the output is V attn ∈R N×d The outputs of multiple heads are spliced and projected into the final category enhanced representation output V enhanced ∈R N×D , and d represents the scaling factor.

[0157] S42, to further enhance the semantic consistency between video features and category semantic embeddings, a video-text alignment loss is calculated by calculating the similarity between the extracted category enhanced representation V enhanced and its corresponding category semantic embedding, and the video-text alignment target is optimized.

[0158] The video-text alignment loss is represented as:

[0159]

[0160] wherein, is the video-text alignment loss, represents the class-enhanced representation of the i-th class that is most relevant to the video, c (j) ∈R D represents the class semantic embedding of the j-th class, sim(·,·) represents the normalized cosine similarity calculation, τ∈T + is a temperature parameter that can be adjusted to control the flexibility of the similarity distribution.

[0161] In this embodiment, steps S3 and S4 are performed simultaneously, and the model is jointly trained and optimized. The video-text alignment target adjustment process of step S4 is only performed during the training process, and the purpose is to optimize the spatio-temporal feature representation of the support sample, so that the video feature can be better matched with the class semantic embedding.

[0162] S5, the spatio-temporal feature of the to-be-identified sample is extracted as a query sample, and the query sample is matched with the class prototype based on the optimized prototype matching process, so as to realize the identification of the action class of the query set under the condition of few samples.

[0163] Step S5 is used to complete the final class prediction in the inference stage, specifically:

[0164] For the video sample of an unseen class, first extract its spatio-temporal feature, then calculate the similarity between the focused time sequence prototype representation corresponding to the spatio-temporal feature and each class prototype corresponding to the support set optimized in step S3, and generate a class probability distribution p few-shot to represent the matching confidence of the class to which it belongs. Finally, the optimal class is selected as the recognition result according to the probability distribution, so as to realize the action class prediction of the query sample under the condition of few samples. This inference process relies on the semantic-temporal alignment capability obtained by joint modeling in the training stage, and realizes the effective cooperation of semantic understanding and action dynamic modeling.

[0165] The preferred embodiments of the present application are described in detail above. It should be understood that those skilled in the art can make many modifications and changes without creative labor according to the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning, or limited experiment by those skilled in the art on the basis of the prior art according to the concept of the present application shall be within the scope of protection determined by the claims.

Claims

1. A small sample action recognition method based on semantic-temporal adaptive representation learning, characterized in that, The method comprises the following steps: In each round of scenario learning task, randomly sample action categories from the training video dataset to construct task sample pairs containing support sets and query sets; Obtain the time-dependent semantic category description generated by the offline language model, obtain the category semantic embedding using the text encoder, and extract the spatio-temporal features of the support samples and query samples using the visual encoder; Align the spatio-temporal features of the support samples with the corresponding category semantic embedding based on the explicit semantic guided focus mechanism to obtain the semantic enhanced spatio-temporal features, and sequentially perform multi-frequency sub-sequence construction, one-way modeling based on the action-specific dynamic time modeling module, first feature fusion, two-way modeling based on the action center unified time modeling module, and second feature fusion on the semantic enhanced spatio-temporal features and the spatio-temporal features of the query samples to obtain the focused time sequence prototype representation of the support samples and the query samples, cluster the focused time sequence prototype representation of all support samples in the support set to obtain the category prototype, calculate the matching distance between the focused time sequence prototype representation of the query sample and the category prototype to generate a matching probability distribution, calculate the few-shot matching loss based on the matching probability distribution, and optimize the prototype matching process; Perform fine-grained video-text alignment on the spatio-temporal features of the support samples and the category semantic embedding using the frame-level cross-modal attention mechanism to obtain the category enhanced representation, calculate the video-text alignment loss based on the similarity between the category enhanced representation and the corresponding category semantic embedding, optimize the video-text alignment target, and adjust the spatio-temporal feature representation of the support set samples; Extract the spatio-temporal features of the to-be-recognized sample as the query sample, match it with the category prototype based on the optimized prototype matching process, and realize the recognition of the action categories in the query set under the few-shot condition.

2. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal adaptive representation learning. The explicit semantic guided focus mechanism aligns the spatio-temporal features of the support samples with the corresponding category semantic embedding to obtain the semantic enhanced spatio-temporal features, specifically: Let the spatio-temporal feature of the k-th support sample be The corresponding class semantic embedding is T k ∈R D where F denotes the number of video frames and D denotes the feature embedding dimension. After normalizing the category semantic embedding, a standardized text representation is obtained: T norm = LayerNorm(T k ) where LayerNorm denotes a normalization process, T norm denotes a standardized text representation of the class semantic embedding; Calculate the time sequence semantic weight of the spatio-temporal features of the support samples and the standardized text representation of the category semantic embedding to obtain the frame-level semantic correlation distribution: wherein W video-text is a temporal semantic weight, representing the degree of association of each frame with the current semantic; Broadcast the time sequence semantic weight in the channel dimension, and perform element-wise weighting and residual fusion on the spatio-temporal features of the support samples to generate the semantic enhanced spatio-temporal features: wherein, ⊙ represents multiplication operation, denotes semantic enhanced spatio-temporal features.

3. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The multi-frequency sub-sequence construction is specifically: to simulate the action change pattern under different time granularities, the semantic enhanced spatio-temporal features and the spatio-temporal features of the query sample are constructed into multi-frequency sub-sequences to capture the dynamic features of the action in each time scale: Define a set of time steps W = {w1, w2, ..., w n }, where n is the number of time steps. For each time step w∈W, a set of subsequences with different starting offsets is constructed from the semantically enhanced spatiotemporal features and the spatiotemporal features of the query samples: where o denotes a starting offset, semantic enhanced spatio-temporal features corresponding sub-sequences under step size w and starting offset o, spatio-temporal features v of the query sample q corresponding sub-sequences under step size w and starting offset o, semantic enhanced spatio-temporal features and sub-sequences of spatio-temporal features of the query sample, respectively.

4. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The one-way modeling of the action-specific dynamic time series modeling module is specifically: in order to extract the local dynamic change characteristics in each subsequence, the one-way modeling of each subsequence is performed by using the action-specific dynamic time series modeling module, and the representation result under the time causal structure is obtained, wherein the action-specific dynamic time series modeling module takes the subsequence corresponding to the semantic enhanced space-time feature and the subsequence corresponding to the space-time feature of the query sample as input respectively, adopts the discretized one-way state space modeling operation TSSM to establish the expression relationship of the system state evolution with time, and obtains the corresponding local modeling output: where TSSM fwd (·) denotes the state space modeling operation under the forward time flow, respectively denote the local modeling results of the semantic enhanced spatio-temporal features and the sub-sequence corresponding to the spatio-temporal features of the query sample.

5. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The first feature fusion is specifically: to unify the local modeling results of sub-sequences obtained under different step lengths to a consistent time scale and realize cross-offset and cross-frequency feature fusion, the local modeling results of sub-sequences are upsampled and integrated in the time dimension to form an intermediate representation with a global time sequence structure, wherein, For supporting the semantic enhanced spatio-temporal feature in each step w under the local modeling result Up-sampling to the length F consistent with the original video frame number by interpolation operation, and splicing and fusing in the time dimension, and then with the semantic enhanced spatio-temporal feature Residual superposition is performed to obtain the corresponding first multi-frequency fusion feature For the local modeling result of the spatio-temporal features of the query video at each step w Only performing upsampling and splicing operations, the corresponding second multi-frequency fusion features are obtained Upsample(·) represents time interpolation on each sub-sequence to align the number of frames to F; Concat(·) represents the fusion of multiple offset sub-sequences along the time dimension.

6. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The bidirectional modeling of the action center unified timing modeling module is specifically: in order to enhance the perception ability of the global dependency structure of the timing modeling, a bidirectional state space modeling mechanism is introduced, the forward state space modeling module and the reverse state space modeling module in the action center unified timing modeling module are used for forward and reverse timing coding fusion, and the unified modeling of the semantic continuity before and after the action is realized; the first multi-frequency fusion feature obtained through the first feature fusion and the second multi-frequency fusion feature respectively construct the time reversal sequence thereof: In the time forward direction, input the forward state space modeling module; in the time reverse direction, input the reverse state space modeling module, output the result again to reverse the order and fuse with the forward result to obtain a final bidirectional modeling result: where TSSM fwd (·) denotes the state space modeling operation in time forward and reverse respectively; Reverse(·) denotes the reverse operation in time dimension, bwd (·) denotes the state space modeling operation in time forward and reverse respectively; Reverse(·) denotes the reverse operation in time dimension, denote the bidirectional modeling results of the semantic enhanced spatio-temporal features and the spatio-temporal features of the query samples respectively.

7. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The second feature fusion is specifically: to unify the modeling results at different time steps, weighted fusion is performed on the bidirectional modeling results at multiple frequencies, and a global context guided channel adjustment mechanism is combined to generate a final focused temporal prototype representation: At all steps w in W, the bidirectional modeling results corresponding to the semantic enhanced spatio-temporal features and the spatio-temporal features of the query sample are respectively aggregated and fused to obtain a multi-scale fusion output: where α w is the aggregated weight of the step size w, and Refine(·) denotes a channel adaptive adjustment function that adjusts the weight distribution of each channel according to global feature statistical information, respectively represent the focused timing prototype representation of the support sample and the query sample.

8. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The focused temporal prototype representations of all support samples in the support set are clustered to obtain class prototypes, and a matching probability distribution is generated by calculating the matching distance between the focused temporal prototype representation of the query sample and the class prototype, specifically: Focal temporal prototypical representation supporting K support samples per class c Average pooling is performed and aggregated within the class to obtain the class prototype U for the class c : The focused timing prototype representation of the query sample is computed using the OTAM metric strategy based on optimal transport The matching distance dist(q, c) with the category prototype U c is computed as follows: Where OTAM(·,·) represents an OTAM metric strategy, a forward dynamic cumulative update cost, and <·,·> represents a feature vector inner product; Based on the calculated matching distance dist(q,c), a matching probability distribution of the query sample under all support categories is generated according to the relative matching strength of each category: where p few-shot (y | q) represents the matching probability distribution, dist(q, c i ) represents the matching distance between the focused time sequence prototype representation of the query sample q and the class prototype U ci corresponding to the class i, and N is the number of classes.

9. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The few-shot matching loss is defined in the form of cross-entropy: L few-shot = -log p few-shot (y = y q || q) wherein L few-shot is a few-shot matching loss, y q is a true class label of the query sample, p few-shot is a matching probability distribution.

10. The method of claim 1, wherein the method is a small sample action recognition method based on semantic-temporal self-adaptive representation learning. The process of optimizing the video-text alignment target specifically includes the following steps: The process of optimizing the video-text alignment target specifically includes the following steps: The standardized text representation T obtained after normalization based on category semantic embedding norm Construct a query matrix based on the frame-level spatio-temporal features V of the support set and query set videos s and V q Form a key-value pair V ∈ {V s ,V q}, perform multi-head attention calculation to achieve fine-grained video-text alignment, and output the category enhanced representation V enhanced ∈ R N×D N is the number of categories, and D represents the feature embedding dimension; By enhancing the extracted category representation V enhanced A similarity calculation between its corresponding category semantic embedding Video-text alignment loss, optimize the video-text alignment target, adjust the spatio-temporal feature representation of the support set sample, the video-text alignment loss is represented as: wherein, is the video-text alignment loss, denotes the class-enhanced representation of the i-th class that is most relevant in the video, c (j) ∈ R D denotes the class semantic embedding of the j-th class, sim(·, ·) denotes the normalized cosine similarity computation, τ ∈ R + is a temperature parameter that can be adjusted to control the flexibility of the similarity distribution.

Images