Zero-shot skeleton action recognition method based on dynamic frequency domain division and time-frequency fusion

Abstract

The application discloses a zero-shot skeleton action recognition method based on dynamic frequency domain division and time-frequency fusion, and belongs to the technical field of computer vision and artificial intelligence. The application aims to solve the technical problems of incomplete skeleton feature extraction, poor text feature adaptability, rough cross-modal alignment and insufficient recognition robustness in the existing zero-shot skeleton action recognition. The technical scheme comprises the following steps: obtaining time domain skeleton features, frequency domain skeleton features and action category text descriptions; performing self-adaptive frequency domain division, EMA smoothing and differential enhancement processing on the frequency domain skeleton features; fusing time-frequency features through a bidirectional cross-attention and a self-adaptive gating mechanism; realizing cross-modal alignment of the fused features and the text features by using a diffusion transformer; and optimizing the model based on a joint loss function, and realizing zero-shot recognition by denoising and similarity matching of multiple noise samples in the inference stage. The application significantly improves the accuracy and generalization ability of zero-shot skeleton action recognition.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and artificial intelligence technology, and in particular relates to a zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion. Background Technology

[0002] Skeleton-based action recognition is an important research direction in the fields of computer vision and artificial intelligence. Skeleton data extracts the joint coordinate information of the human body, removes redundant visual information from traditional RGB video, and has the advantage of privacy protection. With its lightweight storage space, it has become one of the research hotspots in the field of action recognition.

[0003] As applications expand, the number of action categories increases dramatically. Traditional supervised skeleton motion recognition methods require a large amount of labeled motion skeleton data, which is costly and difficult to cover all action categories. Against this backdrop, zero-shot skeleton motion recognition technology has emerged. Its core objective is to achieve generalized recognition of action categories through text descriptions. It utilizes skeleton and text data of known action categories and text data of unknown action categories to achieve accurate recognition of unknown action categories. Its key feature is that it does not require creating skeleton data for new action categories; recognition can be achieved solely through text descriptions. This effectively overcomes the limitations of supervised methods and has stronger practical application value and scenario adaptability.

[0004] Existing zero-shot skeleton action recognition methods mainly utilize cross-modal alignment, feature learning, and generative models. They extract skeleton features through graph convolutional networks (GCNs) and extract text features through contrastive language image pre-training models (CLIPs). Alignment relationships are established between skeleton visual features and action semantic features. Skeleton features and text features of similar actions are clustered and aligned to form category prototypes, reducing feature distribution differences. In the recognition stage, the similarity between the skeleton features to be recognized and all text features is calculated, and the action category with the highest similarity is selected as the recognition result. However, existing methods still have significant drawbacks and limitations in practical applications: skeleton feature extraction is incomplete, utilizing only spatiotemporal coordinate information; text feature extraction has poor adaptability, employing general contrastive language image pre-training models, which are better suited to general visual-text associations but not optimized for the motion characteristics of skeleton actions, failing to accurately extract fine-grained motion semantics from the action text description; the clustering alignment method is coarse, resulting in low accuracy of category prototypes; and the similarity calculation method is simplistic, leading to insufficient recognition robustness. Summary of the Invention

[0005] Purpose of the invention: In order to solve the problems of cross-modal alignment between skeleton data and text description, incomplete skeleton feature extraction, poor text adaptability to skeleton actions, and coarse clustering alignment, this invention adopts an adaptive frequency domain partitioning method, a mechanism of cross-attention fusion of skeleton spatiotemporal domain features and frequency domain features, a single-layer text adapter, and a zero-sample skeleton action recognition method with diffusion model for modal alignment.

[0006] Technical solution: The present invention provides a zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion, comprising the following steps:

[0007] Step 1: Obtain temporal skeleton features, frequency skeleton features, action category text description, and diffusion time step;

[0008] Step 2: Map the temporal skeleton features, frequency skeleton features, and action category text descriptions to the shared feature space to obtain temporal embedding features, frequency embedding features, and text embedding features. Simultaneously, embed the features of the diffusion time step to obtain the time step embedding features.

[0009] Step 3: Perform adaptive frequency domain enhancement processing on the frequency domain skeleton features. Through frequency division, smoothing and differential enhancement strategies, the enhanced frequency domain features are obtained.

[0010] Step 4: Input the temporal embedded features and the enhanced frequency domain features into the cross-modal fusion module for fusion. Cross-modal fusion is performed through bidirectional cross-attention and adaptive gating mechanism to obtain fused features;

[0011] Step 5: The conditional vector formed by superimposing the fused features, text embedding features, and time step embedding features is input into the diffusion transformer for cross-modal joint processing to achieve feature alignment.

[0012] Step 6: Construct a joint loss function based on the output of the diffusion transformer, and optimize the model parameters during training;

[0013] Step 7: In the zero-shot inference stage, generate and denoise multiple noise samples for the skeleton features to be tested. By matching the similarity between the denoised features and the text features of unseen categories, zero-shot action recognition is completed.

[0014] Further, step 3 specifically involves: performing frequency domain enhancement processing on the frequency domain skeleton features, specifically: applying discrete cosine transform to the frequency domain skeleton features to obtain frequency domain features, adaptively calculating frequency division points based on the spectral energy distribution of the frequency domain features, smoothing the frequency division points using an exponential moving average algorithm, dividing the frequency domain features into low-frequency components and high-frequency components according to the smoothed division points, applying different enhancement coefficients to each component, and then merging them to obtain the enhanced frequency domain features.

[0015] Furthermore, the adaptive calculation of frequency division points based on the spectral energy distribution of the aforementioned frequency domain characteristics specifically involves:

[0016] Calculate the average energy of each frequency component in the frequency domain characteristics;

[0017] Calculate the percentage of accumulated energy from low frequency to the current frequency relative to the total accumulated energy.

[0018] When the cumulative energy ratio reaches a preset threshold for the first time, the corresponding frequency index is determined as the initial dividing point; wherein the preset threshold ranges from 0.25 to 0.35.

[0019] Furthermore, the frequency division points are smoothed using an exponential moving average algorithm, following the formula:

[0020]

[0021] in, The smoothed division point of the current step. These are the smoothed division points from the previous step. The initial partition point is calculated in the current step, and α is the smoothing coefficient, with a value ranging from 0.85 to 0.95.

[0022] The frequency domain features are divided into low-frequency and high-frequency components according to the smoothed division points, and then different enhancement coefficients are applied to each component before merging them in the following manner:

[0023] The enhancement process for low-frequency components is as follows: ;

[0024] The enhancement process for high-frequency components is as follows: ;

[0025] in, For frequency domain features, `split` represents the smoothed split points, and `k` represents the learnable enhancement coefficients. This is the low-frequency gain coefficient. This is the high-frequency gain coefficient. The value is negative to reduce low-frequency components. The value is positive to enhance the high-frequency components.

[0026] Furthermore, step 4 specifically involves: inputting the time-domain embedded features and the enhanced frequency-domain features into the cross-modal fusion module for fusion. The cross-modal fusion module includes bidirectional cross-attention processing and adaptive gating weighting processing. The bidirectional cross-attention processing includes time-domain to frequency-domain cross-attention calculation with time-domain features as the query and frequency-domain features as the key and value, as well as frequency-domain to time-domain cross-attention calculation with frequency-domain features as the query and time-domain features as the key and value.

[0027] The adaptive gated weighting process includes calculating gate weights based on joint features in the time domain and joint features in the frequency domain, and then using these gate weights to weight the cross-attention output before performing a residual connection with the original features. The calculation method for the gate weights in the adaptive gated weighting process is as follows:

[0028]

[0029] in, and These are the time-domain gated weights and the frequency-domain gated weights, respectively, and σ is the sigmoid activation function. and These are time-domain skeleton features and frequency-domain skeleton features, respectively. and This is the weight matrix. and This is the bias vector.

[0030] Furthermore, the cross-modal fusion module also includes a feature refinement unit. The feature refinement unit performs nonlinear transformation on the time-domain skeleton fusion features and frequency-domain skeleton fusion features after gated weighted residual connection through a feedforward network and then performs residual connection again. The feedforward network consists of two linear layers, with the dimension of the middle hidden layer being 3 to 5 times that of the input dimension, and uses the GELU activation function.

[0031] Further, step 5 specifically involves: inputting the conditional vector formed by superimposing the fused features, text embedding features, and time-step embedding features into a diffusion transformer for cross-modal joint processing. The diffusion transformer includes multiple cascaded CrossDiT blocks. Each CrossDiT block generates modulation parameters based on the conditional vector and performs adaptive layer normalization modulation on the fused features and text embedding features. Subsequently, the modulated fused features and text embedding features are concatenated and multi-head self-attention is calculated. The attention output is separated into a skeleton part and a text part, which are then processed by residual connections and feedforward networks, respectively. The number of CrossDiT blocks ranges from 8 to 16 layers, and the number of multi-head self-attention heads in each CrossDiT block ranges from 8 to 16 heads.

[0032] Further, step 6 specifically involves: calculating a joint loss function based on the output of the diffusion converter, wherein the joint loss function includes a denoising reconstruction loss and a cross-modal triplet contrast loss, and the denoising reconstruction loss is the mean square error between the noise prediction result and the actual added noise;

[0033] The formula for calculating the joint loss function is:

[0034]

[0035] in, To reduce noise and reconstruct losses, For cross-modal triplet contrast loss, These are the weighting coefficients for the denoising loss. These are the weighting coefficients for the triplet loss. and The values ​​range from 0.5 to 1.5;

[0036] The formula for calculating noise reconstruction loss is:

[0037]

[0038] in, This represents the denoised skeleton features predicted by the model. This indicates the original, clean skeletal features. This represents the mean squared error, used to calculate the difference between the denoised skeleton features predicted by the model and the original clean skeleton features.

[0039] The formula for calculating the cross-modal triplet contrastive loss is:

[0040]

[0041] in, For anchor point features, For positive sample features, The negative sample features are represented by d(·,·), which is the distance metric function, and margin is the boundary parameter, with a value ranging from 0.1 to 0.5.

[0042] Furthermore, in step 7, the number of noise samples K generated by the skeleton feature to be tested ranges from 2 to 10, and the number of denoising steps in the reverse process performed by the diffusion transformer ranges from 20 to 100.

[0043] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0044] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0045] The skeleton-text matching method based on a diffusion model provided in this invention achieves significant technical effects through the following approaches: Improved accuracy: Utilizing the progressive denoising mechanism and cross-modal triplet loss function of the diffusion model, leading recognition accuracy is achieved in zero-shot settings on multiple benchmark datasets; Enhanced generalization ability: Through adaptive time-frequency feature fusion and frequency domain enhancement strategies, the model possesses stronger cross-class and cross-dataset generalization capabilities; Improved robustness: By leveraging multi-noise sample inference and EMA smoothing mechanisms, the model's resistance to noisy data and input fluctuations is improved; Enhanced practicality: Recognition can be achieved without additional training for new categories, lowering the application threshold and maintenance costs; Optimized interpretability: The adaptive frequency partitioning mechanism provides intuitive interpretability, aiding in understanding the model's decision-making basis. The technical effects of this invention provide an efficient, accurate, and practical solution for the field of zero-shot skeleton action recognition, possessing significant academic value and broad application prospects. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0047] Figure 2 It is a visualization of the spatiotemporal and frequency domain characteristics;

[0048] Figure 3 This is a schematic diagram of the frequency domain enhancement module;

[0049] Figure 4 This is a schematic diagram of the cross-modal fusion module;

[0050] Figure 5 This is a schematic diagram of the diffusion converter module. Detailed Implementation

[0051] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0052] This invention proposes a zero-shot skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion, aiming to improve the completeness of skeleton feature extraction, text feature adaptability, and modal alignment efficiency, thereby effectively improving the accuracy of zero-shot skeleton action recognition. The core of this method lies in constructing a joint architecture that includes multimodal feature processing and diffusion denoising alignment, such as... Figure 1 As shown, it specifically includes:

[0053] S1. Obtain temporal skeleton features, frequency skeleton features, and text descriptions of action categories;

[0054] S2. Map the above features to the shared latent space respectively;

[0055] S3. Deeply fuse time-domain and frequency-domain skeleton features through cross-attention and adaptive gating mechanisms;

[0056] S4. Using a diffusion transform based on time step and text conditional modulation, perform cross-modal joint processing on fused features and text features;

[0057] S5. During the training phase, the model parameters are optimized by combining the denoising reconstruction loss and the cross-modal triplet comparison loss.

[0058] S6. In the inference stage, based on the multi-noise sample generation strategy, the test skeleton features are matched with the text descriptions of unseen categories to achieve zero-sample action recognition.

[0059] The system of this invention comprises the following six modules in order of data flow:

[0060] 1. The input processing module includes a skeleton feature acquisition unit: used to acquire temporal and frequency domain skeleton features to be processed; a text description encoding unit: used to encode the input action category text description through a contrastive language-image pre-trained model, outputting sequence-level text features and pooled text features; and a diffusion time step generation unit: used to randomly sample and generate diffusion time steps as conditional variables to control the diffusion noise level. The output of the skeleton feature acquisition unit is connected to the feature embedding module and the frequency domain enhancement module, respectively; the output of the text description encoding unit is connected to the feature embedding module; and the output of the diffusion time step generation unit is connected to the feature embedding module.

[0061] The raw skeleton data in the skeleton feature acquisition unit: Let the input be a sequence of joint coordinates, with the shape as follows. Where T is the time step, J is the number of joints, and C is the coordinate dimension (usually 2 or 3). The overall original skeleton data is represented as follows:

[0062]

[0063] in, Let represent the coordinates of the key points at time step t. Spatiotemporal skeleton features are extracted using a pre-trained graphical convolutional network. The formula for extracting frequency domain skeleton features is:

[0064]

[0065] Here, DCT stands for Discrete Cosine Transform, typically using an orthogonal normalization mode. The frequency domain skeleton features are obtained through this transformation. Visualization of the two types of features is as follows: Figure 2 As shown.

[0066] In the text description encoding unit, let the original text description representation be T. The formula for extracting text features by comparing the language image pre-trained model is as follows:

[0067]

[0068] in, To provide a pre-trained model for contrastive language images. Obtain the text features after pooling.

[0069] The diffusion time-step generation unit is a core component of the diffusion model, responsible for controlling the degree of noise addition and the progress of the denoising process. Firstly, This indicates that a time step t is randomly sampled from a uniform distribution, ranging from 0 to 1. Between these steps, the discrete time step t is transformed into a continuous feature vector. By encoding time step t using cosine and sine functions of different frequencies, frequency embedding can capture the nonlinear relationship between time steps, making it more effective than simple linear coding. The frequency embedding is further processed by a multilayer perceptron (MLP) to generate the final time-step embedding vector.

[0070] 2. Frequency domain enhancement module, such as Figure 3 As shown, it includes an adaptive frequency partitioning unit: calculating the average energy and cumulative energy ratio of each frequency component, and determining the corresponding frequency point as the high-low frequency partitioning point when the cumulative energy ratio reaches a preset threshold; an exponential moving average (EMA) smoothing unit: using the exponential moving average algorithm to smooth the partitioning point of the current iteration step and the historical partitioning point to prevent the partitioning point from oscillating violently during the training process; and a frequency component enhancement unit: based on the smoothed partitioning point, applying different learnable enhancement coefficients to the low-frequency component and the high-frequency component respectively, and merging them to output the enhanced frequency domain skeleton feature to the skeleton feature embedding unit.

[0071] For adaptive frequency division, the average energy of each frequency component is first calculated using the following formula:

[0072]

[0073] in, The f-th frequency component represents the frequency characteristic. Then, the proportion of accumulated energy from the low frequency to the current frequency is calculated to determine the frequency division point; the formula is:

[0074]

[0075] Where N is the total number of frequency components. The frequency point at which the accumulated energy ratio first reaches the threshold is used as the boundary between high and low frequencies, as shown in the formula:

[0076]

[0077] in, The preset energy ratio threshold, This is the initial partition point calculated in the current step.

[0078] The exponential moving average smoothing unit, after determining the split points, uses the exponential moving average to smooth the split points, reducing fluctuations during training and improving model stability. The calculation formula is as follows:

[0079]

[0080] in, This is the smoothing coefficient, usually taken as 0.9.

[0081] The frequency component enhancement unit enhances the low-frequency and high-frequency components with different coefficients, as shown in the formula:

[0082]

[0083] Where split represents the smoothed split point, and k is the enhancement coefficient. As a low-frequency enhancement factor, For high-frequency enhancement factors, different enhancement factors allow the model to adjust the relative importance of high and low frequencies according to the characteristics of the action. The enhanced high and low frequency components are then reassembled to form a complete enhanced frequency domain feature. By employing adaptive frequency partitioning and differentiated enhancement strategies, this technology addresses the shortcomings of traditional frequency domain processing methods, such as poor adaptability and limited detail capture capabilities. It automatically adjusts processing strategies based on the frequency characteristics of the action, thereby improving the efficiency of frequency domain information utilization.

[0084] 3. Cross-modal fusion module, such as Figure 4 As shown, the system includes a time-frequency feature fusion unit, which performs layer normalization on the embedded spatiotemporal and frequency domain features and calculates bidirectional cross-attention to achieve spatiotemporal frequency information interaction; an adaptive gating mechanism unit, which concatenates the original features and cross-attention features in the channel dimension and dynamically calculates the spatiotemporal and frequency domain gating weights through a fully connected layer and a sigmoid activation function; and a feature refinement unit, which performs weighted fusion of features based on the gating weights and performs nonlinear refinement through a feedforward network, and finally concatenates the features in the sequence dimension to output the joint fused features.

[0085] The time-frequency feature fusion unit performs bidirectional cross-attention calculation on spatiotemporal and frequency domain features, achieving both spatiotemporal attention to frequency domain information and frequency domain attention to spatiotemporal information. When calculating the cross-attention from the spatiotemporal domain to the frequency domain, the spatiotemporal domain features are used as queries, and the frequency domain features are used as keys and values. The calculation formula is as follows:

[0086]

[0087] The specific formula for calculating cross attention is:

[0088]

[0089] in, To query features, and Key-value features , , and For learnable parameters, For attention head dimension.

[0090] The adaptive gating mechanism unit first processes the original features , With cross-attention output , To splice, The joint feature representation is obtained. , Then, gating weights are calculated using linear transformation and the sigmoid activation function to control the contribution of cross-attention features. The specific formula is as follows:

[0091]

[0092] in, , This is the weight matrix. , This is the bias vector.

[0093] The feature refinement unit adaptively weights the cross-attention features using gating weights, and then performs residual connections with the original features to achieve a dynamic feature fusion process.

[0094]

[0095] in, This represents an element-wise multiplication operation. The fused features are first normalized using a layer, and then subjected to a nonlinear transformation via a feedforward network to enhance their expressive power. The formula is as follows:

[0096]

[0097] Features are further refined through residual connections, preserving original information while enhancing expressive power. The formula is as follows:

[0098]

[0099] The refined time-domain and frequency-domain features are concatenated along the sequence dimension. This forms the final fusion characteristic.

[0100] 4. Diffusion converter module, such as Figure 5 As shown, it includes multiple cascaded CrossDiT processing blocks. Each block includes: an adaptive layer normalization unit: after adding the time-step feature vector with the pooled text features, it maps the scaling and offset parameters through a multilayer perceptron and performs affine modulation on the fused features and sequence text features respectively; a multi-head self-attention unit: concatenates the modulated fused features and text features in the sequence dimension, inputs them into the multi-head self-attention layer for global cross-modal information interaction, and then splits them back into skeleton features and text features according to the dimension; and a residual and feedforward network: performs adaptive layer normalization and FFN processing on the split features again, and outputs the final feature representation through residual connections.

[0101] The adaptive layer normalization unit processes time-step embeddings and text pooling features through a multilayer perceptron, generating modulation parameters for layer normalization. After layer normalization of the fused skeleton features, scaling parameters are used for modulation to enhance the expressive power of the features. After layer normalization of the text features, scaling parameters are used for modulation to ensure that the text features and skeleton features are in the same feature space. The specific formula is as follows:

[0102]

[0103] in, This represents the scaling parameter. Indicates the gating parameter, subscript This indicates the bulls' self-attention. Indicates a feedforward network. Represents text features.

[0104] Multi-head self-attention units concatenate modulated skeleton features and text features along the sequence dimension to form a joint feature representation. A multi-head self-attention mechanism is applied to the joint features to capture the dependencies between the skeleton and text features. The system generates query, key, and value vectors through linear transformation, calculates attention weights and applies them to the value vectors, and finally outputs the attention output through linear transformation, separating the attention output into skeleton feature parts and text feature parts, thus maintaining the independence of the two modalities.

[0105] The residual and feedforward network performs layer normalization and modulation on skeleton and text features, followed by nonlinear transformation through a feedforward network to enhance feature expressiveness. The feedforward network consists of two linear transformation layers and a GELU activation function, achieving complex nonlinear feature transformations. The specific formula is as follows:

[0106]

[0107] The original skeleton features, original text features, attention output, and feedforward network output are weighted by gating weights and then summed to achieve residual connections, thus preserving the original information while fusing new features.

[0108]

[0109] 5. Training Module, Noise Scheduling Unit: Based on the diffusion time step, Gaussian noise is added to the original skeleton features according to a preset noise scheduling strategy (such as linear or cosine scheduling); Joint Loss Function Unit: Configured to calculate the weighted sum of two losses: Denoising Reconstruction Loss: Calculates the mean squared error (MSE) between the denoised skeleton features predicted by the diffusion transformer and the noise-free original skeleton features; Cross-Modal Triple Contrast Loss: Calculates the spaced triplet loss using the fused skeleton features as anchors, the text features corresponding to the real category as positive samples, and the text features of other categories in the same batch as negative samples; Model Optimization Unit: Updates the model network parameters based on the joint total loss using the backpropagation algorithm.

[0110] The noise scheduling unit adds Gaussian noise based on the clean skeleton features of the diffusion time step. ,in, For noise scheduling parameters, It is standard Gaussian noise. Using linear noise scheduling, from linearly increase to To control the degree of noise addition.

[0111] The joint loss function unit includes the calculation of denoising reconstruction loss and cross-modal triplet loss. First, the denoising reconstruction loss is calculated by determining the mean squared error between the denoised features predicted by the model and the clean skeleton features, ensuring the model can accurately predict the original features. The formula is:

[0112]

[0113] Where MSE represents mean squared error. Then comes the cross-modal triplet loss, which uses the fused skeleton features as anchors to calculate the distance difference with positive samples (corresponding to text features) and negative samples (other text features), enhancing the cross-modal alignment effect. The specific formula is:

[0114]

[0115] in, This represents the distance metric between features. These are the boundary parameters. The denoising reconstruction loss and the cross-modal triplet loss are weighted and summed. To balance the learning objectives of the two tasks, among which, and The loss weights control the relative importance of the two losses.

[0116] The model optimization unit updates the model parameters using gradient descent. ,in, For learning rate, Let be the gradient of the total loss with respect to the parameters. Cosine learning rate scheduling is used, as shown in the formula:

[0117]

[0118] From the initial learning rate Gradually decay to 0 to improve training stability.

[0119] 6. Zero-shot inference module, including a multi-noise sample generation unit: randomly generating K different initial noise samples for the input test skeleton features; a denoising feature acquisition unit: inputting the K noise samples into the trained diffusion transformer module respectively, denoising under the condition of unseen category text description, and obtaining K denoised skeleton features; a matching and prediction unit: calculating the dot product similarity matrix between the K denoised skeleton features and each unseen category text feature, and after Softmax normalization, selecting the unseen category with the highest average matching score as the final zero-shot prediction result.

[0120] Existing frequency domain processing typically relies on manually set fixed cutoff frequencies. This invention automatically optimizes the high- and low-frequency division points by calculating the spectral energy distribution and creatively introduces an EMA mechanism to stabilize the division boundaries. Combined with differential enhancement coefficients, it can adaptively amplify key frequency details for different actions (such as slow "push" actions and fast "swing" actions), significantly improving the model's robustness in recognizing subtle movements.

[0121] Instead of directly concatenating traditional features, it captures the spatiotemporal coupling relationship through bidirectional cross-attention from the time domain to the frequency domain and from the frequency domain to the time domain. It also uses an adaptive gating network to dynamically allocate time-frequency weights based on the characteristics of the current input sample, thus avoiding interference from invalid information.

[0122] This invention introduces a multi-noise sample generation and similarity aggregation strategy during the inference phase to overcome prediction fluctuations caused by the randomness of single diffusion. It achieves zero-sample generalization to new categories without increasing training parameters or retraining costs. Unlike the simple projection alignment in existing technologies, this invention innovatively binds the denoising process of skeleton features to textual conditions. Through adaptive layer normalization in the diffusion transformer, textual semantics are injected as conditions into the denoising trajectory. The progressive nature of noise prediction gradually eliminates modal differences, effectively solving the semantic gap between skeleton coordinates and natural language.

[0123] Example

[0124] This embodiment provides a specific implementation of a skeleton-text matching method based on a diffusion model for zero-shot skeleton action recognition on the NTU RGB+D 60 dataset. The hardware environment for this embodiment is a computing server equipped with an NVIDIA GeForce RTX4090 24GB graphics processor, 128GB of memory, and a 13th Gen Intel(R) Core(TM) i7-13700K processor. The software environment consists of an Ubuntu 24.04 operating system, a Python 3.9 interpreter, a PyTorch 2.4.0 deep learning framework, and a CUDA 13.0 computing platform.

[0125] This embodiment uses the NTU RGB+D 60 publicly available skeleton action recognition dataset as the experimental subject. This dataset contains 60 action categories and 56,880 action samples, with each action sample containing a sequence of coordinates of 25 joints in 3D space. In this embodiment, the 60 action categories are divided into 55 known categories and 5 unknown categories, where the known categories are used for model training and the unknown categories are used for zero-shot testing and validation.

[0126] The original skeleton data is preprocessed as follows: First, the coordinates of all relevant nodes are centered with the center of the human torso as the reference point to eliminate the influence of human position offset; then, the coordinates are globally standardized using the mean and standard deviation of the training set; features are extracted from the skeleton data through a pre-trained graph convolution model; finally, a spatiotemporal skeleton feature tensor with shape (batch size, 1, sequence length 300, feature dimension 256) is obtained.

[0127] For each action category, a standardized text description is generated, with the description template in the form of "a person is \[action\_description]". A pre-trained CLIP-ViT-L / 14 text encoder is used to encode each text description, generating sequence-level text features with dimensions (batch size, 35, 768) and pooled text features with dimensions (batch size, 2048).

[0128] The model constructed in this embodiment includes a feature embedding module, a cross-modal fusion module, a diffusion converter module, and an output layer connected in sequence, and also includes a frequency domain enhancement module connected in parallel with the feature embedding module.

[0129] The feature embedding module includes a temporal skeleton feature embedding unit, a frequency domain skeleton feature embedding unit, a text feature embedding unit, and a time step embedding unit. The temporal skeleton feature embedding unit is a linear layer with an input dimension of 256 and an output dimension of 768; the frequency domain skeleton feature embedding unit is also a linear layer with an input dimension of 256 and an output dimension of 768; the text feature embedding unit consists of two linear layers, the first layer mapping 2048-dimensional pooled text features to 768 dimensions; the time step embedding unit maps discrete time steps to a 256-dimensional frequency space through sinusoidal position encoding, and then maps them to a 768-dimensional conditional vector space through two layers of MLP.

[0130] The cross-modal fusion module includes a time-frequency feature fusion unit, an adaptive gating mechanism unit, and a feature refinement unit. The time-frequency feature fusion unit comprises two cross-attention subunits, respectively implementing information interaction from the time domain to the frequency domain and from the frequency domain to the time domain. Each cross-attention subunit includes a query projection layer, a key projection layer, a value projection layer, and an output projection layer, with 12 attention heads, each with a dimension of 64. The adaptive gating mechanism unit comprises two gating networks, receiving joint time-domain features and joint frequency-domain features as input, respectively. After linear transformation and a sigmoid activation function, each network outputs gating weights between 0 and 1. The feature refinement unit comprises two feedforward networks, each consisting of two linear layers with an intermediate hidden layer dimension of 3072 (four times 768), employing the GELU activation function.

[0131] The diffusion transformer module consists of 12 cascaded CrossDiT blocks. Each CrossDiT block includes two adaptive layer normalization subunits, one multi-head self-attention subunit, and two feedforward network subunits. The adaptive layer normalization subunit generates six modulation parameters (including three scaling parameters and three gating parameters) based on the conditional vector, which are used to modulate the normalization process of the multi-head self-attention and feedforward networks, respectively. The multi-head self-attention subunit concatenates the modulated skeleton features and text features along the token dimension and performs joint self-attention calculation. After the calculation is completed, the skeleton part and the text part are separated and output separately.

[0132] The frequency domain enhancement module includes a DCT transform unit, an adaptive frequency partitioning unit, a frequency component enhancement unit, and an EMA smoothing unit. The DCT transform unit uses orthogonally normalized discrete cosine transform to convert the time-domain skeleton features to the frequency domain. The adaptive frequency partitioning unit calculates the energy distribution of each frequency component, and determines the high-low frequency partitioning point when the cumulative energy ratio reaches a threshold of 0.3. The frequency component enhancement unit multiplies the low-frequency components by a coefficient for enhancement, and multiplies the high-frequency components by a coefficient for enhancement.

[0133] The hyperparameters of the model in this embodiment are configured as follows: 256 input channels, 768 hidden layer dimensions, 12 Transformer layers, 12 attention heads, MLP expansion ratio of 4.0, frequency embedding dimension of 256, and text sequence length of 35.

[0134] The training process in this embodiment specifically includes the following steps:

[0135] Step S1: Obtain a batch of clean skeleton features from the training data loader. Its shape is (256, 1, 300, 256). At the same time, the text description corresponding to this batch is obtained and encoded by CLIP text encoder to obtain sequence-level text features. and pooling text features .

[0136] Step S2: For the clean skeleton features Frequency domain skeleton features are obtained by applying discrete cosine transform. .

[0137] Step S3: Randomly sample the diffusion time step t from the uniform distribution U(0,1000), and determine the direction of the diffusion based on this time step. Adding Gaussian noise yields noisy skeleton features. Noise addition follows the formula: ,in Standard Gaussian noise, It increases linearly from 0.0001 to 0.02 with time step t.

[0138] Step S4: Transfer the noisy skeleton features Temporal embedding features are obtained by inputting temporal skeleton feature embedding units. .

[0139] Step S5: The frequency domain skeleton features The input frequency domain enhancement module processes the data. Specifically, it first confirms that the frequency domain features are in the state after DCT transformation; then, it adaptively calculates the frequency split point based on the spectral energy distribution, that is, first calculates the average energy E(f) of each frequency component, then calculates the cumulative energy ratio C(f), and finds the minimum frequency index that makes C(f) not less than the threshold of 0.3 for the first time as the split point; then, it applies the EMA smoothing formula. The division points are smoothed; finally, the frequency domain features are divided into low-frequency and high-frequency components according to the division points, and then multiplied by the enhancement coefficients respectively before merging to obtain the enhanced frequency domain features. .

[0140] Step S6: The enhanced frequency domain features Input frequency domain skeleton feature embedding unit to obtain frequency domain embedding features .

[0141] Step S7: Embed the time-domain features and frequency domain embedding features The input is processed by the cross-modal fusion module. Specifically, firstly, layer normalization is performed on the two types of features respectively; then, bidirectional cross-attention is used to achieve information exchange between the time domain and the frequency domain, and between the frequency domain and the time domain, to obtain the cross-attention output. and Next, the original features and their corresponding cross-attention outputs are concatenated and then input into the gating network, where they are activated by a Sigmoid algorithm to obtain the gating weights. and Then, the cross-attention output is weighted using gating weights and added to the original features to obtain the fused features. and Finally, the fused features are refined using a feedforward network and then concatenated along the sequence dimension to obtain the combined features. Its shape is (256, 600, 768).

[0142] Step S8: The pooled text features Input text feature embedding unit to obtain text conditional features The sequence-level text features Text sequence features are obtained by adding learnable positional encoding. The diffusion time step t is input into the time step embedding unit to obtain the time step embedding feature. The conditional vector is obtained by adding the time-step embedded features to the text conditional features. .

[0143] Step S9: Combine the features and text sequence features The conditional vector c is sequentially input into 12 CrossDiT blocks for processing. Within each CrossDiT block, the conditional vector c is activated by SiLU and linearly transformed to generate 6 modulation parameters. These 6 modulation parameters are used to perform adaptive layer normalization modulation on the skeleton features and text features, respectively. The modulated skeleton features and text features are concatenated along the token dimension and then input into a multi-head self-attention unit for joint attention calculation. The attention outputs are separated and then residually connected to the original features. The residually connected features are again normalized by adaptive layers and then input into a feedforward network for processing. The output of the feedforward network is then residually connected to the unmodulated features for a second time, completing the processing of a single CrossDiT block.

[0144] Step S10: Separate the output of the 12th CrossDiT block into time-domain and frequency-domain components, and project them back into the 256-dimensional space through the final output layer to obtain the noise prediction result. and .

[0145] Step S11: Calculate the denoising and reconstruction loss Where MSE represents the mean square error function, This refers to the real noise added in step S3.

[0146] Step S12: Calculate the cross-modal triplet loss Specifically, the average value of the temporal portion of the output of the 10th CrossDiT block along the sequence dimension is taken as the anchor feature. Pooled text features corresponding to the actual action categories are taken as positive sample features. Pooled text features from other samples within the same batch are used as negative sample features. According to the formula Calculate the triplet loss, where d(·,·) represents the Euclidean distance function and the margin is 0.2.

[0147] Step S13: Calculate the total loss , where λ1=1.0, λ2=1.0.

[0148] Step S14: Based on total loss The gradient is calculated and clipped (maximum norm 1.0), then the AdamW optimizer is used to update all learnable parameters of the model. The initial learning rate of the AdamW optimizer is 1×10⁻⁶. -4 The weighted decay coefficient is 0.01, and the first moment estimates the decay rate. β 1 = 0.9, second moment estimate of attenuation rate β 2 = 0.999. The learning rate is scheduled using a cosine annealing strategy, starting from an initial value of 1 × 10. -4 Gradually decaying to a minimum value of 1×10 -6 .

[0149] Step S15: Repeat steps S1 to S14 for a total of 50,000 iterations to complete model training. Record the training loss every 2,000 steps, evaluate the accuracy on the validation set every 50 epochs, and save the checkpoint with the highest validation accuracy as the optimal model.

[0150] The zero-shot inference process in this embodiment specifically includes the following steps:

[0151] Step R1: Load the optimal model weights saved during the training phase. Generate text descriptions for the five unseen action categories and encode them using the CLIP text encoder to obtain text feature vectors for each of the five categories, forming the unseen category text feature matrix. Its shape is (5, 2048).

[0152] Step R2: For the skeleton features to be tested, independently generate 3 noise samples that follow a standard Gaussian distribution. , , .

[0153] Step R3: Perform the inverse DDPM denoising process on each noise sample. Specifically, the number of steps decreases progressively from step 1000 to step 1. A total of 50 effective denoising calculations are performed. This process is repeated on each of the three noise samples to obtain three sets of denoised features. , , .

[0154] Step R4: Average the three groups of denoised features in the batch and time dimensions, and further average them in the feature dimension to obtain the final denoised feature vector. Its shape is (1, 256).

[0155] Step R5: Calculate the denoised feature vector Feature matrix of unseen category texts Dot product similarity of features of each category.

[0156] Step R6: Perform Softmax normalization on the similarity scores to obtain the category prediction probability distribution. .

[0157] Step R7: Select the category with the highest predicted probability as the final prediction result. .

[0158] The diffusion process progressively optimizes the skeleton feature representation through multi-step denoising, enabling the feature space to smoothly transition to a state aligned with the text semantic space. This progressive optimization mechanism effectively avoids the feature space abrupt change problem common in traditional methods, improving the stability of cross-modal matching. Experimental results show that, with zero-shot settings (unseen 5 classes) on the NTU RGB+D 60 dataset, the Top-1 accuracy of this invention reaches 87.24%, an improvement of approximately 0.75 percentage points compared to the baseline method; with zero-shot settings (unseen 10 classes) on the NTU RGB+D 120 dataset, the Top-1 accuracy reaches 75.88%, an improvement of approximately 1.73 percentage points compared to the baseline method.

[0159] Table 1. Accuracy of four partition settings on the NTU RGB+D 60 and NTU RGB+D 120 datasets.

[0160]

[0161] The present invention demonstrates a significant improvement in accuracy across four different partitioning methods for the NTU RGB+D 60 and NTU RGB+D 120 datasets.

[0162] The technical advantages of this invention make it applicable to a wide range of practical scenarios: Intelligent monitoring systems: In the field of security monitoring, this invention can identify new abnormal behaviors beyond predefined action categories without the need to collect large amounts of labeled data for model retraining, significantly reducing system maintenance costs. Human-computer interaction systems: In human-computer interaction applications, users can customize new gestures and add text descriptions, and the system can quickly learn to recognize the gestures, greatly improving the flexibility of the interaction system and the user experience. Medical rehabilitation assessment: In the field of rehabilitation medicine, doctors can customize personalized rehabilitation action assessment standards for each patient. The system can automatically identify the quality of the patient's action completion, assisting in the development of precise rehabilitation plans. Sports movement analysis: In sports movement analysis, coaches can define new tactical or technical movements, and the system can quickly identify the athlete's execution, providing real-time technical feedback.

[0163] In summary, the skeleton-text matching method based on the diffusion model provided by this invention achieves significant technical effects through the following approaches: Improved accuracy: Utilizing the progressive denoising mechanism and cross-modal triplet loss function of the diffusion model, leading recognition accuracy is achieved in zero-shot settings on multiple benchmark datasets; Enhanced generalization ability: Through adaptive time-frequency feature fusion and frequency domain enhancement strategies, the model possesses stronger cross-class and cross-dataset generalization capabilities; Improved robustness: By leveraging multi-noise sample inference and EMA smoothing mechanisms, the model's resistance to noisy data and input fluctuations is improved; Enhanced practicality: Recognition can be achieved without additional training for new categories, lowering the application threshold and maintenance costs; Optimized interpretability: The adaptive frequency partitioning mechanism provides intuitive interpretability, aiding in understanding the model's decision-making basis. The technical effects of this invention provide an efficient, accurate, and practical solution for the field of zero-shot skeleton action recognition, possessing significant academic value and broad application prospects.

[0164] The above embodiments are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and equivalent substitutions without departing from the principle of the present invention. All such improvements and equivalent substitutions to the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion, characterized in that, Includes the following steps: Step 1: Obtain temporal skeleton features, frequency skeleton features, action category text description, and diffusion time step; Step 2: Map the temporal skeleton features, frequency skeleton features, and action category text descriptions to the shared feature space to obtain temporal embedding features, frequency embedding features, and text embedding features. Simultaneously, embed the features of the diffusion time step to obtain the time step embedding features. Step 3: Perform adaptive frequency domain enhancement processing on the frequency domain skeleton features. Through frequency division, smoothing and differential enhancement strategies, the enhanced frequency domain features are obtained. Step 4: Input the temporal embedded features and the enhanced frequency domain features into the cross-modal fusion module for fusion. Cross-modal fusion is performed through bidirectional cross-attention and adaptive gating mechanism to obtain fused features; Step 5: The conditional vector formed by superimposing the fused features, text embedding features, and time step embedding features is input into the diffusion transformer for cross-modal joint processing to achieve feature alignment. Step 6: Construct a joint loss function based on the output of the diffusion transformer, and optimize the model parameters during training; Step 7: In the zero-shot inference stage, generate and denoise multiple noise samples for the skeleton features to be tested. By matching the similarity between the denoised features and the text features of unseen categories, zero-shot action recognition is completed.

2. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 1, characterized in that, Step 3 specifically involves: performing frequency domain enhancement processing on the frequency domain skeleton features. Specifically, this involves: applying discrete cosine transform to the frequency domain skeleton features to obtain frequency domain features; adaptively calculating frequency division points based on the spectral energy distribution of the frequency domain features; smoothing the frequency division points using an exponential moving average algorithm; dividing the frequency domain features into low-frequency components and high-frequency components according to the smoothed division points; applying different enhancement coefficients to each component; and then merging the components to obtain the enhanced frequency domain features.

3. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 2, characterized in that, The frequency division points are adaptively calculated based on the spectral energy distribution of the aforementioned frequency domain characteristics, specifically as follows: Calculate the average energy of each frequency component in the frequency domain characteristics; Calculate the percentage of accumulated energy from low frequency to the current frequency relative to the total accumulated energy. When the cumulative energy ratio reaches a preset threshold for the first time, the corresponding frequency index is determined as the initial dividing point; wherein the preset threshold ranges from 0.25 to 0.

35.

4. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 2, characterized in that, The frequency division points are smoothed using an exponential moving average algorithm, following the formula: ； in, The smoothed division point of the current step. These are the smoothed division points from the previous step. The initial partition point is calculated in the current step, and α is the smoothing coefficient, with a value ranging from 0.85 to 0.

95. The frequency domain features are divided into low-frequency and high-frequency components according to the smoothed division points, and then different enhancement coefficients are applied to each component before merging them in the following manner: The enhancement process for low-frequency components is as follows: ; The enhancement process for high-frequency components is as follows: ; in, For frequency domain features, `split` represents the smoothed split points, and `k` represents the learnable enhancement coefficients. This is the low-frequency gain coefficient. This is the high-frequency gain coefficient. The value is negative to reduce low-frequency components. The value is positive to enhance the high-frequency components.

5. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 1, characterized in that, Step 4 specifically involves: inputting the temporal embedded features and the enhanced frequency domain features into the cross-modal fusion module for fusion. The cross-modal fusion module includes bidirectional cross-attention processing and adaptive gating weighting processing. The bidirectional cross-attention processing includes time-domain to frequency-domain cross-attention calculation with time-domain features as queries and frequency-domain features as keys and values, as well as frequency-domain to time-domain cross-attention calculation with frequency-domain features as queries and time-domain features as keys and values. The adaptive gated weighting process includes calculating gate weights based on joint features in the time domain and joint features in the frequency domain, and then using these gate weights to weight the cross-attention output before performing a residual connection with the original features. The calculation method for the gate weights in the adaptive gated weighting process is as follows: ； in, and These are the time-domain gated weights and the frequency-domain gated weights, respectively, and σ is the sigmoid activation function. and These are time-domain skeleton features and frequency-domain skeleton features, respectively. and This is the weight matrix. and This is the bias vector.

6. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 5, characterized in that, The cross-modal fusion module also includes a feature refinement unit. The feature refinement unit performs nonlinear transformation on the time-domain skeleton fusion features and frequency-domain skeleton fusion features after gated weighted residual connection through a feedforward network and then performs residual connection again. The feedforward network consists of two linear layers, with the dimension of the middle hidden layer being 3 to 5 times that of the input dimension, and uses the GELU activation function.

7. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 1, characterized in that, Step 5 specifically involves: inputting the conditional vector formed by superimposing the fused features, text embedding features, and time-step embedding features into a diffusion transformer for cross-modal joint processing. The diffusion transformer includes multiple cascaded CrossDiT blocks. Each CrossDiT block generates modulation parameters based on the conditional vector and performs adaptive layer normalization modulation on the fused features and text embedding features. Subsequently, the modulated fused features and text embedding features are concatenated and multi-head self-attention is calculated. The attention output is separated into a skeleton part and a text part, which are then processed by residual connections and feedforward networks, respectively. The number of CrossDiT blocks ranges from 8 to 16 layers, and the number of multi-head self-attention heads in each CrossDiT block ranges from 8 to 16 heads.

8. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 1, characterized in that, Step 6 specifically involves: calculating a joint loss function based on the output of the diffusion converter. The joint loss function includes a denoising and reconstruction loss and a cross-modal triplet contrast loss. The denoising and reconstruction loss is the mean square error between the noise prediction result and the actual added noise. The formula for calculating the joint loss function is: ； in, To reduce noise and reconstruct losses, For cross-modal triplet contrast loss, These are the weighting coefficients for the denoising loss. These are the weighting coefficients for the triplet loss. and The values ​​range from 0.5 to 1.5; The formula for calculating the denoising and reconstruction loss is: ； in, This represents the denoised skeleton features predicted by the model. This indicates the original, clean skeletal features. This represents the mean squared error, used to calculate the difference between the denoised skeleton features predicted by the model and the original clean skeleton features. The formula for calculating the cross-modal triplet contrastive loss is: ； in, For anchor point features, For positive sample features, The negative sample features are represented by d(·,·), which is the distance metric function, and margin is the boundary parameter, with a value ranging from 0.1 to 0.

5.

9. The zero-sample skeleton action recognition method based on dynamic frequency domain partitioning and time-frequency fusion according to claim 1, characterized in that, In step 7, the number of noise samples K generated by the skeleton feature to be tested ranges from 2 to 10, and the number of denoising steps in the reverse process performed by the diffusion transformer ranges from 20 to 100.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

Images