System and method for driving hands and body postures of human body through audio

Abstract

The invention discloses a system and a method for driving hands and body postures of a human body through audio, and aims to solve the problems of difficulty in rhythm and semantic decoupling and lack of expressive force of actions in the prior art. The core of the scheme lies in a double-path parallel mapping module: one path is a rhythm-motion synchronization path which efficiently maps audio physical characteristics into a'dynamic envelope vector 'for controlling motion dynamics; the second path is an emotion-semantic coding path, and the audio content is coded into a posture prototype sequence representing a posture shape by using a depth model; and then, a cross-channel attention fusion module uses the dynamic envelope vector as a query to perform dynamic weighted fusion on the attitude prototype sequence to generate a structured attitude potential representation. Finally, the attitude decoder synthesizes the representation into a smooth, natural and highly synchronous three-dimensional human skeleton animation with the audio. According to the method, through a decoupling and cooperation mechanism, the music, diversity and reality sense of the generated attitude are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the fields of computer graphics, deep learning and artificial intelligence, and specifically relates to a system and method for generating high-quality and expressive three-dimensional human posture animation from audio signals. Background Technology

[0002] Existing technologies for audio-driven pose generation, even those employing deep learning, generally suffer from the following deep-seated technical bottlenecks: Entangled Representation Learning: Traditional end-to-end models (such as RNNs / LSTMs) attempt to map audio features indiscriminately to pose outputs. However, the rhythmic information (when to move) and content information (what action to perform) contained in audio signals are inherently two different types of information. Learning them entangled together makes it difficult for the model to optimize both objectives simultaneously: the generated actions may be rhythmically accurate but semantically empty, or the poses may be meaningful but out of sync with the music beat ("matching the emotion, but not the beat").

[0003] Lack of micro-dynamic control: Existing methods often generate motions that look "close enough" on a macroscopic level, but lack detailed expressiveness. For example, for a heavy drumbeat, the motion should have a clear burst-hold-decay process, rather than a vague sway. Existing models struggle to capture and reproduce such microsecond-level dynamic changes.

[0004] Pattern collapse and repetitiveness: Due to the single training objective, many generative models are prone to "mode collapse", that is, generating a few similar actions for different types of music, lacking diversity and novelty.

[0005] The premise of this approach is that the movements of an excellent dancer or speaker are the result of their brain's parallel processing of audio. On one hand, their motor system (cerebellum, etc.) reacts almost in real-time, instinctively to rhythm and beat, controlling the speed and intensity of movements; on the other hand, their higher cognitive centers (cerebral cortex) understand the emotions, melody, or semantics of the music, planning body postures with specific meanings. Existing technologies lack this decoupling and coordination mechanism. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0007] The core objective of this invention is to provide a system and method that can simulate cognitive processes and generate human postures that are highly matched with audio in terms of rhythm, dynamics, semantics, and emotion through decoupling and collaborative mechanisms.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: A system for audio-driven human hand and body posture, comprising the following components: a multi-scale audio feature extraction module, configured to receive an audio data stream and extract a hierarchical feature sequence containing local details and global context using a multi-scale temporal convolutional network; a dual-path parallel posture representation mapping module, configured to receive the hierarchical feature sequence and perform the following operations in parallel: a rhythm-motion synchronization path, extracting the beat initiation point intensity and energy envelope low-dimensional physical features from the audio features and inputting them into a first neural network model to generate a time-aligned dynamic envelope vector sequence, which represents the expected motion velocity, acceleration, and energy intensity of the posture at each moment; and an emotion-semantic encoding path, extracting Mel-frequency cepstral coefficients and chroma map high-dimensional content features from the audio features, and... The input is fed into a second neural network model based on a self-attention mechanism to generate a sequence of pose prototype latent vectors, where each vector represents an abstract pose shape related to the audio content. A cross-path attention fusion and decoding module is configured to: at each time step, use a dynamic envelope vector as the query for the attention mechanism; use the sequence of pose prototype latent vectors as the keys and values ​​for the attention mechanism; fuse the dynamic envelope with the most relevant pose prototype by calculating attention weights to generate a structured pose latent representation; input the structured pose latent representation into a pre-trained pose decoder network to generate 3D human skeletal pose data for that time step; and a temporal pose smoothing and rendering module is configured to apply a temporal smoothing filter to the continuous skeletal pose data sequence output by the pose decoder network and drive a 3D human model.

[0009] As a preferred embodiment of the audio-driven human hand and body posture system described in this invention, the multi-scale audio feature extraction module employs a temporal convolutional network with causal convolutional characteristics, ensuring that feature extraction at the current moment does not depend on future audio data when processing real-time audio streams.

[0010] As a preferred embodiment of the audio-driven human hand and body posture system of the present invention, the first neural network model in the rhythm-motion synchronization path is a gated recurrent unit network, which is trained to learn the direct mapping relationship between audio energy changes and human movement explosive force and mildness.

[0011] As a preferred embodiment of the audio-driven human hand and body posture system of the present invention, the second neural network model in the emotion-semantic coding pathway is a Transformer encoder, which captures long-distance dependencies within audio segments through a self-attention mechanism, thereby generating a semantically coherent sequence of posture prototypes.

[0012] As a preferred embodiment of the audio-driven human hand and body posture system described in this invention, the structured posture latent representation is a concatenated vector that includes at least a contextual posture prototype vector generated by cross-path attention fusion and the current dynamic envelope vector, i.e., z. t = [c t d t ], where c t It is the fused posture prototype, d t It is a dynamic envelope.

[0013] As a preferred embodiment of the audio-driven human hand and body posture system described in this invention, the posture decoder network is a multilayer perceptron or an inverse GRU network, which together with a posture encoder constitutes a variational autoencoder architecture and is pre-trained on a large-scale human motion capture dataset to learn a decoupled, structured posture latent space.

[0014] As a preferred embodiment of the audio-driven human hand and body posture system described in this invention, the temporal posture smoothing and rendering module uses a Kalman filter or a Savitzky-Gore filter to post-process the generated skeletal joint angle or position sequence to eliminate high-frequency jitter and ensure the physical realism of the motion.

[0015] To solve the above-mentioned technical problems, the present invention also provides the following technical solution: a method for audio-driven human hand and body posture, using the above-mentioned audio-driven human hand and body posture system, comprising the following steps: S1: Hierarchical feature extraction: Receive audio streams and extract their multi-scale temporal features using a temporal convolutional network; S2: Dual-path parallel mapping: S2.1 (Rhythm Path): Separates beat energy and other information from audio features, and generates a dynamic envelope vector sequence through the first neural network model; S2.2 (Semantic Path): Separate information such as spectral content from audio features, and generate a sequence of potential vectors for pose prototypes through a second neural network model; S3: Collaborative Integration and Decoding: S3.1: For each time step in the sequence, use the dynamic envelope vector of that step as a query to perform cross-path attention operation on the pose prototype latent vector sequence to obtain the fused context pose prototype vector. S3.2: Concatenate the context pose prototype vector with the dynamic envelope vector to form a structured pose latent representation; S3.3: Feed the latent representation into the pose decoder network to generate the single-frame skeleton pose at the current moment; S4: Smoothing and Rendering: Performs temporal filtering and smoothing on the continuously generated bone pose sequence, and uses it to drive the rendering of the 3D model.

[0016] This invention provides a system for audio-driven human hand and body posture, which has the following beneficial effects: 1. The dual-path decoupled architecture can separate and intelligently fuse the rhythmic dynamics and emotional content in the audio. This fundamentally solves the problem of existing technologies that "either miss the beat or produce hollow movements," enabling the generated gestures to be precise and powerful on every beat (rhythmic synchronization) and perfectly match the macro-emotion and semantics of the audio (content relevance), achieving a perfect unity of the two. 2. Through a cross-path attention mechanism, the system can dynamically combine and blend semantic prototypes from the gesture library according to instantaneous rhythm. This combinatorial generation method effectively avoids "pattern collapse" and can create novel, unique, and reasonable action sequences even when faced with similar audio, greatly enriching the diversity of actions.

[0017] 3. By employing techniques such as 6D continuous rotation representation and temporal filtering, the final output animation is ensured to be stable, smooth, and free of technical artifacts, achieving professional-grade quality. More importantly, its decoupled latent representation (pose and form + motion dynamics) allows developers or artists to independently intervene in the style and content of the action, providing unprecedented creative freedom and controllability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This invention provides a flowchart of a method for audio-driven human hand and body posture. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0020] This invention provides a system and method that can simulate cognitive processes and generate human postures that are highly matched with audio in terms of rhythm, dynamics, semantics, and emotion through decoupling and collaborative mechanisms.

[0021] To achieve this objective, the technical solution proposed in this invention constructs a dual-path parallel collaborative generation architecture, which decomposes the complex mapping task into two specialized subtasks that ultimately work together through an attention mechanism.

[0022] Specifically, the present invention provides an audio-driven system for human hand and body posture, comprising the following components: The multi-scale audio feature extraction module is configured to receive audio data streams and use a multi-scale temporal convolutional network to extract a hierarchical feature sequence containing local details and global context. The dual-path parallel posture representation mapping module is configured to receive a hierarchical feature sequence and perform the following operations in parallel: a rhythm-motion synchronization path, which extracts the beat initiation intensity and low-dimensional physical features of the energy envelope from the audio features and inputs them into a first neural network model to generate a time-aligned dynamic envelope vector sequence, which represents the expected motion velocity, acceleration, and energy intensity of the posture at each moment; and an emotion-semantic encoding path, which extracts the Mel frequency cepstral coefficients and high-dimensional content features of the chroma map from the audio features and inputs them into a second neural network model based on a self-attention mechanism to generate a posture prototype latent vector sequence, in which each vector represents an abstract posture shape related to the audio content. The cross-path attention fusion and decoding module is configured as follows: at each time step, the dynamic envelope vector is used as the query for the attention mechanism; the pose prototype latent vector sequence is used as the key and value of the attention mechanism; by calculating the attention weight, the dynamic envelope is fused with the most relevant pose prototype to generate a structured pose latent representation; the structured pose latent representation is input into a pre-trained pose decoder network to generate the 3D human skeleton pose data for that time step. The temporal pose smoothing and rendering module is configured to apply a temporal smoothing filter to the continuous skeletal pose data sequence output by the pose decoder network and drive a 3D human model.

[0023] Furthermore, the multi-scale audio feature extraction module employs a temporal convolutional network with causal convolutional properties, ensuring that feature extraction at the current moment does not depend on future audio data when processing real-time audio streams.

[0024] Furthermore, the first neural network model in the rhythm-motion synchronization pathway is a gated recurrent unit network, which is trained to learn the direct mapping relationship between changes in audio energy and the explosiveness and mellowness of human movement.

[0025] Furthermore, the second neural network model in the emotion-semantic coding pathway is a Transformer encoder that captures long-range dependencies within audio segments through a self-attention mechanism, thereby generating a semantically coherent sequence of pose prototypes.

[0026] Furthermore, the structured pose latent representation is a concatenated vector that contains at least the context pose prototype vector generated by cross-path attention fusion and the current dynamic envelope vector, i.e., z. t = [c t d t ], where c t It is the fused posture prototype, d t It is a dynamic envelope.

[0027] Furthermore, the pose decoder network is a multilayer perceptron or an inverse GRU network, which together with a pose encoder forms a variational autoencoder architecture and is pre-trained on a large-scale human motion capture dataset to learn a decoupled, structured pose latent space.

[0028] Furthermore, the temporal pose smoothing and rendering module uses Kalman filters or Savitzky-Gore filters to post-process the generated skeletal joint angle or position sequences to eliminate high-frequency jitter and ensure the physical realism of the motion.

[0029] It should be noted that, in considering this specific plan, the following factors were taken into account: 1. Decoupling Design: The system no longer treats audio features as a whole, but instead "splits" them into two dedicated paths: The Rhythm-Motion Synchronization Pathway is a lightweight and efficient pathway focused on capturing the instantaneous physical characteristics of audio (such as drum beats, tempos, and volume bursts) and mapping them directly to a dynamic cut vector that controls "how" the action moves. This vector is not concerned with the specific form of the action, but only controls its energy, velocity, and acceleration curves. This solves the problem of microscopic dynamic control.

[0030] The emotion-semantic encoding pathway: a deeper and more complex pathway that uses a powerful attention model to analyze the overall content of an audio segment (melody, harmony, timbre, intonation) and encode it into a series of abstract gesture prototype latent vectors that define "what form" of an action. This addresses the issues of semantic coherence and diversity of actions.

[0031] 2. Synergy: The two decoupled information flows are not isolated. This invention introduces a cross-pathway attention mechanism as a bridge for collaboration. At each critical moment when action needs to be generated (such as a beat), the "dynamic envelope vector" output by the rhythmic path acts as a "commander" (as the query of the attention mechanism), examining the "pose prototype library" generated by the semantic path (as the key / value). It decides whether to choose an energetic pose prototype or a gentle one based on the current energy intensity, and assigns it the corresponding dynamics. This is similar to a dancer who, upon hearing a downbeat, not only knows to exert force but also knows that this force should be used in the "punching" motion, not just "shrugging."

[0032] 3. Structured Decoding: The fused information constitutes a structured pose latent representation, clearly containing two parts: "what action to take" (pose prototype) and "how to take this action" (dynamic envelope). A specially trained pose decoder is responsible for "translating" this clear instruction into the final skeletal coordinates. This decoder is trained on large-scale motion capture data using a variational autoencoder (VAE) approach, and its latent space is designed to be decoupled and structured, thus ensuring the accuracy and realism of the decoding.

[0033] It should be noted that the key modules of this invention are specifically as follows: 1. Multi-scale audio feature extraction module Instead of using a simple MFCC, this invention employs a pre-trained Temporal Convolutional Network (TCN).

[0034] Input: Raw audio waveform or Mel spectrogram.

[0035] Structure: Composed of multiple stacked residual blocks, each block contains two layers of dilated causal convolutions. The dilated convolution design allows the receptive field to grow exponentially, thus capturing long-term dependencies in audio without adding too many parameters. Causal convolution ensures that processing is real-time, with the output at time t depending only on information up to and including time t.

[0036] Output: A hierarchical feature sequence H = {h1, h2, ..., h T This sequence contains both low-frequency envelope information and high-frequency detail information of the audio.

[0037] It should be noted that: Multiscale audio features H = {h1, h2, ..., h T} Name: Hierarchical Audio Feature Sequence.

[0038] Definition: The feature vector ht output at each time step t after processing by a multi-scale temporal convolutional network (TCN).

[0039] Physical / Mathematical Significance: h t It is not a single feature, but a composite vector containing information about the audio at different time scales. Due to the dilated convolution structure in TCN, shallow convolutions extract high-resolution, short-term features (such as the transient characteristics of a single attack), while deep convolutions have a larger receptive field and extract low-resolution, long-term features (such as the outline of a melody or the color of harmony). Therefore, h t The different parts inside the vector correspond to: Microscopic dynamic information: such as the onset, impact, and decay of a note.

[0040] Mesoscopic structural information: such as rhythmic patterns and timbre variations.

[0041] Macro-level semantic information: such as melody direction and emotional atmosphere.

[0042] Example of dimensionality and numerical range: ht can be a 512-dimensional floating-point vector. For example, the first 128 dimensions might primarily encode transient information, the middle 256 dimensions encode the spectrum and timbre, and the last 128 dimensions encode long-term context information. The numerical values ​​are normalized and typically fall within the range of [-1, 1].

[0043] Function: It serves as a unified, high-quality information source for subsequent dual-path parallel modules. It avoids the tediousness and information loss of manually designing and splicing multiple isolated features (such as MFCC, Chroma, RMS), and provides rich and structured input for subsequent decoupling.

[0044] 2. A dual-path parallel attitude representation mapping module receives the feature sequence H and processes it in separate streams.

[0045] 2.1 Rhythm-Motion Synchronization Pathway Input: Low-dimensional physical features extracted from H or directly calculated, such as Onset Strength Signal (OSS) and Root Mean Square Energy (RMSE).

[0046] Model: A gated recurrent unit (GRU) network containing approximately 64 units. The lightweight structure of the GRU makes it well-suited for processing such low-dimensional time-series signals, enabling it to quickly capture instantaneous changes in energy and predict their dynamic trends.

[0047] Output: A dynamic envelope vector sequence D = {d1, d2, ..., d...} T}. Each d t It is a low-dimensional vector (such as 4-dimensional), whose elements can be interpreted as: [velocity, acceleration, amplitude scaling, motion duration].

[0048] It should be noted that the dynamic envelope vector d t (Constitutes sequence D) Name: Dynamic Envelope Vector.

[0049] Definition: In the rhythm-motion synchronization pathway, a low-dimensional vector generated by the first neural network (GRU) at time step t.

[0050] Physical / Mathematical Significance: d t It is a direct description of the kinematic properties of posture; it does not concern itself with the "shape" of the posture, but only controls the "mode of motion." A specific 4D implementation d t = [v, a, p, τ] as follows: v (Velocity): A scalar representing the expected average angular velocity of the skeletal joints at the current moment. High v values ​​correspond to fast, intense movements; low v values ​​correspond to slow, gentle movements. It is directly driven by the rate of change of audio energy (such as onset strength).

[0051] a (Acceleration): A scalar representing the rate of change of velocity. A large positive a value corresponds to the onset of an explosive action (such as a drumbeat); a negative a value corresponds to the deceleration or termination of the action.

[0052] p (Power / Amplitude Scaling): A scalar representing the spatial amplitude of the movement. A high p value results in a larger amplitude of the generated posture (e.g., a wider range of arm swings); a low p value produces a more refined, restrained movement. It is highly correlated with the root mean square energy (RMSE) of the audio.

[0053] τ (Duration / Timing): A scalar representing the expected duration of the current action state, or the interval until the next key action. It is inferred from the beat cycle of the audio.

[0054] Example of dimensionality and numerical range: A 4-dimensional floating-point vector. v and a can be any real numbers (usually normalized), and p and τ are positive real numbers.

[0055] Function: As a "motion instruction," it provides precise rhythmic and dynamic control for pose generation. It is a query in the cross-path attention mechanism, actively seeking a pose shape that matches the current motion state.

[0056] 2.2 Emotion-Semantic Encoding Pathway Input: High-dimensional content features extracted from H, mainly MFCC and Chroma-like parts.

[0057] Model: A Transformer encoder with 6 layers and 8 heads. Its internal self-attention mechanism enables the model to analyze the context of the entire audio segment (e.g., the past 2 seconds), understanding the direction of the melody, changes in harmony, or fluctuations in tone, rather than just looking at the current frame.

[0058] Output: A sequence of potential vectors for the pose prototype S = {s1, s2, ..., s} T}. Each s t It is a high-dimensional vector (e.g., 256-dimensional) that does not directly correspond to a pose, but rather is a point in a pre-trained pose latent space that represents an abstract body pose prototype, such as an "open", "tucked", "powerful", or "questioning" pose.

[0059] It should be noted that the pose prototype latent vector s t (Constitutes sequence S) Name: Pose Prototype Latent Vector.

[0060] Definition: In the sentiment-semantic coding pathway, a high-dimensional vector generated by the second neural network (Transformer encoder) at time step t.

[0061] Physical / Mathematical Significance: s t It's not a specific pose, but rather the coordinates of the pose within a pre-trained, semantic latent space. This latent space is obtained by training a variational autoencoder (VAE) on a large amount of human motion data. Within this space: Similar points correspond to similar postures. For example, all "arms outstretched" postures, regardless of their amplitude or orientation, will cluster in a certain area of ​​the potential space.

[0062] Vector operations have semantic meaning. For example, the direction from the "standing" vector to the "sitting" vector represents the semantic path of the action of "sitting".

[0063] s t What is encoded is the abstract "form" or "intention" of the posture. For example, a passionate violin solo might be encoded as an s_t vector pointing to a "tension, upward stretching" posture region; while a deep cello might be encoded as an s_t vector pointing to an "inward curling" posture region.

[0064] Example of dimension and numerical range: A 256-dimensional floating-point vector that follows a standard normal distribution (a characteristic of VAEs).

[0065] Function: To form a dynamic, synchronized gesture "repository" with the audio content. It is the key and value in the cross-channel attention mechanism, waiting to be queried and selected by rhythmic instructions.

[0066] 3. Cross-path attention fusion and decoding module 3.1 Synergistic Integration: This is the key step in combining information from the two pathways.

[0067] For each time step t, we perform a standard attention calculation: Attention(Q, K, V).

[0068] Query: Q = W q * d t , where d t It is a dynamic envelope vector from the rhythm pathway, W q It is a learnable linear projection matrix.

[0069] Key and Value: K = W k * S and V = W v * S, where S is the entire pose prototype sequence from the semantic pathway, W k W v It is a learnable projection matrix.

[0070] Calculation: Context t = Softmax((Q * K T ) / sqrt(dim k)) * V.

[0071] Output: A context pose prototype vector c t This vector is a dynamically weighted sum of the entire pose prototype sequence S, with the weights determined by the current tempo dynamics d. t The decision was made. This achieved the synergistic effect of "using rhythm to select and modulate semantics".

[0072] It should be noted that the projection matrix W q W k W v Name: Query, Key, Value Projection Matrices.

[0073] Definition: Three learnable linear transformation matrices used in the attention module.

[0074] Physical / Mathematical Significance: Their function is to project the input vector into different subspaces to play different roles in the attention mechanism.

[0075] W q : The dynamic envelope vector d t Projecting onto the "query space". This operation can be understood as: "translating 'motion instructions' into the form of 'questions'". For example, projecting a 4D d... t Transform it into a 64-dimensional query vector so that it can be used to perform a dimension-matched dot product operation with the 64-dimensional "key" vector.

[0076] W k : The pose prototype s t Projecting onto the "key space". This operation can be understood as: "Assigning a 'capability label' to each pose prototype to answer questions." This "label" describes which dynamics the pose prototype is best suited to be combined with.

[0077] W v : The pose prototype s t Projecting onto the "value space". This operation can be understood as: "Preparing the content of the pose prototype itself, and providing it out once selected".

[0078] Inventive detail: In this invention, d t and s t Originating from different pathways, they have different physical meanings, therefore W q and W k W vIts role is to project these two types of heterogeneous information into a unified semantic space that can be compared and integrated. This is a key step in "collaboration".

[0079] Dimension example: If d t It is 4-dimensional, s t It is 256-dimensional, the dimension of the attention head (dim). k If W is 64-dimensional, then W q The dimension is 4x64, W k and W v The dimensions are all 256x64.

[0080] 3.2 Structured representation construction: combining the fused information.

[0081] z t = c t || d t (|| indicates vector concatenation). z t It is the complete posture command at that moment, which includes both "form" and "dynamics".

[0082] It should be noted that the context pose prototype vector c t Name: Contextual Pose Prototype Vector.

[0083] Definition: The output of the cross-path attention mechanism is a weighted sum of value vectors V.

[0084] Physical / Mathematical Significance: c t At time t, dynamically adjust d according to the current rhythm. t A completely new pose prototype is dynamically synthesized from the entire pose prototype sequence S. It is not simply a selection of a single pose prototype from S. i Instead, multiple s i According to its relationship with d t The matching degree (attention weight) is fused together.

[0085] Example: Suppose the audio is a strong drumbeat superimposed on a soft melody. t It will be a high-energy vector. The semantic pathway S may simultaneously contain both the "powerful punch" prototype and the "gentle extension" prototype. d t As a query, the key for the "powerful punch" prototype will be given high weight, while the key for the "gentle stretch" prototype will be given low weight. The final c tIt will primarily consist of the "powerful punch" prototype, but may also incorporate a touch of "stretching" to create a unique posture that is both powerful and graceful.

[0086] Dimension example: Same dimension as the value vector V, for example, 64 dimensions (if multi-head attention is used, it is composed of multiple 64-dimensional vectors concatenated, for example, 512 dimensions).

[0087] Function: It achieves the final fusion of rhythm and semantics, generating a specific "blueprint" of posture that is both in line with the rhythmic dynamics and full of emotional content.

[0088] It should be noted that, Structured attitude latent representation z t Name: Structured Pose Latent Representation.

[0089] Definition: A vector formed by concatenating the context pose prototype c_t and the original dynamic envelope d_t. t = c t || d t .

[0090] Physical / Mathematical Significance: This is the most complete single-frame pose generation instruction sent to the final decoder. It contains two decoupled parts: c t Part tells the decoder: "What kind of pose shape to generate" (e.g., a "celebration" pose prototype).

[0091] d t Part tells the decoder: "How to perform this form" (e.g., to perform this "celebration" gesture very quickly and with a large amplitude).

[0092] This structured design makes the decoder's task extremely simple and clear, thus ensuring the quality of the output. It's like giving a chef a recipe, c t It is an "ingredients list", d t It's "cooking steps," not just "make a delicious dish."

[0093] Dimension example: 512 (c t ) + 4 (d t = 516 dimensions.

[0094] Function: It serves as the direct input to the pose decoder and acts as a bridge connecting the generation module and the final rendering module.

[0095] 3.3 Attitude Decoding: Model: A pose decoder network, typically a multilayer perceptron (MLP), for example (256+4) -> 512 -> 512 -> (24*6). Its output is a 6D rotational representation of the 24 major joints of the human body (a continuous rotational representation without gimbal lock).

[0096] Training Background: This decoder, together with an encoder, constitutes a variational autoencoder (VAE), pre-trained on motion capture datasets such as LAFAN1 or Human3.6M containing tens of hours of dance, speech, and other movements. The training goal is for the VAE to learn a "good" pose latent space, where similar poses are close together and certain dimensions may correspond to interpretable properties. The generative model of this invention aims to learn how to generate trajectories in this pre-trained, structured latent space.

[0097] It should be noted that the pose output is in (24 x 6) dimensions. Name: 6D Rotation Representation for SkeletalPose.

[0098] Definition: The final result output by the pose decoder at each time step.

[0099] Physical / Mathematical Significance: 24: Represents the 24 key joints that are controlled in the human model. For example: 1 root joint (Hips), 3 spinal joints, 1 cervical joint, 1 head joint, 6 shoulder / elbow / wrist joints on both sides, 6 hip / knee / ankle joints on both sides, etc.

[0100] 6: Represents the rotational state of each joint. Traditional methods use 3D Euler angles or 4D quaternions. This invention uses a 6D continuous rotation representation. This is defined by extracting the first two column vectors (each vector is 3-dimensional, for a total of 6 dimensions) of a 3x3 rotation matrix.

[0101] Why use 6D representation? Euler angles suffer from gimbal lock, and quaternions exhibit the non-uniqueness of q and -q representing the same rotation. Both of these discontinuities pose significant challenges to the regression learning of neural networks. In contrast, 6D representation is continuous, allowing neural networks to stably learn and output this representation. Then, through Gram-Schmidt orthogonalization, a unique 3x3 rotation matrix can be unambiguously recovered. This significantly improves the stability and smoothness of the generated animation.

[0102] 4. Timing-based pose smoothing and rendering module Input: The raw skeletal pose sequence output by the decoder. This sequence may contain high-frequency jitter due to small errors in the model's predictions.

[0103] Processing: A Kalman filter is applied. The joint angles and angular velocities of the skeleton posture are used as state variables, and the decoder output is used as the observations. The Kalman filter can optimally estimate the observations based on the dynamic model of the motion (such as a uniform or uniformly accelerated model), effectively smoothing out noise while preserving legitimate fast motion.

[0104] Output: A smoothed, physically more stable, and visually more fluid skeletal animation sequence used to drive the final rendering of the 3D model.

[0105] The technical solution of this invention fundamentally solves the problem of entangled representation in existing technologies by introducing a cognitive science-inspired dual-path decoupling and collaborative attention mechanism. This architecture enables the model to simultaneously consider the micro-rhythmic dynamics of actions and the macro-contextual coherence, thereby generating unprecedented, high-quality audio-driven gesture animation.

[0106] Additionally, the present invention also provides a method for audio-driven human hand and body posture, and the system for audio-driven human hand and body posture includes the following steps: S1: Hierarchical feature extraction: Receive audio streams and extract their multi-scale temporal features using a temporal convolutional network; S2: Dual-path parallel mapping: S2.1 (Rhythm Path): Separates beat energy and other information from audio features, and generates a dynamic envelope vector sequence through the first neural network model; S2.2 (Semantic Path): Separate information such as spectral content from audio features, and generate a sequence of potential vectors for pose prototypes through a second neural network model; S3: Collaborative Integration and Decoding: S3.1: For each time step in the sequence, use the dynamic envelope vector of that step as a query to perform cross-path attention operation on the pose prototype latent vector sequence to obtain the fused context pose prototype vector. S3.2: Concatenate the context pose prototype vector with the dynamic envelope vector to form a structured pose latent representation; S3.3: Feed the latent representation into the pose decoder network to generate the single-frame skeleton pose at the current moment; S4: Smoothing and Rendering: Performs temporal filtering and smoothing on the continuously generated bone pose sequence, and uses it to drive the rendering of the 3D model.

[0107] To verify the superiority of the present invention, "Audio-Driven Human Pose Generation System Based on Dual-Pathway Co-Attention Mechanism" (hereinafter referred to as "the present invention model" or DPCA-Gen, Dual-Pathway Co-Attention Generator), over the prior art, the following experimental verification process was designed.

[0108] I. Experimental Objectives This experiment aims to verify the superior technical effects achieved by the model of this invention in the following core aspects through objective quantitative indicators and subjective user evaluation: Musicality and Synchronization: Verify that the generated gestures are highly synchronized with the audio in terms of rhythm and content.

[0109] Diversity and creativity: Validate the model's ability to generate a wide variety of actions for different inputs, avoiding pattern collapse.

[0110] Motion quality and realism: Verify the smoothness, physical plausibility, and visual realism of the generated animation.

[0111] II. Experimental Setup Dataset: The dataset used is AIST++, an industry-recognized large-scale, high-quality 3D dance audio and video dataset. This dataset contains 140 sequences across 10 different music genres, providing paired audio, multi-view video, and accurate 3D human motion capture (MoCap) data. The model was trained using its officially partitioned training set, and evaluated using the test set.

[0112] Baseline Models: To ensure a fair comparison, two baseline models representing different technological approaches were selected. Baseline 1 (Seq2Seq-LSTM): A classic end-to-end sequence-to-sequence model based on Long Short-Term Memory (LSTM) networks. It represents an early, conventional technique for directly mapping audio features to gesture sequences.

[0113] Baseline 2 (Audio2Pose-Transformer): A current mainstream end-to-end generation model based on Transformer. It does not feature the dual-path decoupling design of this invention; instead, it fuses all audio features and directly inputs them into a standard Transformer encoder-decoder architecture.

[0114] Ground Truth: Raw motion capture data recorded by professional dancers in the AIST++ dataset, serving as the gold standard for evaluation.

[0115] Evaluation Metrics: Objective quantitative indicators: Fréchet Gesture Distance (FGD): (Unit: None, lower is better) Measures the similarity between the generated pose distribution and the real pose distribution. Features are extracted using a pre-trained pose autoencoder, and the FGD distance between the two distributions is calculated. A lower FGD indicates that the generated action is closer to a real human pose in style, content, and dynamics.

[0116] Beat Consistency Score (BCS): (Unit: None, Range [0, 1], Higher is better) Measures the alignment between the peak energy of the generated motion and the beat of the music. It calculates the kinetic energy (sum of squares of joint velocities) curve of the motion sequence and detects its peak. BCS is the proportion of the kinetic energy peak falling within a certain time window near the beat. This metric directly reflects the accuracy of "hitting the beat".

[0117] Average Acceleration (unit: rad / s², lower is better): Calculates the average of the second derivatives of all joint angle sequences. This metric measures the smoothness of movement. A lower value indicates a smoother movement with less jitter.

[0118] Motion Diversity: (Unit: None, higher is better) For the same input audio segment, motion sequences are generated multiple times by introducing different random seeds during the generation process. The average Euclidean distance between corresponding joint positions of these sequences is calculated. A higher value indicates better motion diversity generated by the model and avoids mode collapse.

[0119] Subjective evaluation indicators: Conduct a user study. Recruit 30 volunteers with no professional background and show them anonymous animated videos of the same music generated by different models. Ask them to rate the videos on four dimensions: "musicality," "realism," "diversity," and "overall preference" (on a scale of 1-5, with 5 being the best).

[0120] III. Experimental Procedure Model training: Using the AIST++ training set, the model of this invention (DPCA-Gen), Baseline 1 and Baseline 2 were trained sufficiently until the model converged on the validation set.

[0121] Batch generation: Using all 20 audio segments in the AIST++ test set, generate corresponding pose animation sequences for each model. For diversity evaluation, each audio segment is generated 10 times.

[0122] Objective index calculation: For all generated sequences and Ground Truth sequences, run the script to calculate the above four objective quantitative indices (FGD, BCS, Average Acceleration, Diversity) and take the average value.

[0123] Subjective evaluation execution: Five music clips of different styles were randomly selected from the test set. The corresponding videos generated by the model of this invention and the two baseline models were compared pairwise (e.g., this invention vs. Baseline 2), and participants scored and recorded the results.

[0124] IV. Experimental Results and Analysis The objective and subjective evaluation data obtained from the experiment are summarized below.

[0125] Table 1: Evaluation Results of Objective Quantitative Indicators

[0126] Data Analysis: FGD (Far-Rate and Content-Based Matching): The FGD value (0.21) of the model in this invention is significantly lower than that of the two baseline models, indicating that the distribution of the generated actions is closest to the distribution of real human actions, demonstrating its great advantage in content matching and pose realism.

[0127] BCS (Rhythm Synchronization): The BCS value of the model in this invention (0.86) far exceeds all baseline models and is very close to the data of real dancers (0.91). This strongly demonstrates that the dual-path architecture and dynamic envelope vector design of this invention have achieved decisive success in achieving precise rhythm synchronization (beat-matching).

[0128] Average acceleration (smoothness): The average acceleration of the model in this invention (6.72) is the lowest, close to the level of real data (5.98), indicating that the generated motion is the smoothest and most natural, effectively avoiding the jitter and unnatural abrupt movements common in other models. This is thanks to the 6D rotation representation and temporal smoothing module.

[0129] Diversity: The diversity score of the model in this invention (0.45) is more than twice that of Baseline 2, which proves that its cross-path attention fusion mechanism can effectively avoid pattern collapse and generate rich and diverse creative actions for the same input.

[0130] Table 2: Subjective User Research Evaluation Results (Average Score, Maximum Score 5)

[0131] Data Analysis: The subjective evaluation results are highly consistent with the objective data. Across all dimensions, user ratings show an overwhelming preference for the model of this invention. Particularly, near-perfect scores were achieved in "Musicality" (4.7 points) and "Overall Preference" (4.8 points), indicating that the technical effectiveness of this invention is not only significant in terms of data but also brings a substantial improvement to the end-user's actual viewing experience. Users generally reported that the animations generated by this invention "look like they are really dancing to music," "the movements are powerful and not stiff," and "it's different every time I watch it, which is very interesting."

[0132] V. Experimental Conclusions In summary, through rigorous comparative experiments with representative baseline models on standard datasets, the technical solution of this invention demonstrates significant superiority across all key evaluation dimensions. Objective data and subjective evaluations jointly and powerfully demonstrate the superior technical effects claimed by this invention: it successfully generates high-quality human posture animations that are highly synchronized with audio in rhythm and content, exhibit rich and varied movements, and are smooth and realistic, thus solving a core problem in existing technologies.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A system for driving human hand and body gestures with audio, characterized by, The system comprises the following components: a multi-scale audio feature extraction module configured to receive an audio data stream and extract a hierarchical feature sequence containing local details and global context using a multi-scale temporal convolutional network; a dual-path parallel pose representation mapping module configured to receive the hierarchical feature sequence and perform the following operations in parallel: a rhythm-motion synchronization path that extracts beat onset strength and energy envelope low-dimensional physical features from the audio features and inputs them into a first neural network model to generate a time-aligned dynamic envelope vector sequence representing the expected motion speed, acceleration, and energy intensity of the pose at each time step; and an emotion-semantic encoding path that extracts mel-frequency cepstral coefficients and chromagram high-dimensional content features from the audio features and inputs them into a second neural network model based on a self-attention mechanism to generate a pose prototype latent vector sequence, each vector in the sequence representing an abstract pose shape related to the audio content; a cross-path attention fusion and decoding module configured to: at each time step, use the dynamic envelope vector as the query of the attention mechanism; use the pose prototype latent vector sequence as the key and value of the attention mechanism; fuse the dynamic envelope with the most relevant pose prototype by calculating the attention weight to generate a structured pose latent representation; and input the structured pose latent representation into a pre-trained pose decoder network to generate three-dimensional human body skeletal pose data at the time step; a temporal pose smoothing and rendering module configured to apply a temporal smoothing filter to the continuous skeletal pose data sequence output by the pose decoder network and drive a three-dimensional human body model.

2. The system of claim 1, wherein: The temporal convolutional network used by the multi-scale audio feature extraction module has a causal convolutional property, ensuring that the feature extraction at the current time step does not depend on future audio data when processing real-time audio streams.

3. The system of claim 2, wherein: The first neural network model in the rhythm-motion synchronization path is a gated recurrent unit network trained to learn the direct mapping relationship between audio energy changes and human motion explosive force and relaxation degree.

4. The system of claim 3, wherein: The second neural network model in the emotion-semantic encoding path is a Transformer encoder that captures long-range dependencies within an audio segment through a self-attention mechanism, thereby generating a semantically coherent pose prototype sequence.

5. The system for driving human hand and body gestures with audio according to claim 4, wherein: The structured pose latent representation is a concatenation vector containing at least a context pose prototype vector generated by cross-path attention fusion and a current dynamic envelope vector, i.e. z t = [c t , d t ], where c t is the fused pose prototype and d t is the dynamic envelope.

6. The system of claim 5, wherein: The pose decoder network is a multi-layer perceptron or inverse GRU network that, together with a pose encoder, forms a variational autoencoder architecture and is pre-trained on a large-scale human motion capture dataset to learn a decoupled, structured pose latent space.

7. The system of claim 6, wherein: The temporal pose smoothing and rendering module uses a Kalman filter or Savitzky-Golay filter to post-process the generated skeletal joint angle or position sequence to eliminate high-frequency jitter and ensure the physical authenticity of the motion.

8. A method for driving a human hand and body posture by audio, characterized by, The system for driving human hand and body poses from audio comprises the following steps: S1: hierarchical feature extraction: receiving an audio stream and extracting its multi-scale temporal features using a temporal convolutional network; S2: dual-path parallel mapping: S2.1 (Rhythm Pathway) : Separate rhythm energy and other information from audio features, generate dynamic envelope vector sequence through the first neural network model; S2.2 (Semantic Pathway) : Separate spectral content and other information from audio features, generate posture prototype latent vector sequence through the second neural network model; S3: Collaborative fusion and decoding: S3.1: For each time step in the sequence, use the dynamic envelope vector of the step as a query to perform cross-path attention operation on the posture prototype latent vector sequence to obtain the fused context posture prototype vector; S3.2: Concatenate the context posture prototype vector and the dynamic envelope vector into a structured posture latent representation; S3.3: Send the latent representation into the posture decoder network to generate the single-frame skeletal posture at the current time; S4: Smoothing and rendering: Perform temporal filtering and smoothing processing on the continuously generated skeletal posture sequence, and use it to drive the rendering of the three-dimensional model.

Images