A behavior-driven role animation synthesis method and system

Abstract

This invention discloses a behavior-driven character animation synthesis method and system, relating to the field of computer vision technology. It involves acquiring high-quality audio and corresponding 3D character actions, performing feature extraction processing, and processing music theory features, motion morphology data, and acoustic feature vectors based on a cross-modal motion audio model to obtain motion codebook vectors. The method further involves real-time extraction of latent audio vectors from the audio stream data, analyzing emotional data based on acoustic and music theory features, performing emotional priority filtering, and matching the latent audio vectors with vectors in the target emotional sub-codebook based on a probability sampling mechanism to generate preliminary character animation parameters. These preliminary animation parameters are then corrected to obtain final motion rotation parameters. Finally, the motion rotation parameters are used to model a 3D character, and the final animation is output through a low-latency transmission protocol, effectively driving character actions and dynamically changing the audiovisual synesthetic scene atmosphere.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a behavior-driven character animation synthesis method and system. Background Technology

[0002] With the rapid development of artificial intelligence, computer graphics, and virtual digital human technologies, audio-driven 3D character animation generation technology has gradually become an important research direction in fields such as virtual performance, digital human live streaming, game interaction, online education, and immersive content production.

[0003] In existing technologies, feature extraction from input audio is typically performed, followed by the use of deep learning models to predict the character's motion parameters or facial expression parameters, thereby enabling the character to synchronize lip movements, facial expressions, or body movements with the audio. Among related technologies, one type of method focuses on speech-driven facial animation generation, predicting lip shapes, facial muscle movements, or local expression parameters based on acoustic features in the speech signal. While this type of method can achieve good lip-lip synchronization, it is usually limited to the head or facial area, making it difficult to achieve a holistic expression of body skeletal movements, full-body posture, and stage atmosphere. Another type of method attempts to generate character body movements based on music or speech signals, such as predicting dance movements or body postures based on audio features like rhythm and energy. However, this type of method usually only considers the correspondence of a single modality, lacking modeling of emotional semantics. This results in generated movements that, while possibly matching in rhythm, often differ from the audio content in style, emotional expression, and scene atmosphere. Summary of the Invention

[0004] The technical problem solved by this invention is that the existing technology lacks an explicit constraint mechanism for emotion categories, which easily leads to a mismatch between the action and the emotional semantics of the audio. It is difficult to jointly model the historical action state and the music beat points, and it only outputs the character's action itself, ignoring the dynamic adjustment of visual environment parameters based on audio features.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: Firstly, a behavior-driven character animation synthesis method includes the following steps: Step S1: Collect high-quality audio and corresponding 3D character movements and perform feature extraction processing to obtain music theory features, movement morphology data, acoustic feature vectors and visual environment feature data; Step S2: Based on the cross-modal motion audio model, the music theory features, motion morphology data and acoustic feature vectors are processed to obtain the motion codebook vector; Step S3: Extract the audio latent vector of the audio stream to be processed in real time, and analyze the emotional data (emotional category and intensity) based on acoustic features and music theory features. Perform emotional priority filtering, lock the target emotional sub-codebook corresponding to the emotional data in the hierarchical discrete codebook, and match the audio latent vector with the vector in the target emotional sub-codebook based on the probability sampling mechanism to generate preliminary character animation parameters. Step S4: The initial character animation parameters are corrected by inverse kinematics calculation using the FABRIK algorithm to obtain the final motion rotation parameters; Step S5: Based on the current audio feature mapping, the visual environment features are obtained in real time to drive the rendering engine, and the motion rotation parameters are used to model the 3D character. The final animation is then output through a low-latency transmission protocol.

[0006] Preferably, step S1 includes the following sub-steps: Step S11: Synchronously acquire high-quality audio and corresponding 3D character movements to obtain a synchronized dataset; Step S12: Set a fixed time step to slice the high-quality audio, use HuBERT to extract the acoustic feature vectors in the high-quality audio, and use a music information retrieval algorithm to extract the acoustic feature vectors frame by frame to obtain music theory features, which include pitch, energy and spectral centroid. Step S13: Extract and normalize the features of the 3D character's movements to obtain motion morphology data, which includes joint angles, angular velocities, and end effector positions. Step S14: Based on Fechinger's synesthesia theory, establish a mapping rule function from music theory features to visual parameters, and map the music theory features according to the mapping rule function to obtain visual environment feature data.

[0007] Preferably, step S2 includes the following sub-steps: Step S21: Construct a motion encoder by inputting motion pattern data into the motion encoder for processing to obtain the first latent vector; Step S22: Initialize the hierarchical discrete codebook. The hierarchical discrete codebook includes several sentiment sub-codebooks. Each sentiment sub-codebook corresponds to a preset sentiment label. After obtaining the first latent vector from the action morphology data through the motion encoder, the Euclidean distance from the first latent vector to each codebook vector in the sentiment sub-codebook corresponding to the sentiment label is calculated using a vector quantization mechanism. The codebook vector with the closest Euclidean distance is extracted as the target codebook vector. The first latent vector is replaced with the target codebook vector using a pass-through estimator to obtain the quantized first latent vector. The quantized first latent vector is processed by the decoder to reconstruct the predicted action parameters. By minimizing the reconstruction loss between the predicted action parameters and the action morphology data, the parameters of the motion encoder, decoder, and hierarchical discrete codebook are updated. Step S23: Construct an audio encoder, extract the emotional tags of the audio and encode them into emotional vectors through an embedding layer. After feature fusion and concatenation of the emotional vectors with acoustic feature vectors and music theory features, input them into the audio encoder and map them to obtain continuous second latent vectors (the second latent vectors are used to represent audio latent vectors). In the hierarchical discrete codebook, calculate the Euclidean distance between the second latent vector and each codebook vector in the corresponding emotional sub-codebook, and select the codebook vector with the closest distance as the action codebook vector at the current moment.

[0008] Preferably, step S22 further includes the following sub-steps: Step S221: Construct a structured hierarchical discrete codebook and divide it into N sentiment sub-codebooks. Each of these corresponds to one of N preset emotional tags, and each emotional sub-codebook It contains K learnable codebook vectors; Step S222: For each time t, obtain the real sentiment label corresponding to the action data at that time from the training dataset. Traverse the hierarchical discrete codebook, find the sub-codebook corresponding to the real sentiment label, and use it as the sentiment sub-codebook. Calculate the Euclidean distance between the first latent vector and all vectors in the sentiment sub-codebook, find the index of the nearest codebook vector, and use it as the target codebook vector. Replace the continuous first latent vectors with the target codebook vector to obtain the quantized first latent vector. Step S223: Input the quantized first latent vector into the motion decoder to reconstruct the predicted action parameters, and define the total loss function: ; in, For the total loss function, To reconstruct the loss, For actual motion parameters, To predict motion parameters, For the speed of real movements, To predict the speed of the action, The continuous first latent vector output as the motion vector. For the corresponding codebook vector, To stop the gradient, and To balance the weighting coefficients, minimize the total loss function. The gradient descent method is used to jointly update the parameters of the motion encoder, decoder, and hierarchical discrete codebook.

[0009] Preferably, step S23 includes the following sub-steps: Step S231: Obtain acoustic feature vectors and music theory features, extract the emotional tags of the audio, map the emotional tags of the audio into emotional vectors through an embedding layer, and fuse and splice the emotional vectors with the acoustic feature vectors and music theory features to obtain audio features; Step S232: Construct an audio encoder to map the fused audio features into a continuous second latent vector; Step S233: Perform nearest neighbor search by Euclidean distance between the sentiment subcodebook and the second potential vector, select the codebook vector index with the smallest distance, and use the codebook vector index to extract the corresponding codebook vector from the sentiment subcodebook as the action codebook vector at the current moment.

[0010] Preferably, step S3 includes the following sub-steps: Step S31: Receive the target audio to be processed in real time, extract acoustic feature vectors and music theory features, analyze the target audio using an emotion recognition model, and obtain the target emotion tag. Step S32: Perform sentiment priority filtering, lock the sentiment sub-codebook corresponding to the target sentiment tag in the hierarchical discrete codebook, and block other sentiment sub-codebooks; Step S33: After fusing and splicing the acoustic feature vector, music theory feature and emotional vector obtained by encoding the target emotional label of the target audio, the result is input into the audio encoder to obtain the predicted audio latent vector. Step S34: Within the locked sentiment subcodebook, calculate the Euclidean distance between the predicted audio latent vector and each codebook vector, convert the Euclidean distance into a probability distribution, perform weighted random sampling according to the probability distribution, and select a generated codebook vector. The calculation expression for the probability distribution is: ; in, For probability distribution, The predicted latent vector output by the audio encoder. Let k be the codebook vector in the sentiment subcodebook. This is a temperature parameter used to control the randomness of sampling; Step S35: Input the generated codebook vector into the autoregressive decoder, combine it with the action state of the previous frame to generate the preliminary character animation parameters for the current frame, introduce a rhythm gating mechanism to detect the beat points of the audio, and force the decoder to generate action keyframes at the beat points.

[0011] Preferably, step S35 further includes the following sub-steps: Step S351: At the starting time t=0 of the generated sequence, initialize the hidden state of the decoder to a zero vector. For the current time t, construct the input vector of the decoder. The input vector includes the codebook vector generated at the current time t, the action parameters generated at the previous time, and the position code at the current time. Step S352: Input the input vector into the autoregressive decoder, perform forward inference, and predict the character animation parameters at the current moment; Step S353: Introduce a rhythm gating mechanism to align and enhance the timing of character animation parameters. The input audio stream is analyzed in real time using a beat tracking algorithm, and a set of beat moments is output. Based on the set of beat moments, the time difference between the current moment t and the nearest beat point is calculated. When the time difference is less than the preset time tolerance, the current moment is determined to be the key frame moment of the action, and the gate signal is set to 1. When the time difference is greater than the preset time tolerance, the gate signal is set to 0. When the gating signal is 1, motion enhancement processing is applied to the preliminary character animation parameters to obtain the preliminary character animation parameters for the current frame. The calculation expression for the motion enhancement processing is: ; in, For initial character animation parameters, To enhance the coefficient, The rate of change of each action parameter, The time interval between adjacent frames. The current frame character animation parameters before enhancement. These are the character animation parameters from the previous frame; When the gating signal is 0, the character animation parameters remain unchanged; The initial character animation parameters include facial expression parameters and body skeletal parameters.

[0012] Preferably, step S4 specifically includes: Obtain the initial character animation parameters for the current frame, and use the FABRIK algorithm to iteratively solve the initial character animation parameters to obtain the final motion rotation parameters.

[0013] Preferably, step S5 specifically includes: Step S51: Based on the current music theory features and target emotion tags, call the mapping rule function established based on Fetzinger's audiovisual synesthesia theory to map and obtain dynamic visual environment features, including background hue, light intensity and particle characteristics. Step S52: Drive the 3D character model to perform real-time character movement using the final motion rotation parameters. Combine the dynamic visual environment features obtained by mapping to drive the rendering engine in real time, adjust the background color, light intensity and particle effects, and perform composite rendering with the 3D character model. Output the final animation through a low-latency transmission protocol.

[0014] Secondly, a behavior-driven character animation synthesis system includes a feature extraction module, an action audio modeling module, a probability generation module, an iterative correction module, and a rendering output module. The feature extraction module is used to acquire high-quality audio and corresponding 3D character actions for feature extraction processing; The motion audio modeling module is used to construct a generative model that includes a motion encoder, an audio encoder, and a hierarchical discrete codebook. It trains the VQ-VAE motion audio model based on motion morphology data and music theory features to generate motion codebook vectors that represent motion morphology and have emotional discrimination. The probability generation module is used to extract the audio latent vector of the audio stream to be processed in real time, analyze the emotion category and intensity based on acoustic features and music theory features, perform emotion priority screening to lock the target emotion subcodebook, and combine the probability sampling mechanism to map the audio latent vector to the target emotion subcodebook to generate preliminary character animation parameters. The iterative correction module is used to perform inverse kinematics calculation and correction on the initial character animation parameters using the FABRIK algorithm to obtain the final motion rotation parameters. The rendering output module is used to obtain visual environment features based on the current audio feature mapping, drive the rendering engine in real time, model 3D characters using motion rotation parameters, and output the final animation through a low-latency transmission protocol.

[0015] The beneficial effects of this invention are as follows: By simultaneously extracting the music theory features, acoustic features, and morphological data of the audio, and quantizing them into a unified action codebook vector using a cross-modal model, a precise mapping bridge from audio to action is established. This overcomes the problem of action-audio disconnect caused by relying solely on a single audio feature in traditional methods. The generated character actions are highly synchronized with subtle changes in rhythm, intensity, and form in the audio, effectively improving audiovisual consistency. Furthermore, an emotion priority filtering mechanism is introduced, locking a dedicated emotion sub-codebook based on the emotion category of the real-time audio. Combined with a probabilistic sampling mechanism, rather than a single deterministic matching, this ensures that the generated actions conform to the emotional tone of the current audio, avoiding emotional confusion. The probabilistic sampling mechanism enables the generation of slightly different but natural actions even under the same audio input, avoiding mechanical repetition. The process is more complex, making the character performance more vivid. Furthermore, after generating initial animation parameters, the FABRIK algorithm is used for inverse kinematics calculation and correction, effectively solving common physical inconsistencies in purely data-driven methods, such as clipping, joint distortion, foot slippage, or the inability of hands to accurately reach target points. This makes the final movements not only look natural but also conform to ergonomic constraints, ensuring the physical credibility of the animation. Finally, a low-latency transmission protocol is adopted, and the rendering engine is driven in real-time based on audio characteristics to adjust visual environment features, meeting the requirements of real-time interactive scenarios and ensuring pleasing and immediate feedback. This invention uses audio not only to drive character movements but also to dynamically change the synesthetic atmosphere of the scene, enhancing the user's sensory immersion experience and making the entire animation output a harmonious and unified whole in terms of both vision and hearing. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the steps of a behavior-driven character animation synthesis method according to an embodiment of the present invention; Figure 2 This is a basic flowchart of a behavior-driven character animation synthesis system provided in one embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Example 1, referring to Figure 1 This paper provides a behavior-driven character animation synthesis method, which includes the following steps: Step S1: Collect high-quality audio and corresponding 3D character movements and perform feature extraction processing to obtain music theory features, movement morphology data, acoustic feature vectors and visual environment feature data; Step S2: Based on the cross-modal motion audio model, the music theory features, motion morphology data and acoustic feature vectors are processed to obtain the motion codebook vector; Step S3: Extract the audio latent vector of the audio stream to be processed in real time, and analyze the emotional data (emotional category and intensity) based on acoustic features and music theory features. Perform emotional priority filtering, lock the target emotional sub-codebook corresponding to the emotional data in the hierarchical discrete codebook, and match the audio latent vector with the vector in the target emotional sub-codebook based on the probability sampling mechanism to generate preliminary character animation parameters. Step S4: The initial character animation parameters are corrected by inverse kinematics calculation using the FABRIK algorithm to obtain the final motion rotation parameters; Step S5: Based on the current audio feature mapping, the visual environment features are obtained in real time to drive the rendering engine, and the motion rotation parameters are used to model the 3D character. The final animation is then output through a low-latency transmission protocol.

[0019] By simultaneously extracting the music theory features, acoustic features, and morphological data of the audio, and quantizing them into a unified action codebook vector using a cross-modal model, a precise mapping bridge from audio to action is established. This overcomes the problem of traditional methods relying solely on a single audio feature (such as volume or pitch), which leads to a disconnect between action and audio. This ensures that the generated character actions are highly synchronized with subtle changes in rhythm, intensity, and form in the audio, effectively improving audiovisual consistency. An emotion priority filtering mechanism is introduced, locking a dedicated emotion sub-codebook based on the emotion category of the real-time audio. Combined with a probabilistic sampling mechanism, rather than a single deterministic matching, this ensures that the generated actions match the emotional tone of the current audio (such as joy or sadness), avoiding emotional confusion. The probabilistic sampling mechanism allows for the generation of slightly different but natural actions even under the same audio input, avoiding mechanical repetition and making the character... The performance is more vivid; in addition, after generating the initial animation parameters, the FABRIK algorithm is used for inverse kinematics calculation and correction, which effectively solves the physical inconsistencies commonly found in pure data-driven methods, such as clipping, joint distortion, foot slippage, or the inability of the hands to accurately reach the target point. This makes the final movements not only look natural, but also conform to ergonomic constraints, ensuring the credibility of the animation at the physical level; finally, a low-latency transmission protocol is adopted, and the rendering engine is driven to adjust the visual environment features in real time according to the audio characteristics, which meets the requirements of real-time interactive scenarios (such as virtual anchors, game characters, and real-time performance driving), ensuring pleasing and immediate feedback. This invention uses audio not only to drive character movements, but also to dynamically change the scene atmosphere of audiovisual synesthesia, enhancing the user's sensory immersion experience when watching, so that the entire animation output forms a coordinated and unified whole in terms of vision and hearing.

[0020] Step S1 includes the following sub-steps: Step S11: Synchronously acquire high-quality audio and corresponding 3D character movements to obtain a synchronized dataset; Step S12: Set a fixed time step to slice the high-quality audio, use HuBERT to extract the acoustic feature vectors in the high-quality audio, and use a music information retrieval algorithm to extract the acoustic feature vectors frame by frame to obtain music theory features, which include pitch, energy and spectral centroid. Step S13: Extract and normalize features for the 3D character's movements to obtain motion morphology data, which includes joint angles, angular velocities, and end effector positions. Step S14: Based on Fechinger's synesthesia theory, establish a mapping rule function from music theory features to visual parameters, and map the music theory features according to the mapping rule function to obtain visual environment feature data.

[0021] High-quality audio is acquired using multi-channel recording equipment, while the 3D character movements of the performer are simultaneously captured using an optical motion capture system (such as OptiTrack, sampling rate 120fps). A hardware timecode synchronizer (such as SMPTE timecode) is used to ensure that the timestamps of the audio frames and motion frames are strictly aligned, resulting in a synchronized dataset. Specifically, each sample in the synchronized dataset contains: an audio waveform segment aligned for a certain period of time and its corresponding whole-body skeletal joint motion trajectory data. In this embodiment, data covering multiple music styles (classical, jazz, pop, etc.) and multiple emotional types (joyful, sad, exciting, calm, etc.) are collected, totaling approximately 50 hours. Each data point is accompanied by an emotional label (such as "joyful", "sad", etc.) marked by professional annotators as a supervision signal for subsequent training. High-quality audio is sliced ​​using a fixed time step (e.g., 33.3ms per frame). A pre-trained HuberT model is used to extract the acoustic feature vector of each audio slice. Music information retrieval algorithms are then used to extract the acoustic feature vectors frame-by-frame to obtain music theory features. Specifically, these features include pitch, energy, and spectral centroid. Furthermore, a beat tracking algorithm is used to extract rhythm tempo and beat timing sequences for subsequent rhythm gating mechanisms. Feature extraction is also performed on 3D character motion data to obtain motion morphology data. Specifically, the following features are extracted frame-by-frame from the raw motion capture skeletal data and normalized: Joint angles: Extract the three-axis rotation angles of each bone joint. In this embodiment, the character skeleton contains 24 joints, each joint has 3 degrees of freedom of rotation angles, for a total of 72 dimensions. Angular velocity: The angular velocity of each joint is calculated by dividing the difference between the joint angles of adjacent frames by the time step, resulting in a total of 72 dimensions; End effector location: Extract the three-dimensional position coordinates of five end effectors in the world coordinate system: palms of both hands, toes of both feet, and top of the head; a total of 15 dimensions. Z-score normalization is performed on all the above features to make each dimension feature have zero mean and unit variance, eliminating the influence of dimensional differences on model training. The normalized action shape data is a 159-dimensional vector (72+72+15=159 dimensions). Based on Fetzinger's theory of synesthesia, a mapping rule function from music theory features to visual parameters is established. Fetzinger's theory of synesthesia points out that there are cross-modal intuitive associations between human stimulation of different sensory channels. For example, high notes are often perceived as bright colors, low notes are often perceived as dark colors, and fast rhythms are associated with strong visual motion. The mapping rule function is defined as mapping the spectral centroid to the background hue, the rhythm speed to particle effects, and the energy to light intensity. The rule for mapping the spectral centroid to the background tone is as follows: Let the spectral centroid of the current frame be... After normalizing the centroid of the spectrum to the [0,1] interval, it is mapped to the hue values ​​in the HSV color space: ; in, Hue value The minimum value of all spectral centroids. The maximum value of all spectral centroids. =240, which is blue and corresponds to a dark, low-frequency tone. =60, which is yellow, corresponding to bright silver at high frequencies. For example, when the spectral centroid is high (such as 2000Hz), the background color tone is warm (yellow, orange); when the spectral centroid is low (such as 500Hz), the background color tone is cool (blue, purple). The rules for mapping rhythm speed to particle effects are as follows: Let the rhythm speed of the current paragraph be... Particle emission rate and particle velocity Calculate according to the following rules: ; ; in, and These are the reference particle emission rate and the reference particle velocity, respectively. For reference rhythm speed (set to 120 BPM in this embodiment), when the music rhythm speeds up, the particle emission frequency and movement speed increase synchronously to create a visual sense of urgency and dynamism. The rule for mapping energy to light intensity is: Let the short-time energy of the current frame be... Light intensity The mapping rules between the two are as follows: ; in, and These are the lower and upper limits of light intensity, respectively. The non-linear adjustment factor is set to 0.8 in this embodiment to make the light more in line with the non-linear characteristics perceived by the human eye. When the volume suddenly increases (such as when the music enters a climax), the light intensity increases rapidly; when the volume decreases, the light intensity decreases gently. Based on the above mapping rule function, the music theory features extracted in step S12 are mapped frame by frame to obtain visual environment feature data, including the background hue (HSV hue value), light intensity, and particle effect parameters (emission rate, motion speed) corresponding to each frame.

[0022] Step S2 includes the following sub-steps: Step S21: Construct a motion encoder by inputting motion pattern data into the motion encoder for processing to obtain the first latent vector; Step S22: Initialize the hierarchical discrete codebook. The hierarchical discrete codebook includes several sentiment sub-codebooks. Each sentiment sub-codebook corresponds to a preset sentiment label. After obtaining the first latent vector from the action morphology data through the motion encoder, the Euclidean distance from the first latent vector to each codebook vector in the sentiment sub-codebook corresponding to the sentiment label is calculated using a vector quantization mechanism. The codebook vector with the closest Euclidean distance is extracted as the target codebook vector. The first latent vector is replaced with the target codebook vector using a pass-through estimator to obtain the quantized first latent vector. The quantized first latent vector is processed by the decoder to reconstruct the predicted action parameters. By minimizing the reconstruction loss between the predicted action parameters and the action morphology data, the parameters of the motion encoder, decoder, and hierarchical discrete codebook are updated. Step S23: Construct an audio encoder, extract the emotional tags of the audio and encode them into emotional vectors through an embedding layer. After feature fusion and concatenation of the emotional vectors with acoustic feature vectors and music theory features, input them into the audio encoder and map them to obtain continuous second latent vectors (the second latent vectors are used to represent audio latent vectors). In the hierarchical discrete codebook, calculate the Euclidean distance between the second latent vector and each codebook vector in the corresponding emotional sub-codebook, and select the codebook vector with the closest distance as the action codebook vector at the current moment.

[0023] In this embodiment, a motion encoder is constructed. The motion encoder uses a one-dimensional temporal convolutional network. The motion encoder receives a sequence of motion morphology data as input, encodes it along the time dimension, and outputs a first latent vector corresponding to each frame. The first latent vector is the motion latent vector. During the training phase, the first latent vector after quantization is used as the supervision target. The alignment loss function is set as the mean square error between the second latent vector and the first latent vector after quantization. At the same time, the classification loss function is set as the cross-entropy loss between the predicted sentiment label and the real sentiment label. The parameters of the audio encoder are updated by minimizing the weighted sum of the alignment loss and the classification loss.

[0024] Step S22 also includes the following sub-steps: Step S221: Construct a structured hierarchical discrete codebook and divide it into N sentiment sub-codebooks. Each of these corresponds to one of N preset emotional tags, and each emotional sub-codebook It contains K learnable codebook vectors; Step S222: For each time t, obtain the real sentiment label corresponding to the action data at that time from the training dataset. Traverse the hierarchical discrete codebook, find the sub-codebook corresponding to the real sentiment label, and use it as the sentiment sub-codebook. Calculate the Euclidean distance between the first latent vector and all vectors in the sentiment sub-codebook, find the index of the nearest codebook vector, and use it as the target codebook vector. Replace the continuous first latent vectors with the target codebook vector to obtain the quantized first latent vector. Step S223: Input the quantized first latent vector into the motion decoder to reconstruct the predicted action parameters, and define the total loss function: ;

[0025] in, For the total loss function, To reconstruct the loss, For actual motion parameters, To predict motion parameters, For the speed of real movements, To predict the speed of the action, The continuous first latent vector output as the motion vector. For the corresponding codebook vector, To stop the gradient, and To balance the weighting coefficients, minimize the total loss function. The gradient descent method is used to jointly update the parameters of the motion encoder, decoder, and hierarchical discrete codebook.

[0026] In this embodiment, a structured hierarchical discrete codebook is constructed and divided into N=6 emotion sub-codebooks, corresponding to 6 preset emotion tags: joy, sadness, excitement, calmness, anger, and gentleness. Each emotion sub-codebook contains K learnable codebook vectors. All codebook vectors in the hierarchical discrete codebook are initialized uniformly and randomly, and updated using gradient descent during subsequent training. Then, they are vectorized. For each time t, the real emotion tag corresponding to the action data at that time is obtained from the training dataset. The hierarchical discrete codebook is traversed to find the emotion sub-codebook corresponding to the real emotion tag. The continuous first latent vector is replaced with the target codebook vector to obtain the quantized first latent vector. During backpropagation, a pass-through estimator is used to directly copy the quantized gradient to the unquantized first latent vector to solve the problem of non-differentiability of discrete operations. Codebook loss in this embodiment This term is used to drive the codebook vector to converge towards the latent vector output by the encoder. It only updates the codebook vector parameters, therefore... =0.25, because the update rate of the codebook vector needs to be slower than the update rate of the encoder. If the codebook vector is updated too quickly, it will cause the codebook space to oscillate violently during training, making it difficult for the latent vector output by the encoder to be stably mapped to a fixed codebook region, resulting in quantization instability and a decrease in codebook utilization; commitment loss This term is used to constrain the latent vectors output by the encoder to converge towards the assigned codebook vectors. It only updates the encoder parameters. =0.99 is because in the hierarchical discrete codebook architecture, each sentiment sub-codebook contains only a limited number (K=512) of codebook vectors. The latent vectors output by the encoder must be closely clustered in the neighborhood of the corresponding codebook vector to ensure that the quantization error is small enough and the sentiment discrimination is high enough. The large value of γ exerts a strong constraint on the encoder, forcing the continuous latent vectors output by the encoder to be as close as possible to the discrete codebook vectors, thereby reducing information loss during the quantization process. At the same time, since this invention adopts a hierarchical sentiment codebook structure, clear boundaries need to be maintained between different sentiment sub-codebooks. The large commitment loss weight can prevent the latent vectors output by the encoder from falling into the region of the wrong sentiment sub-codebook, enhance the discrimination between different sentiment categories, and ensure the effectiveness of the sentiment priority selection mechanism in the subsequent inference stage. The asymmetric setting of β and γ (β is much larger than γ) forms a training strategy of slow codebook updates and strong encoder constraints. On the one hand, it ensures the stable convergence of the codebook space, and on the other hand, it ensures the close alignment between the encoder output and the codebook vector. The synergistic effect of the two makes the hierarchical discrete codebook have high codebook utilization and strong sentiment discrimination ability after training.

[0027] Step S23 includes the following sub-steps: Step S231: Obtain acoustic feature vectors and music theory features, extract the emotional tags of the audio, map the emotional tags of the audio into emotional vectors through an embedding layer, and fuse and splice the emotional vectors with the acoustic feature vectors and music theory features to obtain audio features; Step S232: Construct an audio encoder to map the fused audio features into a continuous second latent vector; Step S233: Perform nearest neighbor search by Euclidean distance between the sentiment subcodebook and the second potential vector, select the codebook vector index with the smallest distance, and use the codebook vector index to extract the corresponding codebook vector from the sentiment subcodebook as the action codebook vector at the current moment.

[0028] In this embodiment, the acoustic feature vector (768-dimensional) and music theory features (including pitch, energy, and spectral centroid, totaling 3 dimensions) extracted in step S12 are obtained. Emotional tags corresponding to each audio segment are extracted. These emotional tags are mapped to 64-dimensional emotional vectors through an embedding layer (lookup table dimension is N×64=6×64). The emotional vector, acoustic feature vector, and music theory features are concatenated along the feature dimensions to obtain the fused audio features. An audio encoder is then constructed. This audio encoder adopts a multi-layer Transformer structure, containing 6 Transformer coding layers, each layer including an 8-head self-attention mechanism and a feedforward network. The fused audio feature sequence is input into the audio encoder and mapped to obtain a continuous second latent vector. The second latent vector is in sequence form. In step S233 of the training phase, the nearest neighbor search is performed between the sentiment subcodebook and the second latent vector using Euclidean distance. The codebook vector index with the smallest distance is selected. The corresponding codebook vector is extracted from the sentiment subcodebook using the codebook vector index as the action codebook vector at the current moment, which is used to calculate the alignment loss. After training, the audio encoder can map the audio features to the latent space aligned with the action codebook vector, so that semantically similar audio can be matched with semantically reasonable action codebook vectors.

[0029] Step S3 includes the following sub-steps: Step S31: Receive the target audio to be processed in real time, extract acoustic feature vectors and music theory features, analyze the target audio using an emotion recognition model, and obtain the target emotion tag. Step S32: Perform sentiment priority filtering, lock the sentiment sub-codebook corresponding to the target sentiment tag in the hierarchical discrete codebook, and block other sentiment sub-codebooks; Step S33: After fusing and splicing the acoustic feature vector, music theory feature and emotional vector obtained by encoding the target emotional label of the target audio, the result is input into the audio encoder to obtain the predicted audio latent vector. Step S34: Within the locked sentiment subcodebook, calculate the Euclidean distance between the predicted audio latent vector and each codebook vector, convert the Euclidean distance into a probability distribution, perform weighted random sampling based on the probability distribution, and select a generated codebook vector. The expression for calculating the probability distribution is: ; in, For probability distribution, The predicted latent vector output by the audio encoder. Let k be the codebook vector in the sentiment subcodebook. This is a temperature parameter used to control the randomness of sampling; Step S35: Input the generated codebook vector into the autoregressive decoder, combine it with the action state of the previous frame to generate the preliminary character animation parameters for the current frame, introduce a rhythm gating mechanism to detect the beat points of the audio, and force the decoder to generate action keyframes at the beat points.

[0030] To facilitate those skilled in the art in determining appropriate temperature parameters In this embodiment, other parameters are kept constant on the validation set, and different parameters are selected respectively. Multiple random samplings were performed to generate action sequences, and the following metrics were calculated: The codebook relevance metric R measures how closely the sampled codebook vectors match the latent audio vectors. ; in, The latent vector output by the audio encoder. The sampled codebook vector, where a smaller R indicates a stronger semantic relevance; The motion diversity index D, which is the average pairwise distance between generated sequences when the same audio segment is generated S times under different random seeds, is defined as: ; The larger the value of D, the greater the diversity. The beat alignment index B is represented by the probability of the peak speed of the action near the beat point. It is normalized to [0,1], and the larger the value, the better the beat consistency. Joint out-of-bounds ratio V is the percentage of frames that experience joint clipping out of bounds before or after FABRIK correction. A lower ratio indicates better stability. In a specific embodiment, for calibrating temperature parameters On the validation set, the model parameters are kept constant, and the following values ​​are taken respectively. The same audio segment was repeatedly generated S=10 times, and the codebook correlation index R, motion diversity index D, beat alignment index B, and joint over-limit ratio V were statistically analyzed. The results are shown in the table below:

[0031] As shown in the table, if τ is too small, the sampling will be close to the deterministic nearest neighbor, resulting in insufficient diversity. If τ is too large, the distribution will be too flat, which will weaken the semantic association of the audio and reduce the beat consistency. Taking all factors into consideration, the preferred value range for τ is [0.5, 1.2], and the default value can be about 0.8. For real-time interactive scenarios with higher stability requirements, [0.5, 0.9] can be used, while for offline creation, [0.9, 1.2] can be used.

[0032] In this embodiment, by analyzing the emotional category of the target audio in real time, the emotional sub-codebook corresponding to the target emotional tag is locked in the hierarchical discrete codebook, while other emotional sub-codebooks are blocked. This reduces the retrieval range of codebook vectors from all N×K codebook vectors to K codebook vectors of a single emotional sub-codebook. This filtering mechanism eliminates the problem of mismatches across emotional categories at the source, such as avoiding situations where sad music triggers cheerful actions or calm music triggers excited actions. It ensures that the generated action sequence is strictly consistent with the input audio at the emotional semantic level. Simultaneously, the reduction in the retrieval space reduces the computational cost of codebook matching to 1 / N of the original, effectively improving the computational efficiency of the real-time inference stage. The Euclidean distance between the audio latent vector and the codebook vector is converted into a Softmax probability distribution, and... Instead of simply selecting the nearest single codebook vector, weighted random sampling is performed on this probability distribution. This probabilistic sampling mechanism brings two advantages: First, the closer the codebook vector, the greater the probability of it being selected, ensuring a high degree of correlation between the selected codebook vector and the audio semantics, and maintaining the rationality of the generated actions. Second, codebook vectors that are close but not the closest also have a certain probability of being selected, allowing the same audio input to generate action sequences with similar styles but different details at different times. This effectively avoids the problem of monotonous repetition of actions caused by deterministic matching, giving the generated animation a stronger sense of naturalness and artistic expression. In addition, the introduction of the temperature parameter τ provides an adjustable control over the randomness of sampling, allowing users to flexibly adjust the degree of action diversity according to the needs of actual application scenarios.

[0033] Step S35 also includes the following sub-steps: Step S351: At the start time t=0 of the generated sequence, initialize the hidden state of the decoder to a zero vector. For the current time t, construct the input vector of the decoder. The input vector includes the codebook vector generated at the current time t, the action parameters generated at the previous time, and the position code at the current time. Step S352: Input the input vector into the autoregressive decoder, perform forward inference, and predict the character animation parameters at the current moment; Step S353: Introduce a rhythm gating mechanism to align and enhance the timing of character animation parameters. The input audio stream is analyzed in real time using a beat tracking algorithm, and a set of beat moments is output. Based on the set of beat moments, the time difference between the current moment t and the nearest beat point is calculated. When the time difference is less than the preset time tolerance, the current moment is determined to be the key frame moment of the action, and the gate signal is set to 1. When the time difference is greater than the preset time tolerance, the gate signal is set to 0. When the gating signal is 1, motion enhancement processing is applied to the initial character animation parameters to obtain the initial character animation parameters for the current frame. The calculation expression for motion enhancement processing is: ;

[0034] in, These are the character animation parameters after motion enhancement processing for the current frame, i.e., the initial character animation parameters. To enhance the coefficient, The rate of change of each action parameter, i.e. , The time interval between adjacent frames. The decoder directly predicts the character animation parameters output in frame t. These are the character animation parameters from the previous frame; When the gating signal is 0, the character animation parameters remain unchanged; Initial character animation parameters include facial expression parameters and body skeletal parameters.

[0035] The motion parameters that the autoregressive decoder outputs directly at the current time t, before rhythm enhancement, are denoted as the parameters before enhancement. ; The motion parameters enhanced by the rhythm gating are denoted as the enhanced parameters. (i.e., the left-hand side variable of the computational expression for enhancement processing), and use it as the input to the subsequent FABRIK inverse kinematics correction module as the output of the current frame; Furthermore, The preferred approach is to use the output motion parameters from the previous frame, i.e. This ensures that the enhancement effect is continuously applied to the final output sequence and avoids error accumulation. When the gate signal is 0, The two methods are equivalent when there are no keyframes.

[0036] The above formula calculates each dimension of the animation parameters independently using an element-wise operation. During the calculation, the parameters of the current frame before enhancement are first... Enhanced parameters compared to the previous frame Subtracting the adjacent changes yields the total change; dividing by the time interval... Obtain the rate of change of the motion parameters, then multiply by the enhancement factor. Then superimposed on the current frame parameters The enhanced character animation parameters are obtained above. ; When the gating signal is 0, the character animation parameters of the current frame output by the decoder are directly used. As initial parameters for the enhanced character animation, i.e. ; In this embodiment, the body skeleton parameters are represented by Euler angles for joint rotation. The rotation of each joint consists of three Euler angles (such as rotation angles around the X, Y, and Z axes). The above addition and subtraction operations are performed directly on the Euler angle values. Since Euler angles are angle values ​​defined in the real number field, the difference in Euler angles between adjacent frames can accurately represent the trend of joint rotation changes, thus enhancing the coefficient. Multiplying by this rate of change can linearly amplify or enhance the amplitude of the movement, thereby producing a significant enhancement effect on the beat point movement; The initial character animation parameters in this embodiment include facial expression parameters and body skeletal parameters, wherein the body skeletal parameters are represented by local joint Euler angles in the decoder output stage. Specifically, for each joint j in the skeletal system, Euler angle vectors are output in a fixed rotation order (e.g., ZYX or XYZ, consistent between training and inference): ; in, Let be the Euler angle vector at the j-th joint at the current time t. Euler angles are in Euclidean vector form, which allows addition, subtraction and difference operations in all dimensions. Therefore, linear enhancement of motion parameters of this form is used to record the amplitude of the change of each joint and each angle component, highlighting the motion accent or peak at the beat point. To avoid Euler angles wrapping around at ±π, which would cause abnormal amplification of the difference term, this embodiment calculates... The Euler angle sequence is first unwrapped to make the angles of adjacent frames change along continuous branches. After enhancement, the angles are normalized to a preset range (e.g., [-π, π]). Although the enhancement stage uses Euler angles and processes them linearly, when driving 3D character skinning or engine rendering, Euler angles can be converted into rotation matrices or quaternions according to a predetermined rotation order and then used for bone-level transformation. This conversion is a conventional graphics processing and does not involve direct linear addition or subtraction of quaternions. The subsequent FABRIK inverse kinematics correction in step S4 will further eliminate unreasonable poses such as end-effector offset and bone length constraint violation, thereby ensuring that the final output motion rotation parameters can be stably used for 3D character modeling and rendering. In this embodiment, the preset time tolerance is 60ms, and the enhancement coefficient is set to 0.3. The preset time tolerance is used to determine whether the current moment is close enough to the music beat point in order to trigger action keyframe enhancement. The enhancement coefficient is used to control the superposition intensity of the rate of change of action at the beat point. By inputting the codebook vector generated at the current moment, the action parameters generated at the previous moment, and the current position encoding into the autoregressive decoder, the generation of the action in the current frame not only depends on the current audio semantic information, but can also explicitly refer to the historical action state and time position information. This allows the generated action sequence to maintain a continuous transition between adjacent frames, avoiding the abrupt changes, jitter, or frame skipping phenomena caused by the frame-by-frame independent generation method, thereby improving the temporal stability and natural smoothness of the character's actions. Through the beat tracking and time tolerance judgment mechanism, when the time difference between the current moment and the nearest beat point is less than the preset time tolerance, action keyframe enhancement is triggered, making the generated action keyframe enhancement more effective. The animation creates more pronounced posture changes and peaks near the beat. Since the preset time tolerance is preferably 60ms, this time scale is within the acceptable range where the human eye and ear are sensitive to audiovisual synchronization. Therefore, it can ensure that the keyframes of the animation can stably fall near the music beat, and avoid the problem of missing keyframes due to small errors in beat detection. This significantly enhances the audio-visual synchronization effect. By setting an enhancement coefficient (preferably 0.3), the change trend of the current action parameters is superimposed on the action parameters of the current frame at a certain ratio at the beat point. This makes the movements such as arm swings, body swaying, head nodding, and facial expressions more vivid and powerful at the beat, highlighting the rhythmic emphasis of the action. Compared with ordinary continuous motion without enhancement, this scheme can make the animation show stronger explosiveness and visual appeal at key positions such as music climax, drum beats, and heavy beats, thereby improving the stage presence and artistic expression of the character performance.

[0037] Step S4 specifically includes: Obtain the initial character animation parameters for the current frame, and use the FABRIK algorithm to iteratively solve the initial character animation parameters to obtain the final motion rotation parameters.

[0038] In this embodiment, the initial character animation parameters of the current frame are modified by inverse kinematics to eliminate problems such as joint interleaving, end-effector offset, bone stretching, or posture incoordination that may exist in directly generated motion parameters, thereby obtaining the final motion rotation parameters that satisfy the skeletal motion constraints. Since the previous character animation parameters are generated by probability sampling and autoregression, although they can better express the semantics of motion driven by audio, there may still be problems such as changes in bone length, joint position deviation, or posture that does not conform to the laws of human body structure at local joints. By introducing the FABRIK algorithm to iteratively correct the skeletal chain, the length of each bone segment can be kept constant, avoiding limb stretching, compression, or interleaving, thereby improving the rationality of the generated motion in the skeletal structure.

[0039] Step S5 specifically includes: Step S51: Based on the current music theory features and target emotional tags, call the mapping rule function established based on Fetzinger's audiovisual synesthesia theory to map and obtain dynamic visual environment features, including background hue, light intensity, and particle characteristics. Step S52: Drive the 3D character model to perform real-time character movement using the final motion rotation parameters. Combine the dynamic visual environment features obtained by mapping to drive the rendering engine in real time, adjust the background color, light intensity and particle effects, and perform composite rendering with the 3D character model. Output the final animation through a low-latency transmission protocol.

[0040] In this embodiment, target emotion tags are introduced into the visual environment mapping process. Therefore, audio of different emotion categories can trigger differentiated visual atmosphere expressions. For example, cheerful emotions can correspond to higher brightness, warmer tones, and more active particle effects; sad emotions can correspond to lower brightness, cooler tones, and slower, sparser particle effects; and excited emotions can correspond to stronger lighting changes and higher-density particle dynamics. Through this emotion-driven environmental linkage mechanism, the generated animation can not only reflect emotions at the action level but also strengthen emotional expression at the scene level, thereby increasing the audience's emotional perception of the character's performance. Then, the final motion rotation parameters and dynamic visual environment features are used to drive the rendering engine in real time, enabling character movement, background changes, lighting changes, and particle effect changes to update synchronously with the audio input. This method makes the entire animation... The system possesses rapid response capabilities to real-time audio streams, promptly reflecting changes in musical rhythm, volume fluctuations, or emotional shifts in character actions and scene effects. This avoids the audiovisual disconnect caused by traditional static backgrounds or preset effects, enhancing audio-visual synchronization and interactivity. By dynamically adjusting background color, lighting intensity, and particle effects, it breaks through the traditional single-expression method that relies solely on character actions to convey content, adding rich environmental visual layers to the final animation. For example, background color can be used to create an overall atmosphere, changes in lighting intensity can highlight musical climaxes and key beats, and particle effects can enhance rhythm and spatial dynamism. The combined effect of various visual elements and character movement makes the final image more layered, rhythmic, and artistically appealing, making it particularly suitable for scenarios such as virtual singing, stage performances, digital human live streaming, and game cutscenes.

[0041] Example 2, refer to Figure 2 This paper presents a behavior-driven character animation synthesis system, including a feature extraction module, a motion audio modeling module, a probability generation module, an iterative correction module, and a rendering output module. The feature extraction module is used to extract features from high-quality audio and corresponding 3D character movements. The motion audio modeling module is used to build a generative model that includes a motion encoder, an audio encoder, and a hierarchical discrete codebook. It trains the VQ-VAE motion audio model based on motion morphology data and music theory features to generate motion codebook vectors that represent motion morphology and have emotional discrimination. The probability generation module is used to extract the audio latent vector of the audio stream to be processed in real time, analyze the emotion category and intensity based on acoustic features and music theory features, perform emotion priority filtering to lock the target emotion subcodebook, and combine the probability sampling mechanism to map the audio latent vector to the target emotion subcodebook to generate preliminary character animation parameters; The iterative correction module is used to perform inverse kinematics calculation and correction on the initial character animation parameters using the FABRIK algorithm to obtain the final motion rotation parameters; The rendering output module is used to obtain visual environment features based on the current audio feature mapping, drive the rendering engine in real time, model 3D characters using motion rotation parameters, and output the final animation through a low-latency transmission protocol.

[0042] In this embodiment, high-quality audio and corresponding 3D character actions are acquired and processed for feature extraction. The feature extraction module integrates the HuBERT feature extractor and the MIR analysis engine, receives synchronously acquired audio and motion data, and outputs acoustic feature vectors, music theory features, motion morphology data, and visual environment feature data. During the training phase, this module processes offline datasets; during the inference phase, this module processes the input audio stream in real time. The motion audio modeling module is deployed on a GPU training server and is used to build a generative model that includes a motion encoder, an audio encoder, and a hierarchical discrete codebook. This module trains the VQ-VAE motion audio model based on motion morphology data and music theory features, employing a two-stage training strategy: the first stage trains the motion encoder, motion decoder, and hierarchical discrete codebook, learning discrete representations of motion; the second stage trains the audio encoder to achieve cross-modal alignment. Upon completion, it outputs a hierarchical discrete codebook containing emotional discrimination and the trained audio encoder parameters, which are used by the probability generation module during the inference phase. The probability generation module, deployed on the inference server, extracts the latent audio vectors of the audio stream in real time. Based on acoustic and music theory features, it analyzes emotional categories and intensities, performs emotional priority filtering to lock onto the target emotional sub-codebook, and maps the latent audio vectors to the target emotional sub-codebook using a probabilistic sampling mechanism. It then generates initial character animation parameters through an autoregressive decoder combined with a rhythm gating mechanism. The iterative correction module, deployed on the same inference server as the probability generation module, performs inverse kinematic calculations to correct the initial character animation parameters using the FABRIK algorithm. This module receives preliminary character animation parameters from the probability generation module, performs FABRIK iterative solutions on each kinematic chain of the character's entire body to ensure constant bone length and reasonable joint angles, and outputs final motion rotation parameters that satisfy physical constraints. The rendering output module, deployed on the rendering server, is used to obtain visual environment features based on the current audio feature mapping to drive the rendering engine in real time. This module receives the final motion rotation parameters from the iterative correction module and the visual environment feature data from the feature extraction module, drives the 3D character model to move in real time, synchronously adjusts the background color, lighting intensity, and particle effects of the rendering engine, synthesizes and renders the final animation, and streams the final animation in real time via a low-latency transmission protocol.

[0043] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0044] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A behavior-driven character animation synthesis method, characterized in that, Includes the following steps: Step S1: Collect high-quality audio and corresponding 3D character movements and perform feature extraction processing to obtain music theory features, movement morphology data, acoustic feature vectors and visual environment feature data; Step S2: Based on the cross-modal motion audio model, the music theory features, motion morphology data and acoustic feature vectors are processed to obtain the motion codebook vector; Step S3: Extract the audio latent vector of the audio stream data to be processed in real time, and obtain the emotional data based on acoustic features and music theory features. Perform emotional priority filtering, lock the target emotional sub-codebook corresponding to the emotional data in the hierarchical discrete codebook, and match the audio latent vector with the vector in the target emotional sub-codebook based on the vector matching mechanism to generate preliminary character animation parameters. Step S4: The initial character animation parameters are corrected by inverse kinematics calculation using the FABRIK algorithm to obtain the final motion rotation parameters; Step S5: Based on the current audio feature mapping, the visual environment features are obtained in real time to drive the rendering engine, and the motion rotation parameters are used to model the 3D character. The final animation is then output through a low-latency transmission protocol.

2. The behavior-driven character animation synthesis method as described in claim 1, characterized in that, Step S1 includes the following sub-steps: Step S11: Synchronously acquire high-quality audio and corresponding 3D character movements to obtain a synchronized dataset; Step S12: Set a fixed time step to slice the high-quality audio, use HuBERT to extract the acoustic feature vector in the high-quality audio, and use a music information retrieval algorithm to extract the acoustic feature vector frame by frame to obtain music theory features, which include pitch, energy and spectral centroid. Step S13: Extract and normalize the features of the 3D character's movements to obtain motion morphology data, which includes joint angles, angular velocities, and end effector positions. Step S14: Based on Fechinger's synesthesia theory, establish a mapping rule function from music theory features to visual parameters, and map the music theory features according to the mapping rule function to obtain visual environment feature data.

3. The behavior-driven character animation synthesis method as described in claim 1, characterized in that, Step S2 includes the following sub-steps: Step S21: Construct a motion encoder by inputting motion pattern data into the motion encoder for processing to obtain the first latent vector; Step S22: Initialize the hierarchical discrete codebook. The hierarchical discrete codebook includes several sentiment sub-codebooks. Each sentiment sub-codebook corresponds to a preset sentiment label. After obtaining the first latent vector from the action morphology data through the motion encoder, the Euclidean distance from the first latent vector to each codebook vector in the sentiment sub-codebook corresponding to the sentiment label is calculated using a vector quantization mechanism. The codebook vector with the closest Euclidean distance is extracted as the target codebook vector. The first latent vector is replaced with the target codebook vector using a pass-through estimator to obtain the quantized first latent vector. The quantized first latent vector is processed by the decoder to reconstruct the predicted action parameters. By minimizing the reconstruction loss between the predicted action parameters and the action morphology data, the parameters of the motion encoder, decoder, and hierarchical discrete codebook are updated. Step S23: Construct an audio encoder, extract the emotional tags of the audio and encode them into emotional vectors through an embedding layer. After feature fusion and concatenation of the emotional vectors with acoustic feature vectors and music theory features, input them into the audio encoder and map them to obtain continuous second latent vectors (the second latent vectors are used to represent audio latent vectors). In the hierarchical discrete codebook, calculate the Euclidean distance between the second latent vector and each codebook vector in the corresponding emotional sub-codebook, and select the codebook vector with the closest distance as the action codebook vector at the current moment.

4. The behavior-driven character animation synthesis method as described in claim 3, characterized in that, Step S22 further includes the following sub-steps: Step S221: Construct a structured hierarchical discrete codebook and divide it into N sentiment sub-codebooks. Each of these corresponds to one of N preset emotional tags, and each emotional sub-codebook It contains K learnable codebook vectors; Step S222: For each time t, obtain the real sentiment label corresponding to the action data at that time from the training dataset. Traverse the hierarchical discrete codebook, find the sub-codebook corresponding to the real sentiment label, and use it as the sentiment sub-codebook. Calculate the Euclidean distance between the first latent vector and all vectors in the sentiment sub-codebook, find the index of the nearest codebook vector, and use it as the target codebook vector. Replace the continuous first latent vectors with the target codebook vector to obtain the quantized first latent vector. Step S223: Input the quantized first latent vector into the motion decoder to reconstruct the predicted action parameters, and define the total loss function: ； in, For the total loss function, To reconstruct the loss, For actual motion parameters, To predict motion parameters, For the speed of real movements, To predict the speed of the action, The continuous first latent vector output as the motion vector. For the corresponding codebook vector, To stop the gradient, and To balance the weighting coefficients, minimize the total loss function. The gradient descent method is used to jointly update the parameters of the motion encoder, decoder, and hierarchical discrete codebook.

5. The behavior-driven character animation synthesis method as described in claim 3, characterized in that, Step S23 includes the following sub-steps: Step S231: Obtain acoustic feature vectors and music theory features, extract the emotional tags of the audio, map the emotional tags of the audio into emotional vectors through an embedding layer, and fuse and splice the emotional vectors with the acoustic feature vectors and music theory features to obtain audio features; Step S232: Construct an audio encoder to map the fused audio features into a continuous second latent vector; Step S233: Perform nearest neighbor search by Euclidean distance between the sentiment subcodebook and the second potential vector, select the codebook vector index with the smallest distance, and use the codebook vector index to extract the corresponding codebook vector from the sentiment subcodebook as the action codebook vector at the current moment.

6. The behavior-driven character animation synthesis method as described in claim 1, characterized in that, Step S3 includes the following sub-steps: Step S31: Receive the target audio to be processed in real time, extract acoustic feature vectors and music theory features, analyze the target audio using an emotion recognition model, and obtain the target emotion tag. Step S32: Perform sentiment priority filtering, lock the sentiment sub-codebook corresponding to the target sentiment tag in the hierarchical discrete codebook, and block other sentiment sub-codebooks; Step S33: After fusing and splicing the acoustic feature vector, music theory feature and emotional vector obtained by encoding the target emotional label of the target audio, the result is input into the audio encoder to obtain the predicted audio latent vector. Step S34: Within the locked sentiment subcodebook, calculate the Euclidean distance between the predicted audio latent vector and each codebook vector, convert the Euclidean distance into a probability distribution, perform weighted random sampling according to the probability distribution, and select a generated codebook vector. The calculation expression for the probability distribution is: ； in, For probability distribution, The predicted latent vector output by the audio encoder. Let k be the codebook vector in the sentiment subcodebook. This is a temperature parameter used to control the randomness of sampling. The set of all codebook vectors contained in the sentiment subcodebook corresponding to the current target sentiment tag. Let j be the codebook vector in the target sentiment subcodebook. Represents the subcodebook. This indicates that the target sentiment label corresponds to the current target audio. Step S35: Input the generated codebook vector into the autoregressive decoder, combine it with the action state of the previous frame to generate the preliminary character animation parameters for the current frame, introduce a rhythm gating mechanism to detect the beat points of the audio, and force the decoder to generate action keyframes at the beat points.

7. The behavior-driven character animation synthesis method as described in claim 6, characterized in that, Step S35 also includes the following sub-steps: Step S351: At the starting time t=0 of the generated sequence, initialize the hidden state of the decoder to a zero vector. For the current time t, construct the input vector of the decoder. The input vector includes the codebook vector generated at the current time t, the action parameters generated at the previous time, and the position code at the current time. Step S352: Input the input vector into the autoregressive decoder, perform forward inference, and predict the character animation parameters at the current moment; Step S353: Introduce a rhythm gating mechanism to align and enhance the timing of character animation parameters. The input audio stream is analyzed in real time using a beat tracking algorithm, and a set of beat moments is output. Based on the set of beat moments, the time difference between the current moment t and the nearest beat point is calculated. When the time difference is less than the preset time tolerance, the current moment is determined to be the key frame moment of the action, and the gate signal is set to 1. When the time difference is greater than the preset time tolerance, the gate signal is set to 0. When the gating signal is 1, motion enhancement processing is applied to the preliminary character animation parameters to obtain the preliminary character animation parameters for the current frame. The calculation expression for the motion enhancement processing is: ； in, For initial character animation parameters, To enhance the coefficient, The rate of change of each action parameter, The time interval between adjacent frames. The current frame character animation parameters before enhancement. These are the character animation parameters from the previous frame; When the gating signal is 0, the character animation parameters remain unchanged; The initial character animation parameters include facial expression parameters and body skeletal parameters.

8. The behavior-driven character animation synthesis method as described in claim 1, characterized in that, Step S4 specifically includes: Obtain the initial character animation parameters for the current frame, and use the FABRIK algorithm to iteratively solve the initial character animation parameters to obtain the final motion rotation parameters.

9. The behavior-driven character animation synthesis method as described in claim 1, characterized in that, Step S5 specifically includes: Step S51: Based on the current music theory features and target emotion tags, call the mapping rule function established based on Fetzinger's audiovisual synesthesia theory to map and obtain dynamic visual environment features, including background hue, light intensity and particle characteristics. Step S52: Drive the 3D character model to perform real-time character movement using the final motion rotation parameters. Combine the dynamic visual environment features obtained by mapping to drive the rendering engine in real time, adjust the background color, light intensity and particle effects, and perform composite rendering with the 3D character model. Output the final animation through a low-latency transmission protocol.

10. A behavior-driven character animation synthesis system, applied in a behavior-driven character animation synthesis method as described in any one of claims 1-9, characterized in that, It includes a feature extraction module, a motion audio modeling module, a probability generation module, an iterative correction module, and a rendering output module; The feature extraction module is used to acquire high-quality audio and corresponding 3D character actions for feature extraction processing; The motion audio modeling module is used to construct a generative model that includes a motion encoder, an audio encoder, and a hierarchical discrete codebook. It trains the VQ-VAE motion audio model based on motion morphology data and music theory features to generate motion codebook vectors that represent motion morphology and have emotional discrimination. The probability generation module is used to extract the audio latent vector of the audio stream to be processed in real time, analyze the emotion category and intensity based on acoustic features and music theory features, perform emotion priority screening to lock the target emotion subcodebook, and combine the probability sampling mechanism to map the audio latent vector to the target emotion subcodebook to generate preliminary character animation parameters. The iterative correction module is used to perform inverse kinematics calculation and correction on the initial character animation parameters using the FABRIK algorithm to obtain the final motion rotation parameters. The rendering output module is used to obtain visual environment features based on the current audio feature mapping, drive the rendering engine in real time, model 3D characters using motion rotation parameters, and output the final animation through a low-latency transmission protocol.

Images