Face fusion deformation parameter generation method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a face fusion deformation parameter generation method, electronic equipment and a storage medium, and the method comprises the steps: firstly, obtaining an audio feature, a noise signal corresponding to a current moment, and a reference hidden variable corresponding to a reference expression; then, the audio features, the noise signals and the reference hidden variables are input into a double-flow combination module for position coding and shared attention processing, and a corresponding interactive noise unit, an interactive expression unit and an interactive audio unit are obtained; afterwards, the interactive noise unit, the interactive expression unit and the interactive audio unit are input into a uniflow combination module for fusion attention processing, and a fusion noise unit is obtained; and finally, performing stream matching resolving processing based on the fusion noise unit to obtain fusion deformation hidden variable data corresponding to a preorder moment, and obtaining a face fusion deformation parameter based on the fusion deformation hidden variable data, thereby remarkably improving flexibility and naturalness of the finally generated digital human expression animation and adaptability of a business scene.

Description

Technical Field

[0001] This application relates to the field of visual data processing technology, and in particular to a method for generating facial fusion deformation parameters, an electronic device, and a storage medium. Background Technology

[0002] In the fields of virtual digital humans and computer graphics, using voice-driven 3D facial animation to generate realistic lip movements and expressions is a key technology for enhancing the realism of human-computer interaction. To achieve this goal, related technologies typically employ deep learning-based generative models (such as diffusion models or autoregressive models) for data generation. These methods extract acoustic features from speech signals and combine them with pre-defined emotion category labels (such as "happy," "sad," etc.) as conditional inputs to predict a sequence of facial fusion deformation parameters that change over time, thereby driving the digital human model to exhibit facial movements that match the speech content.

[0003] However, existing control methods mainly rely on abstract emotion labels, which have too coarse an information granularity and make it difficult to finely limit the amplitude of the generated expressions or the details of specific facial regions (e.g., it is impossible to accurately control the opening and closing of a smile or the fine-tuning of specific muscles). As a result, the generated expressions often lack distinctiveness or fail to meet specific business customization needs. In addition, existing generative model architectures often lack effective feature isolation and deep fusion mechanisms when processing multimodal data such as audio, noise, and conditional signals. This can easily cause mutual interference between different modal information, making it difficult for the generated facial fusion deformation parameters to stably follow the expected expression style. Consequently, the generated digital human facial animations are stiff, monotonous, and lack dynamism and accuracy. Summary of the Invention

[0004] This application provides a method for generating facial fusion deformation parameters, an electronic device, and a storage medium, which can improve the agility and accuracy of the generated digital human facial expression animation.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for generating facial fusion deformation parameters, the method comprising: Acquire audio features, the noise signal corresponding to the current moment, and the reference latent variables corresponding to the reference facial expression; The audio features, the noise signal, and the reference latent variable are input into the dual-stream combination module for positional encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit, and interactive audio unit. The interactive noise unit, the interactive facial expression unit, and the interactive audio unit are input into a single-stream combination module for fusion attention processing to obtain a fused noise unit; Flow matching is performed based on the fusion noise unit to obtain the fusion deformation latent variable data corresponding to the previous time step, and facial fusion deformation parameters are obtained based on the fusion deformation latent variable data.

[0006] To achieve the above objectives, a second aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method for generating facial fusion deformation parameters as described in the first aspect.

[0007] To achieve the above objectives, a third aspect of this application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for generating facial fusion deformation parameters as described in the first aspect.

[0008] The method, electronic device, and storage medium for generating facial fusion deformation parameters proposed in this application include: First, acquiring audio features, the noise signal corresponding to the current moment, and the reference latent variable corresponding to the reference expression; then, inputting the audio features, noise signal, and reference latent variable into a dual-stream combination module for position encoding and shared attention processing to obtain corresponding interactive noise units, interactive expression units, and interactive audio units; subsequently, inputting the interactive noise units, interactive expression units, and interactive audio units into a single-stream combination module for fusion attention processing to obtain fusion noise units; finally, performing stream matching calculation based on the fusion noise units to obtain fusion deformation latent variable data corresponding to the previous moment, and obtaining facial fusion deformation parameters based on the fusion deformation latent variable data. This application's embodiments introduce high-dimensional visual feature constraints by obtaining reference latent variables corresponding to reference expressions as explicit generation guidance conditions. This effectively overcomes the shortcomings of existing technologies that rely solely on abstract emotion labels, resulting in coarse control granularity and an inability to finely limit the amplitude of expressions or local details. This achieves high-precision, fine-grained control over facial animation. Simultaneously, by adopting an architecture that connects dual-stream combined modules in series with single-stream combined modules, the dual-stream structure is first used to perform shared attention interaction on the basis of feature isolation of audio features, noise signals, and reference latent variables. Then, a single-stream structure is used for deep fusion. This mechanism effectively solves the feature interference problem in multimodal data processing, ensuring that different modal information can retain their respective characteristics and be fully correlated. This allows the facial fusion deformation parameters generated by stream matching to stably and accurately follow the expected expression style, significantly improving the agility, naturalness, and adaptability of the final generated digital human facial animation.

[0009] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0010] Figure 1 This is a flowchart of a method for generating facial fusion deformation parameters according to an embodiment of this application.

[0011] Figure 2 This is a schematic diagram of audio feature extraction and alignment provided in another embodiment of this application.

[0012] Figure 3 yes Figure 1 The flowchart for step 102.

[0013] Figure 4 yes Figure 3 The flowchart for step 303.

[0014] Figure 5 yes Figure 4 The flowchart for step 403.

[0015] Figure 6 yes Figure 5 The flowchart for step 503.

[0016] Figure 7 This is a schematic diagram illustrating the principle of a two-dimensional rotational position encoding provided in another embodiment of this application.

[0017] Figure 8 yes Figure 5 The flowchart for step 506.

[0018] Figure 9 This is a schematic diagram of a data stream processing procedure in a dual-stream submodule, provided in another embodiment of this application.

[0019] Figure 10 yes Figure 1 The flowchart for step 103.

[0020] Figure 11 yes Figure 10 The flowchart for step 1003.

[0021] Figure 12 This is a schematic diagram of a data stream processing procedure in a single-stream submodule, provided in another embodiment of this application.

[0022] Figure 13 yes Figure 1 The flowchart for step 104.

[0023] Figure 14 yes Figure 1 Another flowchart for step 104.

[0024] Figure 15 This is a schematic flowchart illustrating the generation process of facial fusion deformation parameters according to an embodiment of this application.

[0025] Figure 16 This is an example diagram illustrating the generation of facial fusion deformation parameters according to an embodiment of this application.

[0026] Figure 17 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0030] In the fields of virtual digital humans and computer graphics, using voice-driven 3D facial animation to generate realistic lip movements and expressions is a key technology for enhancing the realism of human-computer interaction. To achieve this goal, related technologies typically employ deep learning-based generative models (such as diffusion models or autoregressive models) for data generation. These methods extract acoustic features from speech signals and combine them with pre-defined emotion category labels (such as "happy," "sad," etc.) as conditional inputs to predict a sequence of facial fusion deformation parameters that change over time, thereby driving the digital human model to exhibit facial movements that match the speech content.

[0031] However, existing control methods mainly rely on abstract emotion labels, which have too coarse an information granularity and make it difficult to finely limit the amplitude of the generated expressions or the details of specific facial regions (e.g., it is impossible to accurately control the opening and closing of a smile or the fine-tuning of specific muscles). As a result, the generated expressions often lack distinctiveness or fail to meet specific business customization needs. In addition, existing generative model architectures often lack effective feature isolation and deep fusion mechanisms when processing multimodal data such as audio, noise, and conditional signals. This can easily cause mutual interference between different modal information, making it difficult for the generated facial fusion deformation parameters to stably follow the expected expression style. Consequently, the generated digital human facial animations are stiff, monotonous, and lack dynamism and accuracy.

[0032] To improve the agility and accuracy of the generated digital human facial animation, this application introduces high-dimensional visual feature constraints by obtaining reference latent variables corresponding to reference expressions as explicit generation guidance conditions. This effectively overcomes the shortcomings of existing technologies that rely solely on abstract emotion labels, resulting in coarse control granularity and an inability to finely limit the amplitude of expressions or local details. This achieves high-precision, fine-grained control over facial animation. Simultaneously, by adopting an architecture that connects dual-stream combined modules to single-stream combined modules, the dual-stream structure is first used to perform shared attention interaction on the basis of feature isolation of audio features, noise signals, and reference latent variables. Then, a single-stream structure is used for deep fusion. This mechanism effectively solves the feature interference problem in multimodal data processing, ensuring that different modal information retains its own characteristics while being fully correlated. This allows the facial fusion deformation parameters generated through stream matching to stably and accurately follow the expected expression style, significantly improving the agility, naturalness, and adaptability to business scenarios of the final generated digital human facial animation.

[0033] In this application, the implementation is based on a deep learning-based neural network architecture, which mainly consists of three parts: a variational autoencoder (VAE) for compressing facial blending deformation data, a module for audio feature extraction, and a flow matching-based diffusion generation model. The VAE model, as the fundamental component for data transformation, includes an encoder based on 1D convolution and a decoder based on 1D deconvolution.

[0034] During implementation, the encoder is responsible for processing the high-dimensional Blendshape coefficient sequence (e.g., ... Compression mapping is used to map the distribution parameters of latent variables to lower dimensions (such as...). ), and through reparameterization techniques ( The generator produces true low-dimensional latent variables to compress reference expressions into the latent space for model processing; the decoder, at the end of the generation process, restores the latent variable data obtained from the flow matching solution back to the original dimensional Blendshape coefficients, thereby driving the digital human animation.

[0035] Based on this, the method for generating facial fusion deformation parameters, the electronic device, and the storage medium in the embodiments of this application will be described in detail below. First, the method for generating facial fusion deformation parameters in the embodiments of this application will be described. (Refer to...) Figure 1 This is an optional flowchart of the method for generating facial fusion deformation parameters provided in the embodiments of this application. Figure 1 The method described may include, but is not limited to, steps 101 to 104. It is also understood that this embodiment... Figure 1 The order of steps 101 to 104 is not specifically limited; the order of steps can be adjusted or certain steps can be added or removed according to actual needs. The method for generating facial fusion deformation parameters provided in this application can be applied to any control system with computing resources (such as servers, computers, etc.).

[0036] Step 101: Obtain the audio features, the noise signal corresponding to the current moment, and the reference latent variables corresponding to the reference expression.

[0037] Step 101 will be described in detail below.

[0038] In some embodiments, in response to a request to generate facial blending deformation parameters, the process first involves acquiring audio features, the noise signal corresponding to the current time step, and the reference latent variable corresponding to the reference expression. In this step, the audio features (Aud-Embed) typically refer to the speech data vector processed by a preprocessing and feature extraction model (such as the Whisper model), used to provide temporal content guidance for lip-sync generation; the noise signal corresponding to the current time step (Noisy-token) refers to the Gaussian noise data or intermediate state data to be denoised at the current time step during the reverse generation process of a stream matching or diffusion model; the reference latent variable (Ref-Latent) corresponding to the reference expression (Ref-Bs) refers to the low-dimensional latent space vector obtained after inputting user-preset facial blending deformation (Blendshape) data with specific style or amplitude limitations into the encoder of a variational autoencoder (VAE) and performing compression mapping. This step aims to prepare multimodal input conditions for the subsequent generative model, establishing a generative foundation driven by audio and constrained by reference expressions.

[0039] The audio features are extracted using a pre-trained Whisper model. The extracted audio data feature frequency is 50Hz, but the latent variable frequency used by the VAE model and the diffusion model is 15Hz. Based on this difference, this application embodiment designs an audio feature alignment algorithm, which mainly includes two steps: 1) Whisper feature stacking. This application scheme uses all 5 layers of Whisper features from the shallowest to the deepest layer as audio features, that is, the initial features are... Then we stack these five feature layers along the channel dimension to obtain stacked features. 2) Audio feature alignment: For a sequence of latent variables, take the stacked features closest to each latent variable on the time scale as the starting point, take four consecutive features, and then stack them on the channel dimension to obtain aligned audio features. .

[0040] Reference Figure 2 This is a schematic diagram illustrating the extraction and alignment of audio features according to an embodiment of this application. Figure 2 As shown in the diagram, the input audio is first processed using a pre-trained Whisper model to extract all five layers of features from shallow to deep (as shown in the attached diagram). Figure 2 The layers are labeled "1st layer features" to "5th layer features" to balance acoustic details and semantic information. Then, the extracted multi-layer features are stacked along the channel dimension to form "stacked features". Finally, in order to solve the problem of inconsistency between audio feature frequencies (e.g., 50Hz) and facial Blendshape latent variable frequencies (e.g., 15Hz), a feature alignment algorithm is adopted. That is, starting from the stacked feature frame closest to each target Blendshape latent variable on the time axis, a certain number (e.g., 4 frames) of features are continuously selected for stacking and mapping again, so as to obtain "aligned features" that are strictly aligned with the Blendshape latent variables in time, which serve as the input conditions for subsequent generative networks.

[0041] Step 102: Input the audio features, noise signal, and reference latent variable into the dual-stream combination module for position encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit, and interactive audio unit.

[0042] Step 102 is described in detail below.

[0043] In some embodiments, after obtaining the audio features, noise signal, and reference latent variable, the audio features, noise signal, and reference latent variable are input into the dual-stream combination module for position encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit, and interactive audio unit.

[0044] In this process, the dual-stream combination module adopts a network architecture that physically isolates the audio modality from the visual modality (noise and reference expression) at the initial stage of input; the positional encoding process specifically refers to using techniques such as Rotation Position Encoding (RoPE) to assign spatiotemporal information to the input data based on modality identification and temporal position, so that the model can distinguish different modalities and time frames; the shared attention processing refers to calculating the correlation weights between different modalities through the attention mechanism while maintaining the independent flow of each modality, thereby outputting feature units that have undergone preliminary information interaction, namely interactive noise units, interactive expression units, and interactive audio units.

[0045] The main diffusion model in this application is generated using flow matching, which is achieved by a dual-stream combination module (including eight cascaded dual-stream Dit modules, i.e., eight cascaded dual-stream sub-modules) and a single-stream combination module (including eight cascaded single-stream Dit modules, i.e., eight cascaded single-stream sub-modules). As mentioned above, the input of MMDit includes noise signals; a reference BS signal—which is manually specified according to the business scenario and then converted into latent variable input by the VAE encoder mentioned above; and audio information—the input audio is obtained by the audio feature extraction module mentioned above.

[0046] The data processing flow in the dual-stream combination module will be described further below.

[0047] Reference Figure 3 The audio features, noise signals, and reference latent variables are input into the dual-stream combination module for position encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit, and interactive audio unit, including the following steps 301 to 303.

[0048] Step 301: Input the reference latent variable and noise signal into the multilayer perceptron for sensing processing to obtain the sensing reference latent variable and sensing noise signal.

[0049] Step 302: Input the audio features into a multilayer perceptron for perceptual processing to obtain the first perceptual audio features.

[0050] Step 303: Input the perceptual reference latent variable, the perceptual noise signal and the first perceptual audio feature into the dual-stream combination module for position encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit and interactive audio unit.

[0051] Steps 301 to 303 are described in detail below.

[0052] Since the noise signal is essentially a "noisy BS latent variable signal", it and the reference BS signal (i.e., the reference latent variable) are processed using the same MLP before being input into the subsequent backbone network for calculation; the audio features are processed using another MLP before being input into the backbone network for calculation.

[0053] Based on this, to perform appropriate data processing in the two-stream combination module, the reference latent variables and noise signals are first input into a multilayer perceptron for perceptual processing to obtain the perceptual reference latent variables and the perceptual noise signals. In this step, the multilayer perceptron (MLP), as a feedforward neural network structure, mainly undertakes the function of feature mapping and dimension alignment. Since the reference latent variables (usually generated by the VAE encoder) and the noise signals (usually random Gaussian noise or intermediate noise) belong to visual or geometrically related modal data, their original feature dimensions may not be consistent with the hidden layer dimensions of the backbone network. Through perceptual processing, the model can project these original inputs into a unified high-dimensional feature space, thereby transforming them into intermediate expressions rich in semantic information, i.e., perceptual reference latent variables and perceptual noise signals, laying the data foundation for subsequent two-stream interaction.

[0054] Furthermore, the audio features are input into a multilayer perceptron for perceptual processing to obtain the first perceptual audio features. Similar to the previous steps, the audio features, as an independent modality, also need to be preprocessed through a dedicated path. This step utilizes a multilayer perceptron (MLP) to perform nonlinear transformation and dimensionality adaptation on the audio data, extracting acoustic features in the speech signal that are strongly correlated with lip-sync, and transforming them into the first perceptual audio features in the same feature space as the visual modality. This processing ensures that the audio modality retains its unique temporal characteristics before entering the attention mechanism, while also possessing interactivity with the visual modality in terms of numerical distribution.

[0055] Next, the perceptual reference latent variable, perceptual noise signal, and first perceptual audio features are input into the dual-stream ensemble module for positional encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive facial expression unit, and interactive audio unit. In this step, the dual-stream ensemble module receives perceptual features of the visual class (reference latent variable, noise) and the auditory class (audio) through parallel network branches. Positional encoding (such as 2D RoPE) plays a crucial role here, giving each input feature a clear modal identifier and temporal positional information, enabling the model to distinguish between "heard" and "seen" content. Shared attention processing means that within the dual-stream structure, although the feedforward network is independent, the attention mechanism allows feature vectors from different modalities to query and aggregate with each other. The final output interactive noise unit, interactive facial expression unit, and interactive audio unit are updated features that have undergone cross-modal information exchange and incorporated contextual dependencies.

[0056] The data processing flow in the dual-stream combination module will be described further below.

[0057] Reference Figure 4 The dual-stream combination module includes multiple cascaded dual-stream sub-modules. The perceptual reference latent variable, perceptual noise signal and first perceptual audio feature are input into the dual-stream combination module for position encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit and interactive audio unit, including the following steps 401 to 404.

[0058] Step 401: Obtain the time step parameters, input the time step parameters into the multilayer perceptron for sensing processing, and obtain the sensing time parameters.

[0059] Step 402: Generate the first modulation scaling bias parameter based on the sensing time parameter.

[0060] Step 403: Based on the cascade order of multiple dual-stream submodules and the first modulation scaling bias parameter, position encoding and shared attention processing are performed on the dual-stream input data in each dual-stream submodule to obtain dual-stream output data; wherein, the dual-stream input data of the first dual-stream submodule is the perceptual reference latent variable, the perceptual noise signal and the first perceptual audio feature, and the dual-stream input data of the other dual-stream submodules is the dual-stream output data of the previous dual-stream submodule.

[0061] Steps 401 to 403 are described in detail below.

[0062] To perform appropriate data processing in the two-stream combination module, the timestep parameter *t* must first be obtained. This timestep parameter is then input into a multilayer perceptron for perceptual processing to obtain the perceptual time parameter. In this step, the timestep parameter is typically a scalar or vector in the diffusion model that indicates the current denoising process (e.g., indicating the *t*th step of the diffusion process). It determines the current noise level and the required denoising intensity. To enable the neural network to effectively understand and utilize this temporal information, it needs to be input into the multilayer perceptron for feature extraction and dimension mapping, transforming it into a high-dimensional, semantically rich embedding vector, i.e., the perceptual time parameter (Timestep embedding). This provides temporal contextual guidance for subsequent network layers.

[0063] Further, the operation of generating the first modulation scaling bias parameters based on the perceptual time parameters is performed. This step aims to implement a time-step-based feature modulation mechanism (often called adaptive layer normalization, AdaLN). The first modulation scaling bias parameters specifically include a coefficient vector used to scale and shift the features, specifically including the first modulation parameter (first scaling parameter Aa, first bias parameter Ba), the second modulation parameter (second scaling parameter Ab, second bias parameter Bb), the third modulation parameter (third scaling parameter Da, third bias parameter Ea), and the fourth modulation parameter (fourth scaling parameter Db, fourth bias parameter Eb), etc. The model calculates these parameters based on the current perceptual time parameters through linear projection and other methods, so as to dynamically adjust the feature distribution in subsequent network layers, ensuring that the statistical properties of the features match the current diffusion time step, thereby controlling the rhythm and direction of generation.

[0064] Then, within the dual-stream combination module, based on the cascaded order of multiple dual-stream submodules and the first modulation scaling bias parameter, positional encoding and shared attention processing are performed on the dual-stream input data in each dual-stream submodule to obtain dual-stream output data. In this process, the dual-stream combination module adopts a deep cascaded architecture, consisting of multiple structurally identical but parameter-independent dual-stream submodules connected end-to-end. The processing logic follows the cascaded order: for the first dual-stream submodule in the link, its received dual-stream input data explicitly consists of the pre-processed perceptual reference latent variables, perceptual noise signals, and first perceptual audio features from the preceding steps; while for each subsequent dual-stream submodule, its input data directly inherits the dual-stream output data from the previous submodule. In the computation of each layer, the dual-stream submodule modulates the features using the generated first modulation scaling bias parameter and performs cross-modal shared attention calculation in conjunction with positional encoding, thereby deepening the interaction and extraction of features layer by layer.

[0065] The data processing flow in a dual-stream submodule will be described further below.

[0066] Reference Figure 5 The first modulation scaling bias parameter includes a first modulation parameter, a second modulation parameter, a third modulation parameter, and a fourth modulation parameter. In each dual-stream submodule, the dual-stream input data is position-encoded and shared attention is processed to obtain dual-stream output data, including the following steps 501 to 507.

[0067] Step 501: In the first dual-stream submodule, the first perceived audio feature is modulated based on the first modulation parameter to obtain the first modulated audio feature, and the fusion deformation feature is modulated based on the second modulation parameter to obtain the first modulated fusion feature. The fusion deformation feature includes the perceived reference latent variable and the perceived noise signal.

[0068] Step 502: Generate an audio attention parameter sequence based on the first modulation audio features, and generate a fusion attention parameter sequence based on the first modulation fusion features.

[0069] Step 503: Perform positional encoding on the audio attention parameter sequence to obtain the encoded audio sequence, and perform positional encoding on the fused attention parameter sequence to obtain the encoded fused sequence.

[0070] Steps 501 to 503 are described in detail below.

[0071] Each two-stream submodule contains two independent Transformer modules. One module accepts tokens generated from blendshape latent variables; the other accepts tokens generated from audio features. Furthermore, all two-stream modules share a global AdaLN (Adaptive LayerNorm) parameter generation module. This module generates modulation parameters shared by all two-stream modules. After the modulation parameters are input into the two-stream modules, each module adds them to its own unique bias parameter to form the final modulation parameters. Each Transformer module consists of an AdaLN module, a qkv generation module, a shared attention computation module, and an MLP module. Residual connections are used for both the attention output and the MLP output.

[0072] Based on this, taking the data execution of the first two-stream submodule as an example, in the first two-stream submodule, the first perceptual audio features (AudioTkoens) are first modulated based on the first modulation parameters (including the first scaling parameter Aa and the first bias parameter Ba) to obtain the first modulated audio features. Then, the fused deformable features (BsTkoens) are modulated based on the second modulation parameters (the second scaling parameter Ab and the second bias parameter Bb) to obtain the first modulated fused features. In this step, the fused deformable features are a set concept, specifically including visually relevant perceptual reference latent variables and perceptual noise signals. This step uses the first and second modulation parameters (including the scaling factor Scale and the bias factor Shift) derived from the time step information to adaptively normalize the feature distributions of the audio stream and the visual stream (i.e., AdaLN), thereby injecting the temporal information of the current diffusion time step into the features of each modality, ensuring that the statistical characteristics of the input data match the current denoising stage.

[0073] Both AdaLN modules accept data input x and two modulation parameters: scale (i.e., four parameters Ab, Aa, Db, and Da) and shift (i.e., four parameters Bb, Ba, Eb, and Ea). They then modulate x according to the modulation formula. This allows for the introduction of information about the time step in the diffusion model for denoising.

[0074] Then, the operations of generating an audio attention parameter sequence based on the first modulated audio features and generating a fused attention parameter sequence based on the first modulated fusion features are performed. This process is to prepare the input vectors required by the self-attention mechanism, that is, to map the modulated features into query vectors, key vectors, and value vectors through a linear projection layer or a Qkv generation module. The audio attention parameter sequence and the fused attention parameter sequence represent these sets of vectors to be relevance calculated. They retain the semantic content of their respective modalities and are ready to enter the attention space.

[0075] The qkv generation module transforms the input signal x into three initial vectors: query, key, and value, through a fully connected layer (Qkv projection). The query and key vectors are then processed by RMSNorm (root mean square normalization) and multimodal 2DROPE to form the final query and key vectors.

[0076] Therefore, further, positional encoding is performed on the audio attention parameter sequence to obtain an encoded audio sequence, and positional encoding is performed on the fused attention parameter sequence to obtain an encoded fused sequence. In this step, positional encoding (i.e., Rotational Positional Encoding, RoPE) is explicitly superimposed on the aforementioned parameter sequences (including the Query and Key vectors). This not only endows each element in the sequence with temporal positional information but may also distinguish modal identities through specific encoding strategies. Through this processing, the encoded audio sequence and the encoded fused sequence numerically carry geometric information of "who I am (modality)" and "where I am (time)," providing crucial spatial constraints for subsequent attention calculations. This process will be further described in detail below.

[0077] Reference Figure 6 The audio attention parameter sequence is positionally encoded to obtain an encoded audio sequence, including the following steps 601 to 607.

[0078] Step 601: Use the query parameter vector and key parameter vector in the audio attention parameter sequence as rotation processing parameter vectors respectively.

[0079] Step 602: Calculate the frequency vector based on the feature dimension of the audio attention parameter sequence, and obtain the rotation angle based on the product of the current position parameter corresponding to the rotation processing parameter and the frequency vector. The current position parameter is generated based on the two-dimensional coordinates constructed from the modality identifier and the temporal position.

[0080] Step 603: Based on the positional order of the vector elements in the rotation processing parameter vector, divide the odd and even positional elements in the rotation processing parameter vector to obtain odd rotation vectors and even rotation vectors.

[0081] Step 604: Multiply the even-numbered rotation vector with the cosine of the rotation angle element by element to obtain an even-numbered cosine value; multiply the even-numbered rotation vector with the sine of the rotation angle element by element to obtain an even-numbered sine value; multiply the odd-numbered rotation vector with the cosine of the rotation angle element by element to obtain an odd-numbered cosine value; multiply the odd-numbered rotation vector with the sine of the rotation angle element by element to obtain an odd-numbered sine value.

[0082] Step 605: Based on the difference between even cosine values ​​and odd sine values, obtain an even rotation encoding vector, and based on the sum of even sine values ​​and odd cosine values, obtain an odd rotation encoding vector.

[0083] Step 606: Based on the merging process of even-numbered and odd-numbered rotation encoding vectors, the rotation position encoding vector is obtained.

[0084] Step 607: Update the audio attention parameter sequence based on the rotation position encoding vector to obtain the encoded audio sequence.

[0085] Steps 601 to 607 are described in detail below.

[0086] The working principle of multimodal 2D ROPE includes adding an extra one-dimensional data dimension to normal one-dimensional data, thereby displaying and prompting the model's purpose and modality. Two-dimensional coordinates (x, y) are used to describe each data point: y corresponds to the position of the data in the current sequence, i.e., for a sequence of length n, y = 0, 1, 2, ..., n-1; x corresponds to the data modality. In this application, x = 0 represents the input noise of the diffusion model; x = 1 represents audio features; x = 2 represents the minimum value reference blendshape; x = 3 represents the average value reference blendshape; and x = 4 represents the maximum value reference blendshape.

[0087] This two-dimensional coordinate is then converted into ROPE (Rotation Position Encoding) and superimposed on the query and key. Specifically, for each modality of data, the two-dimensional coordinates of the entire sequence are first split into two sets of one-dimensional coordinates according to x and y. The rotation position encoding is then calculated according to the following formula, resulting in two position encoding vectors with shape (n, 192). These two vectors are then concatenated along the channel dimension to obtain the final position encoding vector with shape (n, 384).

[0088] Based on this approach, taking the processing of audio attention parameter sequences as an example (the processing flow for fused attention parameter sequences is similar), the query parameter vector and key parameter vector in the audio attention parameter sequence are first treated as rotation processing parameter vectors. In this step, the audio attention parameter sequence is intermediate data prepared for the self-attention mechanism calculation after linear projection. To introduce relative position information in the subsequent dot product attention calculation, the system specifically selects the query parameter vector (representing the query intent) and the key parameter vector (representing the queried attribute) as the objects to be processed, collectively referred to as rotation processing parameter vectors, while the value parameter vector is usually not processed. This is the standard implementation of the Rotation Position Encoding (RoPE) algorithm, which aims to maintain the relative distance relationship between vectors through rotation operations.

[0089] Next, the frequency vector is calculated based on the feature dimension of the audio attention parameter sequence. The rotation angle is then calculated by multiplying the current position parameter corresponding to the rotation processing parameters with the frequency vector. In this step, the frequency vector is typically an inverse frequency generated based on the size of the feature dimension. It is not a single temporal index, but rather a two-dimensional coordinate system constructed based on modal identifiers (e.g., labeling the data as an audio modality) and temporal position (which frame the data is in the sequence). This means that the calculated rotation angle (Theta) not only contains temporal information but also distinguishes different modal spaces through unique coordinate encoding, providing a precise spatial anchor for the mixed computation of multimodal data.

[0090] The schematic formula for the frequency vector is as follows: The schematic formula for the current position parameter is shown below.

[0091]

[0092] The formula for the rotation angle is as follows: .

[0093] Reference Figure 7 This is a schematic diagram illustrating the principle of a two-dimensional rotational position encoding provided in an embodiment of this application. For example... Figure 7 As shown, in order to accurately distinguish data of different modalities and time sequences within a unified attention space, this application assigns a specific two-dimensional coordinate (x, y) to each data frame in the input sequence: the first-dimensional coordinate x serves as a modality ID, used to define the category to which the data belongs, specifically x=0 corresponds to noise signals, x=1 corresponds to audio features, x=2 corresponds to the minimum reference base (BS), x=3 corresponds to the mean reference base (BS), and x=4 corresponds to the maximum reference base (BS); the second-dimensional coordinate y represents the temporal position (Time ID) of the data in its respective modality sequence. The rotational position encoding generated based on this two-dimensional coordinate system enables the model to simultaneously perceive the "identity" (modality) and "position" (time sequence) of the data during shared attention computation, thereby achieving accurate alignment and interaction of cross-modal features.

[0094] Next, based on the positional order of the vector elements in the rotation processing parameter vector, the odd and even position elements in the vector are divided to obtain odd-numbered rotation vectors and even-numbered rotation vectors. This step is a real-number preprocessing for complex field rotation operations. Since the core idea of ​​RoPE is to treat every two adjacent elements in the feature vector as a complex number for rotation, it is necessary to process the elements in the vector with even indexes. (Real part corresponding terms) are extracted as even-numbered rotation vectors. The element at the odd-numbered position. (Imaginary part corresponding terms) are extracted as odd-numbered rotation vectors. This allows high-dimensional vectors to be decomposed into paired subspaces for independent rotation transformations.

[0095] Then, the even cosine value is obtained by performing element-wise multiplication based on the cosine of the even rotation vector and the rotation angle. The even sine value is obtained by multiplying the even rotation vector by the sine of the rotation angle. And the corresponding operations on odd vectors, including odd cosine values. Sine value of odd numbers This step essentially involves calculating the individual component terms in the rotation matrix. Using the rotation angles obtained in step 602, which carry two-dimensional spatiotemporal information, the system calculates their sine and cosine values, and performs broadcast multiplication with the split vector components. These intermediate results (even cosine, even sine, odd cosine, and odd sine values) form the basis of the subsequent geometric rotation transformation operators.

[0096] Furthermore, an even-numbered rotation encoding vector is obtained by performing a calculation based on the difference between even-numbered cosine values ​​and odd-numbered sine values, i.e. And based on the sum of even sine and odd cosine values, the operation of obtaining the odd-numbered rotated encoding vector is performed, i.e. Through this cross-addition and subtraction operation, the model successfully "injects" the rotation angle calculated based on two-dimensional coordinates into the feature vector, causing the even-numbered rotation encoding vector and the odd-numbered rotation encoding vector to undergo a phase shift in numerical terms corresponding to their positions in the multimodal spatiotemporal space.

[0097] Then, based on the merging of even-numbered and odd-numbered rotation code vectors, the rotation position code vector is obtained, i.e. After the components undergo rotation transformation, this step re-interweaves or splices the separated odd and even parts, restoring them to the original vector dimensional structure, thereby generating a complete rotation position encoding vector. This vector now retains both the original semantic feature strength and, through its phase changes, implies the precise positional information of the feature in the multimodal sequence.

[0098] Finally, the audio attention parameter sequence is updated based on the rotated positional encoding vector to obtain the encoded audio sequence. As the end point of the positional encoding process, this step replaces or updates the original query and key vectors with the computed vector containing positional information, generating the final encoded audio sequence. This sequence will then be input into the self-attention module for computation.

[0099] Through steps 601 to 607 above, an improved rotational position encoding process based on a two-dimensional coordinate system is described. By constructing the modality identifier and temporal position into two-dimensional coordinates and mapping them to rotation angles, the problem of position confusion in multimodal Transformers is solved. Under the shared attention mechanism, the model can naturally perceive "whether the current feature is audio or image" and "which frame of the time series it is located in" using dot product operations. This achieves accurate alignment and decoupling of cross-modal features in the same vector space, significantly improving the model's ability to understand long-sequence multimodal data and the consistency of its generation.

[0100] Step 504: Concatenate the encoded audio sequence and the encoded fusion sequence to obtain a concatenation parameter sequence, and input the concatenation parameter sequence into the self-attention module for attention processing to obtain shared attention feature data.

[0101] Step 505: Perform temporal segmentation on the shared attention feature data to obtain segmented audio attention features and segmented fused attention features.

[0102] Step 506: Perform residual connection processing based on the third modulation parameter and the segmented audio attention feature to obtain residual audio features, and perform residual connection processing based on the fourth modulation parameter and the segmented fusion attention feature to obtain residual fusion features.

[0103] Steps 504 to 506 are described in detail below.

[0104] Then, the encoded audio sequence and the encoded fusion sequence are concatenated to obtain a concatenation parameter sequence. This concatenation parameter sequence is then input into the self-attention module for attention processing, resulting in shared attention feature data. This is the core step in the dual-stream submodule to achieve cross-modal interaction. During the attention calculation phase, the system concatenates the audio and visual stream sequences into a long sequence (i.e., the concatenation parameter sequence) in the temporal dimension and sends it to the shared self-attention module. This allows the audio features to "see" noise and reference expressions, and vice versa, thereby calculating the association weights between all positions. The output shared attention feature data contains global context information that has undergone sufficient interactive fusion.

[0105] In the shared self-attention module, it aggregates the query, key, and value vectors generated by the two transformer modules. Specifically, if the q, k, and v vectors generated by transformer 1 have shapes of (n, 384), (n, 384), and the q, k, and v vectors generated by transformer 2 have shapes of (m, 384), (m, 384), then this module first concatenates them temporally to obtain q, k, and v vectors of shape (m+n, 384). The aggregated q, k, and v vectors are then divided into 12 groups along the channel dimension (each group has q, k, and v vectors of shape (m+n, 32)). Within each group, self-attention is calculated using the following formula.

[0106]

[0107] After the calculation is completed, the 12 sets of calculation results are combined to obtain a result vector with the shape (m+n,384).

[0108] Next, the shared attention feature data is temporally segmented to obtain segmented audio attention features and segmented fusion attention features. After completing cross-modal information exchange, in order to maintain the independence of the dual-stream architecture for subsequent streaming processing, the mixed feature data needs to be separated again. This step, based on the sequence length index before splicing, decomposes the shared attention feature data back into the original two streams, obtaining the updated audio part (segmented audio attention features) and visual part (segmented fusion attention features) respectively. The resulting vectors will then be further split into vectors of shape (n, 384) and (m, 384).

[0109] Next, residual connection processing based on the third modulation parameter and the segmented audio attention features is performed to obtain residual audio features. Then, residual connection processing based on the fourth modulation parameter and the segmented fusion attention features is performed to obtain residual fused features. In this step, residual connection processing (typically including feedforward neural network processing and skip connections) is used to address the gradient vanishing problem in deep networks and promote information flow. Simultaneously, the features updated by the attention mechanism are again subjected to time-step-based gating or modulation using the third and fourth modulation parameters to further refine the control of feature activation levels. Finally, residual audio features and residual fused features containing rich contextual information and numerical stability are obtained. The residual connection processing method will be further described below.

[0110] Reference Figure 8 The residual audio features are obtained by performing residual connection processing based on the third modulation parameters and segmented audio attention features, including the following steps 801 to 804.

[0111] Step 801: Perform linear processing on the segmented audio attention features to obtain linear audio features.

[0112] Step 802: Perform product processing based on the first gating parameter and the linear audio feature, and add the first perceptual audio feature to obtain the intermediate residual audio feature.

[0113] Step 803: Modulate the intermediate residual audio features based on the third modulation parameters to obtain the second modulated audio features, and input the second modulated audio features into the multilayer perceptron for perception processing to obtain the second perceived audio features.

[0114] Step 804: Multiply the second perceived audio features based on the second gating parameters and add the intermediate residual audio features to obtain the residual audio features.

[0115] Steps 801 to 804 are described in detail below.

[0116] In some embodiments, the separated vectors are first passed through a fully connected layer within their respective transformer modules, and then multiplied by the gated modulation parameters (i.e., Cb, Ca and subsequently Fb, Fa) before participating in the residual calculation. This result is then modulated by AdaLN once and processed by a two-layer MLP, before being multiplied by the gated modulation parameters again and finally participating in the residual calculation to obtain the final output.

[0117] Based on this, taking the residual connection processing of segmented audio attention features as an example (the processing of segmented and fused attention features is similar), the first step is to perform linear processing on the segmented audio attention features to obtain linear audio features. In this step, the segmented audio attention features are intermediate data that has just undergone temporal splitting and contains cross-modal interaction information. To adapt them back to the feature dimensions of the network backbone, or to fuse multiple subspace features generated by the multi-head attention mechanism, linear processing is required. Through a fully connected layer or matrix multiplication, the feature vector is linearly projected and recombined, thereby outputting linear audio features with a more compact structure and more concentrated semantics, preparing the numerical values ​​for subsequent residual fusion.

[0118] Then, the linear audio features are multiplied based on the first gating parameter (Ca) and added to the first perceptual audio features to obtain intermediate residual audio features. This step constitutes the residual connection structure of the first sub-layer (i.e., the attention layer) in the two-stream submodule. The first gating parameter is typically a learnable weight scalar or vector (e.g., a gating value initialized to zero) used to control the proportion of newly computed attention features, i.e., linear audio features, injected into the backbone network. The system adds the gated-weighted linear audio features to the original input of this submodule (i.e., the first perceptual audio features) to obtain intermediate residual audio features. This design allows the network to maintain an identity mapping in the early stages of training and gradually introduces interactive information as training progresses, effectively preventing the gradient vanishing problem.

[0119] Furthermore, the intermediate residual audio features are modulated based on the third modulation parameters to obtain the second modulated audio features. These second modulated audio features are then input into a multilayer perceptron for perceptual processing to obtain the second perceptual audio features. This step marks the beginning of the second sub-layer (i.e., the feedforward network layer) in the dual-stream submodule. First, the input intermediate residual features are adaptively normalized using the third modulation parameters (including the third scaling parameter Da and the third bias parameter Ea) generated based on the time step, ensuring that the feature distribution conforms to the current diffusion denoising stage. Subsequently, the modulated second modulated audio features are input into the multilayer perceptron (MLP), and a nonlinear activation function is used to enhance the model's feature extraction and representation capabilities, thereby obtaining the deeply semantic second perceptual audio features.

[0120] Then, the second perceived audio features are multiplied based on the second gating parameter, and the intermediate residual audio features are added to obtain the residual audio features. This is the second residual connection within the dual-stream submodule, used to integrate the output of the feedforward network. The system uses the second gating parameter to weight the new features processed by the MLP and adds them again to the input before entering the MLP (i.e., the intermediate residual audio features). This operation ultimately outputs the final result of the dual-stream submodule on the audio branch, namely the residual audio features, ensuring the stable transmission of information flow in the deep network.

[0121] Through steps 801 to 804 above, a dual residual network structure with gating mechanism and time-step modulation is described. By introducing gating residual connections after the attention layer and the feedforward layer, not only can multimodal interaction information and nonlinear features be smoothly injected into the network, but also the feature distribution can be dynamically calibrated through the time-step modulation parameter. This fine structural design greatly improves the training stability of the deep generative model, enabling the model to gradually fuse complex cross-modal contextual information without losing the original audio signal features, thereby generating high-quality driving data.

[0122] Step 507: Based on the residual audio features and residual fusion features, obtain dual-stream output data.

[0123] Step 507 will be described in detail below.

[0124] Finally, based on the residual audio features and residual fusion features, dual-stream output data is obtained. This step concludes the current dual-stream submodule processing flow, merging the two feature streams after modulation, position coding, shared attention interaction, and residual integration, and outputting them as the input to the next cascaded submodule or the final output of the dual-stream stage.

[0125] Reference Figure 9 This is a schematic diagram illustrating a data stream processing procedure within a dual-stream submodule, as provided in an embodiment of this application. Figure 9 As shown, this module mainly consists of a modulation parameter generation module on the left, a dual-stream Transformer processing unit in the middle, and detailed sub-module diagrams on the right. First, the modulation parameter generation module in the upper left corner is responsible for processing time step information. It receives time step t (sinusoidal encoding of t) and generates globally shared "AdaLN basic modulation parameters" through a time step embedder containing MLP, SiLU activation functions, and linear layers. These parameters are then combined with the module-specific "modulation bias" and input into the main AdaLN module to control the scaling and shifting of feature distribution.

[0126] Secondly, the dual-stream Transformer processing module exhibits two parallel processing paths: the left side processes visually relevant "Bs Tokens" (corresponding to the reference latent variable and noise), and the right side processes auditory-related "Audio Tokens." The two paths are structurally symmetrical, each containing two main sub-layers: Attention Sublayer: After input data is normalized by AdaLN, it enters the "qkv generation module" (which integrates RMSnorm and multimodal 2D RoPE positional encoding) to generate query, key, and value vectors. Through "temporal concatenation," that is, the Q, K, and V vectors of the two streams are concatenated in the temporal dimension and then input into the shared "Self-Attention module" for cross-modal interaction. The data after interaction is then restored to two streams through "temporal segmentation," modulated by a Linear layer and gating parameters (such as Cb, Ca), and then connected as residuals.

[0127] Feedforward sublayer: contains a second AdaLN module, an MLP layer, and gating parameters (such as Fb, Fa). The processed features are then fused with the main path again through residual connections.

[0128] Finally, the detailed diagram on the right further illustrates the internal structure of the qkv generation module and the AdaLN module. The qkv generation module explicitly identifies the introduction of "2d rope" (two-dimensional rotational position encoding) when generating the query and key; while the AdaLN module demonstrates the modulation logic that calculates scale * x + shift based on the input parameters.

[0129] Through steps 501 to 507 above, the sophisticated "modulation-interaction-separation" mechanism inside the dual-stream submodule is explained in detail. By splicing before attention calculation and splitting after calculation, the shared attention interaction (feature correlation) of the entire sequence is achieved while maintaining the dual-stream architecture (feature independence). This design enables the audio signal to deeply guide the direction of noise denoising and the deformation trend of the reference expression. At the same time, with the fine control of four sets of modulation parameters, it is ensured that each layer of feature transformation strictly follows the time step logic of the diffusion model, thereby greatly improving the model's ability to model complex multimodal dependencies and the accuracy of the generated results.

[0130] Step 404: Based on the dual-stream output data of the last dual-stream submodule, obtain the interactive noise unit, interactive facial expression unit, and interactive audio unit.

[0131] Step 404 will be described in detail below.

[0132] In some embodiments, after the data stream has been fully processed through all cascaded dual-stream sub-modules, the data output by the last sub-module represents the final processing result of the dual-stream stage. This data includes interactive noise units (Noisy-token), interactive facial expression units (Ref-token), and interactive audio units (Aud-token), which respectively represent the noise modality, facial expression modality, and audio modality features after deep interaction. At this point, while these feature units still maintain their respective modal independence, they have deeply embedded correlation information from other modalities (e.g., the noise contains rhythmic information from the audio and shape constraints from the reference facial expression).

[0133] Through steps 401 to 404 above, a time-step-aware cascaded dual-stream processing mechanism is constructed, significantly improving the depth and accuracy of multimodal feature interaction. By generating the first modulation scaling bias parameter, the model achieves temporal adaptive adjustment of feature distribution, ensuring that the generation process conforms to the temporal logic of the diffusion model. Furthermore, the cascaded design of multiple dual-stream sub-modules enables progressive shared attention interaction between audio, noise, and reference facial expression data across multiple network layers. This "layer-by-layer" interaction method avoids the information bottleneck that may arise from single-layer interaction and effectively prevents premature assimilation or confusion of modal features by maintaining a dual-stream structure at each layer, providing high-quality, highly aligned feature representations for the final fusion of subsequent single-stream modules.

[0134] Through steps 301 to 303 above, a rigorous multimodal feature preprocessing and preliminary interaction mechanism is constructed. By setting independent perceptual processing paths (MLP) for visual data (referencing latent variables and noise) and audio data respectively, the physical isolation and dimensionality normalization of different modal features are effectively achieved, avoiding mutual interference caused by direct mixing of original features. At the same time, the dual-stream combination module is used to perform position-encoded shared attention processing, which enables the model to safely introduce cross-modal contextual information while maintaining the independent representation of each modality. As a result, the final generated interaction unit has both its own modal characteristics and contains the collaborative information required to drive facial deformation, which significantly improves the stability and controllability of the generated model.

[0135] Step 103: Input the interactive noise unit, interactive facial expression unit and interactive audio unit into the single-stream combination module for fusion attention processing to obtain the fused noise unit.

[0136] Step 103 will be described in detail below.

[0137] In some embodiments, after data processing by the dual-stream combination module, the interactive noise unit, interactive facial expression unit, and interactive audio unit are input into the single-stream combination module for fusion attention processing to obtain the fused noise unit. Unlike the aforementioned steps, the single-stream combination module aims to perform deep fusion of multimodal features. Its operation typically involves concatenating the interactive units in the sequence or channel dimension. Fusion attention processing refers to performing self-attention calculation and feedforward network processing on the concatenated mixed sequence, so that the semantic information of the audio, the constraint information of the reference facial expression, and the structural information of the noise signal are fully intertwined. The final output fused noise unit is essentially a high-dimensional feature prediction representation of the noise field or velocity field at the current time step, providing the core basis for subsequent denoising calculation.

[0138] It's important to note that during computation in the single-stream module, the reference BS signal, noise signal, and audio features are concatenated over time before being input into the single-stream module. Ultimately, only the noise signal segment is extracted from the single-stream module's output and processed by the final MLP.

[0139] The data processing flow in the single-stream combination module will be described further below.

[0140] Reference Figure 10 The single-stream combination module includes multiple cascaded single-stream sub-modules. The interactive noise unit, interactive facial expression unit and interactive audio unit are input into the single-stream combination module for fusion attention processing to obtain the fused noise unit, including the following steps 1001 to 1004.

[0141] Step 1001: Input the time step parameters into the multilayer perceptron for sensing processing to obtain the sensing time parameters.

[0142] Step 1002: Generate the second modulation scaling bias parameter based on the sensing time parameter.

[0143] Step 1003: Based on the cascade order of multiple single-stream sub-modules and the second modulation scaling bias parameter, perform fusion attention processing on the single-stream input data in each single-stream sub-module one by one to obtain single-stream output data; wherein, the single-stream input data of the first single-stream sub-module is the interactive splicing data of the interactive noise unit, the interactive expression unit and the interactive audio unit, and the single-stream input data of the other single-stream sub-modules is the single-stream output data of the previous single-stream sub-module.

[0144] Steps 1001 to 1003 are described in detail below.

[0145] The single-stream combination module provided in this application (composed of eight cascaded single-stream sub-modules) mainly consists of a Transformer module without a final fully connected layer, an MLP module without a final fully connected layer, and a shared final fully connected layer. Specifically, the input to the single-stream combination module also includes Bs Tokens and AudioTokens; unlike the processing in the two-stream combination module, these tokens are concatenated in the temporal dimension and then simultaneously input to both the Transformer and MLP modules. The outputs of the Transformer and MLP modules are concatenated in the Channel dimension and then processed by the shared fully connected layer. Similar to the two-stream sub-modules, each single-stream sub-module uses a residual design; all single-stream sub-modules also use a globally shared ADALN modulation parameter generation module and maintain a modulation bias vector within each single-stream sub-module to generate the final modulated signal. The Transformer structure of the single-stream sub-modules is also constructed in the order of "ADALN, qkv generation module, attention calculation". The MLP module is a two-layer MLP structure with the final fully connected layer removed.

[0146] Based on this, the first step involves inputting the timestep parameters into a multilayer perceptron for perceptual processing to obtain the perceptual time parameters. In this step, the timestep parameter serves as a key indicator of the current denoising progress of the diffusion model, determining the generated stage-specific features. To enable the subsequent single-stream network to understand this temporal context, the system utilizes a multilayer perceptron (MLP) to extract features and perform dimensionality expansion mapping on the low-dimensional timestep values, thereby obtaining perceptual time parameters (time embedding) containing rich semantic information, providing a unified temporal benchmark for the entire single-stream combination module.

[0147] Next, the operation of generating the second modulation scaling bias parameters based on the perceptual time parameters is performed. This step aims to construct the adjustment factors required for Adaptive Layer Normalization (AdaLN). The perceptual time parameters obtained by the model are used to calculate a set of scaling and bias coefficients (Shift) to control the feature distribution, namely the second modulation scaling bias parameters (including the fifth scaling parameter Aw and the fifth bias parameter Bw). These parameters will act on each subsequent single-stream submodule to ensure that the statistical characteristics of the activation values ​​of the network can dynamically adapt to the current diffusion time step when performing deep feature fusion, preventing the generation process from deviating from the predetermined trajectory.

[0148] Subsequently, based on the cascaded order of multiple single-stream submodules and the second modulation scaling bias parameter, the single-stream input data is processed with fusion attention in each single-stream submodule to obtain single-stream output data. In this process, the single-stream combination module adopts a cascaded architecture. The single-stream input data of the first single-stream submodule is an interactive spliced ​​data composed of the interactive noise unit, interactive facial expression unit, and interactive audio unit output from the preceding dual-stream module, concatenated along the sequence dimension. This means that the visual and auditory features previously processed separately are merged into a single long sequence here. Then, each submodule modulates the input using the second modulation scaling bias parameter according to the cascaded order and performs full-sequence self-attention calculation (fusion attention processing). Subsequent submodules use the single-stream output data from the previous level as input to delve deeper into the intermodal correlation information.

[0149] The following section will use the processing flow of the first single-stream submodule as an example to further describe the data processing flow in the single-stream submodule.

[0150] Reference Figure 11 In each single-stream submodule, the single-stream input data is subjected to fusion attention processing to obtain single-stream output data, including the following steps 1101 to 1105.

[0151] Step 1101: In the first single-stream submodule, modulation processing is performed based on the second modulation scaling bias parameter and interactive splicing data to obtain modulation splicing features, and a splicing attention parameter sequence is generated based on the modulation splicing features.

[0152] Step 1102: Input the spliced ​​attention parameter sequence into the self-attention module for attention processing to obtain spliced ​​attention feature data.

[0153] Step 1103: Perform linear processing on the interactive splicing data, and then perform activation processing based on the activation function to obtain perceptual splicing feature data.

[0154] Step 1104: Perform dimensional splicing based on the splicing attention feature data and the perception splicing feature data to obtain intermediate splicing features.

[0155] Step 1105: Input the intermediate splicing features into the shared fully connected layer for feature fusion processing to obtain fused single-stream features. Multiply the fused single-stream features by the third gating parameter and add the interactive splicing data to obtain single-stream output data.

[0156] Steps 1101 to 1105 are described in detail below.

[0157] In the first single-stream submodule, modulation processing is first performed based on the second modulation scaling bias parameter and the interactive splicing data to obtain modulation splicing features, and a splicing attention parameter sequence is generated based on the modulation splicing features. In this step, the interactive splicing data is a multimodal long sequence (including Bs Tokens and AudioTokens) that aggregates audio, noise, and reference facial expressions. The system uses the second modulation scaling bias parameter (including the fifth scaling parameter Aw and the fifth bias parameter Bw) generated based on the time step to perform adaptive layer normalization (AdaLN) on the data, injecting the temporal context into the feature distribution; subsequently, the modulated features are mapped to a combination of query, key, and value vectors through a linear layer (i.e., qkv generation), i.e., the splicing attention parameter sequence, to prepare for subsequent capture of long-distance dependencies within the sequence.

[0158] Next, the concatenated attention parameter sequence is input into the self-attention module for attention processing, resulting in concatenated attention feature data. In this step, the self-attention module calculates the relevance weights between each element in the sequence and all other elements. Since the input is a multimodal concatenated sequence, this process allows the audio frame to directly focus on the distribution of noise frames and the geometric features of the reference expression frame, and vice versa. The output concatenated attention feature data is not merely a weighted sum of the original information, but rather a deeply fused feature containing global contextual relationships.

[0159] Synchronously, the system performs linear processing on the interactively stitched data, followed by activation processing based on an activation function to obtain perceptual stitched feature data. This step constitutes the parallel feed-forward branch in the single-stream submodule. Parallel to the attention branch, the system directly performs linear projection and nonlinear activation (such as GeLU or SiLU functions) on the original interactively stitched data (or its copies). This operation focuses on pointwise semantic transformation and nonlinear dimensionality enhancement of features at each location, extracting a more abstract feature representation, i.e., perceptual stitched feature data, to supplement the shortcomings of the attention mechanism in local feature representation.

[0160] Next, an operation is performed to concatenate the attention feature data and the perception feature data to obtain intermediate concatenated features. Unlike the traditional Transformer structure which uses serial processing, this embodiment uses a channel concatenation strategy after parallel processing. The system physically concatenates the attention features that focus on global interactions with the perception features that focus on local semantics in the channel dimension. The resulting intermediate concatenated features are typically twice the size of the original features in the channel dimension, and they retain the essence of information extracted from both different processing paths.

[0161] Finally, the intermediate concatenated features are input into a shared fully connected layer for feature fusion, resulting in fused single-stream features. These fused single-stream features are then multiplied by a third gating parameter and combined with the interactive concatenated data to obtain single-stream output data. In this step, the shared fully connected layer acts as a feature compression and fusion layer, mapping the concatenated high-dimensional features back to the original channel dimensions, forming compact fused single-stream features. Subsequently, a learnable third gating parameter (such as a zero-initialization gating, Cw) is introduced to weight these features and a residual connection is established with the original input. This mechanism allows the model to degenerate to an identity mapping in the early stages of training, gradually introducing the fused incremental information as training progresses, ensuring the stability and convergence speed of deep network training.

[0162] Reference Figure 12 This is a schematic diagram illustrating a data stream processing procedure performed in a single-stream submodule, as provided in an embodiment of this application. Figure 12 The figure illustrates the parallel fusion architecture of the single-stream submodule in this embodiment. As shown, the input to this module is a multimodal hybrid sequence formed by concatenating "Bs Tokens" (containing reference latent variables and noise) and "Audio Tokens" in the temporal dimension. After entering the module, this sequence is divided into two parallel processing paths: the left path is the "Transformer module (without a final fully connected layer)", which uses the modulation parameters (Aw, Bw) generated based on the time step for adaptive layer normalization (AdaLN) and captures global dependencies through qkv generation and Self Attention calculation; the right path is the "MLP module (without a final fully connected layer)", which performs nonlinear feature transformation through a Linear layer and the GeLU activation function.

[0163] like Figure 12As shown, the outputs of the two parallel paths are first concatenated along the channel dimension to form a high-dimensional intermediate feature; then, this feature is input to the bottom shared fully connected layer for deep fusion and dimensionality restoration; finally, the fused feature is multiplied by the modulation gating parameter (Cw) and connected through the residual connection (…). Figure 12 The feature (indicated by the blue arrow in the middle) is added to the original input to output the final single-stream feature. This design ensures that while deeply fusing multimodal information, the uniformity of feature dimensions and stable gradient propagation are maintained through shared linear layers and gating mechanisms.

[0164] Through steps 1101 to 1105 above, a unique "parallel attention-MLP fusion" architecture was designed, which significantly improved the feature processing efficiency of the single-stream module. By executing the self-attention mechanism (capturing global dependencies) and the multilayer perceptron (capturing local nonlinearities) in parallel and then splicing them together, the model can simultaneously acquire long-range correlations and deep semantics of multimodal data within a single layer. In addition, with the shared fully connected layer and gated residual mechanism, this structure not only effectively controls the number of model parameters, but also solves the gradient instability problem that may occur during deep fusion of multimodal features by dynamically adjusting the information injection ratio, thus providing a solid feature foundation for generating high-quality facial deformation parameters.

[0165] Step 1004: Based on the single-stream output data of the last single-stream submodule, obtain the fused noise unit.

[0166] Step 1004 will be described in detail below.

[0167] In some embodiments, after the data stream has completely passed through all cascaded single-stream submodules, the multimodal information has been deeply entangled and fused in the feature space. The system extracts the portion corresponding to the noise dimension from the output of the last submodule (or after final projection), thus obtaining the fused noise unit (Noisy-token). This unit is no longer simply random noise, but a composite feature field that aggregates audio-driven intent and reference facial expression constraint information, which is directly used for subsequent stream matching calculations.

[0168] Through steps 1001 to 1004 above, a cascaded single-stream fusion mechanism based on time-step modulation is constructed, realizing the transformation of multimodal features from "shallow interaction" to "deep fusion". By splicing interactive noise units, interactive facial expression units, and interactive audio units into unified interactive splicing data, the model forcibly breaks the physical barriers between modalities, so that in the fusion attention processing, each frame of data can directly focus on the information of any modality in the entire sequence. With the precise control of the second modulation scaling bias parameter, this architecture effectively "baks" the prosodic details of the audio and the morphological constraints of the reference facial expression into the noise signal, thereby outputting a high-density fusion noise unit, which greatly improves the coherence and controllability of the final generated facial animation.

[0169] Step 104: Perform flow matching calculation based on the fusion noise unit to obtain the fusion deformation latent variable data corresponding to the previous time step, and obtain the facial fusion deformation parameters based on the fusion deformation latent variable data.

[0170] Step 104 will be described in detail below.

[0171] In some embodiments, after obtaining the fusion noise token, flow matching is performed based on the fusion noise token to obtain the fusion deformation latent variable data (Prediction) corresponding to the previous time step (t-1), and facial fusion deformation parameters are obtained based on the fusion deformation latent variable data. In this step, the flow matching solution refers to using the Ordinary Differential Equation (ODE) solver such as the Euler Method based on the Rectified Flow theory to deduce the sample state of the previous time step (i.e., the previous time step) based on the currently predicted velocity field, thereby gradually removing noise; the fusion deformation latent variable data is the clean facial features in the latent space obtained after the solution; finally, the latent variable is restored to the original data space by the decoder to obtain the final facial fusion deformation parameters (i.e., the specific Blendshape coefficients), which are used to directly drive the facial mesh deformation of the digital human.

[0172] The following section first describes how to perform flow matching and processing.

[0173] Reference Figure 13 The flow matching solution is performed based on the fusion noise unit to obtain the fusion deformation latent variable data corresponding to the previous time step, including the following steps 1301 to 1302.

[0174] Step 1301: Input the fusion noise unit into the multilayer perceptron for predictive sensing processing to obtain the flow matching speed corresponding to the current time.

[0175] Step 1302: Input the flow matching velocity into the flow matching solution module for flow matching solution processing to obtain the fusion deformation latent variable data corresponding to the previous time step.

[0176] Steps 1301 to 1302 are described in detail below.

[0177] In some embodiments, before performing flow matching processing on the fused noise token, it is necessary to input the fused noise token into a multilayer perceptron for predictive sensing processing to obtain the flow matching velocity at the current time step. In this step, the fused noise token is a high-dimensional feature representation rich in multimodal contextual information after two-stream interaction and single-stream deep fusion. To transform it into a physically meaningful motion vector, the system inputs it into a dedicated prediction head (i.e., a multilayer perceptron MLP) for dimensionality mapping and regression prediction to calculate the instantaneous rate of change at the current diffusion time step, i.e., the flow matching velocity (Noise_veio). Within the theoretical framework of Rectified Flow, this velocity characterizes the direction and rate of the straight-line trajectory along which the data evolves from the current noise distribution state to the target real data distribution state.

[0178] Next, the flow matching velocity (Noise_veio) is input into the flow matching solution module for flow matching processing to obtain the fused deformed latent variable data corresponding to the previous time step. In this step, the flow matching solution module is essentially an ordinary differential equation (ODE) numerical solver (such as using the Euler Method). It uses the predicted rate of change (velocity) to perform step-by-step deduction along the opposite direction of the time axis (i.e., the denoising direction). Specifically, based on the current noise state and the predicted velocity, the module calculates the potential state that the system should be in at the next discrete time node (i.e., the previous time step), thereby obtaining the denoised intermediate state data, i.e., the fused deformed latent variable data (Prediction). This process simulates the process of gradually "straightening" disordered Gaussian noise and restoring it to ordered facial geometric latent variables.

[0179] In this embodiment, the entire MMDiT (i.e., the diffusion model composed of a dual-flow combination module and a single-flow combination module) outputs the flow matching velocity at time t. In this solution, pure noise is... The sample at time 1 is denoted as BS data without noise. The sample at time 0 is denoted as [sample value]. In a rectified flow, the process of transforming pure noise into real data is [process description needed]. The denoising process, which transforms into x, is modeled as a linear process, meaning the sample at time t is... Therefore, the flow matching velocity at time t in this application can be expressed as: MMDit outputs the predicted value of this speed. Based on this predicted velocity value, the Euler method can be used to predict the sample value of the previous moment. Specifically, this application uniformly discretizes the time interval from 0 to 1 into 25 moments: 0.04, 0.08, ..., t-1, t, t+1, ... 1. Then, from... as well as The noise at time t-1 can then be calculated. .

[0180] Through steps 1301 to 1302 above, by accurately predicting the flow matching speed from the deep fusion features and combining it with the ODE solver for deterministic temporal evolution, high-quality facial deformation latent variables can be reconstructed efficiently and stably. Compared with the traditional diffusion model, the flow matching method based on the velocity field can usually provide a flatter generation trajectory, which means that a lower approximation error can be obtained with the same number of generation steps, or the number of inference steps can be significantly reduced while ensuring quality, thereby achieving efficient and high-fidelity digital human facial animation parameter generation.

[0181] The following section will further describe how to obtain facial fusion deformation parameters.

[0182] Reference Figure 14 The facial fusion deformation parameters are obtained based on the latent variable data of fusion deformation, including the following steps 1401 to 1403.

[0183] Step 1401: When the current time is greater than zero, the fused deformation latent variable data corresponding to the previous time is used as the new noise signal for the current time, and the current time is decremented by one.

[0184] Step 1402: Based on the audio features, noise signal, and reference latent variables, perform the fusion deformation latent variable data calculation of the previous time step again until the current time step is equal to zero.

[0185] Step 1403: Input the final obtained fusion deformation latent variable data into the decoder for decoding processing to obtain the facial fusion deformation parameters.

[0186] Steps 1401 to 1403 are described in detail below.

[0187] In some embodiments, when the current time is greater than zero, the fusion deformation latent variable data corresponding to the previous time is used as the new noise signal at the current time, and the current time is decremented by one. This step constructs an iterative loop mechanism in the inference process of the flow matching or diffusion model. Since the generation process of the facial fusion deformation parameters in this application is a continuous trajectory that gradually evolves from a pure noise state to a clear data state, as long as the current time t has not yet returned to zero, it means that the denoising process has not ended. At this time, the system will feed back the intermediate state result (i.e., the fusion deformation latent variable data corresponding to the previous time) that was just calculated and represents a more realistic data distribution to the input end, redefine it as the starting point of the next round of calculation (i.e., the new noise signal at the current time), and update the time counter (i.e., decrement the current time by one). This propels the generation process toward the end of the timeline (t=0).

[0188] Next, the system performs a second calculation of the fused and deformed latent variable data from previous time steps based on audio features, noise signals, and reference latent variables, until the current time step equals zero. In this step, the system repeatedly calls the aforementioned dual-stream and single-stream combination module for inference. Although the audio features and reference latent variables, as conditional inputs, maintain semantic guidance throughout the process (their corresponding features may slide with the time window), the core noise signal is continuously updated and cleaned in each iteration. This recursive calculation method ensures that the model can gradually eliminate random disturbances along the predicted velocity field trajectory, progressively recovering the spatiotemporal structure of facial movements until the time step reaches zero (i.e., t=0), marking the complete termination of the stream matching solution process.

[0189] The final fused deformation latent variable data is input into the decoder for decoding processing to obtain the facial fusion deformation parameters. In this step, the fused deformation latent variable data obtained at the end of the iteration is a pure feature representation located in a low-dimensional latent space. In order to transform it into data that can be directly recognized by the graphics engine, it needs to be upsampled and reconstructed using the decoder component in the variational autoencoder (VAE). This process restores the abstract latent features to high-dimensional, physically semantic facial fusion deformation parameters (i.e., blendshape coefficients), which directly correspond to the weight values ​​of each muscle control point on the digital human face mesh.

[0190] Steps 1401 to 1403 above describe the entire process from latent space inference to final parameter output. By constructing an iterative solution closed loop based on decreasing time steps, it is ensured that the generated facial movements strictly follow the optimized trajectory planned by the flow matching theory, thus mathematically guaranteeing the accuracy and stability of the generated results. The final decoding stage successfully breaks down the barrier between efficient latent space computation and practical graphics-driven applications, enabling the abstract features obtained through complex deep learning computation to be transformed into smooth, natural, and highly controllable digital lip-syncing animations.

[0191] Reference Figure 15 This is a schematic block diagram illustrating the generation process of facial fusion deformation parameters provided in an embodiment of this application. For example... Figure 15 The diagram illustrates the overall network architecture and data flow logic of the Multimodal Diffusion Generation Model (MMDiT) provided in this application embodiment. Figure 15 As shown, the model's input receives multiple modal data: reference facial expressions (Ref-Bs) are transformed into reference latent variables (Ref-Latent) via a VAE encoder, and then processed by an MLP along with noise signals (Noisy-token); audio features (Aud-Embed) are processed by a separate MLP; simultaneously, the time step parameter t is used to generate global modulation parameters through the MLP and AdaLN modules. These multimodal features first enter a cascaded "two-stream combination module" (such as...). Figure 15 In the purple module, visual features and audio features interact based on shared attention while maintaining independent flow paths, outputting preliminary fused interaction features (Ref-token, Noisy-token, Aud-token).

[0192] Subsequently, the data stream enters a deep fusion phase. The modal tokens output by the dual-stream module are concatenated and aggregated along the sequence dimension, and then input to the cascaded "single-stream combination module" (such as...). Figure 15 (Green module in the middle). At this stage, all modal information undergoes comprehensive attention calculation and deep mixing within a unified feature space. The single-stream combination module also accepts AdaLN parameter adjustment based on time step t, and after multi-layer processing, outputs a highly fused feature sequence. The system extracts the part corresponding to the noise dimension (e.g., Figure 15 As shown in the red box, the current flow matching velocity (Noise_Velo) is predicted by the MLP at the end.

[0193] Finally, the model enters the flow matching solution and generation stage. The predicted velocity field is input into the "flow matching solution" module, and algorithms such as the Euler method are used to derive the latent variable states of the previous time step. This process forms a closed-loop feedback (…). Figure 15(Blue and green arrow loops): If the current time step t is not equal to 0, the updated latent variable is used as a new noise signal to loop back to the input and continue iterating; if t=0, it indicates that the denoising is completed, and the final latent variable is sent to the "VAE decoder" to be reconstructed and restored to the final facial fusion deformation parameters (Res-Bs), thereby driving the 3D digital human to generate high-quality lip-sync animation.

[0194] Reference Figure 16 This is an example diagram illustrating the generation of facial fusion deformation parameters provided in an embodiment of this application. Figure 16 The diagram illustrates the application scenarios and controllability of voice-driven lip-sync generation in this embodiment. Under the condition of inputting the same voice signal (e.g., having a digital human say "I am precise"), by inputting reference expressions (mean reference BS) representing different emotional states (e.g., sad, normal, happy) as guiding conditions, the generation model of this application can accurately generate three sets of distinct facial animation sequences with strictly synchronized lip movements and audio (e.g., ...). Figure 16 (The evolution process of frame 1, frame 40, frame 80, and the end frame is shown in the figure). This result intuitively verifies that the technical solution successfully achieves fine-grained control over the style and amplitude of facial expressions in the generated results by introducing a reference latent variable as a strong constraint. This allows the digital human to stably present the specific emotional color expected by the user while maintaining accurate lip movements, avoiding the problems of vague or unstable facial expression control in traditional methods.

[0195] Through steps 101 to 104 above, by introducing reference latent variables as strong guiding conditions and adopting a cascaded architecture of "first dual-stream interaction, then single-stream fusion," the problems of coarse granularity of emotion control and mutual interference of multimodal features in existing technologies are effectively solved. The introduction of reference expressions enables precise control over the amplitude and style of the generated results (such as controlling the minimum, maximum, or mean amplitude); while the progressive processing mechanism from dual-stream to single-stream ensures the independence and integrity of different modal features in the initial stage and achieves deep semantic alignment in the later stage, thereby generating high-fidelity, highly controllable, and temporally coherent facial animation parameters.

[0196] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the facial fusion deformation parameter generation method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0197] Please see Figure 17 , Figure 17 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1702 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1702 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1702 and is called and executed by the processor 1701 to execute the facial fusion deformation parameter generation method of the embodiments of this application. The input / output interface 1703 is used to implement information input and output; The communication interface 1704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1705 transmits information between various components of the device (e.g., processor 1701, memory 1702, input / output interface 1703, and communication interface 1704); The processor 1701, memory 1702, input / output interface 1703 and communication interface 1704 are connected to each other within the device via bus 1705.

[0198] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described method for generating facial fusion deformation parameters.

[0199] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0200] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0201] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0202] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0203] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0204] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0205] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0207] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0209] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating facial fusion deformation parameters, characterized in that, The method includes: Acquire audio features, the noise signal corresponding to the current moment, and the reference latent variables corresponding to the reference facial expression; The audio features, the noise signal, and the reference latent variable are input into the dual-stream combination module for positional encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit, and interactive audio unit. The interactive noise unit, the interactive facial expression unit, and the interactive audio unit are input into a single-stream combination module for fusion attention processing to obtain a fused noise unit; Flow matching is performed based on the fusion noise unit to obtain the fusion deformation latent variable data corresponding to the previous time step, and facial fusion deformation parameters are obtained based on the fusion deformation latent variable data.

2. The method for generating facial fusion deformation parameters according to claim 1, characterized in that, The step of inputting the audio features, the noise signal, and the reference latent variable into a dual-stream combination module for positional encoding and shared attention processing to obtain corresponding interactive noise units, interactive facial expression units, and interactive audio units includes: The reference latent variable and the noise signal are input into a multilayer perceptron for sensing processing to obtain the sensing reference latent variable and the sensing noise signal. The audio features are input into a multilayer perceptron for perceptual processing to obtain the first perceptual audio features. The perceptual reference latent variable, the perceptual noise signal, and the first perceptual audio feature are input into a dual-stream combination module for position encoding and shared attention processing to obtain the corresponding interactive noise unit, interactive expression unit, and interactive audio unit.

3. The method for generating facial fusion deformation parameters according to claim 2, characterized in that, The dual-stream combination module includes multiple cascaded dual-stream sub-modules. The step of inputting the perceptual reference latent variable, the perceptual noise signal, and the first perceptual audio feature into the dual-stream combination module for position encoding and shared attention processing to obtain corresponding interactive noise units, interactive facial expression units, and interactive audio units includes: The time step parameters are obtained and input into the multilayer perceptron for sensing processing to obtain the sensing time parameters. Based on the sensing time parameters, a first modulation scaling bias parameter is generated; Based on the cascading order of the multiple dual-stream submodules and the first modulation scaling bias parameter, position encoding and shared attention processing are performed on the dual-stream input data in each dual-stream submodule to obtain dual-stream output data. Wherein, the dual-stream input data of the first dual-stream submodule is the perceptual reference latent variable, the perceptual noise signal and the first perceptual audio feature, and the dual-stream input data of the other dual-stream submodules is the dual-stream output data of the previous dual-stream submodule; Based on the dual-stream output data of the last dual-stream submodule, the interactive noise unit, the interactive facial expression unit, and the interactive audio unit are obtained.

4. The method for generating facial fusion deformation parameters according to claim 3, characterized in that, The first modulation scaling bias parameter includes a first modulation parameter, a second modulation parameter, a third modulation parameter, and a fourth modulation parameter. The step of performing position encoding and shared attention processing on the dual-stream input data in each dual-stream submodule to obtain dual-stream output data includes: In the first dual-stream submodule, the first perceived audio feature is modulated based on the first modulation parameter to obtain the first modulated audio feature, and the fusion deformation feature is modulated based on the second modulation parameter to obtain the first modulated fusion feature. The fusion deformation feature includes the perceived reference latent variable and the perceived noise signal. An audio attention parameter sequence is generated based on the first modulation audio feature, and a fusion attention parameter sequence is generated based on the first modulation fusion feature; The audio attention parameter sequence is subjected to positional encoding to obtain an encoded audio sequence, and the fused attention parameter sequence is subjected to positional encoding to obtain an encoded fused sequence; The encoded audio sequence and the encoded fusion sequence are concatenated to obtain a concatenation parameter sequence, and the concatenation parameter sequence is input into the attention module for attention processing to obtain shared attention feature data; The shared attention feature data is temporally segmented to obtain segmented audio attention features and segmented fused attention features; Residual connection processing is performed based on the third modulation parameters and the segmented audio attention features to obtain residual audio features, and residual connection processing is performed based on the fourth modulation parameters and the segmented fusion attention features to obtain residual fusion features; The dual-stream output data is obtained based on the residual audio features and the residual fusion features.

5. The method for generating facial fusion deformation parameters according to claim 4, characterized in that, The step of performing positional encoding on the audio attention parameter sequence to obtain an encoded audio sequence includes: The query parameter vector and the key parameter vector in the audio attention parameter sequence are respectively used as rotation processing parameter vectors; Based on the feature dimensions of the audio attention parameter sequence, a frequency vector is calculated, and the rotation angle is obtained by multiplying the current position parameter corresponding to the rotation processing parameter with the frequency vector. The current position parameter is generated based on two-dimensional coordinates constructed from modality identifier and temporal position. Based on the positional order of the vector elements in the rotation processing parameter vector, the odd and even positional elements in the rotation processing parameter vector are divided to obtain odd rotation vectors and even rotation vectors. An even cosine value is obtained by multiplying the even rotation vector with the cosine of the rotation angle element by element; an even sine value is obtained by multiplying the even rotation vector with the sine of the rotation angle element by element; an odd cosine value is obtained by multiplying the odd rotation vector with the cosine of the rotation angle element by element; and an odd sine value is obtained by multiplying the odd rotation vector with the sine of the rotation angle element by element. Based on the difference between the even cosine value and the odd sine value, an even rotation encoding vector is obtained, and based on the sum of the even sine value and the odd cosine value, an odd rotation encoding vector is obtained. Based on the merging process of the even-numbered rotation encoding vector and the odd-numbered rotation encoding vector, the rotation position encoding vector is obtained; The audio attention parameter sequence is updated based on the rotation position encoding vector to obtain the encoded audio sequence.

6. The method for generating facial fusion deformation parameters according to claim 4, characterized in that, The residual connection processing based on the third modulation parameters and the segmented audio attention features to obtain residual audio features includes: The segmented audio attention features are linearly processed to obtain linear audio features; The intermediate residual audio feature is obtained by multiplying the first gating parameter with the linear audio feature and adding the first perceived audio feature. Based on the third modulation parameter, the intermediate residual audio feature is modulated to obtain the second modulated audio feature, and the second modulated audio feature is input into a multilayer perceptron for perception processing to obtain the second perceived audio feature. The second perceived audio feature is multiplied based on the second gating parameter, and then the intermediate residual audio feature is added to obtain the residual audio feature.

7. The method for generating facial fusion deformation parameters according to claim 1, characterized in that, The single-stream combination module includes multiple cascaded single-stream sub-modules. The step of inputting the interactive noise unit, the interactive facial expression unit, and the interactive audio unit into the single-stream combination module for fusion attention processing to obtain a fused noise unit includes: The time step parameters are input into the multilayer perceptron for sensing processing to obtain the sensing time parameters. Based on the sensing time parameters, a second modulation scaling bias parameter is generated; Based on the cascade order of the multiple single-stream sub-modules and the second modulation scaling bias parameter, the single-stream input data is subjected to fusion attention processing in each single-stream sub-module to obtain single-stream output data. The single-stream input data of the first single-stream submodule is the interactive splicing data of the interactive noise unit, the interactive expression unit and the interactive audio unit, and the single-stream input data of the other single-stream submodules is the single-stream output data of the previous single-stream submodule. The fusion noise unit is obtained based on the single-stream output data of the last single-stream submodule.

8. The method for generating facial fusion deformation parameters according to claim 7, characterized in that, The process of performing fusion attention processing on the single-stream input data in each single-stream submodule to obtain single-stream output data includes: In the first single-stream submodule, modulation processing is performed based on the second modulation scaling bias parameter and the interactive splicing data to obtain modulation splicing features, and a splicing attention parameter sequence is generated based on the modulation splicing features; The spliced ​​attention parameter sequence is input into the self-attention module for attention processing to obtain spliced ​​attention feature data; The interactive splicing data is linearly processed, and then activated based on an activation function to obtain perceptual splicing feature data. Based on the splicing attention feature data and the perceptual splicing feature data, dimensional splicing is performed to obtain intermediate splicing features; The intermediate splicing features are input into a shared fully connected layer for feature fusion processing to obtain fused single-stream features. The fused single-stream features are then multiplied by a third gating parameter and added to the interactive splicing data to obtain the single-stream output data.

9. The method for generating facial fusion deformation parameters according to claim 1, characterized in that, The flow matching solution based on the fused noise unit yields the fused deformation latent variable data corresponding to the preceding time step, including: The fused noise unit is input into a multilayer perceptron for predictive sensing processing to obtain the flow matching speed at the current moment. The flow matching velocity is input into the flow matching solution module for flow matching solution processing to obtain the fusion deformation latent variable data corresponding to the preceding time step.

10. The method for generating facial fusion deformation parameters according to claim 1, characterized in that, The process of obtaining facial fusion deformation parameters based on the fusion deformation latent variable data includes: When the current time is greater than zero, the fused deformed latent variable data corresponding to the previous time is used as the new noise signal for the current time, and the current time is decremented by one. Based on the audio features, the noise signal, and the reference latent variable, the fusion deformation latent variable data of the previous time step is calculated again until the current time step is equal to zero. The final obtained latent variable data of fusion deformation is input into the decoder for decoding processing to obtain the facial fusion deformation parameters.

11. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method for generating facial fusion deformation parameters as described in any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for generating facial fusion deformation parameters as described in any one of claims 1 to 10.

Images