Interactive three-dimensional head generation method and apparatus based on interleaved multi-modal context

Abstract

This invention relates to the fields of computer vision and speech processing technology, and particularly to an interactive 3D head generation method and apparatus based on interleaved multimodal context. It primarily addresses the problems of decoupling speech and 3D head motion, lack of context awareness, inconsistent generation results, and insufficient multimodal information fusion, proposing the following technical solution: Step 1, extracting speech signal features; Step 2, fusing head motion features with speech features; Step 3, dividing speech features and 3D head motion features into fixed-length segments; Step 4, inputting multimodal context segments to a multilayer transformer encoder; Step 5, generating a 3D head dynamic sequence; Step 6, adjusting the generation result through a guidance mechanism. This invention unifies the speech and head motion feature spaces to achieve precise adaptation and fusion, and combines multimodal context modeling and diffusion-guided generation to improve the synergy between head dynamics and speech, context awareness, and the consistency and diversity of generation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and speech processing technology, and in particular to an interactive 3D head generation method and apparatus based on interleaved multimodal context. Background Technology

[0002] With the rapid development of artificial intelligence and human-computer interaction technologies, virtual humans and intelligent robots have been widely used in many fields, especially in virtual dialogue systems, intelligent customer service, immersive virtual reality, and digital entertainment. To improve user experience and the naturalness of interaction, generating high-quality 3D virtual human head animations has become one of the key technologies.

[0003] Traditional 3D head generation methods are typically based on static expressions or preset animation templates. These methods cannot adequately adapt to complex human-computer dialogue and emotional interaction scenarios, and the generated head movements often lack naturalness and personalization. In dynamic dialogue scenarios, the movement of the 3D head needs to be closely integrated with the speech content, emotional changes, and dialogue context to provide a more natural and realistic user interaction experience. However, existing technologies have shortcomings in the following aspects: Existing 3D head generation models typically treat speech and head movements as independent modules, failing to fully consider the interaction between them. Because this correlation is ignored, the generated 3D head movements often lack synchronization and coordination with the speech content.

[0004] Most existing head generation methods fail to adequately consider contextual information in multi-turn dialogues. During long dialogues, head movements need to be dynamically adjusted based on the user's interaction history with the interactive entity, and traditional methods often struggle to model this context-dependent behavior.

[0005] Many existing methods fail to effectively guarantee the continuity and stability of head movements when generating them, resulting in abrupt or unnatural jumps in head movements. This problem is particularly noticeable in scenarios with significant changes in emotion or tone of voice.

[0006] Some existing methods attempt to generate 3D head motion by combining speech and head movement information, but they often use simple feature fusion methods and fail to effectively handle the time alignment and information interaction between different modalities, resulting in insufficient coordination between the generated head movements and speech content.

[0007] Therefore, achieving natural synchronization between speech and head movements, effective integration of contextual information, and coherence of the generated results during the generation of 3D head animations has become a significant challenge in current technological development. In view of this, this invention proposes an interactive 3D head generation method and apparatus based on interleaved multimodal context. Summary of the Invention

[0008] The purpose of this invention is to address the problems in the prior art, such as decoupling of speech and 3D head motion, lack of context awareness, non-coherent generation results, and insufficient fusion of multimodal information, by proposing an interactive 3D head generation method and apparatus based on interleaved multimodal context.

[0009] The technical solution of this invention: an interactive 3D head generation method based on interleaved multimodal context, comprising the following steps: Step 1: Extract the speech signal features of the user and the interactive object, and map the speech signal to the same time frame as the 3D head motion signal through time alignment; Step 2: Perform feature space transformation on the 3D head motion parameters of the user and the interactive object so that the transformed head motion features and speech features are in the same feature space, thus completing the adaptation and fusion of the two. Step 3: The voice features and 3D head motion features of the user and the interactive entity are divided into multiple fixed-length segments in chronological order and interleaved into multiple multimodal context segments; Step 4: Input the interleaved multimodal context fragments into the multilayer transformer encoder, process them, and generate a fused feature representation that contains the correlation information between speech and head movement; Step 5: Predict the masked parts in the head features of the interactive entity using diffusion modeling methods, and generate a 3D dynamic head sequence that conforms to the speech and interaction context. Step Six: During the sampling phase, the generated results are adjusted through a guidance mechanism to control the balance between speech consistency and generation diversity of 3D head features.

[0010] Optionally, the speech signal feature extraction in step one is completed by a pre-trained speech model, and the time frame rate of the speech signal after alignment with the head motion signal is consistent.

[0011] Optionally, the head motion parameters in step two can be feature-mapped using a multilayer perceptron-based encoder.

[0012] Optionally, the segment described in step three includes four modal features: user speech, interactive speech, user head movement, and interactive head movement. The fixed-length segment segment is adapted to the frame rate of the 3D head motion, ensuring that the segment time granularity matches the interaction rounds.

[0013] Optionally, the multi-layer transformer encoder in step four uses a causal attention mechanism to perform bidirectional feature alignment within the same round, and only allows historical information to be passed to the current round between different rounds.

[0014] Optionally, in step five, the diffusion modeling method uses a conditional denoising network to reconstruct the masked head features and generate three-dimensional head parameters that conform to the speech semantics and interaction context. The conditional denoising network uses real head parameters as a benchmark. The real head parameters are added with forward noise to construct training samples and learn the noise removal rules.

[0015] Optionally, the guidance mechanism in step six replaces the user token with a forged token based on a fixed probability and adjusts the generated 3D head features during the sampling process.

[0016] The present invention also proposes an interactive 3D head generation device based on interleaved multimodal context. The interactive 3D head generation device adopts the above-mentioned interactive 3D head generation method and includes a speech coding unit, a head coding unit, a multimodal fusion unit, a diffusion reconstruction unit, and a sampling control unit. The generation device is used to complete the joint modeling of speech and 3D head motion, feature fusion, and generation and optimization of head actions.

[0017] Optionally, the speech coding unit is used to extract and align the speech features of the user and the interactive entity; The head encoding unit is used to map 3D head parameters to a unified feature space; The multimodal fusion unit is used to perform the interleaving and processing of multimodal contexts to generate fused features; The diffusion reconstruction unit is used to predict and reconstruct the masked 3D head parameters based on the fusion features; The sampling control unit is used to perform guided sampling to adjust the diversity and consistency of the generated results.

[0018] In summary, this application includes at least one of the following beneficial technical effects: This invention transforms the 3D head motion parameters of the user and the interactive object into a feature space, so that the head motion features and speech features are in the same feature space. This breaks through the technical barrier of the mismatch between the dimensions of speech and head motion features and their independent processing in traditional methods, laying the foundation for the deep integration of the two and effectively solving the problem of insufficient coordination between head movements and speech content. The generated 3D head dynamics are highly synchronized with the speech semantics. Furthermore, based on the speech and head motion features in a unified feature space, the features are segmented and interleaved in time sequence into multimodal context segments. Combined with the processing of a multi-layer transformer encoder with causal attention mechanism, it not only achieves bidirectional feature alignment of speech and head motion within the same round, but also preserves the historical information transmission path between multiple rounds of dialogue. This enables the model to fully capture the contextual association of human-computer interaction, and the generated head dynamics can be dynamically adjusted with the progress of the dialogue, avoiding the problem of abrupt actions caused by lack of context awareness. Furthermore, the head features of the masked interactive entity are reconstructed by using a diffusion modeling method with a conditional denoising network. The model is optimized based on the real head parameters, ensuring the coherence and naturalness of the three-dimensional head dynamic sequence. At the same time, a guidance mechanism is introduced in the sampling stage. The token replacement strategy balances speech consistency and generation diversity, which avoids the homogenization of the generated results and ensures the matching degree between head movements and speech semantics, thereby improving the realism and personalization of virtual human interaction. Furthermore, this invention replaces the complex and inefficient feature fusion methods in traditional approaches with an integrated design of unified feature space transformation and interleaved context construction. It can achieve deep fusion of speech and head motion features without the need for additional feature adaptation modules. At the same time, the modular design of each step makes the method easy to deploy and adaptable to various human-computer interaction scenarios such as virtual dialogue and digital entertainment, and has strong generalization ability and practical value. In summary, this invention achieves precise adaptation and fusion by unifying the speech and head motion feature space, and improves the synergy, context awareness, and generation coherence and diversity of head dynamics and speech by combining interleaved multimodal context modeling and diffusion-guided generation. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the training phase structure according to an embodiment of the present invention; Figure 2 This is a flowchart of the reasoning stage in an embodiment of the present invention; Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] The components of the embodiments of the invention described and shown in the accompanying drawings can typically be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. Example

[0022] Combination Figure 1 and Figure 2 As shown, the interactive 3D head generation method based on interleaved multimodal context proposed in this invention includes the following steps: Step 1: Extract the speech signal features of the user and the interactive entity. Speech signal feature extraction is completed through a pre-trained speech model, and the speech signal is mapped to the same time frame as the 3D head motion signal through time alignment; ensure that the time frame rate of the speech signal and the head motion signal are consistent after alignment. Step 2: Perform feature space transformation on the 3D head motion parameters of the user and the interactive object so that the transformed head motion features and speech features are in the same feature space, thus completing the adaptation and fusion of the two; the head motion parameters are feature mapped through an encoder based on a multilayer perceptron. Step 3: The voice features and 3D head motion features of the user and the interactive entity are divided into multiple fixed-length segments in chronological order and interleaved into multiple multimodal context segments; The segments contain four modal features: user speech, interactive speech, user head movement, and interactive head movement. The fixed-length segment segment length is adapted to the frame rate of the three-dimensional head movement to ensure that the segment temporal granularity matches the interaction rounds. Step 4: Input the interleaved multimodal context segments into the multilayer transformer encoder. The multilayer transformer encoder uses a causal attention mechanism to perform bidirectional feature alignment within the same round. Only historical information is allowed to be passed to the current round between different rounds. The multilayer transformer encoder processes the interleaved multimodal context segments and generates a fused feature representation, which contains the correlation information between speech and head movement. Step 5: Predict the masked part of the interactive entity's head features using the diffusion modeling method to generate a 3D head dynamic sequence that conforms to the speech and interaction context; The diffusion modeling method uses a conditional denoising network to reconstruct the masked head features and generate 3D head parameters that conform to the speech semantics and interaction context. The conditional denoising network uses real head parameters as a benchmark. The real head parameters are added with forward noise to construct training samples and learn the noise removal rules. Step Six: During the sampling phase, the generated results are adjusted through a guidance mechanism to control the balance between speech consistency and generation diversity of the 3D head features. The guidance mechanism replaces the user token with a forged token based on a fixed probability and adjusts the generated 3D head features during the sampling process.

[0023] This embodiment also proposes an interactive 3D head generation device based on interleaved multimodal context. The generation device includes a speech coding unit, a head coding unit, a multimodal fusion unit, a diffusion reconstruction unit, and a sampling control unit. The generation device is used to complete the joint modeling of speech and 3D head motion, feature fusion, and generation and optimization of head actions. The speech coding unit is used to extract and align the speech features of the user and the interactive entity; the head coding unit is used to map the 3D head parameters to a unified feature space; the multimodal fusion unit is used to perform the interleaving and processing of multimodal contexts to generate fused features; the diffusion reconstruction unit is used to predict and reconstruct the masked 3D head parameters based on the fused features; and the sampling control unit is used to perform guided sampling to adjust the diversity and consistency of the generated results.

[0024] This embodiment details the complete implementation process of an interactive 3D head generation method based on interleaved multimodal context, including the training and inference phases. The specific process is as follows: Training phase (e.g.) Figure 1 (as shown) Data and parameter settings: Set the duration of the dialogue segment to be... (seconds), speech sampling rate (Hz), head motion frame rate (Frames / Second), the 3D head parameter dimension is ; Get user voice Interactive voice User header parameters Interactive header parameters .

[0025] Speech coding: using pre-trained speech models Extract speech features and interpolate them to the frame rate over time. And projected onto dimension From the unified token space, we obtain: in, For linear or multilayer perceptron projection. Indicates resampling to Let the user and the interactive entity's voice token sequences be respectively denoted as , Head motion coding: encoding head parameters Input Head Encoder Obtain a token from the same dimension: Let the user and interaction entity header token sequences be respectively denoted as , ; This represents the encoded characteristics of the header parameters.

[0026] Interleaved context construction: based on fixed duration (seconds) divided into The segment, the first Each segment is defined as:

[0027]

[0028]

[0029] in, This represents a slice of the user's language token sequence, specifically the language tokens from index (i+1)cfh to icfh. Constructing round-level context sequences using interleaved functions:

[0030] Where ( ); in, and This represents the above sequence segment; Represents an interleaved sequence of language and head; For the first A set of multimodal tokens for a round.

[0031] Causal multimodal fusion: Input encoding network with round-by-round causal attention Add position embedding The fusion features were then obtained: in, For the first The wheel fusion feature has the following causal constraints: attention within the same wheel is unrestricted; attention across wheels is only allowed to historical wheels.

[0032] Diffusion Reconstruction Training: Randomly occlude a portion of the interactive entity's head token; the occluded token is denoted as... Its features after fusing context are Prediction is performed using a conditional denoising network. and with Prediction format optimization: in, These are actual head parameters. This is the signal after adding forward noise to the actual head parameters. For frame-level position embedding, This is a denoising prediction network.

[0033] Classifier free-guided training: using probability The user-side token is replaced with a pseudo-token to form an unconditional branch for joint training, thereby obtaining an adjustable conditional dependency strength.

[0034] Reasoning stage (e.g.) Figure 2 As shown): Round input construction: in the first round The system retrieves user voice, interactive voice, and user head parameters, and then performs the same encoding process as steps two and three to obtain... , , ; Set the position of the interactive entity's header token to a masking sequence ( Fill in the current round's token set. Context caching and input sequence: retaining the previous... Wheel token set Construct the input sequence The head position of the interaction object in the historical round in the cache remains occluded and is not replaced with the predicted value. Feature extraction fusion: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Input fusion network to obtain And extract the features corresponding to the occlusion location. .

[0035] Conditional denoising generation: Input the diffusion module and press the diffusion step. Iteratively generate the header parameters of the current round of interaction:

[0036] Guided sampling adjustment: using unconditional features With guidance coefficient Perform linear interpolation prediction: Get the output of the current round. Then, it proceeds to the next round of processing; among which, This refers to the unconditional fusion feature obtained after replacing the user-side token with a pseudo-token. Used to adjust the strength and diversity of context dependencies; guided sampling adjustment is an optional step.

[0037] The above specific embodiments are merely optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. An interactive 3D head generation method based on interleaved multimodal context, characterized in that, Includes the following steps: Step 1: Extract the speech signal features of the user and the interactive object, and map the speech signal to the same time frame as the 3D head motion signal through time alignment; Step 2: Perform feature space transformation on the 3D head motion parameters of the user and the interactive object so that the transformed head motion features and speech features are in the same feature space, thus completing the adaptation and fusion of the two. Step 3: The voice features and 3D head motion features of the user and the interactive entity are divided into multiple fixed-length segments in chronological order and interleaved into multiple multimodal context segments; Step 4: Input the interleaved multimodal context fragments into the multilayer transformer encoder, process them, and generate a fused feature representation that contains the correlation information between speech and head movement; Step 5: Predict the masked parts in the head features of the interactive entity using diffusion modeling methods, and generate a 3D dynamic head sequence that conforms to the speech and interaction context. Step Six: During the sampling phase, the generated results are adjusted through a guidance mechanism to control the balance between speech consistency and generation diversity of 3D head features.

2. The interactive 3D head generation method based on interleaved multimodal context according to claim 1, characterized in that, The speech signal feature extraction in step one is completed by a pre-trained speech model, and the time frame rate of the speech signal after alignment with the head motion signal is consistent.

3. The interactive 3D head generation method based on interleaved multimodal context according to claim 1, characterized in that, In step two, the head motion parameters are feature-mapped using a multilayer perceptron-based encoder.

4. The interactive 3D head generation method based on interleaved multimodal context according to claim 1, characterized in that, The segment mentioned in step three includes four modal features: user speech, interactive speech, user head movement, and interactive head movement. The fixed-length segment segment is adapted to the frame rate of the 3D head motion, ensuring that the segment time granularity matches the interaction rounds.

5. The interactive 3D head generation method based on interleaved multimodal context according to claim 1, characterized in that, The multi-layer transformer encoder in step four uses a causal attention mechanism to perform bidirectional feature alignment within the same round, and only allows historical information to be passed to the current round between different rounds.

6. The interactive 3D head generation method based on interleaved multimodal context according to claim 1, characterized in that, In step five, the diffusion modeling method uses a conditional denoising network to reconstruct the masked head features and generate three-dimensional head parameters that conform to the speech semantics and interaction context. The conditional denoising network uses real head parameters as a benchmark. The real head parameters are added with forward noise to construct training samples and learn the noise removal rules.

7. The interactive 3D head generation method based on interleaved multimodal context according to claim 1, characterized in that, The guidance mechanism in step six replaces the user token with a forged token with a fixed probability and adjusts the generated 3D head features during the sampling process.

8. An interactive 3D head generation device based on interleaved multimodal context, employing the interactive 3D head generation method as described in any one of claims 1-7, characterized in that, The interactive 3D head generation device includes a speech coding unit, a head coding unit, a multimodal fusion unit, a diffusion reconstruction unit, and a sampling control unit, which are used to complete the joint modeling of speech and 3D head motion, feature fusion, and generation and optimization of head movements.

9. The interactive 3D head generation device based on interleaved multimodal context according to claim 8, characterized in that, The speech coding unit is used to extract and align the speech features of the user and the interactive entity; The head encoding unit is used to map 3D head parameters to a unified feature space; The multimodal fusion unit is used to perform the interleaving and processing of multimodal contexts to generate fused features; The diffusion reconstruction unit is used to predict and reconstruct the masked 3D head parameters based on the fusion features; The sampling control unit is used to perform guided sampling to adjust the diversity and consistency of the generated results.

Images