Multi-modal asynchronous fusion mouth action generation method, device and system based on space-time attention
By employing a multimodal asynchronous fusion method based on spatiotemporal attention, the problems of lighting variations and environmental robustness in the generation of virtual avatar mouth movements are solved, achieving high-precision synchronization and adaptive alignment, generating natural and accurate mouth animations suitable for mobile VR devices and AR glasses.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-27
- Publication Date
- 2026-04-10
AI Technical Summary
Existing technologies suffer from problems when generating mouth movements in virtual characters, such as changes in lighting, facial occlusion, motion blur, poor environmental robustness, inability to perceive tongue position, and slight tooth exposure. These issues result in insufficient pronunciation accuracy and emotional expression. Furthermore, audiovisual fusion methods cannot achieve precise cross-modal alignment, leading to unnatural phenomena such as animation jitter and jumping.
A multimodal asynchronous fusion method based on spatiotemporal attention is adopted. Facial image and audio features are acquired and encoded using a spatiotemporal attention encoder. Cross-modal asynchronous fusion is then performed to generate mouth movements. A learnable delay compensation mechanism and adversarial training strategy are introduced to achieve high-precision synchronization and adaptive alignment.
It achieves millisecond-level high-precision synchronization in changing environments, generates natural and accurate mouth animations, has strong generalization ability and robustness, adapts to complex real-world scenarios, reduces computational complexity, and is suitable for mobile VR devices and AR glasses.
Smart Images
Figure CN121837464A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of electric digital data processing, and particularly relates to a multi-modal asynchronous fusion mouth movement generation method, device and system based on space-time attention. BACKGROUND
[0002] In virtual reality socialization, metaverse and online education and other frontier applications, constructing a real and natural virtual image is the core of creating an immersive user experience, among which high-fidelity, low-latency mouth movement synchronous reproduction plays a crucial role in accurately conveying speech content and expressing rich emotions and emotions. However, the existing technology still faces a series of key technical challenges in achieving this goal.
[0003] At present, the mainstream technical solution mainly relies on pure visual analysis, pure audio driving or simple audio-visual fusion method, but all have significant limitations.
[0004] The pure visual analysis method captures the user's mouth image sequence through the camera and drives the virtual image using computer vision algorithms, which has the limitation of being severely affected by light changes, face occlusion and motion blur interference, unstable feature extraction, poor environmental robustness, and due to the inability to perceive the internal oral state such as tongue position, teeth exposure, etc. which is crucial for pronunciation accuracy, resulting in incorrect mouth shape generation for specific phonemes, and when the user's head moves significantly or quickly, the tracking performance of single-view vision also decreases sharply.
[0005] The pure audio driving method only takes the speech signal as input, and generates the mouth shape through the mapping model of acoustic features to action parameters, which leads to the fact that similar acoustic phonemes may correspond to different mouth shapes, resulting in blurred and averaged generated animation, lacking accurate details, and when the audio is noisy or unclear, the system cannot correct it with visual information, and it is easy to produce error output, and it is difficult to generate mouth expressions unrelated to speech, limiting emotional expression.
[0006] For simple audiovisual fusion methods, existing technologies typically employ early feature stitching or post-processing using basic recurrent neural networks. However, these methods fail to deeply model the complex relationships between modalities. For instance, due to differences in sensor hardware sampling, data transmission, and processing pipelines, there is a non-fixed time delay between visual and audio frames, usually ranging from tens to hundreds of milliseconds. Using fixed delay compensation or simple synchronization assumptions cannot adapt to the dynamic changes in real-world scenes, resulting in perceptible misalignment between generated lip movements and speech. Furthermore, due to the non-linear, context-dependent, and complex mapping relationship between phonemes and visual pixels, existing methods cannot achieve fine, adaptive cross-modal alignment, limiting the fusion effect. In addition, existing solutions are mostly limited to short-term window analysis and lack effective modeling of long-range dependencies. Mouth movements have strong spatiotemporal coherence and dynamism, and the current frame state is highly dependent on historical motion trajectories, leading to unnatural phenomena such as jitter and jumps in the generated animation. Summary of the Invention
[0007] This invention addresses the problems existing in the prior art and provides a method, device, and system for generating mouth movements based on spatiotemporal attention and multimodal asynchronous fusion. It effectively handles the asynchronicity of audiovisual signals, fully models the spatiotemporal dependence of mouth movements, and achieves accurate cross-modal alignment, thereby generating more natural and accurate virtual image mouth animations.
[0008] The technical solution adopted in this invention is a multimodal asynchronous fusion mouth action generation method based on spatiotemporal attention, which obtains facial image sequences and audio feature sequences;
[0009] Dynamic visual features are obtained based on the facial image sequence;
[0010] The dynamic visual features are encoded using a spatiotemporal attention encoder to obtain visual context features, and the audio feature sequence is encoded to obtain audio context features.
[0011] The visual context features and the audio context features are asynchronously fused across modalities to obtain fused features;
[0012] Mouth movements are generated based on the fused features.
[0013] Preferably, key points of the mouth are detected based on the facial image sequence, and geometric features of the mouth are extracted; the ROI of the mouth is obtained based on the key points of the mouth, and the appearance features of the mouth are extracted through a three-dimensional convolutional neural network; dynamic visual features are obtained based on the geometric features and appearance features of the mouth.
[0014] Preferably, the mouth geometry features include the vertical opening of the lips, the horizontal width of the lips, the asymmetry of the corners of the mouth, and the protrusion of the lips.
[0015] Preferably, the spatiotemporal attention encoder is an encoder based on a multi-head self-attention mechanism, which adds learnable positional codes to the input dynamic visual feature sequence.
[0016] Preferably, a mask matrix is introduced during the calculation of the attention mechanism to constrain the attention weights between different spatiotemporal locations.
[0017] Preferably, the visual feature sequence and the audio feature sequence are time-aligned with a time offset, and bidirectional attention features are calculated based on the aligned two features. The aligned visual context features and bidirectional attention features are then weighted and fused to complete cross-modal asynchronous fusion.
[0018] Preferably, the time offset is a trainable parameter whose value is optimized during training using gradient descent.
[0019] Preferably, the generation process of mouth movements is optimized using an adversarial training strategy during the training phase.
[0020] A multimodal asynchronous fusion mouth action generation device based on spatiotemporal attention includes:
[0021] The data acquisition unit is used for facial image sequences and audio feature sequences;
[0022] A feature extraction unit is used to obtain dynamic visual features based on facial image sequences;
[0023] The encoding module is used to encode the dynamic visual features using a spatiotemporal attention encoder to obtain visual context features, and to encode the audio feature sequence to obtain audio context features.
[0024] A cross-modal asynchronous fusion unit is used to perform feature fusion on asynchronous visual context features and audio context features;
[0025] The mouth motion generation unit generates mouth motions based on fused features.
[0026] A multimodal asynchronous fusion mouth motion generation system based on spatiotemporal attention includes a processor and a memory:
[0027] The memory is used to store program code and transmit the program code to the processor;
[0028] The processor is used to execute the spatiotemporal attention-based multimodal asynchronous fusion mouth action generation method according to the instructions in the program code.
[0029] This invention relates to a method, apparatus, and system for generating mouth movements based on spatiotemporal attention and multimodal asynchronous fusion. The method involves acquiring a facial image sequence and an audio feature sequence; acquiring dynamic visual features based on the facial image sequence; encoding the dynamic visual features using a spatiotemporal attention encoder to obtain visual context features, and encoding the audio feature sequence to obtain audio context features; performing cross-modal asynchronous fusion of the visual context features and the audio context features to obtain fused features; generating mouth movements based on the fused features; and implementing the method using an apparatus and system.
[0030] The beneficial effects of this invention are as follows:
[0031] (1) By introducing a learnable delay compensation mechanism and cross-modal attention alignment, the time offset between audio and visual streams can be dynamically and adaptively calibrated instead of relying on fixed compensation values, thereby achieving high-precision synchronization at the millisecond level in variable environments, fundamentally eliminating sensory lag, and effectively solving the inherent asynchronous problem of audiovisual signals.
[0032] (2) By using a Transformer-based spatiotemporal attention encoder, the long-range dependence of mouth movements in time and space is explicitly modeled, which can accurately grasp complex dynamic effects such as co-pronunciation. The generated action sequence is smooth and continuous in time and accurate in space, effectively avoiding the jitter, jump and blur problems of animation.
[0033] (3) The bidirectional cross-modal attention fusion module automatically calculates and allocates the contribution weights of visual and audio features at different times and in different dimensions based on the specific phoneme content and visual context. This achieves non-linear and context-dependent fine alignment between phonemes and visuals, realizes fine cross-modal adaptive fusion, improves feature complementarity, and makes the fused feature information more comprehensive and discriminative.
[0034] (4) Introducing adversarial training strategies effectively improves the model’s generalization ability to unfamiliar users, different accents, noisy environments and partial occlusion, and has strong generalization ability and robustness, adapting to complex real-world scenarios.
[0035] (5) While ensuring the quality of generation, it greatly reduces the computational complexity and memory usage, and can be deployed in mobile VR devices, AR glasses and embedded systems with limited computing power to achieve high frame rate real-time mouth motion capture and driving, and has high practical potential. Attached Figure Description
[0036] Figure 1 This is a flowchart of the method of the present invention;
[0037] Figure 2 This is a schematic block diagram of the device structure of the present invention. Detailed Implementation
[0038] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.
[0039] This invention relates to a multimodal asynchronous fusion mouth motion generation method based on spatiotemporal attention, the method comprising the following steps:
[0040] (1) Obtain facial image sequences and audio feature sequences;
[0041] (2) Obtain dynamic visual features based on the facial image sequence;
[0042] (3) The dynamic visual features are encoded using a spatiotemporal attention encoder to obtain visual context features, and the audio feature sequence is encoded to obtain audio context features;
[0043] (4) Perform cross-modal asynchronous fusion of the visual context features and the audio context features to obtain fused features;
[0044] (5) Generate mouth movements based on the fused features.
[0045] The method will be described below with reference to specific embodiments.
[0046] (1) Obtain facial image sequences and audio feature sequences;
[0047] (2) Obtain dynamic visual features based on the facial image sequence;
[0048] Based on the facial image sequence, key points of the mouth are detected, and geometric features of the mouth are extracted; based on the key points of the mouth, the ROI of the mouth is obtained, and the appearance features of the mouth are extracted through a three-dimensional convolutional neural network; dynamic visual features are obtained based on the geometric features and appearance features of the mouth.
[0049] The geometric features of the mouth include the vertical opening of the lips, the horizontal width of the lips, the asymmetry of the corners of the mouth, and the protrusion of the lips.
[0050] In this embodiment, unlike the traditional approach of mouth feature extraction which relies on either geometric or visual features, the two are combined and introduced into three-dimensional convolution to capture short-term spatiotemporal information.
[0051] Specifically, given a sequence of facial images of length T (frames) First, a lightweight CNN is used to detect N mouth key points in each frame. In this embodiment... ,get,
[0052]
[0053] in, Refers to the set of coordinates of all N keypoints in frame t. These are the trainable parameters for the keypoint detector;
[0054] Geometric features calculated based on key points Including the vertical opening of the lips Horizontal width of lips Asymmetry of the corners of the mouth and the degree of lip protrusion ,satisfy,
[0055]
[0056]
[0057]
[0058]
[0059] in, , , , , From A specific point obtained by indexing or calculation. and The vertical coordinates of the left and right corners of the mouth;
[0060] The vertical opening of the lips is calculated by the Euclidean distance between the midpoint of the upper lip and the midpoint of the lower lip; the horizontal width of the lips is calculated by the Euclidean distance between the left and right corners of the mouth; the asymmetry of the corners of the mouth is calculated by the vertical coordinate difference between the left and right corners of the mouth; and the lip protrusion is calculated by the average distance from all key points to the center point of the lips, reflecting the degree of pouting.
[0061] Simultaneously, the mouth area region of interest (ROI) is cropped centered on key points, and its appearance features are extracted using 3D-CNN.
[0062]
[0063] in, Centered on frame t, the frames before and after it are... A sequence of images showing the mouth ROI in the frame. For the parameters of 3D-CNN, It includes local spatiotemporal texture variation features.
[0064] Final visual features This is the concatenation of the two, and serves as the input for subsequent encoding.
[0065] v t =[ g t ; e t ]
[0066] (3) The dynamic visual features are encoded using a spatiotemporal attention encoder to obtain visual context features, and the audio feature sequence is encoded to obtain audio context features;
[0067] The spatiotemporal attention encoder is an encoder based on a multi-head self-attention mechanism, which adds learnable positional codes to the input dynamic visual feature sequence.
[0068] To model the long-range spatiotemporal dependence of mouth movements, this invention uses a Transformer-based spatiotemporal attention encoder to encode dynamic visual features.
[0069] For visual feature sequences ( The sum of the dimensions of the mouth's geometric features and its visual features is first added with learnable positional encoding. To inject sequence information,
[0070]
[0071] Then, through a multi-head self-attention mechanism, it is calculated that for the i-th head,
[0072]
[0073] in, , , and Let be a trainable matrix, where the attention is calculated as follows:
[0074]
[0075] The outputs of all heads are concatenated and projected again to obtain the encoded visual context features, i.e. , here For the total number of attention heads, This is for outputting the projection matrix.
[0076] As can be seen, a mask matrix M is introduced in the calculation process of the attention mechanism to constrain the attention weights between different spatiotemporal locations. Specifically, M is a spatiotemporal mask matrix used to control the visibility between different spatiotemporal locations, ensuring that the model can learn meaningful physical constraints. For example, it can prevent future frames from paying attention to past frames (causal mask), or constrain excessive interaction between non-adjacent areas in space based on the physical structure of the mouth. This enables the model to learn dependencies that conform to physical laws.
[0077] (4) Perform cross-modal asynchronous fusion of the visual context features and the audio context features to obtain fused features;
[0078] The visual feature sequence and the audio feature sequence are time-aligned with a time offset. Based on the two aligned features, bidirectional attention features are calculated. The aligned visual context features and bidirectional attention features are then weighted and fused to complete cross-modal asynchronous fusion.
[0079] The time offset is a trainable parameter whose value is optimized during training using gradient descent.
[0080] During the training phase, an adversarial training strategy was used to optimize the generation process of mouth movements.
[0081] In this embodiment, it is actually a cross-modal attention mechanism based on learnable delay, where learnable delay compensation is defined as a trainable parameter. This indicates a systematic delay in audio relative to visual information; in the early stages of training, It is initialized to an empirical value, such as 3 frames, and then automatically optimized using gradient descent;
[0082] For visual context features and audio context features (Generated by the audio encoder), first time alignment is performed.
[0083] H v ' = H v [1:T - delta H a ' = H a [1 + delta: T]
[0084] That is, for Extracting visual features Frame, to Then extract the audio features. frame, and Aligned in the time dimension and all of length . sequence, The t-th frame and The t-th frame in the sequence is time-aligned, corresponding to the t-th frame and (t+δ)-th frame in the original sequence, respectively.
[0085] Then calculate the bidirectional attention features:
[0086] Visual-to-audio attention (visual as query, aligned audio as key and value) is satisfied.
[0087]
[0088]
[0089]
[0090] Here This represents the complementary information retrieved from audio features by visual features; similarly, the attention from audio to visual is calculated. Ultimately, the two attentional information sets are fused through a gating mechanism while preserving the original visual context features.
[0091]
[0092] in, This is a learnable gating parameter, with a value between 0 and 1, used to control the balance between the two types of attentional contributions. The residual connections help maintain the integrity of the original visual information and stabilize training.
[0093] To further improve the naturalness of the generated actions, a GAN-based training strategy is introduced:
[0094] generator The loss function is:
[0095] L G = lambda rec ‖ G(x) - y ‖ 1 + lambda adv E[ - D(G(x))
[0096] Including reconstruction losses and combat losses E[ - D(G(x)) , The fusion features are the input. Targeting real mouth movements and The weighting coefficients are used to balance the losses.
[0097] Discriminator We employ Wasserstein GAN (WGAN-GP) loss with gradient penalty, the loss function being:
[0098] L D =E[ D(G(x)) - E[ D(y) + lambda gp D(G(x)) E[( ‖ ∇ x D(y) D( x Figure 1 ) ‖ 2 - 1 ) 2 ]
[0099] in, E[ Figure 1 - E[ Figure 1 , It is a random interpolation of real data and generated data. This is a gradient penalty term used to force the decision maker to... The gradient norm at a certain point is close to 1, indicating a stable training process. It is the gradient penalty coefficient.
[0100] (5) Generate mouth movements based on the fused features.
[0101] The present invention also relates to a multimodal asynchronous fusion mouth action generation device based on spatiotemporal attention, comprising:
[0102] The data acquisition unit is used for facial image sequences and audio feature sequences;
[0103] A feature extraction unit is used to obtain dynamic visual features based on facial image sequences;
[0104] The encoding module is used to encode the dynamic visual features using a spatiotemporal attention encoder to obtain visual context features, and to encode the audio feature sequence to obtain audio context features.
[0105] A cross-modal asynchronous fusion unit is used to perform feature fusion on asynchronous visual context features and audio context features;
[0106] The mouth motion generation unit generates mouth motions based on fused features.
[0107] This invention also relates to a multimodal asynchronous fusion mouth action generation system based on spatiotemporal attention, including a processor and a memory:
[0108] The memory is used to store program code and transmit the program code to the processor;
[0109] The processor is used to execute the spatiotemporal attention-based multimodal asynchronous fusion mouth action generation method according to the instructions in the program code.
[0110] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0111] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes The function specified in one or more boxes.
[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. One or more processes and / or boxes The steps of the function specified in one or more boxes.
[0114] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0115] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A multimodal asynchronous fusion mouth action generation method based on spatiotemporal attention, characterized in that: Obtain facial image sequences and audio feature sequences; Dynamic visual features are obtained based on the facial image sequence; The dynamic visual features are encoded using a spatiotemporal attention encoder to obtain visual context features, and the audio feature sequence is encoded to obtain audio context features. The visual context features and the audio context features are asynchronously fused across modalities to obtain fused features; Mouth movements are generated based on the fused features.
2. The method for generating mouth movements based on spatiotemporal attention using multimodal asynchronous fusion according to claim 1, characterized in that: Based on the facial image sequence, key points of the mouth are detected, and geometric features of the mouth are extracted; Based on the key points of the mouth, the ROI of the mouth is obtained, and the appearance features of the mouth are extracted by a three-dimensional convolutional neural network. Dynamic visual features are obtained based on the geometric and visual features of the mouth.
3. The method for generating mouth movements based on spatiotemporal attention using multimodal asynchronous fusion according to claim 2, characterized in that: The geometric features of the mouth include the vertical opening of the lips, the horizontal width of the lips, the asymmetry of the corners of the mouth, and the protrusion of the lips.
4. The method for generating mouth movements based on spatiotemporal attention using multimodal asynchronous fusion according to claim 1, characterized in that: The spatiotemporal attention encoder is an encoder based on a multi-head self-attention mechanism, which adds learnable positional codes to the input dynamic visual feature sequence.
5. The method for generating mouth movements based on spatiotemporal attention using multimodal asynchronous fusion according to claim 4, characterized in that: The attention mechanism incorporates a mask matrix during computation to constrain attention weights across different spatiotemporal locations.
6. The method for generating mouth movements based on spatiotemporal attention using multimodal asynchronous fusion according to claim 1, characterized in that: The visual feature sequence and the audio feature sequence are time-aligned with a time offset. Based on the two aligned features, bidirectional attention features are calculated. The aligned visual context features and bidirectional attention features are then weighted and fused to complete cross-modal asynchronous fusion.
7. The method for generating mouth movements based on spatiotemporal attention using multimodal asynchronous fusion according to claim 6, characterized in that: The time offset is a trainable parameter whose value is optimized during training using gradient descent.
8. The method for generating mouth movements based on spatiotemporal attention using multimodal asynchronous fusion according to claim 1, characterized in that: During the training phase, an adversarial training strategy was used to optimize the generation process of mouth movements.
9. A multimodal asynchronous fusion mouth action generation device based on spatiotemporal attention, characterized in that: include: The data acquisition unit is used for facial image sequences and audio feature sequences; A feature extraction unit is used to obtain dynamic visual features based on facial image sequences; The encoding module is used to encode the dynamic visual features using a spatiotemporal attention encoder to obtain visual context features, and to encode the audio feature sequence to obtain audio context features. A cross-modal asynchronous fusion unit is used to perform feature fusion on asynchronous visual context features and audio context features; The mouth motion generation unit generates mouth motions based on fused features.
10. A multimodal asynchronous fusion mouth motion generation system based on spatiotemporal attention, characterized in that: Including the processor and memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the multimodal asynchronous fusion mouth action generation method based on spatiotemporal attention as described in any one of claims 1 to 8 according to the instructions in the program code.