Audio global perception based animation generation method and device, equipment and medium

Abstract

This invention relates to the field of artificial intelligence technology, and discloses an animation generation method, apparatus, device, and medium based on global audio perception. The method includes: acquiring driving audio and a reference portrait image, and encoding them respectively to obtain multi-scale audio features and image features; enhancing motion information related to audio temporal sequence in the image features through global audio context modeling to obtain enhanced image features; predicting head motion parameters and facial expression motion parameters based on the multi-scale audio features; synchronously adjusting the amplitude of the head motion parameters and facial expression motion parameters based on the acquired adjustment factors to obtain adjusted head parameters and adjusted facial expression parameters; and generating a portrait animation video based on the enhanced image features and the adjusted parameters. This invention can be applied to fintech and healthcare scenarios, achieving precise driving and natural synchronization of audio with portrait animation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an animation generation method, apparatus, device, and medium based on global audio perception. Background Technology

[0002] Audio-driven portrait animation generation technology aims to achieve realistic animation effects with synchronized audio and video by driving the facial expressions and head movements of virtual avatars through input audio signals. The technical challenge in this field lies in how to extract sufficiently rich motion control information from the audio to support the generation of long-duration, highly natural-looking animations.

[0003] In existing technologies, one approach segments the audio signal into short time segments according to a time window and performs local feature matching between each segment and the corresponding video frame to generate facial movements frame by frame. However, this local matching strategy means that each frame can only perceive the limited adjacent audio context, failing to capture the long-term variations in tone, rhythm, and speech rate inherent in the audio. This results in generated facial expressions and head movements that are monotonous and lack natural dynamic fluctuations. Furthermore, another approach introduces visual aids such as moving frames and overlapping windows to address inter-frame jitter and transition discontinuities, constraining temporal consistency through additional visual information. However, this approach not only introduces interference from non-audio information, compromising the purity of audio-visual synchronization, but also significantly increases the computational burden of model training and inference.

[0004] In long video generation tasks, the aforementioned limitations are further amplified, manifesting as increased jitter between clips and quality issues such as identity feature drift. In scenarios such as remote video banking and online customer service in the fintech field, and remote consultations and mental health counseling in the healthcare field, the stable generation of portrait animation over a long period and the expression of natural facial expressions have a crucial impact on the interactive experience. The aforementioned deficiencies of existing technologies are particularly prominent in these demanding scenarios. Summary of the Invention

[0005] This invention provides an animation generation method, apparatus, device, and medium based on global audio perception, to solve the technical problem of how to achieve precise driving and natural synchronization of audio in portrait animation.

[0006] Firstly, an animation generation method based on global audio awareness is provided, including: In response to acquiring the driving audio and the reference portrait image, the driving audio is encoded to obtain multi-scale audio features, and the reference portrait image is encoded to obtain image features; Based on the multi-scale audio features and the image features, global audio context modeling is used to enhance the motion information related to audio temporal sequence in the image features, resulting in enhanced image features. Based on the multi-scale audio features, head motion parameters and facial expression motion parameters are predicted, wherein the head motion parameters and facial expression motion parameters are used to decouple the facial motion to be generated into head motion and facial expression motion. Based on the obtained adjustment factor, the amplitude of the head motion parameters and the facial expression parameters are adjusted synchronously to obtain the adjusted head parameters and adjusted facial expression parameters. Based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters, a portrait animation video is generated.

[0007] Secondly, an animation generation device based on global audio awareness is provided, comprising: The encoding unit is configured to, in response to acquiring the driving audio and the reference portrait image, encode the driving audio to obtain multi-scale audio features, and encode the reference portrait image to obtain image features; An enhancement unit is used to enhance the motion information related to audio temporality in the image features based on the multi-scale audio features and the image features, through global audio context modeling, to obtain enhanced image features; The decoupling unit is used to predict head motion parameters and facial expression motion parameters based on the multi-scale audio features, wherein the head motion parameters and facial expression motion parameters are used to decouple the facial motion to be generated into head motion and facial expression motion; An adjustment unit is used to simultaneously adjust the amplitude of the head motion parameters and the facial expression parameters based on the acquired adjustment factor, so as to obtain the adjusted head parameters and the adjusted facial expression parameters. The generation unit is used to generate a portrait animation video based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for generating animation based on global audio perception.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method for generating animation based on global audio perception.

[0010] The aforementioned scheme, implemented through a global audio-based animation generation method, apparatus, computer device, and storage medium, deeply integrates multi-scale audio features with image features via global audio context modeling. This enables the model to perceive and drive long-term variations in tone, rhythm, and speech rate within the audio, thereby generating facial movements highly matched to the audio content. During this process, head motion parameters and facial expression motion parameters are predicted independently, allowing for precise control of head movements and facial expression changes, avoiding the problem of singular motion caused by coupling these two aspects in traditional methods. Furthermore, based on enhanced image features and adjusted motion parameters, highly synchronized audio and visual images with natural and coherent motion can be generated for portrait animation videos. The combined effect of these techniques achieves precise driving and natural synchronization of audio with portrait animation. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for an animation generation method based on global audio perception, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating an animation generation method based on global audio perception in one embodiment of the present invention. Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S1; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method of step S1; Figure 5 yes Figure 2 A flowchart illustrating a specific implementation method of step S2; Figure 6 yes Figure 2 A flowchart illustrating a specific implementation method of step S3; Figure 7 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 8 yes Figure 2 A flowchart illustrating a specific implementation of step S5; Figure 9 This is a schematic diagram of an animation generation device based on global audio perception in one embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] This invention provides an animation generation method based on global audio awareness, which can be applied to applications such as... Figure 1In this application environment, the client communicates with the server via a network. The server can receive driving audio and a reference portrait image from the client, encode the driving audio to obtain multi-scale audio features, and encode the reference portrait image to obtain image features. Based on the multi-scale audio features and the image features, global audio context modeling is used to enhance the motion information related to audio temporal sequence in the image features, resulting in enhanced image features. Based on the multi-scale audio features, head motion parameters and facial expression motion parameters are predicted, whereby the head motion parameters and facial expression motion parameters are used to decouple the facial motion to be generated into head motion and facial expression motion. Based on the obtained adjustment factors, the amplitudes of the head motion parameters and facial expression motion parameters are simultaneously adjusted to obtain adjusted head parameters and adjusted facial expression parameters. Based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters, a portrait animation video is generated. The portrait animation video is then fed back to the client. In this invention, for scenarios requiring audio-driven virtual avatar generation, such as virtual anchors, online education, remote consultations, and mental health counseling, a global audio perception and motion decoupling control scheme can be utilized. First, a context-enhanced audio learning module extracts global audio features. Then, a motion decoupling controller and an audio-bucket mapping network achieve independent control and automatic prediction of head and facial movements. Finally, adjustment factors flexibly control the dynamic range of the animation. This effectively avoids the problems of audio-visual asynchrony, monotonous movements, and lack of natural variation found in traditional methods, greatly improving the quality and naturalness of the generated animation and enhancing the interactive experience. Furthermore, the client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, as well as intelligent terminal devices with display and interactive capabilities, such as intelligent companion robots, smart home control panels, and smart TVs. Furthermore, the server can be implemented using a dedicated server or a server cluster composed of multiple servers. The invention will now be described in detail through specific embodiments.

[0015] Please see Figure 2 As shown, Figure 2 A flowchart illustrating an animation generation method based on global audio awareness, provided in an embodiment of the present invention, includes the following steps: S1, in response to acquiring the driving audio and the reference portrait image, the driving audio is encoded to obtain multi-scale audio features, and the reference portrait image is encoded to obtain image features.

[0016] In specific implementation, in response to receiving the driving audio and the reference portrait image sent by the client, the server invokes a pre-trained audio encoder to process the driving audio. The audio encoder employs a lightweight neural network architecture, such as a self-attention mechanism based on a convolutional neural network or a transformer architecture, capable of extracting rich audio features while maintaining low computational overhead. Furthermore, during the encoding process, the audio encoder performs multi-layer convolution or attention calculations on the audio signal, extracting feature representations at different time scales layer by layer. Shallow networks capture short-term acoustic features, such as pitch, energy, and formants; deep networks capture long-term contextual features, such as intonation curves, rhythm patterns, and speech rate variations. These features at different scales collectively constitute the multi-scale audio features.

[0017] Furthermore, the server invokes an image encoder to process the reference portrait image. The image encoder employs a deep convolutional neural network-based architecture, such as a residual network or an efficient network, to perform a series of convolution, pooling, and non-linear activation operations on the input image, mapping the original pixel data into a high-dimensional feature vector or feature map. This feature map contains structural information, texture information, and spatial layout information of the face, i.e., the image features.

[0018] In one alternative implementation, the image encoder may employ a deep convolutional neural network-based architecture, such as a residual network or an efficient network, and may be pre-trained on a large image dataset. Specifically, the final classification layer of the pre-trained network may be removed, and the feature map output from the last feature extraction stage may be used as the image feature. This feature map encapsulates the structural information, geometric layout, and texture details of the face.

[0019] See Figure 3 In some preferred embodiments, encoding the driving audio to obtain multi-scale audio features includes: S21, preprocessing the driving audio to obtain an audio signal sequence; S22, extracting multi-scale features from the audio signal sequence using a pre-trained audio encoder to capture long-term contextual information in the driving audio and obtain multi-scale features, wherein the long-term contextual information includes at least one of tone, rhythm, and speech rate; S23, combining the multi-scale features to obtain multi-scale audio features.

[0020] In specific implementation, the server preprocesses the driving audio to obtain an audio signal sequence. The preprocessing operations may include one or more of the following: noise reduction filtering, using frequency domain or time domain filtering algorithms to remove ambient background noise; silence removal, using energy detection algorithms to remove silent segments at the beginning and end of the audio and long silence intervals between speech segments; resampling, uniformly converting the audio signal sampling rate to a preset target sampling rate, such as 16 kHz; pre-emphasis, boosting high-frequency components to compensate for the attenuation of the speech signal in the high-frequency band; framing and windowing, dividing the continuous audio signal into short time frames and multiplying each frame by a window function to reduce spectral leakage. After the above preprocessing, the original audio is converted into a standardized, structured audio signal sequence.

[0021] Furthermore, the server performs multi-scale feature extraction on the audio signal sequence using a pre-trained audio encoder to capture long-term contextual information in the driving audio, obtaining multi-scale features. The long-term contextual information includes at least one of intonation, rhythm, and speech rate. In an optional implementation, the pre-trained audio encoder can employ a self-supervised learning model based on a convolutional neural network or Transformer architecture, pre-trained on a large-scale speech dataset. It extracts deep acoustic and semantic features at different time scales through multi-layer self-attention computation or convolution operations to construct multi-scale features that capture short-term acoustic details and long-term contextual information. The encoder performs multi-layer convolution or attention computation on the audio signal sequence, extracting features at different time scales layer by layer. Shallow networks capture short-term acoustic features such as pitch, formant frequencies, and short-term energy; mid-layer networks capture syllable-level features; and deep networks capture sentence-level long-term contextual features such as intonation curves, rhythm patterns, speech rate variations, and emotional tendencies. The output of the pre-trained audio encoder is a multi-scale feature sequence, i.e., the multi-scale features.

[0022] Furthermore, the server combines the multi-scale features to obtain multi-scale audio features. The combination method can be one or more of the following: concatenation after alignment along the time dimension, pooling operation, attention-weighted fusion, or a feature pyramid network. The feature vector or feature map obtained after combination is the multi-scale audio feature.

[0023] This embodiment employs a lightweight pre-trained audio encoder to extract multi-scale features from the driving audio, effectively capturing long-term contextual information such as intonation, rhythm, and speech rate. This information is crucial for driving natural and rich facial expressions and head movements in virtual avatars. Compared to existing methods that only use local audio features, the multi-scale audio features provided in this embodiment contain rich information ranging from short-term acoustic details to long-term semantic context, enabling the model to more accurately understand the emotions and intentions behind the speech content, thereby generating more expressive and engaging portrait animations.

[0024] See Figure 4 In some preferred embodiments, encoding the reference portrait image to obtain image features includes: S31, performing facial region detection on the reference portrait image and extracting facial region images; S32, performing spatial feature extraction on the facial region images to obtain the image features containing facial structure information.

[0025] In practice, the server performs facial region detection on the reference portrait image and extracts the facial region image. The server calls a pre-trained face detection algorithm, such as an algorithm based on a multi-task cascaded convolutional network or a single-pass multi-box detection network, to detect all faces in the image and output the bounding rectangle coordinates of each face region. Based on the detected face bounding boxes, the server crops the original image, separating the regions containing the faces from the background to obtain the facial region image.

[0026] Furthermore, the server performs spatial feature extraction on the facial region image to obtain image features containing facial structural information. The server then inputs the facial region image into an image encoder. The image encoder is a pre-trained feature extractor based on a deep convolutional neural network, such as a residual network, mobile network, or high-efficiency network. The image encoder performs a series of convolution, batch normalization, activation function, and pooling operations on the facial region image, mapping the input 3D image matrix into a high-dimensional feature vector or feature map. This feature map is a highly condensed representation of facial structural information, containing the geometric layout of the face, organ positions, contour features, and texture details.

[0027] This embodiment obtains high-quality facial structural features, i.e., image features, by performing facial region detection and spatial feature extraction on a reference portrait image. These features contain key geometric and texture information of the face, providing precise spatial anchors for subsequent audio-driven processing. Furthermore, in audio-driven portrait animation generation, the feature representation of the facial region determines whether the model can accurately understand and align the various parts of the face. This embodiment uses a face detection algorithm to accurately crop the facial region, eliminating background interference, and extracts multi-layer spatial features through a deep convolutional neural network, ensuring that the extracted image features have sufficient discriminative power and information content, guaranteeing the consistency of the character's identity and the accuracy of the facial structure in the generated animation.

[0028] S2, based on the multi-scale audio features and the image features, enhance the motion information related to audio temporal sequence in the image features through global audio context modeling to obtain enhanced image features.

[0029] In specific implementation, the server uses a context-enhanced audio learning module to implement the modeling process. This audio learning module employs a spatiotemporal audio cross-attention mechanism, simultaneously injecting the audio embeddings from the multi-scale audio features into the spatial attention layer and temporal attention layer of the diffusion network. In the spatial attention layer, the audio embedding serves as the query vector, and the image features serve as the key and value vectors, respectively. A spatial attention map is calculated through cross-attention. This spatial attention map guides the model to focus on facial regions within the image features, such as the mouth, eyes, and eyebrows—areas closely related to speech expression—avoiding interference from background noise in the generation process. Furthermore, in the temporal attention layer, the audio embedding also serves as the query vector, and the temporal sequence of image features serves as the key and value vectors, respectively. A temporal attention map is calculated through cross-attention. This temporal attention map uses the temporal audio embedding to guide inter-frame motion alignment, ensuring that the generated animation frames are precisely synchronized with the audio signal in time. Through the aforementioned spatial and temporal audio cross-attention mechanism, the motion information related to the audio temporal sequence in the image features is significantly enhanced, resulting in the enhanced image features.

[0030] See Figure 5In some preferred embodiments, the step of enhancing the motion information related to audio temporal sequence in the image features by global audio context modeling based on the multi-scale audio features and the image features to obtain enhanced image features includes: S41, performing interactive calculation between the audio embedding in the multi-scale audio features and the spatial dimension of the image features to obtain spatial interactive features; S42, performing interactive calculation between the audio embedding in the multi-scale audio features and the temporal dimension of the image features to obtain temporal interactive features; S43, fusing the spatial interactive features and the temporal interactive features to generate the enhanced image features.

[0031] In specific implementation, the server performs interactive calculations between the audio embeddings in the multi-scale audio features and the spatial dimensions of the image features to obtain spatial interactive features. The server implements this interactive calculation through a spatial audio cross-attention layer. First, the audio embeddings are mapped to query vectors, and the image features are mapped to key vectors and value vectors. Then, the dot product of the query vector and the key vector is calculated, and after scaling and softmax normalization, a spatial attention weight matrix is ​​obtained. Each element of this weight matrix represents the degree of attention the audio embeddings pay to a specific spatial location in the image. Further, the spatial attention weight matrix is ​​multiplied by the value vectors to obtain the weighted spatial interactive features. These spatial interactive features represent the degree of attention the audio information pays to different regions in the image, guiding the model to focus its attention on the face region within the image features.

[0032] Furthermore, the server performs interactive calculations between the audio embeddings in the multi-scale audio features and the temporal dimension of the image features to obtain temporal interactive features. The server implements this interactive calculation through a temporal audio cross-attention layer. First, the audio embeddings are mapped to query vectors, and the sequence formed by expanding the image features along the temporal dimension is mapped to key and value vectors. Then, the dot product of the query vector and the key vector is calculated, and after scaling and softmax normalization, a temporal attention weight matrix is ​​obtained. Each element of this weight matrix represents the degree of attention the audio embedding pays to a specific time frame in the image sequence. Further, the temporal attention weight matrix is ​​multiplied by the value vector to obtain the weighted temporal interactive features. These temporal interactive features utilize temporal audio embeddings to guide inter-frame motion alignment, ensuring that the generated animation frames are precisely synchronized with the audio signal in time.

[0033] Furthermore, the server fuses the spatial interaction features and the temporal interaction features to generate the enhanced image features. The fusion method can employ element-wise addition, feature concatenation, or attention fusion. The fused features contain both the spatial focus information and the temporal alignment information of the audio guidance, i.e., the enhanced image features.

[0034] This embodiment achieves deep interaction and fusion of audio and visual features by simultaneously injecting audio embedding into the spatial attention layer and temporal attention layer of the diffusion network. The spatial audio cross-attention mechanism enables the model to focus on facial regions related to speech expression, avoiding interference from background noise; the temporal audio cross-attention mechanism uses temporal audio embedding to guide inter-frame motion alignment, ensuring precise temporal synchronization of lip movements, facial expression changes, and audio signals. Furthermore, through the synergistic effect of these two mechanisms, this embodiment achieves high-precision audio-visual synchronization and natural audio-driven facial expression generation without any visual motion cues, significantly outperforming traditional methods that rely on local matching.

[0035] S3, based on the multi-scale audio features, predict head motion parameters and facial expression motion parameters, wherein the head motion parameters and facial expression motion parameters are used to decouple the facial motion to be generated into head motion and facial expression motion.

[0036] In specific implementation, the facial movements to be generated refer to all dynamic facial changes that the virtual avatar corresponding to the reference portrait image will exhibit in the portrait animation video, driven by the driving audio. These changes specifically include head translation, rotation, and swaying, as well as facial expression changes caused by the contraction and relaxation of facial muscles. To achieve independent control of the aforementioned facial movements based on audio features, this embodiment uses a motion decoupling controller to predict motion parameters and analyze dimensions. The motion decoupling controller is a combination of functional modules implemented by a software program, specifically including a parameter analysis unit for constructing motion quantification indicators and a parameter prediction unit for performing inference prediction.

[0037] Furthermore, during the training phase, the parameter parsing unit of the motion decoupling controller uses training video clips as input to construct the motion bucket parameters. The constructed motion bucket parameters are not a single numerical value, but rather a multi-dimensional intensity control vector. For each training video clip, the parameter parsing unit first obtains the facial bounding box coordinates for each frame using a face detection algorithm, and calculates the motion variance of the bounding box center in the horizontal and vertical directions. The horizontal variance reflects the intensity of the head's left-right swaying, and the vertical variance reflects the intensity of the head's up-down nodding. The parameter parsing unit directly uses the horizontal and vertical variances as two components of the head motion bucket parameters for that clip, thereby constructing a two-dimensional control vector. These head motion bucket parameters are used to control the translational amplitude of the head in the horizontal and vertical directions. Simultaneously, the parameter parsing unit obtains the coordinates of multiple facial key points in each frame using a facial key point detection algorithm. For each specific key point group, it calculates the relative displacement variance. For example, for the corner of the mouth key point group, it calculates the variance of the lateral relative displacement between adjacent frames to reflect the intensity of the smile; for the brow bone key point group, it calculates the variance of the vertical relative displacement to reflect the intensity of the eyebrow lift. The parameter parsing unit combines the relative displacement variances of each key point group into a multi-dimensional vector, which serves as the expression motion bucket parameter for that segment. The expression motion bucket parameter is used to control the intensity of the expression features corresponding to different key point groups. To clarify the logical positioning of the above construction process, it should be specifically noted that the head motion bucket parameter and expression motion bucket parameter constructed by the parameter parsing unit during the training phase are always used as supervision labels in the offline training phase and do not participate in the actual online inference calculation. In the subsequent inference phase, the audio-bucket mapping network will take the multi-scale audio features as input, and after regression training, directly output head motion intensity information and expression motion intensity information. The output results are the head motion parameters and expression motion parameters in the online inference phase.

[0038] Furthermore, upon entering the inference phase, the server directly inputs the multi-scale audio features generated in step S1 into the trained audio-bucket mapping network. After undergoing multiple internal nonlinear transformations, the network outputs independent prediction results, which correspond to head motion intensity information and facial expression motion intensity information, respectively. The head motion intensity information is defined as the head motion parameters, and the facial expression motion intensity information is defined as the facial expression parameters. It should be noted that both the head motion parameters and the facial expression parameters are multi-dimensional intensity control vectors. Taking facial expression as an example, a complete facial expression requires simultaneous control of multiple muscle groups, such as the stretching of the corners of the mouth and the contraction of the orbicularis oculi muscle. Therefore, the facial expression parameters contain multiple dimensions, each corresponding to a specific muscle group intensity control value. Similarly, the head motion parameters can also contain multiple dimensions to control the amplitude of head movement in different directions.

[0039] The two types of parameter vectors output by the audio-bucket mapping network are directly fed into the subsequent motion intensity adjustment step in the data stream, and are synchronously scaled by the adjustment factor, thereby completing the decoupled control of head movement and facial expression movement.

[0040] See Figure 6 In some preferred embodiments, the step of predicting head motion parameters and facial expression motion parameters based on the multi-scale audio features includes: S51, inputting the multi-scale audio features into an audio-bucket mapping network, and having the audio-bucket mapping network predict the head motion bucket parameters and the facial expression bucket parameters; S52, using the predicted head motion bucket parameters as the head motion parameters and the predicted facial expression bucket parameters as the facial expression motion parameters.

[0041] In practical implementation, to achieve independent and precise control of the virtual avatar's movement, this embodiment explicitly decouples the facial motion to be generated into head motion parameters and facial expression motion parameters in terms of parameter control through a motion decoupling controller. Explicit decoupling is achieved by constructing two independent parameter prediction paths and quantification indicators, corresponding to head translational motion and facial muscle expression motion respectively, allowing the intensity of each motion to be adjusted independently in subsequent steps, rather than being mixed together. Specifically, the head motion parameters are controlled by head motion bucket parameters, which control the amplitude of head translation; the facial expression motion parameters are controlled by facial expression bucket parameters, which control the intensity of facial expression changes.

[0042] Furthermore, the server inputs the multi-scale audio features into an audio-bucket mapping network, which automatically predicts the head motion bucket parameters and the facial expression bucket parameters. The audio-bucket mapping network is a lightweight neural network, such as a multilayer perceptron or an attention-based model. This network receives the multi-scale audio features as input, undergoes a series of nonlinear transformations, and outputs two independent parameters, corresponding to the head motion bucket parameters and the facial expression bucket parameters, respectively. The training process of this network is as follows: using a large amount of audio-video pairing data, the audio is encoded into multi-scale audio features, and the motion in the video is calculated as motion bucket parameters as supervision labels. The network is trained using a regression loss function to learn the mapping relationship from audio features to motion bucket parameters.

[0043] In one optional implementation, the audio-bucket mapping network can employ a multilayer perceptron or a lightweight network structure based on an attention mechanism. It includes an input layer, at least one hidden layer, and an output layer. The output layer outputs the predicted values ​​of the head motion bucket parameters and the facial expression bucket parameters. During training, a regression loss function such as mean squared error can be used, and parameters are updated through a momentum optimizer. The training data is a pre-collected audio-video pairing dataset. The audio portion is encoded to obtain multi-scale audio features, and the video portion is subjected to motion analysis to obtain head motion bucket parameter labels and facial expression bucket parameter labels. Specifically, the head motion bucket parameter labels can be obtained by calculating the motion variance of the center coordinates of the facial bounding box in the video, and the facial expression bucket parameter labels can be obtained by calculating the relative motion variance of the coordinates of facial key points in the video. After sufficient training on large-scale audio-video pairing data, the network can learn the mapping relationship from audio features to bucket parameters.

[0044] Furthermore, the server uses the predicted head motion bucket parameters as head motion parameters and the predicted facial expression bucket parameters as facial expression parameters.

[0045] This embodiment explicitly separates facial motion into head motion and facial expression motion using a motion decoupling controller, and controls each through independent motion bucket parameters, achieving fine-grained motion control. The head motion bucket parameters are derived from the variance of the facial bounding box, accurately reflecting the magnitude of head translation; the facial expression bucket parameters are derived from the relative motion variance of facial key points, precisely characterizing the intensity of expression changes. This decoupling design allows the model to independently adjust the amplitude of head swaying and facial expression changes, greatly enriching the dynamic range and expressiveness of the animation. Furthermore, through an audio-bucket mapping network, the system can automatically predict the best-matching motion bucket parameters based on the input audio content, achieving end-to-end automated prediction from audio to motion parameters, simplifying user operation and improving generation efficiency.

[0046] S4. Based on the obtained adjustment factor, the amplitude of the head motion parameters and the facial expression parameters are adjusted synchronously to obtain the adjusted head parameters and adjusted facial expression parameters.

[0047] In practice, the adjustment factor is a scaling coefficient used to control the overall dynamic range. The server receives the adjustment factor setting value sent by the user through the client. For example, when the user selects the "mild" mode, the adjustment factor is 0.5; when the user selects the "natural" mode, the adjustment factor is 1.0; and when the user selects the "strong" mode, the adjustment factor is 2.0. The server applies the adjustment factor to the head motion parameters and the facial expression parameters respectively, adjusting their amplitudes synchronously in the same proportion to obtain the adjusted head parameters and the adjusted facial expression parameters.

[0048] See Figure 7 In some preferred embodiments, the step of synchronously adjusting the amplitude of the head motion parameters and the facial expression parameters based on the obtained adjustment factor to obtain the adjusted head parameters and adjusted facial expression parameters includes: S61, receiving the adjustment factor input by the user; S62, applying the adjustment factor to the head motion parameters and the facial expression parameters respectively, and synchronously adjusting the amplitude of the head motion parameters and the facial expression parameters in the same proportion; S63, using the adjusted head motion parameters as the adjusted head parameters and the adjusted facial expression parameters as the adjusted facial expression parameters.

[0049] In practice, the server receives the adjustment factor input by the user through the client. The client provides a user interface, such as a slider, numeric input box, or voice command on a graphical user interface, to receive a value set by the user. The adjustment factor is a non-negative real number, and its value typically corresponds to the dynamic range desired by the user. For example, when the user selects the "mild" mode, the adjustment factor is 0.5; when the user selects the "natural" mode, the adjustment factor is 1.0; and when the user selects the "strong" mode, the adjustment factor is 2.0.

[0050] Furthermore, the server applies the adjustment factor to both the head movement parameter and the facial expression parameter, adjusting their amplitudes synchronously at the same ratio. For example, if the head movement parameter is 10, the facial expression parameter is 5, and the adjustment factor is 2, then the adjusted head movement parameter is 20, and the facial expression parameter is 10. Through this operation, the amplitudes of head movement and facial expression are amplified or reduced at the same ratio, ensuring their coordination and consistency.

[0051] Furthermore, the server uses the adjusted head motion parameters as the adjusted head parameters and the adjusted facial expression motion parameters as the adjusted facial expression parameters.

[0052] This embodiment introduces a unified scaling factor as an adjustment factor, providing users with a convenient and precise means of controlling the overall dynamic range of the animation. Users do not need to adjust the intensity of head movements and facial expressions separately; they only need to set a simple value or select a level through the client to achieve a full range of adjustments, from gentle to intense and exaggerated. Furthermore, this design allows the same model to flexibly adapt to the needs of different scenarios. For example, a gentle mode can be used in online education or mental health counseling scenarios, while an intense mode can be used in virtual anchor or game character interaction scenarios, thus significantly broadening the application scope of this method.

[0053] S5. Based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters, a portrait animation video is generated.

[0054] In specific implementation, the server takes the enhanced image features, adjusted head parameters, and adjusted facial expression parameters obtained in the aforementioned steps as input and performs a joint denoising generation operation. To simultaneously ensure the high fidelity of a single-frame image and the temporal continuity of the video sequence, this embodiment employs a strategy combining spatial and temporal denoising. The spatial domain refers to the spatial structure dimension within a single-frame image, involving the distribution of pixels on a two-dimensional plane, edge contours, texture details, and the geometrical positional relationships of facial features; the temporal domain refers to the temporal dimension unfolding along the time axis between multiple frames, involving the continuity of content between adjacent frames, the smoothness of motion, and the naturalness of dynamic transitions.

[0055] During joint denoising generation, the server first performs spatial domain denoising on the enhanced image features. This spatial domain denoising uses a convolutional neural network or Transformer architecture to perform multi-layer convolution and attention calculations on the enhanced image features to eliminate noise artifacts in single-frame images and restore spatial details in facial regions, ensuring that each portrait image frame is clear, complete, and consistent in spatial structure and identity features. Simultaneously, the server performs temporal domain denoising on the enhanced image features. This temporal domain denoising introduces a temporal attention mechanism or 3D convolution operations to perform temporal correlation calculations on features between adjacent frames, eliminating inter-frame jitter and maintaining the continuity of motion information, ensuring a smooth transition in the generated video sequence. Furthermore, in the above-mentioned joint spatial and temporal denoising process, the server embeds the adjusted head parameters and adjusted facial expression parameters as control conditions into the denoising network. Specifically, the control conditions are fused with denoising features through an adaptive normalization layer or a conditional convolutional layer, so that the denoising network can actively apply head movements and facial expressions synchronized with the audio in the image sequence while restoring the spatial details and temporal coherence of the image, in accordance with the requirements of the control conditions.

[0056] After multiple rounds of joint denoising iterations, the server obtains a denoised feature that integrates motion control information. This denoised feature contains both clear and complete spatial information of the portrait image and temporal motion information synchronized with the audio. Finally, based on the denoised feature with integrated motion control information, the server generates a portrait animation video frame by frame using an image decoder. Each frame in the video is clear and complete in spatial structure, and the frames are coherent and natural in the temporal dimension, with the overall motion precisely synchronized with the driving audio. The server then feeds the generated portrait animation video back to the client for playback or display.

[0057] This embodiment achieves high-quality, highly synchronized, and highly efficient portrait animation generation by constructing a unified generation framework entirely based on audio-driven principles. First, this embodiment employs a context-enhanced audio learning module, embedding audio into the spatial and temporal attention layers of a diffusion network through a spatiotemporal audio cross-attention mechanism. This allows the model to perceive long-term variations in tone, rhythm, and speech rate within the audio, thereby generating natural facial expressions that match the speech content. This avoids the problems of monotonous movements and lack of natural variation caused by traditional local matching methods. Furthermore, this embodiment explicitly separates facial movements into head movements and facial expression movements through a motion decoupling controller, controlling each through independent motion bucket parameters, greatly improving the flexibility and controllability of the animation. Further, this embodiment introduces an adjustment factor, allowing users to set a simple value on the client side to achieve a full range of dynamic adjustments, from gentle and restrained to intense and exaggerated, significantly broadening its practical value in fields such as virtual anchors, online education, and content creation. Finally, this embodiment completely eliminates reliance on auxiliary conditions such as visual motion information, reducing computational complexity, supporting real-time inference, and achieving higher operating efficiency.

[0058] See Figure 8 In some preferred embodiments, generating a portrait animation video based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters includes: S71, using the adjusted head parameters and the adjusted facial expression parameters as control conditions, performing joint spatial and temporal denoising processing on the enhanced image features to obtain denoised features; S72, generating the portrait animation video based on the denoised features.

[0059] In specific implementation, the server inputs the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters into the time-aware positional offset fusion denoising module. The denoising module is built on a diffusion model architecture, which is a generative probabilistic model. Its basic principle is to generate images by simulating the noise addition and denoising process of data distribution. Specifically, the diffusion model first adds noise to the image step by step through a forward process, transforming the image into a pure noise distribution; then, through a reverse process, iteratively removes noise from the pure noise to recover a clear image with the target data distribution characteristics. In this embodiment, the denoising module uses the reverse denoising process of the diffusion model as its core generation mechanism, possessing the ability to jointly process spatial and temporal information.

[0060] Furthermore, in specific execution, the server first initializes a noisy image sequence based on pure noise data and uses the enhanced image features as a guiding condition. During processing, the denoising module executes the reverse denoising process of the diffusion model, gradually recovering the target image sequence from the pure noise through multiple iterations. The number of iterations is a preset value, the specific value of which can be determined according to factors such as the configuration during the model training phase and the image resolution of the actual application scenario, and can be reasonably set by those skilled in the art without creative effort.

[0061] Furthermore, in each iteration, the denoising module performs the following operations sequentially: First, using the noisy image sequence of the current iteration step and the enhanced image features as input, spatial domain denoising is performed through the convolutional layer and attention layer of the denoising module to eliminate noise artifacts within a single frame image and restore spatial details of the face region, resulting in intermediate features after spatial domain denoising; Second, using the intermediate features after spatial domain denoising as input, temporal domain denoising is performed through the temporal attention mechanism or 3D convolution operation of the denoising module to temporally correlate features between adjacent frames. The first step involves a combined computation to eliminate inter-frame jitter and maintain the continuity of motion information, resulting in intermediate features after temporal denoising. The second step uses these intermediate features as input, and the adjusted head parameters and adjusted facial expression parameters as control conditions. These control conditions are then embedded into the network layer of the denoising module through an adaptive normalization layer or a conditional convolutional layer, fusing the control conditions with the intermediate features to obtain the output features of the current iteration step with fused control conditions. These output features retain spatial details and temporal continuity while also applying motion information synchronized with the audio according to the control conditions.

[0062] Furthermore, the above three steps are performed cyclically in each iteration. After a preset number of iterations, the denoising module outputs the denoised features. These denoised features are the final features obtained after spatial domain denoising, temporal domain denoising, and motion control condition embedding. They contain both clear and complete spatial information of the portrait image and temporal motion information synchronized with the audio. Based on these denoised features, the denoising module generates the portrait animation video frame by frame.

[0063] Furthermore, the denoising module generates the portrait animation video frame by frame based on the denoised features. Specifically, the denoising module calls an image decoder and uses an upsampling convolutional layer to convert the denoised features into frame-by-frame image pixel data, forming a continuous image sequence, which constitutes the portrait animation video. Each frame in the video is clear and complete in spatial structure, and the frames are coherent and natural in the temporal dimension, with the overall motion precisely synchronized with the driving audio.

[0064] Furthermore, the server will send the generated portrait animation video back to the client, which will then play or display it.

[0065] This embodiment employs a time-aware positional offset fusion denoising module to deeply fuse image features with motion control parameters during the denoising process. While restoring image details, this module actively embeds head motion parameters and facial expression motion parameters as control conditions, achieving joint optimization of image generation and motion control. This approach ensures that the generated animated video is temporally coherent, jitter-free, and features smooth and natural transitions between frames. Furthermore, because motion control conditions are directly embedded in the generation process, the model can more accurately execute motion commands predicted from the audio, thus achieving high-precision audio-visual synchronization. The final generated portrait animation video features high resolution, high naturalness, and high synchronization, meeting the demands of high-quality content creation.

[0066] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0067] Specific application examples of this invention are as follows: In the fintech field, this invention can be applied to virtual customer service assistants in remote video banking systems. When a user initiates a business inquiry through the client, the server first acquires the driving audio and a reference portrait image. The driving audio is voice content synthesized by the system based on the business scenario, such as "Hello, how can I help you?" or "Your large transfer has been successfully processed." The reference portrait image is the virtual customer service avatar specified by the user. The server performs global audio perception on the driving audio, extracting long-term contextual information such as tone, rhythm, and speech rate, and generating virtual customer service facial expressions and head movements that highly match the voice content. For example, when the driving audio is played with an enthusiastic and cheerful rhythm, the server predicts larger head movement bucket parameters and strong facial expression bucket parameters through an audio-bucket mapping network, driving the virtual customer service to convey friendliness with slightly larger smiles and nods; when the driving audio content is a serious risk warning, the server predicts lower head movement bucket parameters and gentler facial expression bucket parameters, driving the virtual customer service to play with a steady and focused posture. Through a motion decoupling controller, the server can independently control the virtual customer service representative's head movements (such as nodding in confirmation) and facial expressions (such as smiling), ensuring seamless switching between polite gestures and informative communication. Simultaneously, financial institutions can dynamically adjust the overall dynamic range of the virtual customer service representative based on business scenario needs, such as choosing a gentle mode for general business inquiries and a strong mode for financial product presentations to enhance persuasiveness. This solution does not rely on any visual motion cues; it generates highly natural virtual customer service animations solely by driving audio, significantly improving the interactive experience and professionalism of remote financial services.

[0068] In the healthcare field, this invention can be applied to virtual psychologist avatars in remote mental health counseling platforms. When a patient shares their psychological distress with the platform via a client, the server generates reassuring voice content to respond to the patient based on a preset counseling strategy. This voice content is the driving audio, such as, "I understand how you feel right now. Can you share what happened with me?" The server performs global context awareness on the driving audio, extracting long-term information such as tone, rhythm, and speech rate, and generates highly synchronized facial expressions and head movements of the virtual psychologist with the driving audio content. For example, when the driving audio is played at a gentle, slow pace, the server predicts lower head movement bucket parameters and gentle facial expression bucket parameters through an audio-bucket mapping network, driving the virtual doctor to respond with slow nodding and gentle smiling expressions, creating a safe and stable communication atmosphere. Through a motion decoupling controller, the system can independently control the virtual doctor's head tilt angle and facial expression intensity, thereby accurately presenting nonverbal communication skills under different psychological counseling strategies. This solution does not rely on any visual motion cues; it generates highly natural virtual doctor animations simply by driving audio, significantly enhancing the sense of presence and therapeutic effect of remote psychological counseling.

[0069] In one embodiment, an animation generation device 40 based on global audio perception is provided, which corresponds one-to-one with an animation generation method based on global audio perception described in the above embodiments. For example... Figure 9 As shown, the audio-based global awareness-based animation generation device 40 includes: Encoding unit 41 is configured to encode the driving audio to obtain multi-scale audio features and encode the reference portrait image to obtain image features in response to acquiring the driving audio and the reference portrait image. Enhancement unit 42 is used to enhance the motion information related to audio time sequence in the image features based on the multi-scale audio features and the image features, through global audio context modeling, to obtain enhanced image features; The decoupling unit 43 is used to predict head motion parameters and facial expression motion parameters based on the multi-scale audio features, wherein the head motion parameters and facial expression motion parameters are used to decouple the facial motion to be generated into head motion and facial expression motion. The adjustment unit 44 is used to simultaneously adjust the amplitude of the head motion parameters and the facial expression parameters based on the acquired adjustment factor, so as to obtain the adjusted head parameters and the adjusted facial expression parameters. The generation unit 45 is used to generate a portrait animation video based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters.

[0070] In some preferred embodiments, encoding the driving audio to obtain multi-scale audio features includes: The driving audio is preprocessed to obtain an audio signal sequence; Multi-scale feature extraction is performed on the audio signal sequence by a pre-trained audio encoder to capture long-term contextual information in the driving audio and obtain multi-scale features. The long-term contextual information includes at least one of tone, rhythm and speech rate. The multi-scale features are combined to obtain multi-scale audio features.

[0071] In some preferred embodiments, encoding the reference portrait image to obtain image features includes: Facial region detection is performed on the reference portrait image to extract the facial region image; Spatial feature extraction is performed on the facial region image to obtain the image features containing facial structure information.

[0072] In some preferred embodiments, the step of enhancing the motion information related to audio temporal sequence in the image features by global audio context modeling based on the multi-scale audio features and the image features to obtain enhanced image features includes: The spatial interaction features are obtained by interactively calculating the audio embedding in the multi-scale audio features and the spatial dimension of the image features. The audio embedding in the multi-scale audio features is interactively calculated with the temporal dimension of the image features to obtain the temporal interactive features; The enhanced image features are generated by fusing the spatial interaction features and the temporal interaction features.

[0073] In some preferred embodiments, the prediction of head motion parameters and facial expression motion parameters based on the multi-scale audio features includes: The multi-scale audio features are input into an audio-bucket mapping network, which then predicts the head motion bucket parameters and the facial expression bucket parameters. The predicted head movement barrel parameters are used as the head movement parameters, and the predicted facial expression barrel parameters are used as the facial expression parameters.

[0074] In some preferred embodiments, the step of simultaneously adjusting the amplitude of the head motion parameters and the facial expression parameters based on the obtained adjustment factor to obtain the adjusted head parameters and adjusted facial expression parameters includes: Receive the adjustment factor input by the user; The adjustment factor is applied to the head motion parameter and the facial expression parameter respectively, and the amplitude of the head motion parameter and the facial expression parameter is adjusted synchronously in the same proportion; The adjusted head motion parameters are used as the adjusted head parameters, and the adjusted facial expression motion parameters are used as the adjusted facial expression parameters.

[0075] In some preferred embodiments, generating a portrait animation video based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters includes: Using the adjusted head parameters and the adjusted facial expression parameters as control conditions, the enhanced image features are subjected to joint spatial and temporal denoising processing to obtain denoised features; The portrait animation video is generated based on the denoised features.

[0076] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side animation generation method based on global audio perception.

[0077] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an audio-based global perception-based animation generation method.

[0078] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of an audio-based global awareness-based animation generation method provided in any of the above embodiments.

[0079] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of an audio-based global perception-based animation generation method provided in any of the above embodiments.

[0080] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0081] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0083] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0084] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An animation generation method based on global audio perception, characterized in that, include: In response to acquiring the driving audio and the reference portrait image, the driving audio is encoded to obtain multi-scale audio features, and the reference portrait image is encoded to obtain image features; Based on the multi-scale audio features and the image features, global audio context modeling is used to enhance the motion information related to audio temporal sequence in the image features, resulting in enhanced image features. Based on the multi-scale audio features, head motion parameters and facial expression motion parameters are predicted, wherein the head motion parameters and facial expression motion parameters are used to decouple the facial motion to be generated into head motion and facial expression motion. Based on the obtained adjustment factor, the amplitude of the head motion parameters and the facial expression parameters are adjusted synchronously to obtain the adjusted head parameters and adjusted facial expression parameters. Based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters, a portrait animation video is generated.

2. The animation generation method based on global audio perception according to claim 1, characterized in that, The encoding of the driving audio to obtain multi-scale audio features includes: The driving audio is preprocessed to obtain an audio signal sequence; Multi-scale feature extraction is performed on the audio signal sequence by a pre-trained audio encoder to capture long-term contextual information in the driving audio and obtain multi-scale features. The long-term contextual information includes at least one of tone, rhythm and speech rate. The multi-scale features are combined to obtain multi-scale audio features.

3. The animation generation method based on global audio awareness according to claim 1, characterized in that, The process of encoding the reference portrait image to obtain image features includes: Facial region detection is performed on the reference portrait image to extract the facial region image; Spatial feature extraction is performed on the facial region image to obtain the image features containing facial structure information.

4. The animation generation method based on global audio awareness according to claim 1, characterized in that, The process involves enhancing the motion information related to audio temporal sequence in the image features based on the multi-scale audio features and the image features, through global audio context modeling, to obtain enhanced image features, including: The spatial interaction features are obtained by interactively calculating the audio embedding in the multi-scale audio features and the spatial dimension of the image features. The audio embedding in the multi-scale audio features is interactively calculated with the temporal dimension of the image features to obtain the temporal interactive features; The enhanced image features are generated by fusing the spatial interaction features and the temporal interaction features.

5. The animation generation method based on global audio awareness according to claim 1, characterized in that, The head motion parameters and facial expression motion parameters predicted based on the multi-scale audio features include: The multi-scale audio features are input into an audio-bucket mapping network, which then predicts the head motion bucket parameters and the facial expression bucket parameters. The predicted head movement barrel parameters are used as the head movement parameters, and the predicted facial expression barrel parameters are used as the facial expression parameters.

6. The animation generation method based on global audio awareness according to claim 1, characterized in that, The process of simultaneously adjusting the amplitude of the head motion parameters and the facial expression parameters based on the obtained adjustment factor to obtain the adjusted head parameters and adjusted facial expression parameters includes: Receive the adjustment factor input by the user; The adjustment factor is applied to the head motion parameter and the facial expression parameter respectively, and the amplitude of the head motion parameter and the facial expression parameter is adjusted synchronously in the same proportion; The adjusted head motion parameters are used as the adjusted head parameters, and the adjusted facial expression motion parameters are used as the adjusted facial expression parameters.

7. The animation generation method based on global audio awareness according to claim 1, characterized in that, The process of generating a portrait animation video based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters includes: Using the adjusted head parameters and the adjusted facial expression parameters as control conditions, the enhanced image features are subjected to joint spatial and temporal denoising processing to obtain denoised features; The portrait animation video is generated based on the denoised features.

8. An animation generation device based on global audio perception, characterized in that, include: The encoding unit is configured to, in response to acquiring the driving audio and the reference portrait image, encode the driving audio to obtain multi-scale audio features, and encode the reference portrait image to obtain image features; An enhancement unit is used to enhance the motion information related to audio temporality in the image features based on the multi-scale audio features and the image features, through global audio context modeling, to obtain enhanced image features; The decoupling unit is used to predict head motion parameters and facial expression motion parameters based on the multi-scale audio features, wherein the head motion parameters and facial expression motion parameters are used to decouple the facial motion to be generated into head motion and facial expression motion; An adjustment unit is used to simultaneously adjust the amplitude of the head motion parameters and the facial expression parameters based on the acquired adjustment factor, so as to obtain the adjusted head parameters and the adjusted facial expression parameters. The generation unit is used to generate a portrait animation video based on the enhanced image features, the adjusted head parameters, and the adjusted facial expression parameters.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images