A voice-driven talking portrait generation method and an electronic device

Abstract

The application discloses a speech-driven talking portrait generation method and electronic equipment, wherein the method comprises the following steps: acquiring audio features, spatial features and image features; inputting the audio features and the spatial features into an audio-spatial coding module, aligning the audio features and the spatial features by dynamically adjusting the spatial position of the audio features and the facial region, and obtaining spatially perceived lip features; inputting the audio features and the image features into an audio-guided dynamic fusion module, calculating audio-perceived channel attention and spatial attention by using a dynamic linear layer, and obtaining dynamic fusion features; acquiring reference key points, inputting the reference key points and the audio features into an expression optimization module, fusing the reference key points and the audio reference points, and obtaining key point features; and inputting the lip perception features, the dynamic fusion features and the key point features into a trained generation network to generate video frames. The method realizes a higher-quality, more natural and stable speech-driven portrait synthesis effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and particularly relates to a speech-driven talking portrait generation method and an electronic device. BACKGROUND

[0002] Speech-driven talking portrait synthesis is an important research direction in computer vision and graphics, aiming to generate high-quality facial images that are precisely synchronized with given audio. With the continuous development of technology, the research and application in this field have gradually expanded, mainly focusing on synthesizing lip movements of real people or virtual characters through audio driving, providing high-quality facial synthesis effects for movies, animations, games, virtual reality, and other various applications.

[0003] In recent years, methods based on neural radiance fields (NeRF) have been widely applied to speech-driven talking portrait synthesis tasks due to their significant advantages in image rendering. NeRF generates high-quality images by constructing a three-dimensional radiance field, particularly excelling in details, naturalness, and lighting processing of facial images, breaking through the resolution limitations of traditional methods. Although these methods have achieved some results, they still face a series of technical challenges, especially in improving lip synchronization accuracy and image fidelity.

[0004] Currently, many existing NeRF-based methods focus more on optimizing lip synchronization accuracy, but often ignore the overall coherence of facial expressions. This approach, while improving lip synchronization to some extent, can lead to distortion of facial expressions, greatly reducing the visual quality and user experience of the final generated video. In addition, existing methods often fail to fully consider the details of other facial regions, making the overall effect still not coordinated enough to meet the requirements of high-quality video generation.

[0005] In summary, existing technologies mainly suffer from the following technical problems: 1) Insufficient lip synchronization accuracy: Although existing methods can achieve lip synchronization to some extent, due to incomplete audio-to-image mapping learning, the synchronization between audio and lips in the synthesized result still has a significant deviation. This deviation leads to asynchrony between the generated image and audio, thus affecting the naturalness and realism of the synthesized portrait. 2) Facial expression distortion and instability: Current methods usually only focus on optimizing lip synchronization accuracy, but ignore the expression changes in other areas of the face. Since changes in facial expressions involve the coordination and smooth transition of multiple areas, inaccurate facial key point information may lead to facial expression distortion or abrupt changes, especially when the expression transition between adjacent frames is relatively drastic. This distortion and abrupt changes make the generated portrait exhibit unnatural expression fluctuations, further damaging the realism and smoothness of the visual effect, and reducing the overall effect and watchability of the generated video. 3) Identity consistency: In existing voice-driven portrait synthesis methods, cross-modal alignment of audio and visual features remains a technical challenge. Directly applying audio features to the generation process often leads to changes in facial features, making it impossible to maintain consistent identity information, thereby reducing the generalization ability and practicality of the synthesized portrait across different identities.

[0006] Therefore, how to balance lip synchronization and the naturalness of overall facial expressions, and improve the overall realism and stability of synthesized videos, remains a key issue that urgently needs to be addressed in the field of voice-driven portrait synthesis. Summary of the Invention

[0007] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a high-quality, highly synchronous voice-driven speech portrait generation method, electronic device and medium.

[0008] The first technical solution adopted in this invention is:

[0009] A method for generating a speech-driven portrait includes the following steps:

[0010] Obtain audio features f based on the input audio. a Spatial features f are obtained from the input image. x Image features F are obtained from reference video frames;

[0011] Audio features f a and spatial features f x The input audio-spatial coding module dynamically adjusts the spatial position of audio features and facial regions to align audio features with spatial features, thereby obtaining spatially perceptive lip features f. lip ;

[0012] Audio features f aThe image features F are input to the audio-guided dynamic fusion module, which uses a dynamic linear layer to calculate the channel attention and spatial attention of audio perception to obtain the dynamic fusion features F″.

[0013] Obtain reference key point L ref Reference key point L ref and audio features f a The input expression optimization module fuses reference key points and audio reference points to obtain key point features;

[0014] The lip perception features, dynamic fusion features, and key point features are input into the trained generative network to generate video frames.

[0015] Furthermore, the audio-spatial coding module operates as follows:

[0016] The audio features are repeatedly expanded to make their dimensions consistent with those of the spatial features;

[0017] The repeated expanded audio features are concatenated with spatial features, and the attention weight W is obtained after processing. aud_att ;

[0018] According to attention weight W aud_att Each channel of the audio features is reweighted to obtain spatially perceptual lip features f. lip .

[0019] Furthermore, the attention weight W aud_att The expression is:

[0020] h = ReLU(W1[f a ;f x ]+b1)

[0021] W aud_att =σ(W2h+b2)

[0022] The lip feature f lip The expression is:

[0023] f lip =f a ⊙W aud_att

[0024] Where (W1,b1) and (W2,b2) are the weight matrix and bias vector of the multilayer perceptron (MLP); ⊙ represents the Hadamard product.

[0025] Furthermore, the dynamic fusion module operates as follows:

[0026] The image features F are input into the dynamic fusion module to calculate the channel attention weights A. ch ;

[0027] The channel attention weight A is obtained by performing element-wise multiplication on the image feature F and the channel attention weight A ch

[0028] The visual feature F' is input into a dynamic linear layer to obtain a spatial attention map A sp

[0029] The spatial attention map A is multiplied by the visual feature F' to obtain a dynamic fusion feature F". sp

[0030] Further, the dynamic fusion module comprises a dynamic perception machine DyMLP; the dynamic perception machine DyMLP is composed of two dynamic linear layers with a LeakyReLU activation function in between;

[0031] The image feature F is input into the dynamic perception machine DyMLP to obtain the channel attention weight A ch , and the calculation formula is as follows:

[0032] A ch =Sigmoid(DyMLP(F))

[0033] DyMLP(x)=DyLinear1(LeakyReLU(DyLinear2(x)))

[0034] In the formula, Sigmoid represents the Sigmoid function, DyLinear1 is the first dynamic linear layer, and DyLinear2 is the second dynamic linear layer;

[0035] The calculation formula of the visual feature F' is as follows:

[0036]

[0037] In the formula, represents element-wise multiplication;

[0038] The calculation formula of the spatial attention map A sp is as follows:

[0039]

[0040] The calculation formula of the dynamic fusion feature F" is as follows:

[0041]

[0042] Wherein, DyLinear3 is the third dynamic linear layer.

[0043] Further, the dynamic linear layer is used to adjust the visual feature and calculate the dynamic audio perception attention, and its expression is:​​

[0044] x out = DyLinear θ (x in )

[0045] wherein, and denote the input vector and output vector of the dynamic linear layer, respectively;

[0046] The trainable parameters of the dynamic linear layer are denoted as The number of parameters is reduced using matrix decomposition to reduce the training cost, i.e., decomposing the parameter matrix θ into and where M a is the parameter matrix after the dynamic reshaping operation of the audio feature f a , which transforms the input from to S is a static learnable matrix, and K is the dimension of the parameter matrix obtained by audio feature reconstruction;

[0047] The parameters of the dynamic linear layer are reconstructed by multiplying M a and S, and the expression is as follows:

[0048] θ = M a S

[0049] M a = Reshape(f a )

[0050] where the parameters are not shared between different dynamic linear layers.

[0051] Further, the working mode of the expression optimization module is as follows:

[0052] Take the reference key points L ref and the audio feature f a as the input of the key point encoder, and output the generated key points L audio aligned by audio correction;

[0053] Use the weighted KNN strategy based on distance and area weight to weight the similarity and spatial distance between key points, fuse the key point information from the reference video and the prediction model, and generate the final key points;

[0054] Obtain the key point features according to the generated key points.

[0055] Further, the weighted KNN strategy based on distance and area weight is used to weight the similarity and spatial distance between key points, fuse the key point information from the reference video and the prediction model, and generate the final key points, including:

[0056] The reference key points L ref and the predicted key points L audio are divided into regions, respectively, to obtain a set of lip key points R lip , a set of eye key points R eye , a set of nose key points R nose , a set of eyebrow key points R brow , and a set of face contour key points R face .

[0057] For each key point i, the Euclidean distance d i between the reference key point and the predicted key point is calculated:

[0058] d i =||L audio,i -L ref,i ||2

[0059] According to the Euclidean distance d i , the deviation between the predicted key point and the reference key point is quantified to guide the weighted fusion process.

[0060] An initialization weight w r is set for each region, and all weights are normalized to obtain normalized weights The weight of each key point i is determined by the region R r(i) to which it belongs:

[0061]

[0062] The reference key points and the audio reference points are fused to generate the expression of the key points:

[0063]

[0064] where L final is the final generated fused key point set, represents the distance-based weight, the smaller the distance, the greater the weight, w i represents the region weight of each key point.

[0065] The fused key point set L final is encoded into a key point feature representation.

[0066] Further, the total loss L of the generation network during training is expressed as follows:

[0067]

[0068] wherein, The MSE loss of each pixel value is specifically expressed as:

[0069]

[0070] In the formula, C represents the color value of each pixel in the generated image, C gt represents the color value of each pixel in the reference image, and I represents the pixel set of the image.

[0071] The L2 loss between the real key points and the predicted key points is the overall reconstruction constraint:

[0072]

[0073] In the formula, T represents the frame number of the image, K represents the number of face key points in each frame of the image, L ref ={l1, l2, l3, …, l T} represents the extracted real face key points, represents the predicted face key points.

[0074] The reconstruction loss between the optimized key points and the reference key points is:

[0075]

[0076] The temporal consistency loss is used to enhance the continuity between frames and avoid the situation that the key point positions are different due to a large time span, so as to improve the naturalness and fluency of the generated results. The temporal consistency loss is defined as follows:

[0077]

[0078] In the formula, t and t+1 represent the time of adjacent two frames.

[0079] The second technical solution adopted by the present application is:

[0080] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the voice-driven talking portrait generation method as described above.

[0081] The third technical solution adopted by the present application is:

[0082] A computer readable storage medium, the storage medium has at least one instruction, at least one program, a code set or instruction set stored therein, the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to realize a voice-driven talking portrait generation method as described above.

[0083] The fourth technical solution adopted by the present application is:

[0084] A computer program product or computer program, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer readable storage medium. The processor of the computer equipment can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer equipment executes the above method.

[0085] The beneficial effects of the present application are: the present application realizes high-precision alignment between audio features and lip movement by introducing an audio-spatial coding module; the fusion process of audio and visual features is optimized through the audio-guided dynamic fusion module, avoiding the problems of facial expression distortion and instability; the audio-guided dynamic fusion module proposed by the present application can accurately fuse audio and visual features, ensuring that the influence of audio features on lips and facial expressions is effectively guided while the identity features of the face are maximally preserved. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0087] Figure 1 is the overall framework diagram of a high-quality, high-synchronization voice-driven talking portrait generation method in the embodiments of the present application;

[0088] Figure 2 is a structural diagram of an audio-guided dynamic fusion module in the embodiments of the present application;

[0089] Figure 3 is a structural diagram of an expression optimization module based on face key points in the embodiments of the present application;

[0090] Figure 4 is a step flowchart of a voice-driven talking portrait generation method in the embodiments of the present application. DETAILED DESCRIPTION

[0091] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explaining the description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0092] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and cannot be understood as indicating or implying that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0093] In the description of the present application, several meanings are one or more, and multiple meanings are two or more. Greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0094] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0095] Term explanation:

[0096] NeRF: Neural Radiance Field, neural radiance field.

[0097] Embodiment 1

[0098] As Figure 4 shown, the present embodiment provides a high-quality and high-synchronization voice-driven talking portrait generation method, which includes the following steps:

[0099] S1, obtaining audio features f a from input audio, obtaining spatial features f x from input images, and obtaining image features F from reference video frames;

[0100] S2, combining audio features f a and spatial features f xThe input audio-spatial encoding module aligns the audio features with the spatial features by dynamically adjusting the spatial positions of the audio features and the facial regions, to obtain spatially-aware lip features f lip ;

[0101] S3, inputting the audio features f a and the image features F into an audio-guided dynamic fusion module, calculating the audio-aware channel attention and spatial attention by using a dynamic linear layer, to obtain dynamic fusion features f";

[0102] S4, obtaining reference key points L ref ; ref and the audio features f a inputting the reference key points L a and the audio features f x into an expression optimization module, fusing the reference key points and the audio reference points, to obtain key point features;

[0103] S5, inputting the lip-aware features, the dynamic fusion features, and the key point features into the trained generative network, to generate video frames.

[0104] To solve the technical problems of insufficient lip synchronization accuracy, distorted and unstable facial expressions, and identity consistency in existing speech-driven lip portrait synthesis methods, this embodiment proposes a novel framework, as shown in Figure 1 The framework combines three core modules: an audio-spatial encoding module, an expression optimization module based on facial key points, and an audio-guided dynamic fusion module. The following will explain these three core modules in detail in combination with the drawings and specific embodiments.

[0105] (1) Audio-spatial encoding module

[0106] To solve the problem of insufficient lip synchronization accuracy, an audio-spatial encoding module is introduced. This module accurately captures the audio-driven lip movement features and maps them to spatial coordinates, thereby generating high-quality lip movement representations.

[0107] The audio-spatial encoding module takes spatial features and audio features as input. First, the input audio is resampled, framed, feature extracted, and encoded, and finally encoded into audio features f a . At the same time, the NeRF framework maps the input image to a three-dimensional space and samples on the light rays with the camera optical center as the origin, thereby obtaining a series of coordinates (x, y, z) of the sampling points and their corresponding directions. These low-dimensional spatial coordinate information is then encoded and converted into high-dimensional spatial feature vectors f xThe existing methods usually simply concatenate the audio features and the spatial features, and then input them into a multi-layer perception (MLP) to control the synthesis of lip movements. However, this simple feature concatenation and encoding method fails to fully capture the complex correlation between the audio features and the spatial features, resulting in poor audio synchronization of the lip movements in the synthesis results. To this end, the present embodiment proposes an audio-spatial encoding module, which aims to further enhance the correlation between the audio features and the spatial features in the synthesis of lip movements.

[0108] First, the audio features are repeatedly expanded to have the same dimension as the spatial features. Then, the repeatedly expanded audio features and the spatial features are connected through a multi-layer perception to obtain attention weights W aud_att , as shown in the formula:

[0109] h = ReLU (W1[f a ; f x ]+b1)

[0110] W aud_att = σ (W2h+b2)

[0111] The attention weights W aud_att are used to re-weight each channel of the audio features to obtain spatially-aware lip features f lip :

[0112] f lip = f a ⊙W aud_att

[0113] where (W1, b1) and (W2, b2) are the weight matrix and bias vector of the MLP. ⊙ represents the Hadamard product.

[0114] This module can achieve high-precision alignment of audio features and spatial features by dynamically adjusting the spatial position of the audio features and the facial regions, so that the generated lip movements are highly consistent with the audio changes, avoiding the lip asynchronous problem in traditional methods.

[0115] (2) Audio-guided dynamic fusion module

[0116] To address the problems of facial feature distortion and identity consistency, the present embodiment designs an audio-guided dynamic fusion module, as shown in Figure 2

[0117] ​Existing audio-driven talking-head synthesis methods usually rely on static multi-layer perceptrons. However, since the parameters of these MLPs are randomly initialized, the driving audio cannot provide additional guidance information during the generation process. In addition, the influence of the audio sequence on the visual information is dynamically changing, and the synthesized portrait is only associated with the dynamic features of the driving audio in the lip region, so the visual features must be properly adjusted to adapt to the changes in the audio sequence. To solve this problem, a dynamic linear layer is first introduced, which learns a new linear transformation to adjust the visual features and calculate the dynamic audio perceptual attention. The expression of the dynamic linear layer can be represented as:

[0118] x out =DyLinear θ (x in )

[0119] where and represent the input vector and output vector of the dynamic linear layer, respectively. The trainable parameters of the dynamic linear layer are represented as To reduce the training cost, matrix decomposition is used to reduce the number of parameters. Specifically, the parameter matrix θ is decomposed into and M a is the parameter matrix after the dynamic reshaping operation of the audio feature f a , which transforms the input from to S is a static learnable matrix. The parameters of the dynamic linear layer can be reconstructed by multiplying M a and S, which can be represented as:

[0120] θ=M a S

[0121] M a =Reshape(f a )

[0122] where different dynamic linear layers do not share parameters.

[0123] The key to audio-guided fusion is the dynamic perceptual machine DyMLP, which consists of two dynamic linear layers with a LeakyReLU activation function in between. The invention uses dynamic linear layers to calculate the channel attention and spatial attention of audio perception, thereby obtaining the fusion features. The reference video frame is feature extracted to obtain the image feature F. The image feature is input into the dynamic MLP, and the output value is normalized to [0, 1] through the Sigmoid function. The calculation formula of the channel attention weight A ch is as follows:

[0124] A ch= Sigmoid(DyMLP(F))

[0125] DyMLP(x)=DyLinear1(LeakyReLU(DyLinear2(x)))

[0126] A ch and F are element-wise multiplied to obtain the channel-refined visual feature F':

[0127]

[0128] The purpose of channel attention is to activate the visual feature channels related to the driving audio.

[0129] To generate the spatial attention map, another dynamic linear layer is used to reduce the channel dimension and learn the region of interest of the audio. The calculation process is as follows:

[0130] A sp = Sigmoid(DyLinear3(F'))

[0131]

[0132] where A sp is the spatial attention map, and F" is the activated dynamic fusion feature. The goal of spatial attention is to activate the spatial image information related to the driving audio.

[0133] This module ensures that the generated fine-grained lip movement can maintain high identity consistency by guiding the network to pay dual attention to the lips and the overall face. Specifically, the model learns the dynamic mapping relationship between audio and lip movement through the refinement and fusion of channel and spatial domain features. Under the guidance of audio features, this module can efficiently fuse two-dimensional visual features, achieving excellent cross-modal alignment. This not only ensures the precise synchronization of the lips, but also maximizes the preservation of visual features unrelated to the driving audio, effectively ensuring the identity consistency of the generated video and significantly improving the overall effect and stability of the generated video.

[0134] (3) Expression optimization module based on face key points

[0135] To solve the problem of distorted facial expressions, the present invention proposes an expression optimization module based on face key points. It is used to extract the underlying facial key point information from the input audio, as shown in Figure 3 First, key point extraction is performed on the face image to obtain a set of 68 reference key points L ref These key points include eye, mouth, nose, and contour regions of the face. These key points are then combined with the audio feature f aTogether, as the input of the key point encoder, the output is the generated key point L after audio correction alignment audio .

[0136] The predicted face key points can accurately reflect the dynamic characteristics of the lips changing with the audio. However, due to the noise in the audio signal, the predicted face key points will inevitably produce some errors, which may cause the generated face expression to be distorted or have unnatural jumps. In order to enhance the dynamic expression ability while maintaining the stability and identity consistency of facial features, the present application proposes a key point fusion strategy based on weighted K-nearest neighbors (Weighted K-Nearest Neighbors, WKNN). This strategy fuses the key point information from the reference video and the prediction model by weighting the similarity and spatial distance between the key points to optimize the final face expression generation effect. Specifically, by dynamically assigning weights to each face key point, the relative position between the key points and their importance to the final result are weighted to adjust their contribution to the final generation result. The relatively stable facial regions (such as eyes, nose, etc.) have higher weights to maintain the consistency of individual features, while the lips and other regions that change greatly with speech are appropriately adjusted in weight to more accurately match the dynamic changes driven by audio. This method not only effectively suppresses the key point deviation caused by noise, but also improves the stability and accuracy of facial features while ensuring natural and smooth expression, making the generated face expression more realistic and consistent.

[0137] First, the reference key points L ref ∈R 68×2 and the predicted key points L audio ∈R 68×2 are regionally divided, respectively obtaining the lip key point set R lip , the eye key point set R eye , the nose key point set R nose , the eyebrow key point set R brow and the face contour key point set R face , and the region label can be represented by the formula:

[0138]

[0139] For each key point i, the Euclidean distance d i between the reference key point and the predicted key point is calculated:

[0140] d i =||L audio,i -L ref,i ||2

[0141] By this distance metric, the deviation between the predicted keypoint and the reference keypoint can be quantified, further guiding the weighted fusion process.

[0142] Then, set an initialization weight w for each region r (where r e {lip, eye, nose, brow, face}) and dynamically adjust the weight according to the distance d i and the importance of each region during the training process to achieve an adaptive weighting process. For example, the nearby keypoint has a greater impact on the prediction result, while the keypoint far from the target point has a smaller contribution, so as to ensure the smooth transition of local features. In this generation task, the goal of the model is to keep the facial regions stable except for the lip region, while ensuring that the lip keypoint has a higher influence and can adapt to the changes driven by the audio. Therefore, the weights w of the regions other than the lip region others are increased to ensure their stability in the optimization process, so that they can maintain the consistency of the portrait features under different audio inputs. For the lip region, since its keypoint needs to be highly adapted to the dynamic changes driven by the speech, its weight w lip is appropriately reduced to make it more flexible during optimization, thereby avoiding being constrained by other facial regions and ensuring the naturalness and accuracy of the lip movement. In training, the region weight coefficient is set as a learnable hyperparameter, so that the model can automatically optimize the weight distribution during the training process to adapt to the facial dynamic features driven by different audios. To ensure the stability of the weight optimization process, first normalize all weights, and the specific formula is as follows:

[0143]

[0144] The normalized weight satisfies and ensures that the weight is always positive. The weight of each keypoint i is determined by the region R r(i) to which it belongs:

[0145]

[0146] Using the weighted KNN strategy based on distance and region weight, the reference keypoint and the audio reference point are fused to generate the final keypoint, and the keypoint fusion formula is as follows:

[0147]

[0148] where L final is the final generated fusion keypoint set, represents the distance-based weight, the smaller the distance, the greater the weight, w i represents the region weight of each keypoint. The fused keypoint set L finalBy a series of convolutional layers, fully connected layers and other operations, the key point feature representation is encoded.

[0149] The key point fusion strategy realizes precise control of the key point optimization process by dynamically adjusting the weights of different facial regions, ensuring that the optimized key points not only meet the dynamic changes driven by audio, but also reasonably allocate weights between different regions to balance local flexibility and overall stability. The expression optimization module based on facial key points effectively reduces the expression jump problem caused by key point errors in generated videos, avoids unnecessary jitter or distortion, and makes facial movements smoother and more coherent, thereby improving the realism and quality of the synthesis results.

[0150] Finally, the lip perception feature, dynamic fusion feature and key point feature jointly participate in modulating portrait generation, training the overall network, and the total loss The expression of the total loss is as follows:

[0151]

[0152] wherein, is the MSE loss of each pixel value, and the specific expression is:

[0153]

[0154] is the L2 loss between the real key points and the predicted key points, which is the overall reconstruction constraint:

[0155]

[0156] T represents the number of frames of the image, K represents the number of facial key points in each frame of the image, L ref ={l1, l2, l3, …, l T} represents the extracted real facial key points, represents the predicted facial key points.

[0157] is the reconstruction loss between the optimized key points and the reference key points:

[0158]

[0159] is the temporal consistency loss, which is used to enhance the coherence between frames and avoid the situation that the key point positions are different due to large time span, thereby improving the naturalness and fluency of the generated results. The temporal consistency loss is defined as follows:

[0160]

[0161] where t and t+1 represent the time of adjacent two frames respectively.

[0162] In summary, the method of the present application solves the problems of insufficient lip synchronization accuracy, facial expression distortion and instability, and identity consistency, achieving higher quality, more natural and stable speech-driven portrait synthesis effect, greatly improving the visual realism and application value of the synthesized video. Compared with the prior art, at least the following advantages and beneficial effects are included:

[0163] 1) Improve the accuracy of lip synchronization: by introducing an audio-spatial coding module, high-precision alignment between audio features and lip movements is achieved.

[0164] 2) Optimize facial expression: through the audio-guided dynamic fusion module, the fusion process of audio and visual features is optimized, avoiding facial expression distortion and instability.

[0165] 3) Maintain identity consistency: the audio-guided dynamic fusion module proposed in the present application can accurately fuse audio and visual features, ensuring that the influence of audio features on lips and facial expressions is effectively guided while maximizing the preservation of facial identity features.

[0166] Embodiment 2

[0167] The embodiment of the present application also provides an electronic device, which includes a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize a speech-driven speaking portrait generation method as shown in Figure 4

[0168] It can be understood that the memory can include random access memory (RAM) and read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing each of the above methods, etc.; the data storage area can store data created according to the use of the server, etc.

[0169] ​The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Alternatively, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but be realized by a single chip.

[0170] Since the electronic device is an electronic device corresponding to the speech-driven talking portrait generation method of the embodiment of the application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0171] Embodiment 3

[0172] The embodiment of the application also provides a computer readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set are stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to realize a speech-driven talking portrait generation method as shown in Figure 4

[0173] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be instructed by programs to relevant hardware, and the programs can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0174] Since the storage medium is a storage medium corresponding to the voice-driven talking portrait generation method of the embodiments of the present application, and the problem solving principle of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0175] Embodiment 4

[0176] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the voice-driven talking portrait generation method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein the executable computer program codes or "codes" for executing various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0177] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, can be used to implement the hardware: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth, or a combination of them.

[0178] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0179] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for generating a speech-driven portrait, characterized in that, Includes the following steps: Obtain audio features from input audio. Spatial features are obtained from the input image. Image features are obtained from reference video frames. ; Audio features and spatial features The input audio-spatial coding module dynamically adjusts the spatial position of audio features and facial regions to align audio features with spatial features, thereby obtaining spatially perceptive lip features. Audio features and image features The input audio-guided dynamic fusion module utilizes a dynamic linear layer to calculate the channel attention and spatial attention of audio perception, thereby obtaining dynamic fusion features. Obtaining key reference points Key points will be referenced and audio features The input expression optimization module fuses reference key points and audio reference points to obtain key point features; The lip-sensing features, dynamic fusion features, and key point features are input into the trained generative network to generate video frames. The dynamic fusion module operates as follows: Image features Input to the dynamic fusion module, calculate channel weights. ; Channel attention weight and image features Element-wise multiplication is performed to obtain the visual features of channel refinement. ; visual features Input a dynamic linear layer and compute the spatial attention map. Spatial attention diagram and visual features Perform multiplication to obtain dynamic fusion features. .

2. The method for generating a speech-driven portrait according to claim 1, characterized in that, The audio-spatial coding module operates as follows: The audio features are repeatedly expanded to make their dimensions consistent with those of the spatial features; The attention weights are obtained by concatenating the repeated expanded audio features with the spatial features and then processing them. ; Based on attention weight Each channel of the audio features is reweighted to obtain spatially perceptual lip features. .

3. The method for generating a speech-driven portrait according to claim 2, characterized in that, The attention weight The expression is: The lip features The expression is: in, and These are the weight matrix and bias vector of a multilayer perceptron (MLP). This represents the Hadama product.

4. The method for generating a speech-driven portrait according to claim 1, characterized in that, The dynamic fusion module includes a dynamic perceptron DyMLP; the dynamic perceptron DyMLP consists of two dynamic linear layers with a LeakyReLU activation function in between. Image features Input the dynamic perceptron DyMLP to calculate the channel attention weights. The calculation formula is as follows: In the formula, express function, For the first dynamic linear layer, This is the second dynamic linear layer; Visual features The calculation formula is as follows: In the formula, To represent element-wise multiplication; Spatial attention diagram The calculation formula is as follows: Dynamic fusion features The calculation formula is as follows: in, This is the third dynamic linear layer.

5. The method for generating a speech-driven portrait according to claim 1, characterized in that, The dynamic linear layer is used to adjust visual features and compute dynamic audio-aware attention; its expression is: In the formula, and These represent the input and output vectors of the dynamic linear layer, respectively. The trainable parameters of a dynamic linear layer are represented as follows: ; Use matrix factorization to reduce the number of parameters, thereby reducing training costs, i.e., the parameter matrix Decomposed into and ,in It is based on audio characteristics The parameter matrix after dynamic reshaping operation It is a statically learnable matrix. This represents the dimension of the parameter matrix reconstructed from audio features; The parameters of the dynamic linear layer are obtained by... and Reconstructing by multiplication, the expression is as follows: Different dynamic linear layers do not share parameters.

6. The method for generating a speech-driven portrait according to claim 1, characterized in that, The expression optimization module works as follows: Reference key points and audio features As input to the keypoint encoder, the output is the generated keypoint after audio correction and alignment. ; A weighted KNN strategy based on distance and region weights is used to weight the similarity and spatial distance between keypoints, and keypoint information from reference videos and prediction models is fused to generate the final keypoints; Keypoint features are obtained from the generated keypoints.

7. The method for generating a speech-driven portrait according to claim 6, characterized in that, The method employs a weighted KNN strategy based on distance and region weights to weight the similarity and spatial distance between keypoints, fusing keypoint information from reference videos and prediction models to generate the final keypoints, including: For reference key points and predict key points The regions were divided to obtain separate sets of key points for the lips. Key points of the eyes Key points of the nose Key points of eyebrows and facial contour key points set ; For each key point Calculate the Euclidean distance between the reference keypoint and the predicted keypoint. : According to Euclidean distance Quantify the deviation between the predicted key points and the reference key points to guide the weighted fusion process; Set an initial weight for each region All weights are normalized to obtain the normalized weights. Each key point The weight is determined by its region. Decide: The expression for the generated keypoint, which combines reference keypoints and audio reference keypoints, is as follows: in, This indicates distance-based weights; the smaller the distance, the greater the weight. Indicates the regional weight of each key point; The merged key point set The encoding is used to represent key point features.

8. The method for generating a speech-driven portrait according to claim 1, characterized in that, The total loss during generator network training The expression is as follows: in, The MSE loss for each pixel value is specifically expressed as: In the formula, This represents the color value of each pixel in the generated image. This represents the color value of each pixel in the reference image. A set of pixels representing an image; Between true keypoints and predicted keypoints Loss, as a constraint on overall reconstruction: In the formula, Indicates the number of frames in the image. This indicates the number of facial landmarks in each frame of the image. This represents the extracted key points of a real human face. This represents the predicted facial landmarks; To optimize the reconstruction loss between key points and reference key points: The time consistency loss is defined as follows: In the formula, and These represent the times of two adjacent frames.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 8.

Images