Digital human lip shape video generation method and device

Abstract

The embodiment of the application discloses a digital person lip video generation method and device, relates to the technical field of digital people, and is used for solving the problems of low image resolution and low lip synchronization accuracy. The method comprises the following steps: in the case that the audio features corresponding to the target voice and the mouth features corresponding to any one frame of video image in the reference video are acquired, the audio features and the mouth features are input into a preset first neural network to output a target mouth image sequence; the reference video is filtered through a preset negative film selection strategy to obtain a negative film image; and the negative film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital person lip video image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital humans, in particular to a digital human lip video generation method and device. BACKGROUND

[0002] The rise of digital human technology is reshaping human-computer interaction and content production in various fields such as entertainment, media, education, medical care, and social interaction. As human-computer interaction demands continue to rise, certain application scenarios require higher authenticity and naturalness in human-computer interaction. One of the key criteria for authenticity and naturalness of digital humans is the natural synchronization of lip movements and speech. Lip video generation, as a key link between digital human speech and visual expression, deeply affects the interaction quality and user experience of digital humans.

[0003] Currently, in the field of voice-driven digital human generation, existing technologies are mainly based on audio-visual cross-modal generation methods, that is, by extracting speech features and fusing them with visual information, mouth animation synchronized with speech is generated. However, the latent space fusion of existing audio-visual cross-modal generation methods requires simultaneous optimization of multiple objectives, such as completing both audio-mouth matching and image fusion tasks, which can easily lead to optimization conflicts, resulting in high model complexity and low generated image resolution. In addition, discriminators in existing methods are also susceptible to non-mouth region interference and negative interference, making the digital human images output by the discriminators unable to adapt to different pronunciation states of the target speech, resulting in low lip synchronization accuracy. SUMMARY

[0004] To address the problems of high model complexity, low generated image resolution, and low lip synchronization accuracy in existing digital human audio-visual cross-modal generation methods, one or more embodiments of the present application provide a digital human lip video generation method and device.

[0005] One or more embodiments of the present application employ the following technical solutions:

[0006] In a first aspect, one or more embodiments of the present application provide a digital human lip video generation method, which includes:

[0007] After obtaining the audio features corresponding to the target speech and the mouth features corresponding to any frame of video image in the reference video, input them into a preset first neural network to output a target mouth image sequence;

[0008] Filter the reference video by a preset negative film selection strategy to obtain a negative film image;

[0009] The film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population type video image.

[0010] Optionally, in one or more embodiments of the present application, the input is input into a preset first neural network to output a target mouth image sequence, specifically comprising:

[0011] The mouth features are enhanced through a self-attention mechanism of the preset first neural network to obtain enhanced mouth features;

[0012] The enhanced mouth features and the audio features are fused through a cross-attention mechanism of the preset first neural network to output mixed features;

[0013] The mixed features are reconstructed through a preset multi-layer image decoder to obtain a target mouth image sequence matched with the target voice.

[0014] Optionally, in one or more embodiments of the present application, the enhanced mouth features and the audio features are fused through a cross-attention mechanism of the preset first neural network to output mixed features, specifically comprising:

[0015] The enhanced mouth features are transformed into a key matrix and a value matrix, and the audio features are transformed into a query matrix;

[0016] Attention weight calculation is performed according to the query matrix and the key matrix to obtain a weight matrix;

[0017] The value matrix is aggregated through the weight matrix to output mixed features.

[0018] Optionally, in one or more embodiments of the present application, the reference video is filtered through a preset film selection strategy to obtain a film image, specifically comprising:

[0019] A voice stage corresponding to the target voice is determined; wherein the voice stage includes a normal voice stage and a closed mouth stage.

[0020] A reference mouth opening degree sequence of the reference video and a target mouth opening degree sequence of the target mouth image sequence are obtained to dynamically match the reference mouth opening degree sequence and the target mouth opening degree sequence to obtain a matching result.

[0021] The reference video is subjected to film screening in combination with the voice stage and the matching result to obtain a film image.

[0022] Optionally, in one or more embodiments of the present application, the reference mouth opening degree sequence and the target mouth opening degree sequence are dynamically matched to obtain a matching result, specifically comprising:

[0023] An absolute opening degree difference of each frame pair between the reference mouth opening degree sequence and the target mouth opening degree sequence is obtained.

[0024] An accumulated distance corresponding to each frame pair is obtained through the absolute opening degree difference of each frame pair, and an accumulated distance matrix is generated.

[0025] The best matching path is determined as the matching result through the accumulated distance matrix.

[0026] Optionally, in one or more embodiments of the present application, the reference video is film screened in combination with the speech stage and the matching result to obtain a film image, specifically comprising:

[0027] If it is determined that the speech stage is a normal speech stage, the reference video is film screened according to the matching result to obtain an initial film image.

[0028] If it is determined that the speech stage is a closed mouth stage, a pre-stored closed mouth image is obtained as the initial film image.

[0029] The mouth region of the initial film image is covered to obtain the film image.

[0030] Optionally, in one or more embodiments of the present application, the film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population type video image, specifically comprising:

[0031] The target mouth image sequence and the film image are channel spliced to obtain a total input tensor.

[0032] The total input tensor is input into a preset second neural network for fusion processing to output a digital population type video image, so as to generate a mouth shape video through consecutive frame digital population type images.

[0033] Optionally, in one or more embodiments of the present application, after the input into the preset first neural network to output the target mouth image sequence, the method further comprises:

[0034] The current mouth feature of the target mouth image sequence is extracted through a preset image encoder, and the similarity between the current mouth feature and the audio feature is obtained.

[0035] According to the similarity, a synchronization discriminant loss of the audio feature and the current mouth feature is determined, and the preset first neural network is iteratively adjusted through the synchronization discriminant loss.

[0036] Optionally, in one or more embodiments of the present application, before inputting the same into the preset first neural network to output the target mouth image sequence, the method further comprises:

[0037] Optionally, in one or more embodiments of the present application, before inputting the same into the preset first neural network to output the target mouth image sequence, the method further comprises:

[0038] Optionally, in one or more embodiments of the present application, before inputting the same into the preset first neural network to output the target mouth image sequence, the method further comprises:

[0039] Optionally, in one or more embodiments of the present application, before inputting the same into the preset first neural network to output the target mouth image sequence, the method further comprises:

[0040] In a second aspect, one or more embodiments of the present application provide a digital person lip video generation device, the device comprising:

[0041] The acquisition unit is configured to, in a case where the audio feature corresponding to the target speech and the mouth feature corresponding to any one frame of video image in the reference video are acquired, input the same into a preset first neural network to output a target mouth image sequence.

[0042] The screening unit is configured to screen the reference video by using a preset negative film selection strategy to obtain a negative film image.

[0043] The output unit is configured to input the negative film image and the target mouth image sequence into a preset second neural network for fusion processing to output a digital person lip video image.

[0044] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects:

[0045] By decoupling the digital human lip video generation process, the optimization conflict problem when the traditional audio lip matching and image fusion two tasks are completed at the same time is avoided, the algorithm complexity is significantly reduced, and the generated resolution is improved. The mouth features and audio features are input into the preset first neural network for first stage processing. Compared with the method of using complete human face image in the traditional method, the interference of non-mouth area to lip synchronization is reduced, and the synchronization discrimination accuracy of the model is improved. The film selection is performed through the preset film selection strategy, so that the selected film image fits the opening and closing requirements of the target voice, and ensures that the lip shape meets the expectation. The film image is used as the digital human face reference, and the target mouth image sequence is processed based on the preset second neural network in the second stage, so as to ensure that the lip movement not only meets the voice driving, but also reaches the natural coordination with the original facial micro-expression. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0047] Figure 1 An architecture schematic diagram of a digital human lip video generation provided by an embodiment of the present application;

[0048] Figure 2 A flow schematic diagram of a digital human lip video generation method provided by an embodiment of the present application;

[0049] Figure 3 A first stage flow schematic diagram of a digital human lip video generation provided by an embodiment of the present application;

[0050] Figure 4 A synchronization discrimination training schematic diagram of a preset first neural network in an application scenario provided by an embodiment of the present application;

[0051] Figure 5 A second stage flow schematic diagram of a digital human lip video generation provided by an embodiment of the present application;

[0052] Figure 6 A film image screening flow schematic diagram provided by an embodiment of the present application;

[0053] Figure 7 A structure schematic diagram of a digital human lip video generation device provided by an embodiment of the present application;

[0054] Figure 8A structural schematic diagram of a digital person lip video generation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] An embodiment of the present application provides a digital person lip video generation method and device.

[0056] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and explained below in conjunction with the drawings and embodiments.

[0057] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the general meaning understood by a person with ordinary skill in the art to which the present application belongs. In the present application, "one", "a", "an", "the", "these" and the like similar words do not represent a quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof have the purpose of covering non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and the like similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" refers to two or more. The association between the associated objects is described as "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order for the objects.

[0058] The digital person lip video generation method provided by an embodiment of the present application can be applied in an application environment as shown in Figure 1 The digital person lip video generation method provided by an embodiment of the present application can be applied in an application environment as shown in Figure 1As shown, the application environment can include: a client terminal 101, a communication network 102, a server 103, a database 104. The communication network 102 can serve as a channel for data transmission, providing a communication link for communication between the client terminal 101 and the server 103, so that after the client terminal 101 inputs the target voice and the reference video, it can be transmitted to the server 103 based on the communication network 102, and the digital human lip video generated by the server 103 can also be fed back to the client terminal 101 for display. The server 103 is a server that can provide various services, and in order to improve processing efficiency, the server 103 is connected with multiple servers 103 to form a master-slave server cluster, so that after receiving the information uploaded by the client terminal 101 based on the communication network 102, it can be sent to the corresponding server 103 in the server cluster to input the audio features of the target voice and the mouth features in any frame of the original image into the preset first neural network, and realize the first stage processing. After the first stage processing is completed, the server 103 will process in the second stage, first screen the negative film image in the reference video, obtain the mouth mask image after masking the negative film image, and then input the mouth mask image and the mouth features obtained in the first stage into the preset second neural network, realize the second stage processing to obtain the digital human lip video image. Through the communication network 102, the digital human lip video image is fed back to the client terminal, so as to generate the digital human lip video through the continuous frame digital human lip video image.

[0059] Among them, the client terminal 101 can be a computer, a mobile phone and other hardware that can display digital people, etc., or it can be software installed in the above-mentioned electronic devices. The communication network 102 can include various connection types, such as wired, wireless communication link or optical cable, etc.

[0060] Among them, the database 104 is connected with each server 103 respectively, and can be used to store and manage the data of the preset first neural network and the preset second neural network. The database 104 can be integrated on the server, or placed on the cloud or other network servers.

[0061] As shown Figure 2 The embodiment of the present application provides a flow diagram of a digital human lip video generation method. Figure 2The method shown can be applied to various application scenarios requiring virtual digital human interaction, including but not limited to virtual digital human live broadcast and real-time interaction scenarios, such as improving the immersion and realism of virtual anchors or AI customer service live broadcast by generating high-quality digital human video; film and television animation and game character dubbing scenarios, such as reducing the cost of traditional frame animation or manual adjustment by directly driving the character's mouth shape with audio; remote conferencing and virtual assistant scenarios, such as transmitting only voice signals in a low-bandwidth environment, and the receiving end can save network resources with high-fidelity digital human images; medical rehabilitation and auxiliary communication scenarios, such as helping patients correct pronunciation through visual feedback by generating high-precision mouth shape digital humans. The digital human mouth shape video generation method includes the following steps: S201-S203.

[0062] S201: In the case where the audio features corresponding to the target speech and the mouth features corresponding to any one frame of video image in the reference video are obtained, they are input into the preset first neural network to output a target mouth image sequence.

[0063] In the current field of speech-driven digital human generation, it is common to extract audio features and mask images, and fuse them in the latent space to generate the final image. However, this requires simultaneous completion of audio matching face image generation and image stitching tasks, resulting in high overall complexity. Moreover, existing neural network models such as SyncNet are prone to interference from non-mouth regions such as facial expressions and head poses when discriminating the synchronization of speech and face images, making it difficult for subsequent models to accurately learn the correspondence between mouth shape and speech, affecting the final discrimination accuracy. Therefore, in order to reduce the complexity of model processing and improve the resolution of generated images and discrimination accuracy, the digital human mouth shape video generation process is decoupled into two stages in the embodiments of the present application. As shown in Figure 3 As shown in the first stage flow diagram, in the first stage, after obtaining the audio features corresponding to the target speech and the mouth features corresponding to any one frame of video image in the reference video, the obtained audio features and mouth features are input into the preset first neural network for processing. After obtaining the audio features and mouth features, in order to generate a mouth image sequence that is time-synchronous with the target speech and conforms to the mouth shape of the digital human, as shown in Figure 2As shown, in order to solve the existing underwater space fusion needs to complete audio-mouth matching and image fusion two tasks at the same time, resulting in easy optimization conflict makes the image resolution is low, difficult to meet the demand of high-definition digital human problem. In the first stage of the process of the present application embodiment will be audio features and mouth features input pre-set first neural network, obtain the matching mouth image sequence of target speech. That is, the traditional audio-mouth matching and image fusion are decoupled in the embodiment of the present application, only audio and mouth matching processing in the first stage, so as to input the audio features and mouth features into the pre-set first neural network, obtain the matching mouth image sequence of target speech, avoid task conflict, at the same time, because the consumption of computing resources is reduced, therefore, the details of the mouth image sequence are optimized, and the resolution of the image is improved.

[0064] In some embodiments, before the input pre-set first neural network in step S201 described above to output the target mouth image sequence, the method further comprises:

[0065] S301: input the target speech into the pre-set audio encoder to output the audio features corresponding to the target speech;

[0066] S302: extract any one frame of video image in the reference video to obtain the mouth region image corresponding to the pre-set mouth region of the any one frame of video image;

[0067] S303: input the mouth region image into the pre-set image encoder to extract the mouth features corresponding to the mouth region image.

[0068] That is, by Figure 3 It can be seen that the audio feature extraction process in this process is to extract the high-dimensional audio features of the target speech by the pre-set audio encoder obtained by pre-training. It can be understood that the high-dimensional audio features can represent the time sequence content information of the speech. The extraction process of the mouth features is to input the mouth image of the mouth region of any one frame of video image in the reference video into the pre-set image encoder obtained by pre-training to extract, so as to obtain the mouth features corresponding to any one frame of video image in the reference video.

[0069] In one embodiment, the preset image encoder is composed of a multi-layer convolutional neural network to output low spatial resolution and high-dimensional mouth features. The multi-layer convolutional neural network can convert local details of the mouth image, such as lip contour, interdental space, and mouth corner changes, into high-dimensional vectors, providing a more abundant feature basis for subsequent speech and mouth matching. In addition, the mouth feature extraction is only for the mouth region image, and the low spatial resolution reduces the mixing of irrelevant information such as facial expressions and head poses, and eliminates the need to process large pixel data of high-resolution images when fusing with audio features, significantly reducing the computational load of latent space fusion and improving the model inference speed. In this process, the mouth features corresponding to any frame of video image in the reference video are obtained, the extraction of full-face features in the traditional method is converted to the extraction of mouth features in the mouth region, the input of the preset first neural network is limited to the image features of the mouth image, so that the subsequent preset first neural network model only needs to focus on the corresponding relationship between the mouth features and the audio, without the need to filter the mouth information from the complete face, which reduces the learning cost of invalid features and effectively avoids the interference of non-mouth regions such as eyebrow movement and head rotation when the preset first neural network calculates the audio-visual synchronization, reducing the problem of misjudging asynchronous frames as synchronous frames. In addition, it should be noted that the audio features and mouth features obtained here are converted into a unified feature vector format through the encoder, so that the subsequent stage does not need to process the original audio waveform or image pixels, but only needs to fuse the standardized features in the latent space, reducing the cost of adapting different modal data formats and indirectly improving the efficiency and stability of the generated image.

[0070] In some embodiments, the mouth region image corresponding to the preset mouth region of any frame of video image can be obtained by cropping the mouth region from any frame of video image, and the specific implementation process is as follows:

[0071] The face image data corresponding to different expressions and postures of the current digital human character is collected, and a position sequence corresponding to the recognition of the mouth key point in each face image data is obtained. The position sequence corresponding to each face image data is determined, and the most edge position corresponding to each preset edge direction is determined. The region range of the mouth region in the face image is determined according to the most edge position and the preset boundary expansion value, so as to fix the region range as the preset mouth region. For example: in a certain scene, the lip key points such as the corners of the mouth and the lip peaks of the digital human can be obtained. If four preset edge directions are defined as left, right, up and down, the initial mouth region frame can be generated by determining the outermost key points in each direction. The initial mouth region frame is expanded based on the preset boundary expansion value, so as to obtain the region range of the mouth region in the face image, thereby fixing the region range as the preset mouth region. It should be noted that the preset boundary expansion value is determined according to the expansion ratio corresponding to each preset edge direction.

[0072] In this process, the mouth key points are extracted from the face image data exclusive to the current digital human character, rather than relying on a general face model, which can accurately match the facial structure of the digital human. Then, the outermost key points of the preset edge direction are used to generate an initial region frame, which directly frames the core range of the mouth movement, thereby avoiding the inclusion of too much cheek, chin and other non-mouth regions. In addition, since the initial region frame is generated based on key points only, the initial region frame may be expanded by the preset boundary expansion value to cover the necessary range around the lips, thereby ensuring the integrity of the mouth region image. By fixing the region range of the mouth region, subsequent reference video frames can be cropped without the need to detect key points or adjust the region, thereby achieving fast cropping while ensuring the stability of the mouth region.

[0073] In some embodiments, the audio feature and the mouth feature in step S201 are input into the preset first neural network to output a target mouth image sequence, specifically including:

[0074] S401: enhancing the mouth feature through the self-attention mechanism of the preset first neural network to obtain an enhanced mouth feature;

[0075] S402: fusing the enhanced mouth feature and the audio feature through the cross-attention mechanism of the preset first neural network to output a mixed feature;

[0076] S403: reconstructing the mixed feature through a preset multi-layer image decoder to obtain a target mouth image sequence matched with the target voice.

[0077] Traditional CNNs or RNNs struggle to model long-distance temporal relationships, leading to abrupt or disjointed mouth movements during rapid speech, particularly in handling sustained phonemes such as long vowels, thus hindering the maintenance of stable mouth shapes. To overcome this problem, this embodiment employs a pre-configured self-attention mechanism in a first neural network to capture long-term dependencies in mouth features, thereby outputting enhanced mouth features. By dynamically analyzing the mouth feature relationships across all frames through a self-attention layer, the long-term correspondence between phoneme duration and mouth movement is accurately captured, resulting in a smoother and more natural mouth shape sequence in the temporal dimension, eliminating unreasonable instantaneous jumps.

[0078] Then, through the cross-attention mechanism of the pre-configured first neural network, enhanced mouth features are fused with audio features to output hybrid features. This allows the model to focus on speech-driven mouth shape changes, eliminating interference from irrelevant facial movements and improving the matching accuracy between audio and mouth shape. A pre-configured multi-layer image decoder is then used to reconstruct the hybrid features, obtaining a mouth image sequence that matches the target speech. Specifically, in the process of reconstructing the hybrid features using the pre-configured multi-layer image decoder to obtain the mouth image sequence, the hybrid features are first upsampled and reconstructed based on the pre-configured multi-layer image decoder to obtain multi-scale mouth features at each layer. Then, through gated skip connections, the multi-scale mouth features at each layer are dynamically fused with the features of the corresponding layers of the pre-configured image encoder to obtain a mouth image sequence that matches the target speech, thus ensuring that the generated mouth image sequence is consistent with the image style of the original reference video.

[0079] In a certain application scenario, the process of enhancing mouth features involves using a self-attention layer to enhance these features and capture long-term dependencies within them. The formula for calculating self-attention is shown below:

[0080] The query matrix Q, the key matrix K, and the value matrix V are all included. This represents the transpose of the key matrix. The query matrix and the transpose of the key matrix are then multiplied together. The vector dimension of the key matrix is ​​represented by the Softmax function, which is used to calculate the size of attention required at each position in the original sequence. The final output is the enhanced mouth feature after attention enhancement.

[0081] Furthermore, in some embodiments, the enhanced mouth features are fused with the audio features through the cross-attention mechanism of the pre-set first neural network to output a mixed feature, specifically including:

[0082] S501: Transform the enhanced mouth features into a key matrix and a value matrix, and transform the audio features into a query matrix;

[0083] S502: Calculate attention weights based on the query matrix and the key matrix to obtain a weight matrix;

[0084] S503: Aggregate the value matrix using the weight matrix to output mixed features.

[0085] Traditional methods, which simply concatenate audio and visual features, struggle to distinguish the master-slave relationship between the two modalities, leading to weakened speech-driven effects and lip-syncing that is easily dominated by visual features, deviating from the speech content. Therefore, to obtain hybrid features that integrate audio-driven information and reference the visual context of the mouth, enabling lip-syncing to better align with the speech input, this specification's embodiments transform enhanced mouth features into key and value matrices to provide visual context. Audio features are then transformed into a query matrix to represent the speech-driven visual feature requirements. Attention weights are calculated based on the query and key matrices to obtain a weight matrix. The value matrix is ​​then aggregated using the weight matrix to output the hybrid features. This process, by aggregating the value matrix transformed from enhanced mouth features using attention weights, achieves dynamic recombination of visual features based on attention weights. This preserves the speaker's personalized mouth features while injecting speech-driven motion trajectories, improving the lip-syncing accuracy of the digital human.

[0086] Furthermore, in some embodiments, after inputting audio features and mouth features into a preset first neural network to output a target mouth image sequence, the method further includes:

[0087] S601: Extract the current mouth features of the target mouth image sequence through a preset image encoder to obtain the similarity between the current mouth features and the audio features;

[0088] S602: Based on the similarity, determine the synchronization discrimination loss between the audio feature and the current mouth feature, and iteratively adjust the preset first neural network using the synchronization discrimination loss.

[0089] Currently, neural network models used as discriminators are easily affected by non-mouth regions when synchronizing speech and facial images, making it difficult for the model to accurately learn the correspondence between lip movements and speech, thus affecting the final discrimination accuracy. Furthermore, neural network model training requires simultaneous optimization of multiple losses, such as reconstruction loss, SyncNet loss, and adversarial loss, and their gradient conflicts lead to convergence difficulties. Therefore, to overcome this problem and improve lip-sync accuracy, after obtaining the target lip image sequence matching the target speech based on the above steps S401-S403, as follows... Figure 4The method involves extracting current mouth features from a target mouth image sequence using a pre-set image encoder to obtain the similarity between the current mouth features and audio features. This similarity is then used as the synchronization discrimination loss between the audio features and the current mouth features, and the pre-set first neural network is iteratively adjusted based on this synchronization discrimination loss. In one feasible embodiment, this similarity is cosine similarity. It is understood that a higher similarity score indicates greater synchronization. Iteratively adjusting the pre-set first neural network using this synchronization discrimination loss enables the model to learn more accurate mouth synchronization in the first stage. This process establishes a quantifiable synchronization feedback mechanism by calculating the similarity between the generated mouth features and the input audio in real time. Compared to traditional fixed-threshold detection methods, this method can capture fine-grained changes in the audio-visual relationship, providing precise guidance for model optimization. Simultaneously, it avoids the convergence difficulties caused by gradient conflicts in optimizing multiple losses simultaneously during traditional training, thus improving the discrimination accuracy of the subsequent pre-set first neural network model.

[0090] S202: By using a preset film selection strategy, the reference video is filtered to obtain film images.

[0091] After completing the audio-lip-sync matching in the first stage based on the above steps S201, the following steps are required: Figure 5 The second stage of the image fusion process is shown. Existing methods rely on the input film during the fusion stage. Due to the lack of a dynamic film selection mechanism, reference frames are typically selected randomly or in a fixed manner, resulting in the film not adapting to different pronunciation states. Furthermore, if the lip shape in the film does not match the target lip shape—for example, the target needs to close their mouth but the film shows an open mouth—the fused lip shape will not close correctly. Even if the speech requires closing the mouth, the generated lip shape will still be slightly open due to the influence of the film. The lip shape state in the film, such as smiling or upturned corners of the mouth, may also interfere with the generation of the target lip shape. Therefore, to solve this film interference problem and the forced closure problem, this embodiment uses a preset film selection strategy to screen the video images of the reference video to obtain film images. This process, through the preset film selection strategy, dynamically filters matching films, ensuring that the lip shape in the film is consistent with the physiological state of the target speech.

[0092] In one feasible embodiment, the preset film selection strategy can combine the user ID corresponding to the target speech with the voice emotion tag corresponding to the target speech to further filter personalized films corresponding to the user ID and the voice emotion tag, thereby flexibly adapting to the needs of different styles of lip-sync generation and improving the scene adaptability of digital human lip-sync video generation.

[0093] In some of these embodiments, such as Figure 6As shown in step S203 above, the reference video is filtered using a preset film selection strategy to obtain film images, specifically including:

[0094] S701: Determine the speech stage corresponding to the target speech; wherein, the speech stage includes: normal speech stage and closed-mouth stage;

[0095] S702: Obtain the reference mouth opening sequence of the reference video and the target mouth opening sequence of the target mouth image sequence, so as to dynamically match the reference mouth opening sequence and the target mouth opening sequence to obtain a matching result;

[0096] S703: Combining the voice stage with the matching result, perform negative screening on the reference video to obtain negative images.

[0097] To accurately distinguish between normal pronunciation and the closed-mouth phase, such as the brief closure after the plosive / p / , this embodiment first determines the speech phase corresponding to the target speech. The speech phase includes a normal speech phase and a closed-mouth phase. Then, the mouth opening and closing degree of each frame of the reference video image is quantized to obtain a reference mouth opening and closing degree sequence. Simultaneously, the mouth opening and closing degree of each frame of the target mouth image sequence is quantized to obtain a target mouth opening and closing degree sequence. A preset dynamic matching method is then used to align and match the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result. Furthermore, to reliably bypass complex latent space control through forced mouth closure and accurately achieve the closed-mouth state with simple post-processing, ensuring the mouth shape matches expectations, this embodiment combines the speech phase determined in the above process with the matching result to filter each frame of the reference video, thereby obtaining the film image.

[0098] In one feasible embodiment, determining the speech stage corresponding to the target speech can be achieved by: determining the speech stage corresponding to the target speech based on the speech duration and audio energy. It is understood that using the speech duration and audio energy corresponding to the target speech can adapt to different speech rates and accents, thereby avoiding lip-sync errors and reducing the unnaturalness of audio-visual asynchrony.

[0099] Furthermore, in some embodiments, the reference mouth opening sequence and the target mouth opening sequence are dynamically matched to perform film screening on the reference video based on the matching result, thereby obtaining film images. Specifically, this includes:

[0100] S801: Obtain the absolute opening degree difference between each frame pair between the reference mouth opening degree sequence and the target mouth opening degree sequence;

[0101] S802: Obtain the cumulative distance corresponding to each frame pair by the difference in absolute opening degree of each frame pair, and generate a cumulative distance matrix;

[0102] S803: Determine the best matching path as the matching result using the cumulative distance matrix.

[0103] To significantly reduce the negative impact of mouth opening information on the film when the target speech processing is in a closed-mouth state by selecting a film with suitable mouth opening degree, this improves the ability to generate accurate closed-mouth images when mouth closure is required, solves the problem of digital humans not being able to close their mouths, and improves visual effects. In the embodiments of this specification, the absolute difference in mouth opening degree between each frame pair between the obtained reference mouth opening degree sequence and the target mouth opening degree sequence is considered. It should be noted that before this process, the mouth opening degree of the mouth shape sequence in each frame of the reference video and the mouth opening degree of each frame of the target mouth image sequence are pre-calculated. The calculation method is as follows: Let the coordinates of the N key points of the mouth in each frame of the mouth key point detection model be set P, and the mouth opening degree metric (O) be defined as follows:

[0104] ,in and These represent k pairs of key points corresponding to the upper and lower lips, The distance is Euclidean. Then, the absolute difference in openness between all frame pairs in the two sequences is calculated:

[0105] ,in, For absolute opening / closing difference, T is the length of the reference video frame and S is the length of the generated video frame.

[0106] Then, by using the difference in absolute opening degree between each frame pair, the cumulative distance corresponding to each frame pair is obtained, generating a cumulative distance matrix (dynamic programming table). Cumulative distance matrix initialization. The remaining elements It can be obtained from the recursive formula:

[0107] ,in, The cumulative distance matrix value is in the (i-1)th row and jth column. The cumulative distance matrix value is in the i-th row and j-1-th column. Let be the cumulative distance matrix value in row i-1 and column j-1. Then, backtrack from (T,S) to (1,1) of the distance matrix, select the path W that minimizes the cumulative distance as the best matching path, and determine the best matching path as the matching result.

[0108] This process quantifies the matching degree of a single frame by the difference in absolute opening degree and evaluates the matching degree of the entire sequence by combining the cumulative distance matrix. This avoids the temporal discontinuity problem caused by relying solely on single-frame matching in existing methods, ensuring that the selected film images closely match the mouth opening and closing requirements of the target speech. Furthermore, the optimal matching path is selected based on the minimum overall cumulative distance, which minimizes the overall deviation between the reference video and the target speech in the opening degree sequence. This provides highly matched film images for subsequent lip shape generation, effectively mitigating the mouth closure problem caused by the mismatch between the film lip shape state and the target state, thus improving the robustness of lip shape control.

[0109] Furthermore, in some embodiments, by combining the voice stage with the matching result, the reference video is subjected to negative screening to obtain a negative image, specifically including:

[0110] S901: If it is determined that the speech stage is a normal speech stage, then the reference video is screened according to the matching result to obtain an initial image.

[0111] S902: If it is determined that the speech stage is the closed-mouth stage, then the pre-stored closed-mouth image is obtained as the initial film image;

[0112] S903: Cover the mouth area of ​​the initial film image to obtain the film image.

[0113] During this screening process, if the speech stage is determined to be a normal speech stage, then the reference video is screened according to the best matching path corresponding to the matching result to obtain the negative image. That is, for the j-th frame of the generated sequence, the index of its corresponding best reference frame is... for:

[0114] Where argmin is the index number corresponding to the minimum value, and the constraint condition is... This indicates that only reference frame i in the valid alignment path W determined by the algorithm is considered. Based on the above process, the reference video segment that best matches the opening and closing change pattern of the current generation target is obtained as the initial film image for generating the mouth shape image at that moment.

[0115] If the speech stage is determined to be a closed-mouth stage, such as the beginning, end, or silence segment of the target speech, a pre-stored closed-mouth image is acquired as the negative image to achieve forced closure. Then, the mouth area of ​​this initial negative image is masked to obtain the original negative image. By masking the mouth area of ​​the negative image using a mask, the interference of the original mouth shape on the generated result is eliminated while fully preserving the key facial context, which helps in the subsequent generation of a natural-looking digital human.

[0116] In one feasible embodiment, the mouth region of the initial film image is masked to obtain the film image. This can be achieved as follows: the mouth region of the initial film image is masked using a binary mask to obtain a mouth mask image of the initial film image as the film image. A binary mask is an image containing only two pixel values, "0" and "1". In this process, the "1" region of the binary mask corresponds to the mouth region of the film image, thus masking the mouth region, while the "0" region corresponds to the non-mouth region of the film image, thus preserving the non-mouth region. After masking the film image using a binary mask, the final mouth mask image only retains image information in the non-mouth regions, while the mouth region... Figure 5 The image shown is blocked.

[0117] S204: Input the negative image and the target mouth image sequence into a preset second neural network for fusion processing and output a digital mouth shape video image.

[0118] like Figure 5 As shown, in this embodiment, the film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital mouth-printing video image. Through the above process, the mouth region of the initial film image is masked, eliminating the interference of the original mouth shape on the generated result while fully preserving the key facial context. Then, the voice-driven target mouth image sequence and the film image are input into the second neural network for fusion processing. This achieves dynamic adjustment of the fusion weights between the film context and the generated mouth shape, ensuring that the pronunciation action naturally integrates into the original facial expression, improving the overall image quality, and obtaining a high-resolution digital mouth-printing video.

[0119] In some embodiments, step S203 above, which involves inputting the film image and the target mouth image sequence into a preset second neural network for fusion processing to output a digital mouth-printing video image, specifically includes:

[0120] S1001: Perform channel-by-channel stitching of the target mouth image sequence with the film image to obtain the total input tensor;

[0121] S1002: The total input tensor is input into the preset second neural network for fusion processing, and the digital lip-sync video image is output so as to generate a lip-sync video through consecutive frames of digital lip-sync images.

[0122] Since the target mouth image sequence obtained through the above steps contains the core content of dynamic lip movements, while the film image contains other features from non-mouth areas, it is necessary to perform channel-wise concatenation of the target mouth image sequence and the mouth mask image to obtain the total input tensor. It should be noted that this concatenation process is performed along the channel dimension, obtaining a composite tensor containing spatiotemporal information as the total input tensor. This total input tensor is then input into a pre-set second neural network. After encoding and decoding processing by the pre-set second neural network, while repairing the mask area, it adaptively fuses the new mouth shape with the facial features of the film image, outputting a seamlessly connected high-resolution digital lip-sync video image. This allows for the generation of lip-sync video from consecutive frames of digital lip-sync images, ensuring that the mouth movements are both speech-driven and naturally coordinated with the original facial micro-expressions.

[0123] In one embodiment of this application, the second stage can be implemented based on the following process: First, the input data includes a target mouth image sequence and a film image. The target mouth image sequence is input as five frames via a sliding window, including frame t and the two frames before and after it, to reduce inter-frame jitter. Additionally, the film image is the original head image of frame t, whose mouth region is masked using a binary mask. Then, the input data, i.e., the five mouth images, is concatenated with the masked head image channels to form a total input tensor. A U-Net-like encoder-decoder structure is then employed, using skip connections to preserve spatial details during processing. A super-resolution module is integrated end-to-end at the decoder end to jointly optimize repair and resolution enhancement, ultimately generating a repaired high-resolution head image of frame t, where the mouth blends naturally with the surrounding area.

[0124] This application also provides a digital human lip-sync video generation device, such as... Figure 7 As shown, a digital human lip-syncing video generation device includes:

[0125] The acquisition unit 701 is used to input the audio features corresponding to the target speech and the mouth features corresponding to any frame of video image in the reference video into a preset first neural network to output a target mouth image sequence.

[0126] The filtering unit 702 is used to filter the reference video using a preset negative selection strategy to obtain negative images;

[0127] The output unit 703 is used to input the film image and the target mouth image sequence into a preset second neural network for fusion processing and output a digital mouth shape video image.

[0128] In some embodiments, the input is given to a preset first neural network to output a target mouth image sequence, specifically including:

[0129] The mouth features are enhanced by the self-attention mechanism of the pre-set first neural network to obtain enhanced mouth features;

[0130] The enhanced mouth features and the audio features are fused using the cross-attention mechanism of the pre-set first neural network to output a mixed feature;

[0131] The mixed features are reconstructed by a pre-set multi-layer image decoder to obtain a target mouth image sequence that matches the target speech.

[0132] In some embodiments, the enhanced mouth features and the audio features are fused through the cross-attention mechanism of the preset first neural network to output a mixed feature, specifically including:

[0133] The enhanced mouth features are transformed into a key matrix and a value matrix, and the audio features are transformed into a query matrix;

[0134] An attention weight matrix is ​​obtained by calculating the attention weights based on the query matrix and the key matrix.

[0135] The value matrix is ​​aggregated using the weight matrix to output mixed features.

[0136] In some embodiments, the reference video is filtered using a preset film selection strategy to obtain film images, specifically including:

[0137] Determine the speech stage corresponding to the target speech; wherein, the speech stage includes: normal speech stage and closed-mouth stage;

[0138] The reference mouth opening sequence of the reference video and the target mouth opening sequence of the target mouth image sequence are obtained, and the reference mouth opening sequence and the target mouth opening sequence are dynamically matched to obtain a matching result.

[0139] By combining the voice stage with the matching result, the reference video is screened to obtain the negative image.

[0140] In some embodiments, the reference mouth opening sequence and the target mouth opening sequence are dynamically matched to obtain a matching result, specifically including:

[0141] Obtain the absolute mouth opening difference between each frame pair between the reference mouth opening sequence and the target mouth opening sequence;

[0142] By obtaining the difference in absolute opening degree of each frame pair, the cumulative distance corresponding to each frame pair is obtained, and a cumulative distance matrix is ​​generated.

[0143] The optimal matching path is determined as the matching result using the cumulative distance matrix.

[0144] In some embodiments, combining the speech stage with the matching result, the reference video is subjected to negative screening to obtain a negative image, specifically including:

[0145] If the speech stage is determined to be a normal speech stage, the reference video is screened according to the matching results to obtain an initial image.

[0146] If the speech stage is determined to be a closed-mouth stage, then a pre-stored closed-mouth image is obtained as the initial negative image.

[0147] The mouth area of ​​the initial film image is covered to obtain the film image.

[0148] In some embodiments, the film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital mouth-texture video image, specifically including:

[0149] The target mouth image sequence is stitched together with the film image through channels to obtain the total input tensor;

[0150] The total input tensor is input into a preset second neural network for fusion processing, and the output is a digital lip-sync video image, so as to generate a lip-sync video from consecutive frames of digital lip-sync images.

[0151] In some embodiments, after inputting the image into a preset first neural network to output a sequence of target mouth images, the method further includes:

[0152] The current mouth features of the target mouth image sequence are extracted by a preset image encoder, and the similarity between the current mouth features and the audio features is obtained.

[0153] Based on the similarity, a synchronization discrimination loss between the audio feature and the current mouth feature is determined, and the preset first neural network is iteratively adjusted using the synchronization discrimination loss.

[0154] In some embodiments, before inputting it into a pre-configured first neural network to output a target mouth image sequence, the method further includes:

[0155] The target speech is input into a preset audio encoder, and the audio features corresponding to the target speech are output.

[0156] Extract any frame of video image from the reference video, and obtain the mouth region image corresponding to the preset mouth region of the arbitrary frame of video image;

[0157] The mouth region image is input into a preset image encoder to extract the mouth features corresponding to the mouth region image.

[0158] This specification also provides an embodiment of a digital human lip-sync video generation device, such as... Figure 8 As shown, a digital human lip-syncing video generation device includes:

[0159] At least one processor; and,

[0160] A memory communicatively connected to the at least one processor; wherein,

[0161] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to: perform any of the above-described methods for generating lip-sync videos of digital humans.

[0162] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0163] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] The above description is merely one or more embodiments of this application and is not intended to limit this application. For those skilled in the art, various modifications and variations can be made to one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this application should be included within the scope of the claims of this application.

Claims

1. A method for generating lip-sync videos of digital humans, characterized in that, The method includes: After obtaining the audio features corresponding to the target speech and the mouth features corresponding to any frame of video image in the reference video, they are input into a preset first neural network to output a sequence of target mouth images. By using a preset negative selection strategy, the reference video is filtered to obtain negative images; The film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital mouth shape video image; Before inputting it into a pre-configured first neural network to output a sequence of target mouth images, the process also includes: Extract any frame of video image from the reference video, and obtain the mouth region image corresponding to the preset mouth region of the arbitrary frame of video image; specifically, collect facial image data of different expressions and postures corresponding to the current digital human character, and obtain the position sequence of the key points of the mouth in each facial image data; determine the position sequence and preset edge direction in each facial image data, and determine the outermost edge position corresponding to each preset edge direction; determine the area range of the mouth region in the facial image based on each outermost edge position and preset boundary expansion value, thereby fixing this area range as the preset mouth region; wherein, the preset boundary expansion value is determined according to the expansion ratio corresponding to each preset edge direction; By using a preset film selection strategy, the reference video is filtered to obtain film images, specifically including: The speech stage corresponding to the target speech is determined; wherein the speech stage is determined based on the speech time and audio energy corresponding to the target speech, including: normal speech stage and closed-mouth stage; The reference mouth opening sequence of the reference video and the target mouth opening sequence of the target mouth image sequence are obtained, and the reference mouth opening sequence and the target mouth opening sequence are dynamically matched to obtain a matching result. By combining the voice stage with the matching result, the reference video is screened to obtain the negative image; Specifically, combining the voice stage with the matching result, the reference video is subjected to negative screening to obtain negative images, including: If the speech stage is determined to be a normal speech stage, the reference video is screened according to the matching results to obtain an initial image. If the speech stage is determined to be a closed-mouth stage, then a pre-stored closed-mouth image is obtained as the initial negative image. The mouth area of ​​the initial film image is covered to obtain the film image.

2. The method for generating lip-sync video of a digital human according to claim 1, characterized in that, The image is input into a pre-set first neural network to output a sequence of target mouth images, specifically including: The mouth features are enhanced by the self-attention mechanism of the pre-set first neural network to obtain enhanced mouth features; The enhanced mouth features and the audio features are fused using the cross-attention mechanism of the pre-set first neural network to output a mixed feature; The mixed features are reconstructed by a pre-set multi-layer image decoder to obtain a target mouth image sequence that matches the target speech.

3. The method for generating lip-sync video of a digital human according to claim 2, characterized in that, The enhanced mouth features and the audio features are fused using the cross-attention mechanism of the pre-set first neural network to output a mixed feature, specifically including: The enhanced mouth features are transformed into a key matrix and a value matrix, and the audio features are transformed into a query matrix; An attention weight matrix is ​​obtained by calculating the attention weights based on the query matrix and the key matrix. The value matrix is ​​aggregated using the weight matrix to output mixed features.

4. The method for generating lip-sync video of a digital human according to claim 1, characterized in that, The reference mouth opening sequence and the target mouth opening sequence are dynamically matched to obtain a matching result, specifically including: Obtain the absolute mouth opening difference between each frame pair between the reference mouth opening sequence and the target mouth opening sequence; By obtaining the difference in absolute opening degree of each frame pair, the cumulative distance corresponding to each frame pair is obtained, and a cumulative distance matrix is ​​generated. The optimal matching path is determined as the matching result using the cumulative distance matrix.

5. The method for generating lip-sync video of a digital human according to claim 1, characterized in that, The film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital mouth-texture video image, specifically including: The target mouth image sequence is stitched together with the film image through channels to obtain the total input tensor; The total input tensor is input into a preset second neural network for fusion processing, and the output is a digital lip-sync video image, so as to generate a lip-sync video from consecutive frames of digital lip-sync images.

6. The method for generating lip-sync video of a digital human according to claim 1, characterized in that, After inputting it into a pre-set first neural network to output a sequence of target mouth images, the method further includes: The current mouth features of the target mouth image sequence are extracted by a preset image encoder, and the similarity between the current mouth features and the audio features is obtained. Based on the similarity, a synchronization discrimination loss between the audio feature and the current mouth feature is determined, and the preset first neural network is iteratively adjusted using the synchronization discrimination loss.

7. The method for generating lip-sync video of a digital human according to claim 1, characterized in that, Before inputting it into a pre-set first neural network to output a target mouth image sequence, the method further includes: The target speech is input into a preset audio encoder, and the audio features corresponding to the target speech are output. The mouth region image is input into a preset image encoder to extract the mouth features corresponding to the mouth region image.

8. A digital human lip-sync video generation device, characterized in that, The device includes: The acquisition unit is used to input the audio features corresponding to the target speech and the mouth features corresponding to any frame of video image in the reference video into a preset first neural network to output a target mouth image sequence. The filtering unit is used to filter the reference video using a preset film selection strategy to obtain film images; The output unit is used to input the film image and the target mouth image sequence into a preset second neural network for fusion processing and output a digital mouth shape video image; Before inputting it into a pre-configured first neural network to output a sequence of target mouth images, the process also includes: Extract any frame of video image from the reference video, and obtain the mouth region image corresponding to the preset mouth region of the arbitrary frame of video image; specifically, collect facial image data of different expressions and postures corresponding to the current digital human character, and obtain the position sequence of the key points of the mouth in each facial image data; determine the position sequence and preset edge direction in each facial image data, and determine the outermost edge position corresponding to each preset edge direction; determine the area range of the mouth region in the facial image based on each outermost edge position and preset boundary expansion value, thereby fixing this area range as the preset mouth region; wherein, the preset boundary expansion value is determined according to the expansion ratio corresponding to each preset edge direction; By using a preset film selection strategy, the reference video is filtered to obtain film images, specifically including: The speech stage corresponding to the target speech is determined; wherein the speech stage is determined based on the speech time and audio energy corresponding to the target speech, including: normal speech stage and closed-mouth stage; The reference mouth opening sequence of the reference video and the target mouth opening sequence of the target mouth image sequence are obtained, and the reference mouth opening sequence and the target mouth opening sequence are dynamically matched to obtain a matching result. By combining the voice stage with the matching result, the reference video is screened to obtain the negative image; Specifically, combining the voice stage with the matching result, the reference video is subjected to negative screening to obtain negative images, including: If the speech stage is determined to be a normal speech stage, the reference video is screened according to the matching results to obtain an initial image. If the speech stage is determined to be a closed-mouth stage, then a pre-stored closed-mouth image is obtained as the initial negative image. The mouth area of ​​the initial film image is covered to obtain the film image.

Images