Method and device for generating mouth shape video of digital human

Abstract

The embodiment of the invention discloses a mouth shape video generation method and device of a digital human, relates to the technical field of digital human, and is used for solving the problems of low resolution of an existing generated image and low mouth shape synchronization precision. The method comprises the following steps: inputting audio features corresponding to target voice and mouth features corresponding to any frame of video image in a reference video into a preset first neural network to output a target mouth image sequence under the condition of obtaining the audio features corresponding to the target voice and the mouth features corresponding to any frame of video image in the reference video; screening the reference video through a preset negative film selection strategy to obtain a negative film image; and inputting the negative image and the target mouth image sequence into a preset second neural network for fusion processing, and outputting a digital population type video image.

Description

Technical Field

[0001] The present application relates to the field of digital human technology, and in particular to a method and device for generating a lip-sync video of a digital human. Background Art

[0002] The rise of digital human technology is reshaping human-computer interaction and content production in diverse fields, from entertainment, media, and education to healthcare and social interaction. As the demand for human-computer interaction continues to grow in these scenarios, certain applications place higher demands on the authenticity and naturalness of human-computer interaction. One of the core criteria for determining the authenticity and naturalness of digital humans is the natural synchronization of lip movements and speech. Lip-sync video generation, as a key link between a digital human's voice and visual expression, profoundly impacts the quality of digital human interaction and user experience.

[0003] Currently, in the field of voice-driven digital human generation, existing technologies are mainly based on audio-visual cross-modal generation methods, that is, by extracting voice features and fusing them with visual information, generating lip animations synchronized with the voice. However, the latent space fusion of existing audio-visual cross-modal generation methods requires the simultaneous optimization of multiple objectives, such as the need to complete the two tasks of audio-lip matching and image fusion at the same time, which easily leads to optimization conflicts, resulting in high model complexity and low resolution of generated images. In addition, the discriminator in the existing method is also susceptible to interference from non-lip area and film interference, making the digital human image output by the discriminator unable to adapt to the different pronunciation states of the target voice, resulting in low lip synchronization accuracy. Summary of the Invention

[0004] In response to the problems of high model complexity, low generated image resolution, and low lip synchronization accuracy in existing digital human generation methods based on audio-visual cross-modality, one or more embodiments of the present application provide a method and device for generating a digital human lip-sync video.

[0005] One or more embodiments of the present application adopt the following technical solutions: In a first aspect, one or more embodiments of the present application provide a method for generating a lip-syncing video of a digital human, the method comprising: When the audio features corresponding to the target speech and the mouth features corresponding to any frame of the reference video are obtained, they are input into a preset first neural network to output a target mouth image sequence; The reference video is screened by a preset negative film selection strategy to obtain a negative film image; The negative film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population video image.

[0006] Optionally, in one or more embodiments of the present application, the input is fed into a preset first neural network to output a target mouth image sequence, specifically including: enhancing the mouth feature by using the self-attention mechanism of the preset first neural network to obtain an enhanced mouth feature; fusing the enhanced mouth feature with the audio feature through a cross-attention mechanism of the preset first neural network to output a mixed feature; The mixed features are reconstructed by a preset multi-layer image decoder to obtain a target mouth image sequence that matches the target speech.

[0007] Optionally, in one or more embodiments of the present application, the enhanced mouth features are fused with the audio features through the cross attention mechanism of the preset first neural network to output a mixed feature, specifically including: transforming the enhanced mouth features into a key matrix and a value matrix, and transforming the audio features into a query matrix; Calculate attention weights based on the query matrix and the key matrix to obtain a weight matrix; The value matrix is ​​aggregated by the weight matrix to output a mixed feature.

[0008] Optionally, in one or more embodiments of the present application, the reference video is screened by a preset negative film selection strategy to obtain a negative film image, specifically including: Determining the speech stage corresponding to the target speech; wherein the speech stage includes: a normal speech stage and a closed mouth stage; Obtaining a reference mouth opening and closing degree sequence of the reference video and a target mouth opening and closing degree sequence of the target mouth image sequence, and dynamically matching the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result; The reference video is subjected to negative screening in combination with the speech stage and the matching result to obtain a negative image.

[0009] Optionally, in one or more embodiments of the present application, dynamically matching the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result specifically includes: Obtaining the absolute mouth opening degree difference of each frame pair between the reference mouth opening degree sequence and the target mouth opening degree sequence; The cumulative distance corresponding to each frame pair is obtained through the absolute opening and closing difference of each frame pair, and a cumulative distance matrix is ​​generated; The best matching path is determined as the matching result through the accumulated distance matrix.

[0010] Optionally, in one or more embodiments of the present application, combining the voice stage and the matching result, performing negative screening on the reference video to obtain a negative image specifically includes: If it is determined that the speech stage is a normal speech stage, performing negative screening on the reference video according to the matching result to obtain an initial negative image; If it is determined that the speech stage is a closed-mouth stage, obtaining a pre-stored closed-mouth image as an initial negative image; The mouth area of ​​the initial film image is masked to obtain the film image.

[0011] Optionally, in one or more embodiments of the present application, the negative film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population-type video image, specifically including: Perform channel splicing on the target mouth image sequence and the negative film image to obtain a total input tensor; The total input tensor is input into a preset second neural network for fusion processing, and a digital population-shaped video image is output, so as to generate a lip-syncing video through continuous frames of digital population-shaped images.

[0012] Optionally, in one or more embodiments of the present application, after inputting the preset first neural network to output a target mouth image sequence, the method further includes: Extracting current mouth features of the target mouth image sequence through a preset image encoder, and obtaining similarity between the current mouth features and the audio features; According to the similarity, the synchronization discrimination loss between the audio feature and the current mouth feature is determined, and the preset first neural network is iteratively adjusted according to the synchronization discrimination loss.

[0013] Optionally, in one or more embodiments of the present application, before inputting the preset first neural network to output the target mouth image sequence, the method further includes: Input the target speech into a preset audio encoder and output the audio features corresponding to the target speech; Extracting any one frame of video image from the reference video, and obtaining a mouth region image corresponding to a preset mouth region of the any one frame of video image; The mouth region image is input into a preset image encoder to extract mouth features corresponding to the mouth region image.

[0014] In a second aspect, one or more embodiments of the present application provide a device for generating a lip-sync video of a digital human, the device comprising: an acquisition unit configured to, upon acquiring audio features corresponding to the target speech and mouth features corresponding to any frame of a video image in the reference video, input them into a preset first neural network to output a target mouth image sequence; A screening unit, configured to screen the reference video using a preset negative film selection strategy to obtain a negative film image; The output unit is used to input the film image and the target mouth image sequence into a preset second neural network for fusion processing to output a digital population video image.

[0015] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: By decoupling the digital human lip-syncing video generation process, the optimization conflict problem encountered when simultaneously completing audio lip-syncing and image fusion is avoided, significantly reducing algorithm complexity and improving generation resolution. Mouth and audio features are input into a preset first neural network for first-stage processing. Compared with the traditional method of using complete facial images, this reduces the interference of non-mouth areas on lip synchronization and improves the model's synchronization discrimination accuracy. Film selection is performed using a preset film selection strategy, ensuring that the selected film images match the mouth opening and closing requirements of the target speech, ensuring that the lip shape meets the expected requirements. Using the film image as the digital human facial reference and the target mouth image sequence, the second-stage processing is performed based on a preset second neural network, ensuring that the lip movement is both consistent with the speech drive and naturally coordinated with the original facial micro-expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings: Figure 1 A schematic diagram of the architecture for generating a lip-sync video of a digital human provided in an embodiment of the present application; Figure 2 A flowchart of a method for generating a lip-sync video of a digital human provided in an embodiment of the present application; Figure 3 A schematic diagram of the first stage of the process of generating a lip-sync video of a digital human provided in an embodiment of the present application; Figure 4 A schematic diagram of synchronous discrimination training of a pre-set first neural network in an application scenario provided in an embodiment of the present application; Figure 5A schematic diagram of the second stage of the process of generating a lip-sync video of a digital human provided in an embodiment of the present application; Figure 6 A schematic diagram of a screening process for a film image provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a device for generating a lip-sync video of a digital human provided in an embodiment of the present application; Figure 8 A schematic structural diagram of a device for generating lip-syncing videos of digital humans provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The embodiments of the present application provide a method and apparatus for generating a lip-syncing video of a digital human.

[0018] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0019] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0020] The embodiment of the present application provides a method for generating a digital human lip shape video, which can be applied to Figure 1 In the application environment shown. Figure 1As shown, the application environment may include: client terminal 101, communication network 102, server 103, and database 104. Communication network 102 can serve as a data transmission channel, providing a communication link for communication between client terminal 101 and server 103. After the client terminal 101 records the target speech and reference video, they are transmitted to server 103 via communication network 102. At the same time, the digital human mouth-shaped video image processed by server 103 can also be fed back to the client terminal 101 for the digital human mouth-shaped video display. Server 103 is a server that can provide various services. To improve processing efficiency, server 103 is connected to multiple servers 103 to form a master-slave server cluster. After receiving the information uploaded by client terminal 101 via communication network 102, it can send it to the corresponding server 103 in the server cluster, which then inputs the audio features extracted from the target speech and the mouth features in any frame of the original image into the preset first neural network to achieve the first stage of processing. After the first stage of processing is complete, server 103 begins the second stage of processing by first screening the negative image in the reference video, masking the negative image to obtain a mouth mask image, and then inputting the mouth mask image and the mouth features obtained in the first stage into a pre-set second neural network to complete the second stage of processing and obtain a digital human-shaped video image. The digital human-shaped video image is fed back to the client terminal via communication network 102, thereby generating a digital human lip-syncing video from consecutive frames of the digital human-shaped video image.

[0021] The client terminal 101 may be a computer, mobile phone or other hardware capable of displaying a digital human, or software installed in the electronic devices listed above. The communication network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0022] The database 104 is connected to each server 103 and can be used to store and manage the data of the preset first neural network and the preset second neural network. The database 104 can be integrated on the server, or placed on the cloud or other network servers.

[0023] like Figure 2 The embodiment of the present application provides a flow chart of a method for generating a lip-sync video of a digital human. Figure 2The method shown can be applied to various application scenarios requiring virtual digital human interaction, including but not limited to virtual digital human live broadcast and real-time interaction scenarios. For example, by generating high-quality digital human mouth-shaped videos, the immersiveness and realism of virtual anchors or AI customer service live broadcasts can be enhanced; film and television animation and game character dubbing scenarios, such as directly driving character mouth shapes through audio, reducing the cost of traditional frame animation or manual adjustment; remote conferencing and virtual assistant scenarios, such as transmitting only voice signals in low-bandwidth environments, so that the receiving end can obtain a high-fidelity digital human image in the sound field, saving network resources; medical rehabilitation and assisted communication scenarios, such as by generating high-precision digital human mouth-shaped videos to help patients correct pronunciation through visual feedback. The digital human mouth-shaped video generation method includes the following steps: S201-S203.

[0024] S201: When the audio features corresponding to the target speech and the mouth features corresponding to any frame of video image in the reference video are obtained, they are input into a preset first neural network to output a target mouth image sequence.

[0025] In the current field of voice-driven digital human generation, it is common to extract audio features and mask images and fuse them in the latent space to generate the final image. However, this requires the simultaneous completion of audio matching, facial image generation, and image splicing tasks, resulting in high overall complexity. Moreover, existing neural network models such as SyncNet are easily interfered by non-mouth areas such as facial expressions and head postures when judging the synchronization between speech and facial images, making it difficult for subsequent models to accurately learn the correspondence between mouth shape and speech, affecting the final discrimination accuracy. Therefore, in order to reduce the complexity of model processing and improve the resolution and discrimination accuracy of generated images. In the embodiment of the present application, the digital human lip-sync video generation process is decoupled into two-stage generation. As Figure 3 As shown in the flowchart of the first stage, in the first stage, after obtaining the audio features corresponding to the target speech and the mouth features corresponding to any frame of the reference video, the obtained audio features and mouth features will be input into the preset first neural network for processing. After obtaining the audio features and mouth features, in order to generate a mouth image sequence that is synchronized with the target speech time sequence and conforms to the digital human mouth morphology. Figure 2As shown, in order to solve the problem that the existing latent space fusion needs to complete the two tasks of audio-lip shape matching and image fusion at the same time, it is easy to have optimization conflicts, resulting in low image resolution and difficulty in meeting the needs of high-definition digital humans. In the embodiment of the present application, in the first stage, the audio features and mouth features are input into the preset first neural network to obtain a mouth image sequence that matches the target speech. That is, in the embodiment of the present application, the traditional audio-lip shape matching and image fusion are decoupled, and in the first stage, only the audio and mouth shape matching process is performed, so that the audio features and mouth features are input into the preset first neural network to obtain a mouth image sequence that matches the target speech. While avoiding task conflicts, due to the reduced consumption of computing resources, the detail optimization of the mouth image sequence is achieved, and the image resolution is improved.

[0026] In some embodiments, before inputting the preset first neural network in step S201 to output the target mouth image sequence, the method further includes: S301: Input a target speech into a preset audio encoder, and output audio features corresponding to the target speech; S302: extracting any frame of video image from the reference video, and obtaining a mouth region image corresponding to a preset mouth region of the any frame of video image; S303: Input the mouth region image into a preset image encoder to extract mouth features corresponding to the mouth region image.

[0027] That is, Figure 3 As can be seen, the audio feature extraction process in this process involves extracting high-dimensional audio features of the target speech using a pre-trained preset audio encoder. It is understood that these high-dimensional audio features can represent the temporal content of the speech. The mouth feature extraction process involves inputting the mouth image of any frame of the reference video into the pre-trained preset image encoder for extraction, thereby obtaining the mouth features corresponding to any frame of the reference video.

[0028] In one embodiment, the preset image encoder is composed of a multi-layer convolutional neural network to output low spatial resolution and high-dimensional mouth features. The multi-layer convolutional neural network can convert local details of the mouth image, such as lip contours, interdental spaces, changes in mouth corners, and other detailed information, into high-dimensional vectors, providing a richer feature basis for subsequent matching of speech and mouth shape. In addition, the mouth feature extraction is only for the mouth feature mouth area image, and is low in spatial resolution. Compared with the traditional method of using a complete face image, it reduces the mixing of irrelevant information such as facial expressions and head postures, and makes it unnecessary to process the huge pixel data of the high-resolution image when fused with the audio features. This significantly reduces the computational complexity of latent space fusion and improves the model inference speed. In this process, the mouth features corresponding to any frame of video image in the reference video are obtained, and the traditional method of extracting full-face features is converted into extracting mouth features of the mouth area. The input of the preset first neural network is limited in advance to the image features of the mouth image, so that the subsequent preset first neural network model only needs to focus on the correspondence between mouth features and audio, and there is no need to filter mouth information from the complete face. This not only reduces the learning cost of invalid features, but also effectively avoids the interference of non-mouth areas such as eyebrow raising and head rotation when the subsequent preset first neural network calculates audio and video synchronization, and reduces the problem of misjudging asynchronous frames as synchronous frames. In addition, it should be noted that the audio features and mouth features obtained here are converted into a unified feature vector format through the encoder, so that the subsequent stage does not need to process the original audio waveform or image pixels, but only needs to fuse the standardized features in the latent space, reducing the adaptation cost of different modal data formats and indirectly improving the efficiency and stability of subsequent image generation.

[0029] In some embodiments, obtaining a mouth region image corresponding to a preset mouth region of any frame of video image may be performed by cropping any frame of video image using the preset mouth region to obtain the mouth region image. The specific implementation process is as follows: Facial image data corresponding to different expressions and postures for the current digital human character is collected, and the corresponding position sequence of the identified mouth key points in each facial image data is obtained. The corresponding position sequence and preset edge directions in each facial image data are determined, and the outermost position corresponding to each preset edge direction is determined. The range of the mouth area in the facial image is determined based on each outermost position and a preset boundary extension value, and this range is fixed as the preset mouth area. For example, in a certain scene, lip key points such as the corners of the digital human's mouth and the peak of the lip can be obtained for each image marker. If four preset edge directions are defined as left, right, top, and bottom, an initial mouth area frame can be generated by determining the outermost key points in each direction. This initial mouth area frame is expanded based on the preset boundary extension value to obtain the range of the mouth area in the facial image, and this range is fixed as the preset mouth area. It should be noted that the preset boundary extension value is determined based on the expansion ratio corresponding to each preset edge direction.

[0030] In this process, the mouth key points are extracted through the facial image data unique to the current digital human character, rather than relying on a general facial model, which can accurately match the facial structure of the digital human. Then, the initial area frame is generated by presetting the outermost key points in the edge direction, directly framing the core range of the mouth movement, which can avoid including too many non-mouth areas such as the cheeks and chin. In addition, since the initial area frame is generated only based on key points, it may be due to errors in key point marking, so the area is expanded to cover the necessary range around the lips through the preset boundary extension value to ensure the integrity of the mouth area image. By fixing the area range of the mouth area, there is no need to re-detect key points or adjust the area when cropping any frame of the subsequent reference video, which ensures the stability of the mouth area while achieving fast cropping.

[0031] In some embodiments, step S201 above inputs the audio features and mouth features into a preset first neural network to output a target mouth image sequence, specifically including: S401: enhancing the mouth feature by using the self-attention mechanism of the preset first neural network to obtain an enhanced mouth feature; S402: Fusing the enhanced mouth feature with the audio feature through the cross attention mechanism of the preset first neural network to output a mixed feature; S403: Reconstructing the mixed features through a preset multi-layer image decoder to obtain a target mouth image sequence that matches the target speech.

[0032] Since traditional CNN or RNN is difficult to model long-distance temporal associations, the mouth shape changes suddenly or becomes incoherent when speaking quickly, and is particularly unable to handle sustained phonemes, such as long vowels, which makes it difficult to maintain a stable mouth shape. Therefore, in order to overcome this problem, in the embodiment of the present application, the long-term dependency of mouth features is captured by presetting the self-attention mechanism of the first neural network, thereby outputting enhanced mouth features. The mouth feature relationship of all frames is dynamically analyzed through the self-attention layer, and the long-term correspondence between the phoneme duration and the mouth shape movement is accurately captured, so that the generated mouth shape sequence is smoother and more natural in the time dimension, eliminating unreasonable instantaneous jumps.

[0033] The enhanced mouth features are then fused with the audio features through a cross-attention mechanism pre-installed in the first neural network, outputting a hybrid feature. This allows the model to focus on speech-driven mouth shape changes, eliminating interference from irrelevant facial movements and improving the accuracy of audio-to-mouth matching. The hybrid feature is then reconstructed using a pre-installed multi-layer image decoder to obtain a mouth image sequence that matches the target speech. This process involves first upsampling the hybrid feature based on the pre-installed multi-layer image decoder to obtain multi-scale mouth features at each layer. Gated skip connections are then used to dynamically fuse the multi-scale mouth features at each layer with the features at the corresponding layer of the pre-installed image encoder to obtain a mouth image sequence that matches the target speech, ensuring that the generated mouth image sequence is consistent with the image style of the original reference video.

[0034] In a certain application scenario, the process of acquiring enhanced mouth features is to enhance the mouth features through the self-attention layer, capturing the long-term dependencies within the features. The formula for calculating self-attention is as follows: , where the query matrix Q, key matrix K and value matrix V, represents the transpose of the key matrix, and the query matrix and the transpose of the key matrix are matrix multiplied. Represents the vector dimension of the key matrix. The Softmax function is used to calculate the size of attention required for each position in the original sequence, and finally outputs the enhanced mouth features after attention enhancement.

[0035] Furthermore, in some embodiments, the enhanced mouth features are fused with the audio features through the cross attention mechanism of the preset first neural network to output a mixed feature, specifically including: S501: transforming the enhanced mouth features into a key matrix and a value matrix, and transforming the audio features into a query matrix; S502: Calculate attention weights based on the query matrix and the key matrix to obtain a weight matrix; S503: Aggregate the value matrix through the weight matrix to output a mixed feature.

[0036] Due to the simple splicing of audio and visual features in traditional methods, it is difficult to distinguish the master-slave relationship between the two modalities, resulting in a weakened voice-driven effect. The lip shape generation is easily dominated by visual features and deviates from the voice content. Therefore, in order to obtain a hybrid feature that integrates audio-driven information and references the mouth visual context, the lip shape generation can be more aligned with the voice input. In the embodiment of this specification, the enhanced mouth features are transformed into a key matrix and a value matrix to provide visual context. The audio features are also transformed into a query matrix to represent the voice-driven visual feature requirements. The weight matrix is ​​obtained by calculating the attention weight based on the query matrix and the key matrix. The value matrix is ​​then aggregated using the weight matrix to output the hybrid feature. In this process, by using the attention weight to aggregate the value matrix transformed from the enhanced mouth features, a dynamic reorganization of the visual features based on the attention weight is achieved, which not only retains the speaker's personalized mouth features, but also injects the speech-driven motion trajectory to improve the accuracy of the digital human's lip shape alignment.

[0037] Furthermore, in some embodiments, after inputting the audio features and the mouth features into a preset first neural network to output a target mouth image sequence, the method further includes: S601: extracting current mouth features of the target mouth image sequence using a preset image encoder to obtain similarity between the current mouth features and the audio features; S602: Based on the similarity, determine the synchronization discrimination loss between the audio feature and the current mouth feature, and iteratively adjust the preset first neural network according to the synchronization discrimination loss.

[0038] The current neural network model used as a discriminator is easily interfered by non-mouth areas when synchronizing speech and facial images, making it difficult for the model to accurately learn the correspondence between mouth shape and speech, thereby affecting the final discrimination accuracy. In addition, the training of the neural network model requires the simultaneous optimization of multiple losses such as reconstruction loss, SyncNet loss, and adversarial loss, and the gradient conflicts lead to convergence difficulties. Therefore, in order to overcome this problem and improve the accuracy of lip synchronization, after obtaining the target mouth image sequence that matches the target speech based on the above steps S401-S403, as shown in the following example: Figure 4As shown, a preset image encoder is used to extract the current mouth features of the target mouth image sequence to obtain the similarity between the current mouth features and the audio features. This similarity is then used as the synchronization discrimination loss between the audio features and the current mouth features, thereby iteratively adjusting the preset first neural network based on the synchronization discrimination loss. In one feasible embodiment, the similarity is cosine similarity. It is understood that a higher similarity score indicates greater synchronization. Iteratively adjusting the preset first neural network using this synchronization discrimination loss enables the preset first neural network model corresponding to the first stage to learn more accurate lip synchronization. This process generates the similarity between the mouth features and the input audio through real-time calculation, establishing a quantifiable synchronization feedback mechanism. Compared to traditional fixed threshold detection methods, this method can capture fine-grained changes in the audio-visual correlation, providing precise guidance for model optimization. At the same time, it avoids the gradient conflicts caused by optimizing multiple losses simultaneously during traditional training, which leads to convergence difficulties, thereby improving the discrimination accuracy of the subsequent preset first neural network model.

[0039] S202: Screening the reference video using a preset negative film selection strategy to obtain a negative film image.

[0040] After completing the matching of audio and mouth shape in the first stage based on the above step S201, the following steps are required: Figure 5 The image fusion process of the second stage is shown. The existing method relies on the input film in the fusion stage. Due to the lack of a dynamic film selection mechanism, the reference frame is usually selected in a random or fixed manner, which makes the film unable to adapt to different pronunciation states. If the film mouth shape does not match the target mouth shape, for example, the target needs to close the mouth but the film is open, the fused mouth shape will not be able to close correctly. Even if the voice requires closing the mouth, the generated mouth shape is still slightly open due to the influence of the film. The mouth shape state of the film, such as smiling, raised corners of the mouth, etc., may also interfere with the generation of the target mouth shape. Therefore, in order to solve the problem of film interference and the problem of forced closure, the embodiment of the present application will use a preset film selection strategy to perform film screening on the video image of the reference video to obtain a film image. This process uses a preset film selection strategy to dynamically screen the matching film to ensure that the film mouth shape is consistent with the physiological state of the target voice.

[0041] In a feasible embodiment, the preset negative film selection strategy can combine the user ID corresponding to the target voice and the voice emotion label corresponding to the target voice to further screen personalized negative films corresponding to the user ID and the voice emotion label, thereby flexibly adapting to the needs of lip shape generation in different styles and improving the scene adaptability of digital human lip shape video generation.

[0042] In some embodiments, such as Figure 6The above step S203 screens the reference video using a preset film selection strategy to obtain a film image, specifically including: S701: Determine the speech stage corresponding to the target speech; wherein the speech stage includes: a normal speech stage and a closed mouth stage; S702: Acquire a reference mouth opening and closing degree sequence of the reference video and a target mouth opening and closing degree sequence of the target mouth image sequence, and dynamically match the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result; S703: Based on the voice stage and the matching result, negative screening is performed on the reference video to obtain a negative image.

[0043] In order to correctly distinguish between normal pronunciation and closed-mouth phases, such as the brief closure after the plosive / p / , the embodiment of the present application first determines the speech phase corresponding to the target speech. The speech phases include: normal speech phase and closed-mouth phase. Then, the mouth opening and closing degrees of the mouth shape sequence of each frame of the reference video are quantized to obtain a reference mouth opening and closing degree sequence. Simultaneously, the mouth opening and closing degrees of each frame of the target mouth image sequence are quantized to obtain a target mouth opening and closing degree sequence of each frame of the target mouth image sequence. A preset dynamic matching method is then used to align and match the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result. To bypass complex latent space control through reliable forced mouth closing, the embodiment of the present application combines the speech phase determined in the above process with the matching results to screen each frame of the reference video, thereby obtaining a negative image.

[0044] In one feasible embodiment, the speech stage corresponding to the target speech can be determined by: determining the speech stage corresponding to the target speech based on the speech time and audio energy corresponding to the target speech. It can be understood that the speech time and audio energy corresponding to the target speech can adapt to different speaking speeds and accents, thereby avoiding lip shape errors and reducing the sense of dissonance caused by audio and video asynchrony.

[0045] Furthermore, in some embodiments, the reference mouth opening and closing degree sequence and the target mouth opening and closing degree sequence are dynamically matched to perform negative screening of the reference video according to the matching result to obtain a negative image, specifically including: S801: Obtaining absolute mouth opening degree differences of each frame pair between the reference mouth opening degree sequence and the target mouth opening degree sequence; S802: Obtaining the cumulative distance corresponding to each frame pair based on the absolute opening and closing degree difference of each frame pair, and generating a cumulative distance matrix; S803: Determine the best matching path as the matching result through the accumulated distance matrix.

[0046] In order to significantly reduce the negative impact of the mouth opening information of the film when the target speech is in a closed mouth state by selecting a film with an appropriate opening and closing degree. Improve the ability to generate accurate closed mouth images when the mouth needs to be closed, solve the problem of digital people not being able to close their mouths, and improve the visual effect. In the embodiment of this specification, the absolute opening and closing degree difference of each frame pair between the reference mouth opening and closing degree sequence and the target mouth opening and closing degree sequence will be obtained. It should be noted that before this process, the mouth opening and closing degree of the mouth shape sequence of each frame video image in the reference video and the mouth opening and closing degree of each frame mouth image in the target mouth image sequence will be pre-calculated. The calculation method is to assume that the coordinates of the N key points of the mouth in each frame image output by the mouth key point detection model are marked as a set P, and the opening and closing degree measure (O) is defined as follows: ,in and Respectively represent k pairs of key points corresponding to the upper and lower lips, is the Euclidean distance. Then calculate the absolute opening and closing difference of all frame pairs between the two sequences: ,in, is the absolute opening and closing difference, T is the reference video frame length, and S is the generated video frame length.

[0047] Then, the cumulative distance corresponding to each frame pair is obtained by the absolute opening and closing difference of each frame pair, and the cumulative distance matrix (dynamic programming table) is generated. , the remaining elements It can be obtained by the recursive formula: ,in, is the cumulative distance matrix value of the i-1th row and jth column, is the cumulative distance matrix value of the i-th row and j-1th column, is the cumulative distance matrix value in row i-1 and column j-1. Then, we backtrack from (T, S) of the distance matrix to (1, 1), select the path W that minimizes the cumulative distance as the best matching path, and determine the best matching path as the matching result.

[0048] This process quantifies the matching degree of a single frame through the absolute difference in opening and closing degrees, and combines this with a cumulative distance matrix to evaluate the matching degree of the entire sequence. This avoids the temporal discontinuity problem caused by existing methods that rely solely on single-frame matching, ensuring that the selected film image matches the lip opening and closing requirements of the target speech. Furthermore, the optimal matching path is selected based on the minimum overall cumulative distance, minimizing the overall deviation in the opening and closing sequence between the reference video and the target speech, providing a highly matched film for subsequent lip generation. This effectively alleviates the problem of mouth closure mismatching with the target speech caused by the mismatch between the film's lip shape and the target state, thereby improving the robustness of lip shape control.

[0049] Furthermore, in some embodiments, combining the speech stage and the matching result to perform negative screening on the reference video to obtain a negative image specifically includes: S901: If it is determined that the speech stage is a normal speech stage, performing negative screening on the reference video according to the matching result to obtain an initial negative image; S902: If it is determined that the speech stage is a closed-mouth stage, obtaining a pre-stored closed-mouth image as an initial negative image; S903: Cover the mouth area of ​​the initial film image to obtain the film image.

[0050] During the screening process, if the speech stage is determined to be a normal speech stage, the reference video is screened according to the best matching path corresponding to the matching result to obtain the negative image. That is, for the jth frame of the generated sequence, the index of the corresponding best reference frame is for: , where argmin is the index number corresponding to the minimum value, and the constraint condition = ...

[0051] If the speech phase is determined to be closed-mouth, such as at the beginning, end, or silence of the target speech, a pre-stored closed-mouth image is used as the negative image to enforce closed-mouth processing. The mouth area of ​​this initial negative image is then masked to obtain a negative image. By masking the mouth area of ​​the negative image, the original mouth shape is eliminated from interfering with the generated result while preserving key facial context, facilitating the subsequent generation of a natural-looking digital human.

[0052] In a feasible embodiment, the mouth area of ​​the initial film image is masked to obtain the film image, which can be achieved in the following way: the mouth area of ​​the initial film image is masked based on a binary mask, and the mouth mask image of the initial film image is obtained as the film image. A binary mask is an image that contains only two pixel values ​​of "0" and "1". In this process, the "1" area of ​​the binary mask corresponds to the mouth area of ​​the film image, thereby masking the mouth area, and "0" corresponds to the non-mouth area of ​​the film image, thereby retaining the non-mouth area. After masking the film image with a binary mask, only the non-mouth area in the final mouth mask image retains image information, while the mouth area, such as Figure 5 Shown is shielded.

[0053] S204: Input the negative image and the target mouth image sequence into a preset second neural network for fusion processing to output a digital population video image.

[0054] like Figure 5 As shown, in the embodiment of the present application, the film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population-type video image. Through the above process, the film image of the mouth area of ​​the initial film picture is masked based on the mask, eliminating the interference of the original mouth shape on the generated result while fully retaining the key facial context. Then, the speech-driven target mouth image sequence and the film image are input into the second neural network for fusion processing, realizing the dynamic adjustment of the fusion weight of the film context and the generated mouth shape, ensuring that the pronunciation action is naturally integrated into the original facial expression, and improving the overall image quality, thereby obtaining a high-resolution digital population-type video.

[0055] In some embodiments, in step S203, the negative film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population video image, specifically including: S1001: performing channel splicing on the target mouth image sequence and the negative film image to obtain a total input tensor; S1002: Input the total input tensor into a preset second neural network for fusion processing, and output a digital population-shaped video image, so as to generate a lip-syncing video through continuous frames of digital population-shaped images.

[0056] Since the target mouth image sequence obtained according to the above steps contains the core content of the dynamic lip shape, while the negative image carries other features of the non-mouth area. Therefore, it is necessary to perform channel splicing on the target mouth image sequence and the mouth mask image to obtain the total input tensor. It should be noted that the splicing process is performed along the channel dimension, and a composite tensor containing spatiotemporal information is obtained as the total input tensor. The total input tensor is then input into the preset second neural network. After encoding and decoding processing by the preset second neural network, while repairing the mask area, the facial features of the new mouth shape and the negative image are adaptively fused to output a seamless high-resolution digital population video image, thereby generating a lip shape video through continuous frame digital population images, ensuring that the lip shape movement is both voice-driven and naturally coordinated with the original facial micro-expressions.

[0057] In a certain embodiment of the present application, the second stage can be implemented based on the following process: First, the input data includes a target mouth image sequence input and a negative image input. The target mouth image sequence input is a five-frame sliding window input, and the five frames include the t-th frame and two frames before and after it, so as to reduce inter-frame jitter. In addition, the negative image input is the original head image of the t-th frame, and its mouth area is masked by a binary mask. Then the input data, that is, the five frames of mouth images and the masked head image channels are spliced ​​to form a total input tensor. Thus, a U-Net-like encoder-decoder structure is adopted, and processing is achieved by retaining spatial details through jump connections. The super-resolution module is integrated end-to-end at the decoder end, and the restoration and resolution are jointly optimized to finally generate a restored high-resolution head image of the t-th frame, in which the mouth is naturally integrated with the surrounding area.

[0058] The embodiment of the present application also provides a device for generating a digital human's lip shape video. Figure 7 As shown, a device for generating a lip-sync video of a digital human comprises: The acquisition unit 701 is configured to, upon acquiring audio features corresponding to the target speech and mouth features corresponding to any frame of a video image in the reference video, input them into a preset first neural network to output a target mouth image sequence; A screening unit 702 is configured to screen the reference video using a preset negative film selection strategy to obtain a negative film image; The output unit 703 is used to input the negative image and the target mouth image sequence into a preset second neural network for fusion processing to output a digital population video image.

[0059] In some embodiments, the input is fed into a preset first neural network to output a target mouth image sequence, specifically comprising: enhancing the mouth feature by using the self-attention mechanism of the preset first neural network to obtain an enhanced mouth feature; fusing the enhanced mouth feature with the audio feature through a cross-attention mechanism of the preset first neural network to output a mixed feature; The mixed features are reconstructed by a preset multi-layer image decoder to obtain a target mouth image sequence that matches the target speech.

[0060] In some embodiments, the enhanced mouth features are fused with the audio features through the cross attention mechanism of the preset first neural network to output a mixed feature, specifically including: transforming the enhanced mouth features into a key matrix and a value matrix, and transforming the audio features into a query matrix; Calculate attention weights based on the query matrix and the key matrix to obtain a weight matrix; The value matrix is ​​aggregated by the weight matrix to output a mixed feature.

[0061] In some embodiments, the reference video is screened by a preset negative film selection strategy to obtain a negative film image, specifically including: Determining the speech stage corresponding to the target speech; wherein the speech stage includes: a normal speech stage and a closed mouth stage; Obtaining a reference mouth opening and closing degree sequence of the reference video and a target mouth opening and closing degree sequence of the target mouth image sequence, and dynamically matching the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result; The reference video is subjected to negative screening in combination with the speech stage and the matching result to obtain a negative image.

[0062] In some embodiments, dynamically matching the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result specifically includes: Obtaining the absolute mouth opening degree difference of each frame pair between the reference mouth opening degree sequence and the target mouth opening degree sequence; The cumulative distance corresponding to each frame pair is obtained through the absolute opening and closing difference of each frame pair, and a cumulative distance matrix is ​​generated; The best matching path is determined as the matching result through the accumulated distance matrix.

[0063] In some embodiments, combining the speech stage and the matching result to perform negative screening on the reference video to obtain a negative image specifically includes: If it is determined that the speech stage is a normal speech stage, performing negative screening on the reference video according to the matching result to obtain an initial negative image; If it is determined that the speech stage is a closed-mouth stage, obtaining a pre-stored closed-mouth image as an initial negative image; The mouth area of ​​the initial film image is masked to obtain the film image.

[0064] In some embodiments, the negative film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population video image, specifically including: Perform channel splicing on the target mouth image sequence and the negative film image to obtain a total input tensor; The total input tensor is input into a preset second neural network for fusion processing, and a digital population-shaped video image is output, so as to generate a lip-syncing video through continuous frames of digital population-shaped images.

[0065] In some embodiments, after inputting the preset first neural network to output a target mouth image sequence, the method further includes: Extracting current mouth features of the target mouth image sequence through a preset image encoder, and obtaining similarity between the current mouth features and the audio features; According to the similarity, the synchronization discrimination loss between the audio feature and the current mouth feature is determined, and the preset first neural network is iteratively adjusted according to the synchronization discrimination loss.

[0066] In some embodiments, before inputting the preset first neural network to output the target mouth image sequence, the method further includes: Input the target speech into a preset audio encoder and output the audio features corresponding to the target speech; Extracting any one frame of video image from the reference video, and obtaining a mouth region image corresponding to a preset mouth region of the any one frame of video image; The mouth region image is input into a preset image encoder to extract mouth features corresponding to the mouth region image.

[0067] The embodiment of this specification also provides a device for generating a digital human's lip shape video. Figure 8 As shown, a device for generating a lip-sync video of a digital human, the device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods for generating a digital human's lip-sync video.

[0068] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0069] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The foregoing is merely one or more embodiments of the present application and is not intended to limit the present application. For those skilled in the art, various modifications and variations may be made to one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for generating a digital human lip-sync video, characterized in that: The method comprises: When the audio features corresponding to the target speech and the mouth features corresponding to any frame of the reference video are obtained, they are input into a preset first neural network to output a target mouth image sequence; The reference video is screened by a preset negative film selection strategy to obtain a negative film image; The negative film image and the target mouth image sequence are input into a preset second neural network for fusion processing to output a digital population video image.

2. The method for generating a digital human lip-sync video according to claim 1, wherein: Input it into the preset first neural network to output the target mouth image sequence, specifically including: enhancing the mouth feature by using the self-attention mechanism of the preset first neural network to obtain an enhanced mouth feature; fusing the enhanced mouth feature with the audio feature through a cross-attention mechanism of the preset first neural network to output a mixed feature; The mixed features are reconstructed by a preset multi-layer image decoder to obtain a target mouth image sequence that matches the target speech.

3. The method for generating a digital human lip-sync video according to claim 2, wherein: The enhanced mouth feature is fused with the audio feature through the cross attention mechanism of the preset first neural network to output a mixed feature, specifically including: transforming the enhanced mouth features into a key matrix and a value matrix, and transforming the audio features into a query matrix; Calculate attention weights based on the query matrix and the key matrix to obtain a weight matrix; The value matrix is ​​aggregated by the weight matrix to output a mixed feature.

4. The method for generating a digital human lip-sync video according to claim 1, wherein: The reference video is screened by a preset negative film selection strategy to obtain a negative film image, specifically including: Determining the speech stage corresponding to the target speech; wherein the speech stage includes: a normal speech stage and a closed mouth stage; Obtaining a reference mouth opening and closing degree sequence of the reference video and a target mouth opening and closing degree sequence of the target mouth image sequence, and dynamically matching the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result; The reference video is subjected to negative screening in combination with the speech stage and the matching result to obtain a negative image.

5. The method for generating a digital human lip-sync video according to claim 4, wherein: Dynamically matching the reference mouth opening and closing degree sequence with the target mouth opening and closing degree sequence to obtain a matching result specifically includes: Obtaining the absolute mouth opening degree difference of each frame pair between the reference mouth opening degree sequence and the target mouth opening degree sequence; The cumulative distance corresponding to each frame pair is obtained through the absolute opening and closing difference of each frame pair, and a cumulative distance matrix is ​​generated; The best matching path is determined as the matching result through the accumulated distance matrix.

6. The method for generating a digital human lip-sync video according to claim 4, wherein: Combining the speech stage with the matching result, performing negative screening on the reference video to obtain a negative image specifically includes: If it is determined that the speech stage is a normal speech stage, performing negative screening on the reference video according to the matching result to obtain an initial negative image; If it is determined that the speech stage is a closed-mouth stage, obtaining a pre-stored closed-mouth image as an initial negative image; The mouth area of ​​the initial film image is masked to obtain the film image.

7. The method for generating a digital human lip-sync video according to claim 1, wherein: Inputting the negative image and the target mouth image sequence into a preset second neural network for fusion processing to output a digital population-shaped video image, specifically comprising: Perform channel splicing on the target mouth image sequence and the negative film image to obtain a total input tensor; The total input tensor is input into a preset second neural network for fusion processing, and a digital population-shaped video image is output, so as to generate a lip-syncing video through continuous frames of digital population-shaped images.

8. The method for generating a digital human lip-sync video according to claim 1, wherein: After inputting the preset first neural network to output a target mouth image sequence, the method further includes: Extracting current mouth features of the target mouth image sequence through a preset image encoder, and obtaining similarity between the current mouth features and the audio features; According to the similarity, the synchronization discrimination loss between the audio feature and the current mouth feature is determined, and the preset first neural network is iteratively adjusted according to the synchronization discrimination loss.

9. The method for generating a digital human lip-sync video according to claim 1, wherein: Before inputting the preset first neural network to output a target mouth image sequence, the method further includes: Input the target speech into a preset audio encoder and output the audio features corresponding to the target speech; Extracting any one frame of video image from the reference video, and obtaining a mouth region image corresponding to a preset mouth region of the any one frame of video image; The mouth region image is input into a preset image encoder to extract mouth features corresponding to the mouth region image.

10. A device for generating a digital human's lip-sync video, characterized in that: The device comprises: an acquisition unit configured to, upon acquiring audio features corresponding to the target speech and mouth features corresponding to any frame of a video image in the reference video, input them into a preset first neural network to output a target mouth image sequence; A screening unit, configured to screen the reference video using a preset negative film selection strategy to obtain a negative film image; The output unit is used to input the film image and the target mouth image sequence into a preset second neural network for fusion processing to output a digital population video image.

Images