Normal map generation method and digital human video generation method

Abstract

The invention provides a normal map generation method and a digital human video generation method, and relates to the technical field of digital human video generation, and the method comprises the steps: firstly receiving a human video, and determining a target voice; extracting three-dimensional digital human model data by utilizing the human video; and finally, combining the three-dimensional digital human model data and the target voice to generate a multi-frame normal map. According to the method, the target voice is utilized to assist the generation of the multi-frame normal map, so that the obtained normal map can represent that a three-dimensional digital person has human face details such as mouth shape, facial expression, posture change and the like which are in fit with a real person state when performing target voice pronunciation; and high-quality image data is provided for subsequent generation of natural and vivid digital human videos.

Description

Technical Field

[0001] This invention relates to the field of digital human video generation technology, and in particular to a method for generating normal maps and a method for generating digital human videos. Background Technology

[0002] With the rapid development of digital human video generation technology, existing technologies can generate a digital human video based on a user-uploaded audio clip and a person's photo. In the video, the digital human's mouth shape changes continuously with the audio and makes corresponding facial expressions and gestures.

[0003] Current technology typically involves directly inputting user-uploaded voice and video into a pre-trained video generation model to obtain a digital human video corresponding to the voice. However, in production practice, video generation models with relatively small parameter sizes are usually chosen, and sufficient high-quality data cannot be used to fully train the video generation model. This results in the generated digital human videos having stiff and unnatural expressions and postures, especially in terms of mouth shapes and facial expression details, which differ significantly from videos obtained from actual live-action footage. Summary of the Invention

[0004] This invention provides a method for generating normal maps and a method for generating digital human videos, in order to overcome the deficiencies existing in related technologies.

[0005] This invention provides a method for generating normal maps, comprising: Receive video of a person and determine the target voice; Based on the video of the person, extract three-dimensional digital human model data; Based on the 3D digital human model data and the target speech, a multi-frame normal map is generated.

[0006] According to a normal map generation method provided by the present invention, the three-dimensional digital human model data includes a face model of a two-dimensional human image in each frame of a human video; the step of generating multi-frame normal maps based on the three-dimensional digital human model data and the target speech includes: The target speech is encoded to obtain speech features; Based on the face model and the voice features, the first constraint condition of the Transformer architecture diffusion model is determined, and based on the Transformer architecture diffusion model, the first constraint condition is applied to denoise the face animation noise data corresponding to the person video to generate target face animation data. Based on the target face animation data and the face model, the multi-frame normal map is generated.

[0007] According to a normal map generation method provided by the present invention, facial animation style features are generated based on a reference style video; Based on the target speech, key facial animation data is generated; Based on the face model and the voice features, as well as the face animation style features and / or the key face animation data, the first constraint condition is determined.

[0008] According to a normal map generation method provided by the present invention, the step of generating facial animation style features based on a reference style video includes: Extract reference face animation data from each frame of the reference style image in the reference style video; The reference face animation data is linearly transformed to obtain the linear transformation result; Style features are extracted from the linear transformation results to obtain the facial animation style features of each frame of the reference style image; The facial animation style features are determined based on the facial animation style features of each frame of reference style image.

[0009] According to a normal map generation method provided by the present invention, the step of generating key facial animation data based on the target speech includes: Determine the target text corresponding to the target speech; The target text is input into a pre-trained key face animation generation model to obtain the key face animation data output by the key face animation generation model.

[0010] According to a normal map generation method provided by the present invention, the step of encoding the target speech to obtain speech features includes: The target speech is divided into multiple speech segments; Based on the speech coding module, speech features of the multiple speech segments are extracted.

[0011] According to a normal map generation method provided by the present invention, the step of extracting speech features of the plurality of speech segments based on a speech coding module includes: For any speech segment other than the first speech segment among the multiple speech segments, the speech segment and the tail speech in the previous speech segment are input into the speech encoding module to obtain multi-frame speech features output by the speech encoding module.

[0012] According to a normal map generation method provided by the present invention, the number of frames of the multi-frame speech features is the same as the number of frames of the face animation noise data; The step of denoising the facial animation noise data corresponding to the character video based on the Transformer architecture diffusion model and applying the first constraint to generate target facial animation data includes: Align the multi-frame speech features with the facial animation noise data; After alignment, based on the Transformer architecture diffusion model, the first constraint condition is applied to denoise the face animation noise data to generate the target face animation data.

[0013] According to a normal map generation method provided by the present invention, the three-dimensional digital human model data further includes the original facial animation data of each frame of a two-dimensional human image in a human video; the step of extracting the three-dimensional digital human model data based on the human video includes: The video of the person is input into the Video2Mesh module, where the 3D deformable face model in the Video2Mesh module extracts the face model, and the face animation data extraction unit in the Video2Mesh module extracts the original face animation data.

[0014] The present invention also provides a method for generating digital human videos, comprising: Based on the above normal map generation method, multiple frames of normal maps are generated, and noise is added to the face region of each frame of the two-dimensional human image in the human video to obtain each frame of the noisy image. The multi-frame normal maps are used as the second constraint condition for the large video generation model. Based on the large video generation model, the second constraint condition and the noisy video reference frames in the human video are applied to denoise each frame of the noisy image in the human video to obtain the digital human video.

[0015] A digital human video generation method according to the present invention further includes: The video of the person is segmented to obtain multiple video clips of the person; Based on the normal map generation method, a multi-frame normal map of each character video segment is determined, and for any character video segment, noise is added to the face region of each frame of the two-dimensional character image in the any character video segment to obtain a noisy image of each frame. Using the multi-frame normal maps of any character video segment as the second constraint, and generating a large model based on the video, the second constraint and the video reference frame corresponding to any character video segment are applied to denoise each frame of the noisy image in any character video segment to obtain the denoising result corresponding to any character video segment. The digital human video is generated based on the denoising results corresponding to the multiple video clips of the individuals.

[0016] According to a digital human video generation method provided by the present invention, for any remaining human video segment other than the first human video segment in the human video, the video reference frame corresponding to any remaining human video segment is the last frame of a two-dimensional human image in the preceding human video segment of any remaining human video segment.

[0017] According to a digital human video generation method provided by the present invention, the method involves adding noise to the face region of each frame of a two-dimensional human image in the video to obtain a noisy image for each frame, or adding noise to the face region of each frame of a two-dimensional human image in any video segment to obtain a noisy image for each frame, comprising: For any frame of a two-dimensional character image in the character video or any video segment of the character, extract the facial contour in the two-dimensional character image of the frame, and generate a binary face mask image based on the facial contour. The binary face mask image is filled with noise to obtain a noisy image of any frame of the two-dimensional human image.

[0018] The present invention also provides a normal map generation apparatus, comprising: The receiving module is used to receive video of a person and determine the target voice. The extraction module is used to extract three-dimensional digital human model data based on the person's video; The texture generation module is used to generate multi-frame normal maps based on the 3D digital human model data and the target speech.

[0019] The present invention also provides a digital human video generation device, characterized in that it comprises: The preprocessing module is used to generate multiple frames of normal maps based on the above-mentioned normal map generation method, and to add noise to the face region of each frame of the two-dimensional human image in the human video to obtain a noisy image for each frame. The video generation module is used to use the multi-frame normal maps as the second constraint condition of the large video generation model, and based on the large video generation model, apply the second constraint condition and the noisy video reference frames in the human video to denoise each noisy image in the human video to obtain a digital human video.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the normal map generation method or the digital human video generation method as described above.

[0021] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the normal map generation method or the digital human video generation method as described above.

[0022] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the normal map generation method or the digital human video generation method as described above.

[0023] The normal map generation method and digital human video generation method provided by this invention first receive a person video and determine the target speech; then, using the person video, extract 3D digital human model data; finally, combine the 3D digital human model data and the target speech to generate multi-frame normal maps. This method uses the target speech to assist in the generation of multi-frame normal maps, enabling the obtained normal maps to represent facial details such as mouth shape, facial expressions, and posture changes in a realistic manner when the 3D digital human pronounces the target speech, providing high-quality image data for the subsequent generation of natural and realistic digital human videos. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is one of the flowcharts illustrating a normal map generation method provided by the present invention.

[0026] Figure 2 This is a schematic diagram of the style feature extraction process in a normal map generation method provided by the present invention.

[0027] Figure 3a This is a schematic diagram illustrating the fusion of speech features and key facial animation data in a normal map generation method provided by the present invention.

[0028] Figure 3b This is a schematic diagram illustrating the working principle of the Transformer architecture diffusion model in a normal map generation method provided by this invention.

[0029] Figure 4 This is a schematic diagram of the structure of the Video2Mesh module provided by the present invention.

[0030] Figure 5 This is the second flowchart illustrating a normal map generation method provided by the present invention.

[0031] Figure 6 This is one of the flowcharts of a digital human video generation method provided by the present invention.

[0032] Figure 7 This is the second flowchart of a digital human video generation method provided by the present invention.

[0033] Figure 8 This is a schematic diagram of the structure of a normal map generation device provided by the present invention.

[0034] Figure 9 This is a schematic diagram of the structure of a digital human video generation device provided by the present invention.

[0035] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] Current technologies typically input user-uploaded voice and video directly into a pre-trained video generation model to obtain a digital human video corresponding to the voice. For this model to generate digital human videos that are completely identical to real people, it requires extensive training with massive amounts of high-quality data on a large-scale video generation model over a long period to achieve the desired results. However, this approach is extremely costly to train. The collection, processing, and annotation of massive amounts of high-quality data consume significant human and financial resources, and training the video generation model also requires substantial electricity and computing resources. Furthermore, after the model is trained, deploying and inferring the large-scale video generation model also consumes significant hardware resources, and the inference time is long, requiring users to wait a considerable amount of time after uploading content before the digital human video is generated.

[0038] To reduce costs, in production practice, video generation models with smaller parameter scales are usually chosen, and sufficient high-quality data cannot be used to fully train the video generation models. As a result, the expressions and postures of the digital humans in the generated videos are relatively stiff and unnatural, especially in terms of mouth shape and facial expression details, which are significantly different from videos obtained from actual live-action footage.

[0039] Based on this, this embodiment of the invention provides a normal map generation method to assist in the generation of digital human videos.

[0040] Figure 1 This is a flowchart illustrating a normal map generation method provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes: S11, Receive video of the person and determine the target voice; S12, Extract 3D digital human model data based on human videos; S13 generates multi-frame normal maps based on 3D digital human model data and target speech.

[0041] Specifically, the normal map generation method provided in this embodiment of the invention is executed by a normal map generation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0042] First, step S11 is executed to receive a character video. This character video is provided by the user and serves as a reference for generating multi-frame normal maps, showing the target character's mouth shape, facial expressions, and pose changes. The character video may include multiple frames of 2D character images arranged chronologically. The target character is the same in each frame of the character video, but the image of the target character shown in each frame is slightly different. Therefore, each frame of the character video is equivalent to multiple images of the target character taken.

[0043] The target speech also originates from the user; it can be speech directly provided by the user or converted from target text provided by the user. For example, if the user provides target text, the target speech corresponding to the target text can be generated using text-to-speech (TTS) technology. Here, speech synthesis technology is a mature technology, and the specific choice of which speech synthesis technology is not highly relevant to the core improvement points of this invention embodiment, and will not be described in detail here.

[0044] Then, step S12 is executed to extract 3D digital human model data using the person video. Here, facial key points can be automatically annotated in each frame of the 2D person image in the video, and the coordinates of the facial key points in all 2D person images in the video can be used to extract the 3D digital human model data. This 3D digital human model data extraction model can be a trained neural network model, a Video2Mesh module, or other tools capable of extracting 3D digital human model data; no specific limitations are made here.

[0045] In this embodiment of the invention, the three-dimensional digital human model data may include a face model of each frame of a two-dimensional human image in a video. The face model can represent the shape of the target person's face in each frame of the two-dimensional human image. Here, the face model can be represented in different forms, such as a three-dimensional face mesh or a face coefficient vector. For ease of subsequent processing, the face model can be represented in the form of a face coefficient vector.

[0046] Building upon this, the 3D digital human model data can also include the raw facial animation data of each frame of the 2D human image in the video. The raw facial animation data can characterize the animation data of the target person in each frame of the 2D human image, and can include at least one type of animation data such as pose animation data, mouth shape animation data, and facial expression animation data. Each type of animation data can characterize the dynamic changes of the target person's face from different dimensions.

[0047] It's important to clarify that animation data refers to the spatial coordinate changes of a set of key points (such as skeletal joints and facial control points) over time, used to represent the facial expressions and movements of a target person. It should be understood that in a single frame of a 2D image, the target person is stationary, and the corresponding facial control points remain unchanged. To make the target person make different expressions, the corresponding facial control points must change over time, thus moving the entire face. The spatial coordinate changes of these facial control points over time are recorded as animation data. Therefore, the aforementioned raw human animation data is the spatial coordinate change data of the target person's key points in a video over time. Different types of animation data correspond to different sets of key points, used to represent the dynamic changes of the target person's face from different dimensions.

[0048] Finally, step S13 is executed to generate multi-frame normal maps using the 3D digital human model data and the target speech. Normal mapping is a computer graphics technique that stores 3D normal vector data through red (R), green (G), and blue (B) color channels, enabling the simulation of high-precision geometric details and lighting effects on the surface of low-polygon models.

[0049] Target speech can be used to generate target facial animation data, and the target facial animation data can be input into the normal map generation model to obtain multi-frame normal maps.

[0050] Here, the normal map generation model can replace the original face animation data contained in the 3D digital human model data with the target face animation data to obtain the target 3D digital human model data; alternatively, it can directly fuse the target face animation data with the face model in the 3D digital human model data to obtain the target 3D digital human model data. The target 3D digital human model data allows the digital human to maintain its face model unchanged and dynamically change according to the target face animation data.

[0051] Subsequently, the normal map generation model can use the target 3D digital human model data to generate multi-frame normal maps. For example, offline tools such as OpenGL, DirectX, or RenderDoc can be used to render each frame of data from the target 3D digital human model data to a virtual screen, record the normal direction of each pixel on the virtual screen, and then perform raster interpolation on the normal direction of each pixel to generate multi-frame pixel-level normal maps.

[0052] The number of frames in the generated normal map is directly related to the dynamic changes of the face model in the target 3D digital human model data. These dynamic changes can include actions, expressions, and deformations.

[0053] If the facial model in the target 3D digital human model data exhibits high-frequency dynamic changes, i.e., high-frequency movements such as dramatic changes in facial expressions, then it is necessary to increase the frame rate of the normal map to capture these detailed changes. For example, a smiling expression requires 30 frames per second for the normal map update, while a slow blink requires 10 frames per second.

[0054] The frame rate of the normal map can be the same as the number of face models in the target 3D digital human model data, i.e., the number of frames of the 2D human image in the video, or it can be determined by the keyframes on the timeline. It can be the same as or proportional to the number of keyframes. For example, if 20 keyframes are set within 0-10 seconds, the frame rate of the normal map can be 20 frames. The frame rate of the normal map can also be preset; no specific limitation is made here.

[0055] The normal map generation method provided in this embodiment of the invention first receives a video of a person and determines the target speech; then, it uses the video to extract 3D digital human model data; finally, it combines the 3D digital human model data and the target speech to generate multi-frame normal maps. This method utilizes the target speech to assist in the generation of multi-frame normal maps, enabling the obtained normal maps to represent facial details such as mouth shape, facial expressions, and posture changes that closely resemble those of a real person when the 3D digital human is speaking the target speech, providing high-quality image data for the subsequent generation of natural and realistic digital human videos.

[0056] Based on the above embodiments, the 3D digital human model data includes the face model of each frame of the 2D human image in the video; based on the 3D digital human model data and the target speech, multi-frame normal maps are generated, including: The target speech is encoded to obtain speech features; Based on the face model and speech features, the first constraint of the Transformer architecture diffusion model is determined. Based on the Transformer architecture diffusion model, the first constraint is applied to denoise the face animation noise data corresponding to the person video and generate the target face animation data. Based on the target face animation data and face model, generate multi-frame normal maps.

[0057] Specifically, in the process of generating multi-frame normal maps, the target speech can first be encoded using a speech coding module to obtain speech features. This speech coding module can be built based on the HuBERT model and is therefore also called a HuBERT encoder. Speech features can include low-level acoustic features, phoneme features, semantic features, and rhythmic features, etc. Low-level acoustic features can include volume, energy, etc. Speech features can characterize the features of the target speech from different perspectives and can be used to predict target facial animation data.

[0058] In this embodiment of the invention, a target face animation data is generated using a Transformer architecture diffusion model, utilizing a face model from a 3D digital human model and speech features obtained by encoding the target speech. Here, the face model and speech features can be used to determine the first constraint condition of the Transformer architecture diffusion model. For example, the face model and speech features can be directly used as the first constraint condition, or the face model and speech features can be combined with other information as the first constraint condition. Other information can be set as needed, and no specific limitations are made here.

[0059] The input to the Transformer architecture diffusion model can include a first constraint and facial animation noise data corresponding to the person video. The facial animation noise data can be obtained by adding noise to each frame of the 2D person image in the person video. The number of frames in the facial animation noise data is the same as the number of frames in the person video, and they correspond one-to-one. Each frame in the facial animation noise data is obtained by adding noise to the corresponding 2D person image in the person video. For example, the face region of each frame of the 2D person image in the person video can be extracted, and the face region can be masked to preserve the background region, thus obtaining each frame of the facial animation noise data with added noise.

[0060] The Transformer architecture diffusion model uses a first constraint on the input to iteratively denoise the input facial animation noise data until noise-free target facial animation data is obtained. The Transformer architecture diffusion model includes a first linear transformation layer, a second linear transformation layer, a third linear transformation layer, and a Transformer decoder. In each iteration, the facial model in the first constraint remains unchanged, the denoising steps are continuously reduced, the speech features in the first constraint can be directly input to the Transformer decoder, and other content in the first constraint besides speech features, along with the denoising steps, are linearly transformed by the first linear transformation layer and then input to the Transformer decoder. The facial animation noise data is linearly transformed by the second linear transformation layer and then input to the Transformer decoder. The Transformer decoder processes the input content, and the output result is linearly transformed by the third linear transformation layer to obtain the denoising result of this iteration.

[0061] Subsequently, using the target face animation data obtained from the Transformer architecture diffusion model and the face model in the 3D digital human model data, multi-frame normal maps are generated. For example, the target 3D digital human model data can be directly fused with the face model in the 3D digital human model data, and multi-frame normal maps can be generated using the target 3D digital human model data.

[0062] In this embodiment of the invention, the whole consisting of the aforementioned speech encoding module and the Transformer architecture diffusion model can be referred to as the three-dimensional facial animation generation model.

[0063] In this embodiment of the invention, the target face animation data is generated using a Transformer architecture diffusion model. The Transformer architecture diffusion model combines the sequence modeling advantages of Transformer with the generation capabilities of the diffusion model, enabling it to better understand and learn the inherent patterns of the data. At the same time, it has high robustness to noisy data, ensuring the accuracy and reliability of the target face animation data. This, in turn, improves the accuracy and reliability of multi-frame normal maps, which helps to enhance user satisfaction with digital human videos generated through multi-frame normal maps.

[0064] Based on the above embodiments, the first constraint condition of the Transformer architecture diffusion model is determined based on the face model and voice features, including: Generate facial animation style features based on reference style videos; Generate key facial animation data based on the target speech; Based on the face model and voice features, as well as the face animation style features and / or key face animation data, the first constraint condition is determined.

[0065] Specifically, in order to improve the generation effect of the target face animation data, in addition to including the face model and voice features, the first constraint can also introduce other information, such as face animation style features and at least one of the key face animation data.

[0066] Facial animation style features can be determined using a user-provided reference style video, which may contain T frames of reference style images. The facial animation style features can be a set of style features from each frame of the reference style image within the reference style video.

[0067] Key facial animation data can be generated from target speech, for example, from target text corresponding to target speech. This key facial animation data can be used to represent specific expressions, such as laughter, confusion, seriousness, anger, etc.

[0068] In this embodiment of the invention, by introducing facial animation style features into the first constraint, it can be ensured that the generated target facial animation data has the same style as the reference style video, allowing users to select a suitable reference style video according to the style requirements of the digital human, thereby achieving style control over the generated target facial animation data. By introducing key facial animation data into the first constraint, it can be ensured that the generated target facial animation data has specific expressions associated with the target speech, allowing users to provide suitable target speech according to the expression requirements of the digital human, thereby achieving expression control over the generated target facial animation data.

[0069] Based on the above embodiments, facial animation style features are generated based on a reference style video, including: Extract reference face animation data from each frame of the reference style image in the reference style video; The reference facial animation data is linearly transformed to obtain the linear transformation result; Style features are extracted from the linear transformation results to obtain the face animation style features of each frame of the reference style image; Face animation style features are determined based on the face animation style features of each frame of reference style image.

[0070] Specifically, when determining the style features of facial animation, the Video2Mesh module can be used to extract T-frame reference facial animation data from the T-frame reference style image of the reference style video. The reference facial animation data can include pose, mouth shape, and facial expression, which can be extracted together with adjacent reference style images.

[0071] Subsequently, the T-frame reference face animation data is linearly transformed using a linear transformation layer to obtain the linear transformation result. For example... Figure 2 As shown, the T-frame reference face animation data can be represented as x0, x1, ..., x T-1 Where x0 represents the reference face animation data of frame 1, x1 represents the reference face animation data of frame 2, and x... T-1 This indicates that the reference face animation data is used in frame T.

[0072] Subsequently, style features are extracted from the linear transformation results using a Transformer Encoder to obtain the face animation style features for each frame of the reference face animation data. The face animation style features for each frame can be represented as h0, h1, ..., h... T-1 Where h0 represents the facial animation style features of frame 1, h1 represents the facial animation style features of frame 2, and h... T-1 This represents the facial animation style characteristics of frame T.

[0073] The final face animation style feature, style s, can be obtained by performing average pooling on the face animation style features of each frame.

[0074] In this embodiment of the invention, by extracting facial animation data and style features from each frame of the reference style image in the reference style video and then fusing them, the final facial animation style features can be determined to represent the overall style of the reference style video, providing a more accurate and complete style reference for the Transformer architecture diffusion model.

[0075] Based on the above embodiments, key facial animation data is generated based on the target speech, including: Determine the target text corresponding to the target speech; The target text is input into a pre-trained key face animation generation model to obtain key face animation data output by the key face animation generation model.

[0076] Specifically, when determining key facial animation data, the target text corresponding to the target speech can be determined first. This target text can be provided directly by the user or obtained by converting the target speech provided by the user through Automatic Speech Recognition (ASR) technology.

[0077] The target text is input into a pre-trained Key Animation (KA) generation model. The KA generation model processes the target text to obtain and output key face animation data. The KA generation model can be built based on a large language model and can be trained using text samples and key face animation data samples.

[0078] When the user provides the target speech, the correspondence between the target speech and the key facial animation data can be established by the correspondence between the target speech and the target text.

[0079] In this embodiment of the invention, the introduction of the KA generation model can improve the generation efficiency and accuracy of key facial animation data.

[0080] It should be understood that, in this embodiment of the invention, target speech and target text can be mutually converted using automatic speech recognition and speech synthesis technologies. Specifically, if the user only provides target text, the corresponding target speech can be generated using speech synthesis technology; if the user only provides target speech, the corresponding target text can be converted using automatic speech recognition technology. Therefore, when generating key facial animation data, text-based data needs to be input into the key facial animation generation model; if the user only provides target speech, it can be converted into target text. Similarly, when generating facial animation data, speech-based data needs to be input into the 3D facial animation generation model; if the user only provides target text, it can be converted into target speech.

[0081] Based on the above embodiments, the target speech is encoded to obtain speech features, including: Divide the target speech into multiple speech segments; Based on the speech coding module, speech features of multiple speech segments are extracted.

[0082] Specifically, when encoding the target speech, in order to facilitate the processing of the target speech, the target speech can first be windowed, dividing the target speech into multiple speech segments. Each time window corresponds to one speech segment. The length of the time window can be set as needed, for example, it can be 4 seconds, or other lengths.

[0083] Subsequently, the speech coding module is used to extract speech features from multiple speech segments. For example, each speech segment can be directly input into the speech coding module to obtain the speech features of each speech segment output by the speech coding module.

[0084] In this embodiment of the invention, by dividing the target speech into multiple speech segments and processing them separately, the processing efficiency of the target speech can be improved, and the situation where encoding failure occurs due to an error in processing a certain speech segment can be avoided.

[0085] Based on the above embodiments, speech features of multiple speech segments are extracted using the speech coding module, including: For any speech segment other than the first speech segment among multiple speech segments, input any speech segment and the tail speech in the previous speech segment into the speech coding module to obtain multi-frame speech features output by the speech coding module.

[0086] Specifically, to prevent windowing processing from disrupting the continuity of speech segments, the speech encoding module can input any speech segment along with the tail portion of the previous speech segment when extracting speech features. This allows the speech encoding module to simultaneously perform feature encoding on all input speech segments, thereby obtaining and outputting multi-frame speech features. The length of the tail portion can be set as needed, for example, 1 / 10 or 1 / 8 of the speech segment length.

[0087] For example, if the duration of any speech segment is 4 seconds, corresponding to 240 frames of speech features, then the duration of the last speech segment in the previous speech segment is 0.5 seconds, corresponding to 30 frames of speech features. In this case, the speech encoding module outputs 270 frames of speech features each time.

[0088] In this embodiment of the invention, by splicing the speech of two adjacent speech segments, the speech encoding module can automatically associate adjacent speech segments during speech encoding to ensure the correlation of the speech features of each speech segment.

[0089] Based on the above embodiments, the number of frames of multi-frame speech features is the same as that of facial animation noise data; based on the Transformer architecture diffusion model, and applying the first constraint condition, the facial animation noise data corresponding to the person video is denoised to generate target facial animation data, including: Align multi-frame speech features with face animation noise data; After alignment, based on the Transformer architecture diffusion model, the first constraint condition is applied to denoise the facial animation noise data corresponding to the person video, and generate the target facial animation data.

[0090] Specifically, the facial animation noise data corresponding to the person video can be the same as the number of frames of speech features output by the speech coding module each time. Therefore, when generating target facial animation data, multiple frames of speech features can be aligned with facial animation noise data first, so that the facial animation at a certain moment is only associated with the speech features at that moment, avoiding the problem of audio-visual inconsistency when generating digital human videos later due to the lack of association between speech features and facial animation.

[0091] After alignment, the Transformer architecture diffusion model can be used to apply the first constraint to denoise the face animation noise data and generate the target face animation data.

[0092] like Figure 3a As shown, the speech coding module integrates the Onset Extractor into the Hubert Endcoer. By introducing the Onset Extractor, the energy peaks of the input speech can be detected. Since each sound corresponds to a change in energy level when a person speaks, by detecting the duration of the interval between each energy peak, the duration of the interval between sounds when a person speaks can be determined, thereby determining the speaking rhythm.

[0093] exist Figure 3a In this process, the input to the speech coding module can include the last speech segment (audio1) in the previous time window and the current speech segment (audio2) in the current time window. The output of the speech coding module includes the Tp frame speech features a corresponding to the last speech segment in the previous time window. -Tp ... a -1 And the Tw frame speech features a0, a1, ..., a in the current speech segment Tw-1 .

[0094] On the other hand, the key facial animation data (flags1) corresponding to the previous time window and the key facial animation data (flag2) corresponding to the current time window are both input into a pre-trained vectorization model, which can be an embedding network. The vectorization model can then vectorize flags1 and flags2, transforming the key facial animation data (flags1) corresponding to the previous time window into the vectorized result k of Tp frames. -Tp ..., k -1 And the key face animation data (flag2) corresponding to the current time window is converted into Tw frame vectorization results k0, k1, ..., k Tw-1 This facilitates frame-by-frame alignment of key facial animation data with speech features.

[0095] Subsequently, the multi-frame speech features output by the speech coding module and the multi-frame vectorized results output by the vectorization model are concatenated frame by frame, i.e., by... Figure 3a This is implemented in the C unit. Then, the stitching results are aligned with the face animation noise data, i.e., an alignment mask.

[0096] like Figure 3b As shown, after alignment, the multi-frame speech features output by the speech coding module and the multi-frame vectorization results output by the vectorization model are input together into the Transformer decoder (i.e., Transformer Decoder).

[0097] Meanwhile, the first linear transformation layer (Linear1) of the Transformer architecture diffusion model receives other contents in the first constraint except for speech features and performs linear transformation, such as face model shape β, face animation style features style s, etc.

[0098] The second linear transformation layer (Linear2) receives the facial animation noise data and performs a linear transformation. The second linear transformation layer receives the facial animation noise data of the current frame, that is, the facial animation noise data corresponding to the current time window. Simultaneously, it also receives the tail denoising result of the last part of the speech segment corresponding to the face animation noise data of the previous frame, that is, the tail denoising result of the last part of the speech segment corresponding to the face animation noise data of the previous time window. .

[0099] The outputs of the first and second linear transform layers serve as inputs to the Transformer decoder. The Transformer decoder processes the input, and the processed output is then linearly transformed by the third linear transform layer (Linear3) to obtain the denoising result for this iteration, including the predicted result of the tail denoising result corresponding to the previous frame's face animation noise data. and the denoising results corresponding to the current frame's face animation noise data. .

[0100] During the iteration process, the denoising step t starts from n, that is, the initial value of the denoising step t is n, such as... Figure 3b The phrase "startwith t==n" means that after completing one iteration, the diffusion probability model solver (DPM solver) is used to reduce the denoising step t, and the next iteration is performed until the denoising step t is 0. Figure 3b The "end with t==0" option in the code results in the denoising result corresponding to the noise data of the current frame's face animation. This refers to the target face animation data (motionfeature).

[0101] Based on the above embodiments, the 3D digital human model data also includes the original facial animation data of each frame of the 2D human image in the human video; based on the human video, the 3D digital human model data is extracted, including: The video of the person is input into the Video2Mesh module, where the 3D deformable face model in the Video2Mesh module extracts the face model, and the face animation data extraction unit in the Video2Mesh module extracts the original face animation data.

[0102] Specifically, when extracting 3D digital human model data, both the face model and the original face animation data can be extracted simultaneously. Based on this, such as... Figure 4 As shown, the Video2Mesh module can include 3D Deformable Face Models (3DMM) and a face animation data extraction unit. The 3DMM can extract the face model of each frame of the 2D human image in the video, and the face animation data extraction unit can extract the original face animation data.

[0103] Among them, 3DMM is a common three-dimensional face model. In 3DMM, the shape of each face can be transformed into a combination of coefficients of the basis vectors composed of all faces in the face database, thereby realizing the quantification of face shape, that is, representing the shape of a specific face through face coefficient vectors.

[0104] 3DMM can first automatically annotate the facial key points of each frame of a two-dimensional human image in a video, and then generate a face model through the annotated facial key points, which can also be called a face coefficient vector or a three-dimensional face mesh (3D mesh).

[0105] In this embodiment of the invention, by simultaneously extracting the face model and the original face animation data through different structures of the Video2Mesh module, the face model and the original face animation data can be accurately separated, ensuring the accuracy of the face model.

[0106] Based on the above embodiments, such as Figure 5 As shown, the normal map generation method provided in this embodiment of the invention includes: Receive video of a person and determine the target voice; The Video2Mesh module was used to determine the face model of each frame of a 2D human image in a video. The target speech is converted into target text, and key facial animation data is determined from the target text. Key facial animation data, target speech, facial animation style features, and facial model are all used as inputs to the 3D facial animation generation model. The speech encoding module in the 3D facial animation generation model encodes the target speech to obtain speech features. By using speech features, facial animation style features, and key facial animation data, the first constraint condition of the Transformer architecture diffusion model in the 3D facial animation generation model is determined. Then, by applying the first constraint condition through the Transformer architecture diffusion model, the facial animation noise data corresponding to the character video is denoised to generate the target facial animation data. The target face animation data is input into the normal map generation model. The normal map generation model can fuse the target face animation data with the face model to obtain the target 3D digital human model data, and then use the target 3D digital human model data to generate multi-frame normal maps.

[0107] like Figure 6 As shown, based on the above embodiments, this embodiment of the invention also provides a method for generating digital human videos, including: S21, Based on the normal map generation method provided in the above embodiments, generate multiple frames of normal maps, and add noise to the face region of each frame of the two-dimensional human image in the human video to obtain each frame of the noisy image. S22, the multi-frame normal map is used as the second constraint condition of the large video generation model. Based on the large video generation model, the second constraint condition and the noisy video reference frame in the human video are applied to denoise each frame of the noisy image in the human video to obtain the digital human video.

[0108] Specifically, the digital human video generation method provided in this embodiment of the invention is executed by a digital human video generation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0109] First, step S21 is executed. Given a video of a person and target audio or text provided by the user, the normal map generation method provided in the above embodiments is used to generate multiple frames of normal maps. Simultaneously, the face region of each frame of the two-dimensional person image in the video can be extracted, and noise can be added to the face region using a mask overlay method to obtain a noisy image for each frame.

[0110] Next, step S22 is executed, using the multi-frame normal maps as the second constraint condition for generating a large model from the video, such as... Figure 7As shown, the second constraint, the denoised video reference frames in the character video, and each denoised image frame are all input into the large-scale video generation model. The large-scale video generation model uses the second constraint and the denoised video reference frames in the character video to denoise each denoised image frame in the character video, obtaining a multi-frame normal map. Figure 1 A digital human video is obtained by sorting multiple frames of two-dimensional human images in chronological order.

[0111] The large-scale video generation model employed can be built based on the Transformer architecture diffusion model (DiffusionTransformer), including an encoder, a latent space input layer, a pre-trained Transformer architecture diffusion model, a latent space output layer, and a decoder. The encoder extracts high-dimensional features from the input data, the latent space input layer reduces the dimensionality of these features to decrease the amount of data processed by the model, the Transformer architecture diffusion model iteratively denoises the data, the latent space output layer restores the denoised low-dimensional features to high-dimensional features, and the decoder generates a two-dimensional image of a person corresponding to the high-dimensional features.

[0112] The un-noiseed video reference frame in a person video can be any two-dimensional person image in the video. For example, the first two-dimensional person image in the video can be directly selected as the video reference frame, or other two-dimensional person images can be selected as the video reference frame. There are no specific limitations here.

[0113] The video reference frames are not denoised and can be used to provide environmental information such as background images and lighting for digital human videos.

[0114] The digital human video generation method provided in this embodiment of the invention, since normal maps and digital human videos both belong to image data, uses multi-frame normal maps generated in the above embodiments as the second constraint condition of the large-scale video generation model. By generating digital human videos through this large-scale model, the parameter scale and training requirements of the large-scale model can be reduced. Using a certain amount of high-quality data for training, the mouth shape, facial expressions, and posture changes of the digital human in the generated video can be almost identical to those of a real person, improving the quality of the digital human videos and thus enhancing the user experience. Compared to traditional geometric modeling methods for generating digital human videos, generating digital human videos through normal maps can save more than 90% of computational resources.

[0115] Based on the above embodiments, the digital human video generation method further includes: The video of a person is segmented to obtain multiple video clips of the person; Based on the normal map generation method, multi-frame normal maps for each character video segment are determined, and for any character video segment, noise is added to the face region of each frame of the two-dimensional character image in any character video segment to obtain each frame of the noisy image. Using the multi-frame normal maps of any person's video clip as the second constraint, and based on the large model generated from the video, applying the second constraint and the video reference frame corresponding to any person's video clip, denoising is performed on each frame of the noisy image in any person's video clip to obtain the denoising result corresponding to any person's video clip. A digital human video is generated based on the denoising results of multiple video clips of people.

[0116] Specifically, when a person's video is long, it can be segmented to obtain multiple video clips. This segmentation can be based on the video's duration or frame rate, thematically, or other criteria.

[0117] Then, the above operations are performed on each person's video clip to generate a corresponding digital human video clip. Finally, the digital human video clips are sorted according to the order of their corresponding person video clips to obtain the digital human video.

[0118] For example, the normal map generation method provided in the above embodiments can be used to determine the multi-frame normal map of each character video segment, and for any character video segment, noise can be added to the face region of each frame of the two-dimensional character image in the video segment to obtain each frame of the noisy image.

[0119] Using multi-frame normal maps of any character's video clip as a second constraint, the second constraint, the corresponding video reference frame for any character's video clip, and the noisy image of each frame are input into a large-scale video generation model. The large-scale video generation model then obtains the denoising result for any character's video clip. Finally, the denoising results for each character's video clip are arranged in the order of the video clips to generate a digital human video.

[0120] Here, the video reference frame corresponding to any person's video clip can be any frame of a two-dimensional person image within any person's video clip, or any frame of a two-dimensional person image from the previous person's video clip; no specific limitation is made here.

[0121] In this embodiment of the invention, the video of a person is segmented so that the large video generation model generates only the denoising result corresponding to one video segment at a time. This not only reduces the amount of data processed by the large video generation model, but also avoids adverse effects between the denoising processes of different video segments of a person.

[0122] Based on the above embodiments, for any remaining character video segment other than the first character video segment in a character video, the video reference frame corresponding to any remaining character video segment is the last two-dimensional character image in the preceding character video segment of any remaining character video segment.

[0123] Specifically, in order to maintain the continuity of the digital human video, for any remaining human video segment other than the first human video segment, the video reference frame corresponding to any remaining human video segment can be set as the last two-dimensional human image in the preceding human video segment.

[0124] Based on the above embodiments, noise is added to the face region of each frame of a two-dimensional person image in a person video to obtain a noisy image for each frame, or noise is added to the face region of each frame of a two-dimensional person image in any person video segment to obtain a noisy image for each frame, including: For any frame of a two-dimensional human image in a video or any video clip, extract the facial contour from the two-dimensional human image in the frame and generate a binary face mask image based on the facial contour. Noise is added to a binary face mask image to obtain a noisy image of any frame of a two-dimensional human figure.

[0125] Specifically, when determining each frame of the noisy image, the facial contour in each frame of the two-dimensional human image can be determined first based on the facial region of each frame of the two-dimensional human image, and a binary face mask image can be generated based on the facial contour.

[0126] Subsequently, random noise is used to fill the binary face mask image, and the filled binary face mask image is used to cover the two-dimensional person image, thereby completing the addition of noise to the face region and obtaining each frame of the noisy image.

[0127] In this embodiment of the invention, noise is added to the face area by noise filling and masking, which can achieve rapid noise addition and the process is simple and easy to implement.

[0128] Applying the normal map generation method and digital human video generation method proposed in this invention to software products allows users to generate corresponding multi-frame normal maps and digital human videos simply by uploading a video of a person and a target audio or text. The changes in the person's appearance and posture corresponding to the multi-frame normal maps and digital human videos are identical to those in the user-uploaded video. The changes in mouth shape and facial expressions corresponding to the multi-frame normal maps and digital human videos match the user-uploaded target audio or text, allowing the digital human in the final generated video to speak according to the user-uploaded target audio or text and make corresponding mouth shape and facial expression changes. The expressions and movements are realistic and natural, almost identical to videos obtained from actual live-action filming.

[0129] like Figure 8 As shown, based on the above embodiments, this embodiment of the invention also provides a normal map generation device, including: The receiving module 81 is used to receive video of a person and determine the target voice. Extraction module 82 is used to extract 3D digital human model data based on human videos; The texture generation module 83 is used to generate multi-frame normal maps based on 3D digital human model data and target speech.

[0130] Based on the above embodiments, the normal map generation apparatus provided in this embodiment of the invention, wherein the normal map module is specifically used for: The target speech is encoded to obtain speech features; Based on the face model and speech features, the first constraint of the Transformer architecture diffusion model is determined. Based on the Transformer architecture diffusion model, the first constraint is applied to denoise the face animation noise data corresponding to the person video and generate the target face animation data. Based on the target face animation data and face model, generate multi-frame normal maps.

[0131] Based on the above embodiments, the normal map generation apparatus provided in this embodiment of the invention further includes a normal map module specifically used for: Generate facial animation style features based on reference style videos; Generate key facial animation data based on the target speech; Based on the face model and voice features, as well as the face animation style features and / or key face animation data, the first constraint condition is determined.

[0132] Based on the above embodiments, the normal map generation apparatus provided in this embodiment of the invention further includes a normal map module specifically used for: Extract reference face animation data from each frame of the reference style image in the reference style video; The reference facial animation data is linearly transformed to obtain the linear transformation result; Style features are extracted from the linear transformation results to obtain the face animation style features of each frame of the reference style image; Face animation style features are determined based on the face animation style features of each frame of reference style image.

[0133] Based on the above embodiments, the normal map generation apparatus provided in this embodiment of the invention further includes a normal map module specifically used for: Determine the target text corresponding to the target speech; The target text is input into a pre-trained key face animation generation model to obtain key face animation data output by the key face animation generation model.

[0134] Based on the above embodiments, the normal map generation apparatus provided in this embodiment of the invention further includes a normal map module specifically used for: Divide the target speech into multiple speech segments; Based on the speech coding module, speech features of multiple speech segments are extracted.

[0135] Based on the above embodiments, the normal map generation apparatus provided in this embodiment of the invention further includes a normal map module specifically used for: For any speech segment other than the first speech segment among multiple speech segments, input any speech segment and the tail speech in the previous speech segment into the speech coding module to obtain multi-frame speech features output by the speech coding module.

[0136] Based on the above embodiments, the normal map generation device provided in this embodiment of the invention has the same number of frames for multi-frame speech features and face animation noise data; The normal mapping module is also specifically used for: Align multi-frame speech features with face animation noise data; After alignment, based on the Transformer architecture diffusion model, the first constraint condition is applied to denoise the face animation noise data and generate the target face animation data.

[0137] The normal mapping module is also specifically used for: The video of the person is input into the Video2Mesh module, where the 3D deformable face model in the Video2Mesh module extracts the face model, and the face animation data extraction unit in the Video2Mesh module extracts the original face animation data.

[0138] Specifically, the functions of each module in the normal map generation device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0139] like Figure 9 As shown, based on the above embodiments, this embodiment of the invention also provides a digital human video generation device, including: The preprocessing module 91 is used to generate multiple frames of normal maps based on the normal map generation method provided in the above embodiments, and to add noise to the face region of each frame of the two-dimensional human image in the human video to obtain each frame of the noisy image. The video generation module 92 is used to use multi-frame normal maps as the second constraint condition of the large video generation model, and based on the large video generation model, it applies the second constraint condition and the noisy video reference frame in the human video to denoise each noisy image in the human video to obtain the digital human video.

[0140] Based on the above embodiments, the digital human video generation device provided in this embodiment of the invention further includes a segmentation module, used for: The video of a person is segmented to obtain multiple video clips of the person; Correspondingly, the preprocessing module is also used to: determine the multi-frame normal map of each character video segment based on the normal map generation method, and for any character video segment, add noise to the face region of each frame of the two-dimensional character image in any character video segment to obtain each frame of the noisy image. The video generation module is also used to: take the multi-frame normal map of any person's video segment as the second constraint, and based on the large video generation model, apply the second constraint and the video reference frame corresponding to any person's video segment to denoise each frame of the noisy image in any person's video segment to obtain the denoising result corresponding to any person's video segment; and generate a digital human video based on the denoising results corresponding to multiple person's video segments.

[0141] Based on the above embodiments, the digital human video generation device provided in this embodiment of the invention provides that, for any remaining human video segment other than the first human video segment, the video reference frame corresponding to any remaining human video segment is the last frame of a two-dimensional human image in the preceding human video segment of any remaining human video segment.

[0142] Based on the above embodiments, the digital human video generation apparatus provided in this embodiment of the invention has a preprocessing module specifically used for: For any frame of a two-dimensional human image in a video or any video clip, extract the facial contour from the two-dimensional human image in the frame and generate a binary face mask image based on the facial contour. Noise is added to a binary face mask image to obtain a noisy image of any frame of a two-dimensional human figure.

[0143] Specifically, the functions of each module in the digital human video generation device provided in this embodiment correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment.

[0144] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the normal map generation method or the digital human video generation method provided in the above embodiments.

[0145] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the normal map generation method or the digital human video generation method provided in the above embodiments.

[0147] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the normal map generation method or the digital human video generation method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and no specific limitation is made herein.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating normal maps, characterized in that, include: Receive video of a person and determine the target voice; Based on the video of the person, extract three-dimensional digital human model data; Based on the 3D digital human model data and the target speech, a multi-frame normal map is generated.

2. The normal map generation method according to claim 1, characterized in that, The 3D digital human model data includes the face model of each frame of the 2D human image in the video; the generation of multi-frame normal maps based on the 3D digital human model data and the target speech includes: The target speech is encoded to obtain speech features; Based on the face model and the voice features, the first constraint condition of the Transformer architecture diffusion model is determined, and based on the Transformer architecture diffusion model, the first constraint condition is applied to denoise the face animation noise data corresponding to the person video to generate target face animation data. Based on the target face animation data and the face model, the multi-frame normal map is generated.

3. The normal map generation method according to claim 2, characterized in that, The determination of the first constraint condition for the Transformer architecture diffusion model based on the face model and the voice features includes: Generate facial animation style features based on reference style videos; Based on the target speech, key facial animation data is generated; Based on the face model and the voice features, as well as the face animation style features and / or the key face animation data, the first constraint condition is determined.

4. The normal map generation method according to claim 3, characterized in that, The generation of facial animation style features based on reference style videos includes: Extract reference face animation data from each frame of the reference style image in the reference style video; The reference face animation data is linearly transformed to obtain the linear transformation result; Style features are extracted from the linear transformation results to obtain the facial animation style features of each frame of the reference style image; The facial animation style features are determined based on the facial animation style features of each frame of reference style image.

5. The normal map generation method according to claim 3, characterized in that, The generation of key facial animation data based on the target speech includes: Determine the target text corresponding to the target speech; The target text is input into a pre-trained key face animation generation model to obtain the key face animation data output by the key face animation generation model.

6. The normal map generation method according to claim 2, characterized in that, The encoding of the target speech to obtain speech features includes: The target speech is divided into multiple speech segments; Based on the speech coding module, speech features of the multiple speech segments are extracted.

7. The normal map generation method according to claim 6, characterized in that, The step of extracting speech features from the multiple speech segments based on the speech coding module includes: For any speech segment other than the first speech segment among the multiple speech segments, the speech segment and the tail speech in the previous speech segment are input into the speech encoding module to obtain multi-frame speech features output by the speech encoding module.

8. The normal map generation method according to claim 7, characterized in that, The number of frames in the multi-frame speech features is the same as the number of frames in the face animation noise data; The step of denoising the facial animation noise data corresponding to the character video based on the Transformer architecture diffusion model and applying the first constraint to generate target facial animation data includes: Align the multi-frame speech features with the facial animation noise data; After alignment, based on the Transformer architecture diffusion model, the first constraint condition is applied to denoise the face animation noise data to generate the target face animation data.

9. The normal map generation method according to claim 2, characterized in that, The 3D digital human model data also includes the original facial animation data of each frame of the 2D human image in the video; the extraction of 3D digital human model data based on the video includes: The video of the person is input into the Video2Mesh module, where the 3D deformable face model in the Video2Mesh module extracts the face model, and the face animation data extraction unit in the Video2Mesh module extracts the original face animation data.

10. A method for generating digital human videos, characterized in that, include: Based on the normal map generation method as described in any one of claims 1-9, multiple frames of normal maps are generated, and noise is added to the face region of each frame of the two-dimensional human image in the human video to obtain a noisy image for each frame. The multi-frame normal maps are used as the second constraint condition for the large video generation model. Based on the large video generation model, the second constraint condition and the noisy video reference frames in the human video are applied to denoise each frame of the noisy image in the human video to obtain the digital human video.

11. The digital human video generation method according to claim 10, characterized in that, Also includes: The video of the person is segmented to obtain multiple video clips of the person; Based on the normal map generation method, a multi-frame normal map of each character video segment is determined, and for any character video segment, noise is added to the face region of each frame of the two-dimensional character image in the any character video segment to obtain a noisy image of each frame. Using the multi-frame normal maps of any character video segment as the second constraint, and generating a large model based on the video, the second constraint and the video reference frame corresponding to any character video segment are applied to denoise each frame of the noisy image in any character video segment to obtain the denoising result corresponding to any character video segment. The digital human video is generated based on the denoising results corresponding to the multiple video clips of the individuals.

12. The digital human video generation method according to claim 11, characterized in that, For any remaining video segment of a person other than the first video segment, the video reference frame corresponding to any remaining video segment is the last two-dimensional image of the person in the preceding video segment.

13. The digital human video generation method according to claim 11 or 12, characterized in that, The step of adding noise to the face region of each frame of the two-dimensional character image in the character video to obtain a noisy image for each frame, or the step of adding noise to the face region of each frame of the two-dimensional character image in any character video segment to obtain a noisy image for each frame, includes: For any frame of a two-dimensional character image in the character video or any video segment of the character, extract the facial contour in the two-dimensional character image of the frame, and generate a binary face mask image based on the facial contour. The binary face mask image is filled with noise to obtain a noisy image of any frame of the two-dimensional human image.

14. A normal map generation apparatus, characterized in that, include: The receiving module is used to receive video of a person and determine the target voice. The extraction module is used to extract three-dimensional digital human model data based on the person's video; The texture generation module is used to generate multi-frame normal maps based on the 3D digital human model data and the target speech.

15. A digital human video generation device, characterized in that, include: The preprocessing module is used to generate multiple frames of normal maps based on the normal map generation method as described in any one of claims 1-9, and to add noise to the face region of each frame of the two-dimensional human image in the human video to obtain a noisy image for each frame. The video generation module is used to use the multi-frame normal maps as the second constraint condition of the large video generation model, and based on the large video generation model, apply the second constraint condition and the noisy video reference frames in the human video to denoise each noisy image in the human video to obtain a digital human video.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the normal map generation method as described in any one of claims 1-9, or the digital human video generation method as described in any one of claims 10-13.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the normal map generation method as described in any one of claims 1-9, or the digital human video generation method as described in any one of claims 10-13.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the normal map generation method as described in any one of claims 1-9, or the digital human video generation method as described in any one of claims 10-13.

Images