Method and device for generating digital human video

Abstract

The invention relates to the technical field of digital people, and provides a method and equipment for generating a digital people video, which can be applied to a digital marketing scene. According to the method, the generation of the digital human video is divided into an audio generation process and a video generation process, when the audio is generated, a target text, an emotion parameter and a reference audio are taken as input, and the target audio of the target text is broadcasted by the sound of a reference person in the reference video and a set emotion, so that sound cloning is realized, the degree of violation of the generated digital human video is reduced, and the user experience is improved. And the text features are extracted from two dimensions of characters and pronunciation, so that the pronunciation accuracy during sound cloning is ensured. When the video is generated, the cloned target audio and the cloned reference video are used as input, and the fidelity of the digital human video is improved through expression and mouth shape matching, so that a reference person can quickly and efficiently generate vivid digital human videos with different broadcast contents only by shooting one reference video without shooting one by one, thereby reducing the manufacturing cost of the digital human video, and improving the user experience. And the video production efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital humans, and provides a digital human video generation method and device. BACKGROUND

[0002] Digital humans have been widely used in various industries, such as entertainment, education, and e-commerce. With the gradual improvement of industry demand and user experience, the demand for digital human videos in various industries is also increasing.

[0003] Currently, the generation of digital human videos mainly faces the following problems: 1. The pronunciation is not accurate when playing content involving numbers and unit symbols, such as 250Hz being read as "two five zero H z"; 2. The expression of the digital human is inconsistent with the emotion of the broadcast content, such as the emotion of the reference video being happy, and when the emotion of the broadcast content is sad, the expression of the digital human still remains happy; 3. The digital human's facial expression does not match the broadcast content, such as the digital human's facial expression being closed when broadcasting "ah". These problems can cause the generated digital human video to have a sense of discomfort and poor authenticity. SUMMARY

[0004] The embodiments of the present application provide a digital human video generation method and device to improve the authenticity and efficiency of digital human videos.

[0005] In a first aspect, the embodiments of the present application provide a digital human video generation method, comprising:

[0006] obtaining a target text and a reference video, and separating a reference audio from the reference video;

[0007] inputting the reference audio, the target text, and a preset emotion parameter into a trained audio generation model, generating a target audio in which a reference character broadcasts the target text in a voice according to a text feature of the target text, an emotion feature of the emotion parameter, and an audio feature and a voiceprint feature of the reference character in the reference audio; wherein the text feature is extracted based on a mapping relationship between a character and pronunciation;

[0008] inputting the target audio and the reference video into a trained video generation model, generating an initial digital human video in which the reference character broadcasts the target text according to a lip feature and an expression feature of the reference character in the reference video and a voice emotion feature of the target audio, and aligning an audio feature of the target audio with an image feature of the initial digital human video to obtain a target digital human video.

[0009] The beneficial effects of the above technical solutions are: dividing the digital person video generation into two generation processes of audio and video, in the generation of audio, the voice cloning technology of the digital person is realized by extracting the audio features and voiceprint features of the reference person in the reference audio, so that the target audio of the generated broadcast text retains the original voice characteristics of the reference person in the reference audio, and by combining the parameter features of the emotion parameters set by the target text, the tone is changed on the basis of the original voice color of the reference person in the reference audio, so that the mood of the reference person expressing the target text is controlled, the discomfort degree of the generated digital person video is reduced, and the accuracy of pronunciation of the reference person when broadcasting the target text is improved by extracting the text features of the target text from two dimensions of text and pronunciation. In the generation of video, the facial expression of the reference person in the initial digital person video is matched with the emotion of the target audio by extracting the lip features and expression features of the reference person in the reference video and the voice emotion features of the target audio, and the articulation shape of the reference person in the target digital person video is consistent with the content of the target audio by aligning the voice emotion features of the target audio with the image features of the initial digital person video, so as to improve the fidelity of the digital person. In this way, only one reference video needs to be shot by the reference person, and a realistic digital person video of different target text can be quickly and efficiently generated without shooting one by one, thereby reducing the production cost of digital person video and improving the video production efficiency.

[0010] Optionally, the audio generation model includes an audio encoder, a text encoder, a large language model, and a voiceprint encoder, and the target audio of the target text broadcast by the reference person is generated according to the text features of the target text, the emotion features of the emotion parameters, and the audio features and voiceprint features of the reference person in the reference audio.

[0011] The text features of the target text, the emotion features of the emotion parameters, and the audio features of the reference person in the reference audio extracted by the text encoder are fused to obtain multi-modal features;

[0012] The multi-modal features are input into the large language model to obtain associated semantic features between different modal features;

[0013] The associated semantic features are fused with the voiceprint features of the reference person in the reference audio extracted by the voiceprint encoder, and then decoded to generate the target audio of the target text broadcast by the reference person.

[0014] The beneficial effects of the above technical solutions are: the voice of the reference person in the reference audio is cloned using features of multiple modalities such as text, emotion, and audio, the large language model is used to make the features of multiple modalities pay attention to key information, and the cross-modal semantic correlation is strengthened, thereby solving the voice emotion separation problem and the pronunciation accuracy problem, and then combining the voiceprint features of the reference person in the reference audio, the target audio of the reference person in the reference audio reporting the target text is obtained.

[0015] Optionally, the training process of the audio generation model comprises:

[0016] A training sample data set is constructed, wherein each training sample data comprises original audio, to-be-generated text, pronunciation of the to-be-generated text, emotion parameters of the to-be-generated text, and real audio of the original person reporting the to-be-generated text in the original audio, and the to-be-generated text comprises at least one of Chinese characters, numbers, unit symbols, and English words.

[0017] According to the training sample data set, the text encoder and the large language model included in the audio generation model are trained in multiple rounds of iterations until a first preset stopping condition is met.

[0018] According to the training sample data set, the audio encoder and the voiceprint encoder included in the audio generation model are trained in multiple rounds of iterations until a second preset stopping condition is met.

[0019] According to the training sample data set, the text encoder, the audio encoder, and the voiceprint encoder are trained in multiple rounds of iterations until a third preset stopping condition is met.

[0020] The beneficial effects of the above technical solutions are: when constructing the training sample data set, since each training sample data contains pronunciation and emotion parameters of the to-be-generated text, the problem of inaccurate pronunciation of numbers, unit symbols, etc. during voice cloning is solved, and the problem of consistent voice tone with the original audio is solved, thereby improving the accuracy of pronunciation and emotional expression during voice cloning. Moreover, using the constructed training sample data set, the audio generation model is trained in three stages, thereby reducing the model training complexity, avoiding conflicts between different modalities, and gradually establishing cross-modal feature alignment capabilities. In the first stage, the text encoder and the large language model are trained, thereby specifically improving the understanding ability of the audio generation model for certain special text. In the second stage, the audio encoder and the voiceprint encoder are trained, thereby enabling the model to independently process audio-related encoding capabilities and avoiding interference with text modal features. In the third stage, the text encoder, the audio encoder, and the voiceprint encoder are jointly trained, thereby realizing cross-modal feature alignment and collaborative optimization.

[0021] Optionally, the video generation model comprises a facial expression capturer, a voiceprint encoder, a multi-perception attention mechanism and hash modeling, and the generation of the initial digital human video in which the reference person reports the target text according to the lip features and expression features of the reference person in the reference video and the sound emotion features of the target audio comprises:

[0022] The lip features and expression features of the reference person in the reference video are extracted by the facial expression capturer, and the sound emotion features of the target audio are extracted by the voiceprint encoder;

[0023] The lip features, the expression features and the sound emotion features are weighted by the multi-perception attention mechanism to obtain facial expression features;

[0024] The facial expression features are mapped to an emotion hash code by hash modeling, and an initial digital human video in which the facial expression matches the target audio is generated according to the emotion hash code.

[0025] The above technical solution has the beneficial effects that the collaborative application of the multi-perception attention mechanism and the hash modeling realizes the dual breakthroughs of high-precision emotion mapping and efficient feature processing. Since the multi-perception attention mechanism can correlate the sound emotion features with the expression and lip features in time and space, the compatibility of the digital human expression change and the audio emotion is improved. Moreover, by hash modeling, the facial expression features of multiple modalities are mapped to an emotion hash code in a hash space, supporting fast expression matching across modalities and ensuring the generation efficiency of the digital human video.

[0026] Optionally, the video generation model further comprises a variational autoencoder, an audio encoder and a generation model, and the alignment of the audio features of the target audio with the image features of the initial digital human video to obtain a target digital human video comprises:

[0027] The initial digital human video is decomposed into multiple frames of digital human images, which are respectively input into the variational autoencoder to obtain image features of each frame of digital human image, each image feature comprising a lip movement degree of the reference person;

[0028] The audio features in the target audio are extracted by the audio encoder;

[0029] The audio features are time-series aligned with the lip movement degree corresponding to each frame of digital human image by the generation model to generate a target digital human video in which the lip movement matches the target audio.

[0030] The technical scheme has the beneficial effects that: the variational autoencoder is used to compress high-dimensional image features into low-dimensional mouth movement degrees, so that the mouth changes are retained while the redundant features are removed, and the problems of "audio-visual misalignment" and "mechanical mouth movement" in traditional digital human generation are effectively avoided by aligning the mouth movement degrees in the image with the audio features in the audio in the time dimension.

[0031] Optionally, the target text includes at least one of a character, a number, a unit symbol, and an English word.

[0032] The technical scheme has the beneficial effects that: the audio generation model can be applied to the target text containing various forms of content, and the pronunciation accuracy can be ensured.

[0033] Optionally, the emotion parameter includes an emotion type and an emotion intensity.

[0034] The technical scheme has the beneficial effects that: by setting the emotion type and the emotion intensity for the playing text, the controllability of the digital human emotion is improved on the basis of retaining the reference character voice in the reference audio.

[0035] In a second aspect, an electronic device is provided, including a processor, a memory, and a communication interface, the communication interface, the memory, and the processor are connected through a bus;

[0036] The communication interface is configured to transmit and receive data.

[0037] The memory stores a computer program, and the processor executes the following operations according to the computer program:

[0038] Obtaining a target text and a reference video, and separating a reference audio from the reference video;

[0039] Inputting the reference audio, the target text, and a preset emotion parameter into a trained audio generation model, and generating a target audio of playing the target text in the voice of a reference character according to a text feature of the target text, an emotion feature of the emotion parameter, and an audio feature and a voiceprint feature of the reference character in the reference audio, wherein the text feature is extracted based on a mapping relationship between a character and pronunciation;

[0040] Inputting the target audio and the reference video into a trained video generation model, and generating an initial digital human video of playing the target text by the reference character according to a lip feature and an expression feature of the reference character in the reference video and a voice emotion feature of the target audio, and aligning the audio feature of the target audio with an image feature of the initial digital human video to obtain a target digital human video.

[0041] Optionally, the audio generation model comprises an audio encoder, a text encoder, a large language model, and a voiceprint encoder, and the processor generates target audio in which the target text is broadcast in the voice of the reference person according to the text features of the target text, the emotional features of the emotional parameters, and the audio features and voiceprint features of the reference person in the reference audio, specifically by:

[0042] performing feature fusion on the text features of the target text extracted by the text encoder, the emotional features of the emotional parameters, and the audio features of the reference person in the reference audio extracted by the audio encoder, to obtain multi-modal features;

[0043] inputting the multi-modal features into the large language model to obtain associated semantic features between different modal features;

[0044] decoding the associated semantic features after being fused with the voiceprint features of the reference person in the reference audio extracted by the voiceprint encoder to generate target audio in which the target text is broadcast in the voice of the reference person.

[0045] Optionally, the process of training the audio generation model by the processor comprises:

[0046] constructing a training sample data set, wherein each training sample data comprises original audio, to-be-generated text, pronunciation of the to-be-generated text, emotional parameters of the to-be-generated text, and real audio in which an original person broadcasts the to-be-generated text, and the to-be-generated text comprises at least one of Chinese characters, numbers, unit symbols, and English words;

[0047] performing multi-round iterative training on the audio generation model comprising a text encoder and a large language model according to the training sample data set until a first preset stopping condition is met;

[0048] performing multi-round iterative training on the audio generation model comprising an audio encoder and a voiceprint encoder according to the training sample data set until a second preset stopping condition is met;

[0049] performing multi-round iterative training on the text encoder, the audio encoder, and the voiceprint encoder according to the training sample data set until a third preset stopping condition is met.

[0050] Optionally, the video generation model comprises a facial expression capturer, a voiceprint encoder, a multi-perception attention mechanism, and a hash modeling, and the processor generates an initial digital human video in which the reference person broadcasts the target text according to the lip features and expression features of the reference person in the reference video and the voice emotional features of the target audio, specifically by:

[0051] The lip feature and the expression feature of the reference character in the reference video are extracted by using the facial expression capture, and the sound emotion feature of the target audio is extracted by using the voiceprint encoder;

[0052] The facial expression feature is obtained by weighting the lip feature, the expression feature and the sound emotion feature through a multi-perception attention mechanism.

[0053] The emotion hash code corresponding to the facial expression feature is determined through hash modeling, and an initial digital human video matching the facial expression and the target audio is generated according to the emotion hash code.

[0054] Optionally, the video generation model further includes a variational autoencoder, an audio encoder and a generation model, and the processor aligns the audio feature of the target audio with the image feature of the initial digital human video to obtain a target digital human video, and the specific operation is as follows:

[0055] The initial digital human video is decomposed into multiple frames of digital human images, and each frame of digital human image is input into the variational autoencoder to obtain an image feature of each frame of digital human image, and each image feature includes a lip movement degree of the reference character.

[0056] The audio feature in the target audio is extracted by using the audio encoder.

[0057] The audio feature is time-aligned with the lip movement degree corresponding to each frame of digital human image through the generation model to generate a target digital human video matching the lip movement and the target audio.

[0058] Optionally, the target text includes at least one of a character, a number, a unit symbol and an English word.

[0059] Optionally, the emotion parameter includes an emotion type and an emotion intensity.

[0060] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions can implement the steps of any one of the digital human video generation methods when executed.

[0061] The technical effects brought by any one of the implementation manners of the third aspect to the third aspect can be referred to the technical effects brought by the corresponding implementation manners of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced as follows. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0063] Figure 1 A digital human video generation method flowchart provided by the embodiment of the present application;

[0064] Figure 2 A system architecture diagram of an audio generation model provided by the embodiment of the present application;

[0065] Figure 3 A target audio generation flowchart provided by the embodiment of the present application;

[0066] Figure 4 A training architecture diagram of an audio generation model provided by the embodiment of the present application;

[0067] Figure 5 A digital human video generation flowchart provided by the embodiment of the present application;

[0068] Figure 6 A target digital human video generation flowchart provided by the embodiment of the present application;

[0069] Figure 7 A target digital human video generation effect diagram provided by the embodiment of the present application;

[0070] Figure 8 A system architecture diagram of a video generation model provided by the embodiment of the present application;

[0071] Figure 9 A digital human video generation system provided by the embodiment of the present application;

[0072] Figure 10 A structural diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments described in the present application document, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the technical solutions of the present application.

[0074] All other embodiments obtained by those of ordinary skill in the art based on the example embodiments shown in the present application without creative labor fall within the scope of the present application. In addition, although the disclosure in the present application is introduced according to one or several examples, it should be understood that each aspect of the disclosure can also constitute a complete technical solution independently.

[0075] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the subsequently described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0076] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, for example, those orders other than given in the embodiment illustration or description of the present application can be implemented.

[0077] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not have to be limited to those components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0078] The term "module" used in the present application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or / and software code capable of performing functions related to the element.

[0079] In the current wave of digital marketing sweeping the world, the scale and efficiency of content production have become key technologies for capturing the market. However, traditional video production mode faces problems such as high labor cost and long production cycle, therefore, automated video generation technology centered on digital human broadcasting has emerged, especially in the marketing field of mix-video and grass-roots video, digital human video generation can realize cost reduction and efficiency improvement of content generation.

[0080] The embodiment of the application provides a digital person video generation method, which divides the generation of the digital person video into two generation processes of audio and video. Wherein, in the generation of the audio, the audio features and the voiceprint features of the reference person in the reference audio are extracted, the voice cloning of the digital person is realized, the target audio of the playing text generated retains the original voice characteristics of the reference person in the reference audio, and the parameter features of the emotion parameters set in combination with the target text are used to change the tone on the basis of the original tone of the reference person, so that the emotional expression of the reference person to the target text is controlled, the discomfort degree of the generated digital person video is reduced, and the accuracy of the pronunciation of the digital person is improved by extracting the text features of the target text from two dimensions of the text and the pronunciation. In the generation of the video, the lip features and the expression features of the reference person in the reference video and the voice emotion features of the target audio are extracted, so that the facial expression of the reference person in the initial digital person video matches the content and emotion of the target audio, and the voice emotion features of the target audio are aligned with the image features of the initial digital person video, so that the digital person's lip movement is consistent with the content and emotion of the target audio, thereby improving the realism of the digital person video generation. In this way, in the content production, the reference person only needs to shoot a reference video, and a realistic digital person video of different target text can be quickly and efficiently generated without shooting one by one, thereby reducing the production cost of the digital person video and improving the video production efficiency.

[0081] Referring to Figure 1 A digital person video generation method flowchart is provided in the embodiment of the application, which mainly includes the following steps:

[0082] S101: Obtain the target text and the reference video, and separate the reference audio from the reference video.

[0083] The reference video is pre-recorded by the reference person, and the reference audio of the reference person can be obtained through audio and video classification. The content of the reference audio is different from that of the target text.

[0084] Optionally, the target text can contain different forms of text, such as at least one of Chinese characters, numbers, unit symbols, and English words.

[0085] It should be noted that the separation method of the audio and video is not limited in the embodiment of the application, for example, the existing editing tool can be used for separation, and the AI algorithm can also be used for separation.

[0086] S102: Input the reference audio, the target text and the preset emotion parameters into the trained audio generation model, generate the target audio of the target text played by the reference person according to the text features of the target text, the emotion features of the emotion parameters, and the audio features and the voiceprint features of the reference person in the reference audio.

[0087] By migrating the original voice of the reference character in the reference audio to the target text, voice cloning can be achieved, thereby reducing the digital human video recording process. However, when the target text contains numbers, unit symbols or English words, voice cloning may have problems with inaccurate pronunciation. To this end, the embodiment of the present application builds an audio generation model for multi-modal mixed cloned voice when voice cloning, which takes the target text, the reference audio and the emotion parameter as multi-modal input. Through the learned mapping relationship between the text and the pronunciation (such as pinyin, phonetic symbols, etc.), the pronunciation of numbers, unit symbols and English words can be calibrated, which can be applied to target texts containing various forms of content. Moreover, through the input emotion parameter corresponding to the target text, the tone can be changed based on the original voice of the reference character in the reference video, thereby realizing the voice cloning technology with accurate pronunciation and controllable emotion.

[0088] In an embodiment, the emotion parameter includes emotion type and emotion intensity, wherein the emotion type includes but is not limited to happy, angry, peaceful, surprised, and fearful, etc., each emotion type can correspond to a unique identifier, and the emotion intensity can be a value between 0 and 1, the greater the value, the higher the emotion intensity.

[0089] Taking the input text "240Hz super high brush" as an example, as shown in Table 1, the pinyin and emotion parameter corresponding to the input text are shown, wherein the numbers in the pronunciation of each character represent the tone of pinyin.

[0090] Table 1, Pinyin and emotion parameter of input text

[0091]

[0092] On the basis of the reference audio, an audio generation model for voice cloning is constructed by mixing the target text composed of at least one of Chinese characters, numbers, unit symbols and English words, the pronunciation of the target text and the emotion parameter. In this way, for specific numbers, unit symbols, English words, etc. in the target text, the pronunciation can be calibrated by pinyin or phonetic symbols, thereby controlling the final pronunciation of the digital human, and by inputting the emotion parameter, the tone of the final pronunciation is controlled, thereby controlling the specific emotion expressed by the target text.

[0093] As Figure 2 shown, the system architecture diagram of the audio generation model provided by the embodiment of the present application mainly includes an audio encoder, a text encoder, a large language model and a voiceprint encoder. By inputting the target text, the reference audio and the emotion parameter, the target audio with the same tone as the digital human in the reference audio can be obtained, thereby realizing voice cloning.

[0094] It should be noted that, as 2 is only an example of an audio generation model, and is not a restrictive requirement for the model structure, for example, the number of text encoders can be multiple or one.

[0095] As Figure 3 shown, a schematic diagram of the generation process of the target audio mainly includes the following steps:

[0096] S1021: The text features of the target text extracted by the text encoder, the emotional features of the emotional parameters, and the audio features of the reference character in the reference audio extracted by the audio encoder are fused to obtain multi-modal features.

[0097] The target text, emotional parameters and reference audio are input to the audio generation model in different forms, and the features can be extracted by the corresponding form encoder. The reference audio can be extracted by the audio encoder, the text features are extracted based on the mapping relationship between the text and the pronunciation, and the text features and the emotional features can be extracted by the same text encoder or different text encoders.

[0098] In one embodiment, when performing feature fusion, the features extracted by the audio encoder and the text encoder can be aligned by a mapper, so that the features of different modalities remain consistent in certain specific dimensions. In specific implementation, the audio features and the emotional features are aligned to the feature space of the text features by the mapper, and then fused with the text features to obtain multi-modal features.

[0099] Optionally, the audio encoder includes but is not limited to a Whisper encoder, the text encoder includes but is not limited to a CLIP encoder and a BERT encoder, and the mapper includes a multi-layer perceptron, a cross-attention mechanism, a Q-Former, etc.

[0100] S1022: Input the multi-modal features into a large language model to obtain associated semantic features between different modalities.

[0101] By inputting the multi-modal features into the large language model, the associated semantic features between different modalities are captured, so as to fuse multi-dimensional information to improve the naturalness, accuracy and scene adaptability of the target audio generation.

[0102] S1023: After the associated semantic features are fused with the voiceprint features of the reference character in the reference audio extracted by the voiceprint encoder, the target audio in which the target text is broadcast by the voice of the reference character is generated.

[0103] The voice of each person is unique, and the voiceprint encoder can obtain the unique timbre of the reference person in the reference audio by referring to the audio input. After aligning the associated semantic features output by the large language model and fusing, the target video of the reference person in the reference audio is obtained by decoding the target text through the audio decoder.

[0104] Optionally, the voiceprint encoder includes but is not limited to a WavLM encoder.

[0105] In an embodiment of the present application, the audio generation model uses features of multiple modalities such as text, emotion and audio when cloning the voice of the reference person in the reference audio. The large language model allows the features of multiple modalities to focus on key information, and the cross-modal semantic association is strengthened, thereby solving the voice emotion separation problem and the pronunciation accuracy problem, and then combining the voiceprint features of the reference person in the reference audio, obtaining the target audio of the reference person in the reference audio. The target text is broadcast.

[0106] The audio generation model for generating the target audio can be pre-trained, as shown in Figure 4 The training architecture of the audio generation model is shown in Figure 1, which provides a hybrid modeling voice cloning method that can generate audio with set emotions consistent with the original timbre, and mix Chinese characters, numbers, unit symbols, English words and other text forms to improve pronunciation accuracy.

[0107] First, based on the model architecture, a training sample data set is constructed, wherein each training sample data in the training sample data set includes original audio, to-be-generated text, pronunciation of to-be-generated text, emotion parameters of to-be-generated text, and real audio of the original person in the original audio. Broadcast the to-be-generated text, so that each training sample data contains the pronunciation and emotion parameters of the to-be-generated text, thereby solving the problem of inaccurate pronunciation of numbers, unit symbols and other special characters during voice cloning, and the problem of consistent tone between cloned voice and original audio, and improving the accuracy of pronunciation and emotion expression during voice cloning.

[0108] In one embodiment, the training sample data set includes more than 5000 data, wherein the data containing numbers, unit symbols and other special characters accounts for more than 80%.

[0109] Then, based on the constructed training sample data set, the audio generation model is trained, and the training method is as follows:

[0110] 1. According to the training sample data set, the audio generation model including the text encoder and the large language model is trained in multiple rounds until the first preset stop condition is met.

[0111] In the first stage, the text encoder and the large language model are trained. The to-be-generated text, the text pronunciation and the emotion parameter in the training sample data set are input into the corresponding text encoder, and the output of the text encoder is fused and input into the large language model, so as to improve the understanding ability of the audio generation model for some special characters.

[0112] 2. According to the training sample data set, the audio encoder and the voiceprint encoder included in the audio generation model are trained in multiple rounds of iterations until the second preset stopping condition is met.

[0113] In the second stage, the audio encoder and the voiceprint encoder are trained. The original audio in the training sample data set is input into the audio encoder and the voiceprint encoder respectively, so that the model can independently process the encoding ability related to the audio, and avoid interference with the text modal features.

[0114] 3. According to the training sample data set, the text encoder, the audio encoder and the voiceprint encoder are trained in multiple rounds of iterations until the third preset stopping condition is met.

[0115] In the third stage, the text encoder, the audio encoder and the voiceprint encoder are jointly trained. The to-be-generated text, the text pronunciation, the emotion parameter and the original audio in the training sample data set are input into the corresponding text encoder, audio encoder and voiceprint encoder, respectively, to realize cross-modal feature alignment and collaborative optimization.

[0116] In the embodiments of the present application, the training sample data set is constructed, and the audio generation model is trained in three stages, so as to reduce the model training complexity, avoid the conflict between different modalities, and gradually establish the cross-modal feature alignment capability.

[0117] During the training process, the loss of the audio generation model can be obtained by the broadcast audio output by the model and the real audio. In specific implementation, the broadcast audio generated by the model and the real audio are converted into mel-ton spectrum, and then the difference between the mel-ton spectrum is calculated to obtain the loss of the model, which is expressed as follows:

[0118]

[0119] Wherein, Mel() is the mel-ton spectrum function.

[0120] Through the above training method, the audio generation model with accurate pronunciation of special characters such as numbers, unit symbols and English words and controllable emotion can be obtained.

[0121] It should be noted that during the audio generation model training process, the size of the training sample data set is not limited by the embodiments of the present application, and the first preset stop condition, the second preset stop condition and the third preset stop condition can be different or the same, and the embodiments of the present application do not make any limiting requirements.

[0122] S103: input the target audio and the reference video into the trained video generation model, generate an initial digital human video in which the reference character reports the target text according to the lip features and expression features of the reference character in the reference video and the sound emotion features of the target audio, and align the audio features of the target audio with the image features of the initial digital human video to obtain the target digital human video.

[0123] Based on the reference character in the reference video, a way of emotion-driven digital human is provided, which can generate digital human video with matching facial expressions according to the emotional fluctuations and tone changes of the target audio, such as upturned corners of the mouth when happy and downturned corners of the mouth when angry; in addition, digital human video with teeth alignment can be generated according to the content of the target audio.

[0124] In an embodiment, the video generation model includes a facial expression capturer, a voiceprint encoder, a multi-perception attention mechanism and a hash modeling. The generation process of the facial expression matching initial digital human video is as shown in Figure 5 , mainly including the following steps:

[0125] S1031: the lip features and expression features of the reference character in the reference video extracted by the facial expression capturer, and the sound emotion features of the target audio extracted by the voiceprint encoder.

[0126] The reference video is input into the facial expression capturer to obtain the expression features and lip features of the reference character by extracting facial key points, and the target audio is input into the voiceprint encoder to obtain the sound emotion features of the target audio.

[0127] Optionally, the facial expression capturer can be built based on a convolutional neural network (CNN), and the voiceprint encoder can be the same as the voiceprint encoder in the audio generation model.

[0128] S1032: the lip features, expression features and sound emotion features are weighted by the multi-perception attention mechanism to obtain the facial expression features.

[0129] Since the multi-perception attention mechanism can associate the sound emotion features with the expression features and the lip features in time and space, the fit degree of the digital human expression change and the audio emotion is improved.

[0130] S1033: Determine the emotional hash code corresponding to the facial expression feature through hash modeling, and generate an initial digital human video matching the facial expression and the target audio according to the emotional hash code.

[0131] By mapping the facial expression features combined by multiple modalities to the emotional hash code in the hash space, the high-dimensional features are compressed into binary codes, supporting fast expression matching across modalities and ensuring the generation efficiency of the digital human video.

[0132] In generating the initial digital human video, through the cooperative application of multi-perception attention mechanism and hash modeling, a dual breakthrough of high-precision emotional mapping and efficient feature processing is realized, effectively improving the consistency of the facial expression of the reference character and the emotion and tone of the target audio, thereby improving the realism of the digital human generation.

[0133] In an embodiment, the video generation model further includes a variational autoencoder (VAE), an audio encoder, and a generation model. Therefore, after obtaining the initial digital human video, the audio features of the target audio and the image features of the initial digital human video can be further aligned to obtain a target digital human video consistent with the target audio in terms of the digital human appearance.

[0134] As shown in Figure 6 , the generation process of the target digital human video is as shown in Figure 6 , which mainly includes the following steps:

[0135] S1034: Decompose the initial digital human video into multiple frames of digital human images, and input each frame of digital human image into the variational autoencoder to obtain the image features of each frame of digital human image.

[0136] Unlike traditional autoencoders, the VAE encoder not only outputs the latent vector but also estimates the probability distribution parameters (such as mean and variance) of the latent vector, enabling the model to understand the uncertainty of the data. Therefore, for each frame of digital human image, the image features include the degree of lip movement of the reference character.

[0137] S1035: Extract the audio features in the target audio using the audio encoder.

[0138] The audio encoder in the video generation model can be the same as the encoder in the audio generation model.

[0139] S1036: Align the audio features and the degree of lip movement corresponding to each frame of digital human image in time sequence through the generation model to generate a target digital human video matching the lip movement and the target audio.

[0140] In generating the target digital human video, the high-dimensional image features are compressed into low-dimensional mouth movement degrees by the variational autoencoder, the changes in the mouth shape are retained while the redundant features are removed, and by aligning the mouth movement degrees in the image with the audio features in the audio in the time dimension, the problems such as "misalignment of audio and video" and "mechanical mouth shape" in traditional digital human generation are effectively avoided.

[0141] As shown in Figure 7 , the digital human video generation effect diagram provided by the embodiment of the application, the facial expression of the digital human in the target digital human video and the emotion and content matching of the target text of the mouth shape.

[0142] In an embodiment, the system structure of the video generation model is as shown in Figure 8 , the main facial expression capturer, the voiceprint encoder, the multi-perception attention mechanism, the hash modeling, the variational autoencoder, the audio encoder and the generation model, the reference video and the target audio are modeled by the facial expression capturer and the voiceprint encoder, the facial expression features of the lips, the expression features and the sound emotion features are obtained by using the multi-perception attention mechanism, and based on the hash modeling, the initial digital human video with matched expression is generated, then the digital human image is encoded by using the VAE encoder, the target audio is encoded by using the audio encoder, and the generated model is input into the generated model for further alignment of the image and the audio, and the target digital human video with synchronized mouth and teeth is generated.

[0143] Figure 8 The video generation model in the embodiment can be pre-trained, and the training of the video generation model is performed on the training pair data set, each training sample pair in the training pair data set includes an audio file and a facial expression and mouth shape synchronized video file, wherein the training pair data set includes different expressions and mouth shapes, for example, the audio file and the video file of normal tone, the audio file of happy tone, the video file of mouth up, the audio file of sad tone, the video file of mouth down, etc. After the training of the video generation model is completed by using the training pair data set, when applied, only the target audio and the reference video need to be input, and the digital human video with the facial expression and the emotion change of the target audio and the tone matching, and the mouth shape and the content matching of the target audio can be obtained.

[0144] Referring to Figure 9A digital person video generation system provided by an embodiment of the present application comprises an audio-video separation module, a sound cloning module and a digital person driving module, wherein the audio-video separation module is configured to separate reference audio from a reference video, the sound cloning module is configured to take the reference audio, target text and emotion parameters as input, perform sound cloning through a mixed modeling audio generation module, and output target audio in which the target text is broadcast in the voice of the reference character in the reference audio and has a set emotion; and the digital person driving module takes the classified reference video and cloned target audio as input, performs driving on the facial expression of the reference character in the reference video through a video generation model to realize spatio-temporal alignment of images and audio, and obtains a realistic digital person video with matched expression and lip shape.

[0145] The entire digital person video generation process covers technologies such as sound cloning, emotion matching and lip synchronization, so that the generated target audio is consistent with the original voice of the reference character in the reference video and meets the set emotion, and at the same time, the facial expression of the reference character in the generated target digital person video is consistent with the emotion of the target audio, the lip shape of the reference character is aligned with the content of the target audio, only one reference video is needed to obtain a digital person video with different broadcast content, which is suitable for all digital person video generation needs, meets the generalizability feature in the reliability feature, does not need to shoot the reference character one by one, and the generation process can be stopped by human intervention, meets the controllability feature in the reliability feature, at the same time, can express important factors affecting the generation result in a human understanding way, meets the explainability feature in the reliability feature, has the advantages of low cost, high realism and fast efficiency.

[0146] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0147] Based on the same technical concept, an electronic device is provided by an embodiment of the present application, which can be a notebook computer, a desktop computer or other terminal device, or a server, and can implement the steps of the above-mentioned digital person video generation method and achieve the same technical effect.

[0148] Referring to Figure 10 The electronic device comprises a processor 1001, a memory 1002 and a communication interface 1003, the communication interface 1003, the memory 1002 and the processor 1001 are connected through a bus 1004;

[0149] The communication interface 1003 is configured to transceive data;

[0150] The memory 1002 stores a computer program, and the processor 1001 executes Figure 1 the steps of the method for generating a digital human video.

[0151] It should be noted that Figure 10 This is only an example, and the electronic device can also include memory, power supply, display screen, crystal oscillator, and other conventional devices.

[0152] In the embodiments of the present application, the memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and programs required for running instant messaging functions, etc.; the data storage area can store various instant messaging information and operation instruction sets, etc. The memory can be a volatile memory, such as a random-access memory (RAM); the memory can also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory can be any other medium capable of carrying or storing a desired computer program in the form of an instruction or a data structure and capable of being accessed by a computer, but is not limited thereto. The memory can be a combination of the above memories. The processor can include one or more central processing units (CPUs), GPUs, or digital processing units, etc.

[0153] The specific connection medium between the communication interface, the memory and the processor in the embodiments of the present application is not limited. In the embodiments of the present application, the bus between the communication interface, the memory and the processor is described with a thick line in the figure, and the connection mode between other components is only schematically described and is not limited. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of description, only one thick line is described in the figure, but only one bus or one type of bus is not described.

[0154] The embodiments of the present application also provide a computer-readable storage medium for storing some instructions, which can complete the steps of the method for generating a digital human video in any of the foregoing embodiments when executed.

[0155] The embodiments of the present application also provide a computer program product for storing a computer program, which is used to execute the steps of the method for generating a digital human video in any of the foregoing embodiments.

[0156] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of other systems which are currently developed or later developed. Therefore, the present application is intended to cover all such modifications and variations of this application that are within the scope of the appended claims and their equivalents. It is intended that each element of claim 1 and 2 is independent of one another. No element of claim 1 and 2, or any other claim, is implied to depend on any other element or limitation of claim 1 and 2 or any other claim except where expressly recited in that claim.

[0157] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to this application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0158] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. ​ one or more functions specified in the flowchart and / or block diagram block or blocks.

[0160] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method of generating a digital human video, the method comprising: The method comprises: acquiring a target text and a reference video, and separating a reference audio from the reference video; inputting the reference audio, the target text and a preset emotional parameter into a trained audio generation model, generating a target audio in which the target text is broadcast by a voice of a reference person according to a text feature of the target text, an emotional feature of the emotional parameter, and an audio feature and a voiceprint feature of the reference person in the reference audio, wherein the text feature is extracted based on a mapping relationship between a character and a pronunciation; inputting the target audio and the reference video into a trained video generation model, generating an initial digital human video in which the target text is broadcast by the reference person according to a lip feature and an expression feature of the reference person in the reference video and a voice emotional feature of the target audio, and aligning an audio feature of the target audio with an image feature of the initial digital human video to obtain a target digital human video.

2. The method of claim 1, wherein, The audio generation model comprises an audio encoder, a text encoder, a large language model and a voiceprint encoder, and the generation of the target audio in which the target text is broadcast by the voice of the reference person comprises: performing feature fusion on the text feature of the target text extracted by the text encoder, the emotional feature of the emotional parameter and the audio feature of the reference person in the reference audio extracted by the audio encoder to obtain a multi-modal feature; inputting the multi-modal feature into the large language model to obtain an associated semantic feature between different modalities; decoding the associated semantic feature after fusion with the voiceprint feature of the reference person in the reference audio extracted by the voiceprint encoder to generate the target audio in which the target text is broadcast by the voice of the reference person.

3. The method of claim 1, wherein, The training process of the audio generation model comprises: constructing a training sample data set, wherein each training sample data comprises original audio, a to-be-generated text, a pronunciation of the to-be-generated text, an emotional parameter of the to-be-generated text and a real audio in which an original person broadcasts the to-be-generated text, and the to-be-generated text comprises at least one of Chinese characters, numbers, unit symbols and English words; performing multi-round iterative training on the audio generation model comprising a text encoder and a large language model according to the training sample data set until a first preset stopping condition is met; performing multi-round iterative training on the audio generation model comprising an audio encoder and a voiceprint encoder according to the training sample data set until a second preset stopping condition is met; performing multi-round iterative training on the text encoder, the audio encoder and the voiceprint encoder according to the training sample data set until a third preset stopping condition is met.

4. The method of claim 1, wherein, The video generation model comprises a facial expression capturer, a voiceprint encoder, a multi-perception attention mechanism and hash modeling, and generates an initial digital human video in which the reference character broadcasts the target text according to the lip features and expression features of the reference character in the reference video and the sound emotion features of the target audio, comprising: The lip features and expression features of the reference character in the reference video are extracted by the facial expression capturer, and the sound emotion features of the target audio are extracted by the voiceprint encoder; The lip features, the expression features and the sound emotion features are weighted by the multi-perception attention mechanism to obtain facial expression features; The facial expression features corresponding to the emotion hash code are determined by the hash modeling, and the initial digital human video in which the facial expression matches the target audio is generated according to the emotion hash code.

5. The method of claim 4, wherein, The video generation model further comprises a variational autoencoder, an audio encoder and a generation model, and the audio features of the target audio are aligned with the image features of the initial digital human video to obtain a target digital human video, comprising: The initial digital human video is decomposed into multiple frames of digital human images, which are respectively input into the variational autoencoder to obtain image features of each frame of digital human image, each image feature comprising the lip movement degree of the reference character; The audio features in the target audio are extracted by the audio encoder; The audio features are time-aligned with the lip movement degree corresponding to each frame of digital human image by the generation model to generate a target digital human video in which the lip movement matches the target audio.

6. The method of any one of claims 1-5, wherein, The target text comprises at least one of a character, a number, a unit symbol and an English word.

7. The method of any one of claims 1-5, wherein, The emotion parameter comprises an emotion type and an emotion intensity.

8. An electronic device, comprising: It comprises a processor, a memory and a communication interface, and the communication interface, the memory and the processor are connected through a bus; The communication interface is used for transmitting and receiving data; The memory stores a computer program, and the processor executes the following operations according to the computer program: Obtain the target text and the reference video, and separate the reference audio from the reference video; The reference audio, the target text and the preset emotion parameter are input into the trained audio generation model, and the target audio in which the reference character broadcasts the target text is generated according to the text features of the target text, the emotion features of the emotion parameter, and the audio features and voiceprint features of the reference character in the reference audio; wherein the text features are extracted based on the mapping relationship between characters and pronunciations; The target audio and the reference video are input into the trained video generation model, and an initial digital human video in which the reference character broadcasts the target text is generated according to the lip features and expression features of the reference character in the reference video and the sound emotion features of the target audio, and the audio features of the target audio are aligned with the image features of the initial digital human video to obtain a target digital human video.

9. The electronic device of claim 8, wherein, The audio generation model comprises an audio encoder, a text encoder, a large language model and a voiceprint encoder, and the processor generates target audio in which the target text is broadcast in the voice of the reference person according to the text features of the target text, the sentiment features of the sentiment parameters, and the audio features and voiceprint features of the reference person in the reference audio, specifically as follows: The text features of the target text extracted by the text encoder, the sentiment features of the sentiment parameters, and the audio features of the reference person in the reference audio extracted by the audio encoder are fused to obtain multi-modal features; The multi-modal features are input into the large language model to obtain associated semantic features between different modalities; The associated semantic features are fused with the voiceprint features of the reference person in the reference audio extracted by the voiceprint encoder, and then decoded to generate target audio in which the target text is broadcast in the voice of the reference person.

10. The electronic device of claim 8, wherein, The process of training the audio generation model by the processor is as follows: A training sample data set is constructed, wherein each training sample data comprises original audio, a text to be generated, pronunciation of the text to be generated, sentiment parameters of the text to be generated, and real audio in which an original person broadcasts the text to be generated, and the text to be generated comprises at least one of Chinese characters, numbers, unit symbols and English words; According to the training sample data set, the audio generation model including the text encoder and the large language model is trained in multiple rounds of iteration until a first preset stopping condition is met; According to the training sample data set, the audio encoder and the voiceprint encoder included in the audio generation model are trained in multiple rounds of iteration until a second preset stopping condition is met; According to the training sample data set, the text encoder, the audio encoder and the voiceprint encoder are trained in multiple rounds of iteration until a third preset stopping condition is met.

Images