Digital human audio and video generation method, system, device and medium

Abstract

The present application provides a method, system, device and medium for generating audio and video of a digital human, the method comprising: first, obtaining text information and an initial image; determining the emotion category corresponding to the text information based on an emotion analysis model; identifying the face area in the initial image to obtain the gender category corresponding to the digital human in the initial image; generating audio data corresponding to the digital human based on the emotion category and the gender category; generating an initial video of the digital human based on the audio data and the initial image; inputting the initial video of the digital human into a lip alignment model to obtain an output video; aligning the mouth shape of the digital human in the output video with the audio data. Using the emotion analysis model and image recognition technology, the emotion category of the text information and the gender category of the digital human in the initial image can be accurately identified, and a digital human video with emotional expression can be generated.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a method, system, device and medium for generating digital human audio and video. Background Art

[0002] With the rapid development of computer and artificial intelligence technology, digital human synthesis technology has become a hot topic in many application scenarios. However, the current digital human synthesis process often relies on a large amount of video and audio training data, and the quality and quantity of the data are difficult to guarantee, resulting in uneven quality of the generated digital human image and video.

[0003] In the prior art, a business card production method based on digital human technology is used, which is combined with speech synthesis technology to generate a digital human image business card. The process of generating a digital human in this solution is a black box, the overall process is uncontrollable, and the emotions of the digital human cannot be expressed.

[0004] Therefore, the existing technology still has problems such as large consumption of computing resources and lack of emotional expression when solving the problem of digital human synthesis. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present application provides a method, system, device and medium for generating digital human audio and video, which can accurately identify the emotion category of text information and the gender category of the digital human in the initial image, and generate a digital human video with emotional expression.

[0006] The first aspect of the present application provides a method for generating digital human audio and video, comprising:

[0007] Acquire text information and an initial image; the initial image at least includes a static face area used to generate a digital human;

[0008] Determine the emotion category corresponding to the text information based on the emotion analysis model;

[0009] Identify the face area in the initial image to obtain the gender category corresponding to the digital person in the initial image;

[0010] Generate audio data corresponding to the digital human based on the emotion category and the gender category;

[0011] Generate an initial video of a digital human based on the audio data and the initial image;

[0012] The initial video of the digital human is input into a lip alignment model to obtain an output video; the mouth shape of the digital human in the output video is aligned with the audio data.

[0013] Optionally, determining the emotion category corresponding to the text information based on a sentiment analysis model includes:

[0014] Identifying emotional keywords in the text information;

[0015] Matching the emotion keywords with a preset emotion vocabulary library to obtain a matching result;

[0016] The matching result is input into the sentiment analysis model to obtain the sentiment category corresponding to the text information.

[0017] Optionally, the identifying a face region in the initial image to obtain a gender category corresponding to the digital person in the initial image includes:

[0018] recognizing the initial image input by the user;

[0019] Extracting the face region in the initial image based on an image segmentation algorithm, and outputting the face region image;

[0020] Extracting a high-dimensional feature vector of the face region image;

[0021] The gender category corresponding to the digital person in the initial image is determined based on the high-dimensional feature vector.

[0022] Optionally, extracting a high-dimensional feature vector of the face region image includes:

[0023] Dividing the face region image into a plurality of sub-regions based on a visual conversion model;

[0024] Each sub-region is sequentially identified based on the position coding to obtain a high-dimensional feature vector corresponding to each sub-region.

[0025] Optionally, it includes: preprocessing the initial image, wherein the preprocessing process is used to convert the initial image into a format that can be processed by a deep learning model.

[0026] Optionally, the step of inputting the initial digital human video into a lip alignment model to obtain an output video includes:

[0027] Identifying the mouth position of the digital human in the initial video;

[0028] Generate corresponding lip movement video based on audio data;

[0029] The mouth position of each frame in the initial digital human video is extracted, and the mouth position in the initial digital human video is replaced by the lip movement video corresponding to the current frame to obtain an output video.

[0030] The second aspect of the present application is a digital human audio and video generation system, comprising:

[0031] An acquisition module, used to acquire text information and an initial image; the initial image at least includes a static face area used to generate a digital human;

[0032] A sentiment analysis module, used to determine the sentiment category corresponding to the text information based on a sentiment analysis model;

[0033] An image gender recognition module, used to recognize the face area in the initial image and obtain the gender category corresponding to the digital person in the initial image;

[0034] An audio synthesis module, used for generating audio data corresponding to the digital human based on the emotion category and the gender category;

[0035] A digital human video generation module, used to generate a digital human initial video based on the audio data and the initial image;

[0036] The digital human lip alignment module is used to input the initial digital human video into the lip alignment model to obtain an output video; the mouth shape of the digital human in the output video is aligned with the audio data.

[0037] Optionally, the digital human video generation module includes: an eye recognition module, a facial recognition module, a facial action database, and an action fusion module;

[0038] The eye recognition module is used to recognize the eye region image of the digital human in the initial digital human image;

[0039] The facial recognition module is used to recognize the facial region image of the digital human in the initial digital human image;

[0040] The facial action database is used to store pre-collected facial video data of real people speaking, wherein the facial video data includes video data of real people at multiple angles;

[0041] The action fusion module is used to generate an initial digital human video without audio.

[0042] A third aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute any of the methods described above.

[0043] A fourth aspect of the present application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement any of the methods described above.

[0044] It can be seen that in the first aspect, the present application provides a method for generating audio and video of a digital human. First, obtain text information and an initial image; determine the emotion category corresponding to the text information based on the emotion analysis model; identify the face area in the initial image to obtain the gender category corresponding to the digital human in the initial image; generate audio data corresponding to the digital human based on the emotion category and the gender category; generate an initial video of the digital human based on the audio data and the initial image; input the initial video of the digital human into the lip alignment model to obtain an output video; align the mouth shape of the digital human in the output video with the audio data. Using the emotion analysis model and image recognition technology, the emotion category of the text information and the gender category of the digital human in the initial image can be accurately identified, and a digital human video with emotional expression can be generated.

[0045] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0047] Figure 1 It is a flow chart of a method for generating digital human audio and video according to an embodiment of the present application.

[0048] Figure 2 It is a schematic diagram of the process of determining emotion categories through emotion analysis shown in an embodiment of the present application.

[0049] Figure 3 It is a schematic diagram of a process for identifying the gender category of an image shown in an embodiment of the present application.

[0050] Figure 4 It is a schematic diagram of the process of generating the initial video of a digital human shown in an embodiment of the present application.

[0051] Figure 5 It is a schematic diagram of the process of digital human lip alignment shown in an embodiment of the present application.

[0052] Figure 6 It is a structural schematic diagram of a digital human audio and video generation system shown in an embodiment of the present application.

[0053] Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0055] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0056] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0057] With the rapid development of computer and artificial intelligence technology, digital human synthesis technology has become a hot topic in many application scenarios. Whether it is corporate digital human business cards, virtual digital humans or video conferencing, digital human synthesis technology has demonstrated its huge application potential. However, the current digital human synthesis technology still faces a series of challenges. Existing digital human synthesis schemes, such as business card production methods based on virtual digital human technology, virtual human generation methods based on audio control, realistic virtual human generation methods based on text drive, and lip alignment schemes, have achieved digital human synthesis and driving to a certain extent, but there are still many shortcomings.

[0058] Specifically, existing technologies often rely on a large amount of video and audio training data, and the quality and quantity of the data are difficult to guarantee, resulting in uneven quality of the generated digital human images and videos. In addition, the above-mentioned schemes require a large amount of computing resources, low inference efficiency, a long time to generate digital human videos, and high video memory consumption, which makes it difficult to meet the requirements for efficiency and real-time performance in practical applications. More importantly, when generating digital human videos, the importance of emotional expression is often overlooked, resulting in the generated digital human facial expressions being stiff and unnatural, making it difficult to achieve the desired effect. Therefore, how to integrate text emotions and generate a natural, controllable, efficient, and high-fidelity audio-driven digital human solution has become a problem that needs to be solved urgently.

[0059] In response to the above problems, in the prior art, for example, a business card production method based on virtual digital human technology can be used, combined with speech synthesis technology to generate digital human image business cards. However, the process of generating digital humans in the above scheme is a black box, the overall process is uncontrollable, the emotions of the digital humans cannot be expressed, and the speech synthesis method does not have the function of voice customization. Other schemes use deep learning networks to generate digital human videos through potential diffusion models and multimodality. Although this method improves the quality of digital human videos to a certain extent, it requires a large amount of aligned video and audio data, has high resource consumption, and due to the characteristics of the diffusion model, the generated digital human videos have certain instability.

[0060] In addition, the video and audio are used to output the digital human video containing actions and lip movements, or the video of the digital human voice broadcast is generated through the lip alignment scheme. However, the above schemes cannot generate the digital human image through images, and cannot synthesize audio independently. In the generation process, the emotions in the text cannot be effectively recognized and controlled, resulting in the generated digital human facial expressions being stiff and unnatural, which is difficult to meet the needs of practical applications.

[0061] In summary, existing technologies still have many deficiencies in solving the problem of digital human synthesis, such as strong data dependence, high consumption of computing resources, low reasoning efficiency, lack of emotional expression, and unnatural facial expressions. Therefore, there is a need for an audio-driven digital human solution that can integrate text emotions and generate natural, controllable, efficient, and high-fidelity digital human images, in order to overcome the deficiencies of existing technologies and meet the needs of practical applications.

[0062] In the present application embodiment, Figure 1 As shown, the method for generating digital human audio and video includes:

[0063] S100, acquiring text information and an initial image; the initial image at least includes a static face area used to generate a digital human;

[0064] In the embodiment of the present application, the text information and the initial image may be content input by the user, wherein, for example, the text information may be a natural language description; the user transmits the image information while inputting the text information.

[0065] Specifically, the initial image includes at least the face area, so that the facial expression of the digital human can be adjusted and rendered according to the emotional description in the text information. In addition, the text information can contain specific instructions or descriptions to guide the speech synthesis of the digital human, including intonation, speech speed and emotional color. This allows the digital human to not only have facial expressions that match the text information, but also make sounds that are coordinated with the emotions in the text, thereby achieving a more natural and vivid digital human performance.

[0066] S101, determining the emotion category corresponding to the text information based on a sentiment analysis model;

[0067] In an embodiment of the present application, the emotion analysis model can be a model that determines the emotion expressed by the text based on its content. The model identifies the emotional tendency by analyzing the keywords, phrases and sentence structures in the text information, thereby determining the emotion category expressed by the text information. For example, if positive words such as "happy" and "joyful" appear in the text information, the model will judge it as a positive emotion; if negative words such as "sad" and "disappointed" appear, it will be judged as a negative emotion. In addition, the model can also identify complex emotions in the text information, such as "surprise" and "anger", to achieve more refined adjustments to the digital human's expression and voice. This allows the digital human to more accurately reflect the emotions contained in the text information, thereby improving the user experience.

[0068] Specifically, it is mainly used to identify the emotional information of the input text, such as Figure 2 As shown, the preferred solution for the large language model is to select the language models of the GLM-4 and Qwen2.5 series as the cornerstone.

[0069] S200: Identify emotional keywords in the text information.

[0070] In the embodiment of the present application, the keywords related to emotions are first obtained, and the user uploads the text content broadcast by the digital human through the interface or system interface, and fills it into the first prompt to generate a complete large language model input text. The large language model generates the emotional keyword information contained in the text based on the input information. For example, sad and excited, etc.

[0071] S201, matching the emotion keyword with a preset emotion vocabulary library to obtain a matching result.

[0072] In the embodiment of the present application, the definition of the emotion keyword contained in the input text is obtained, and the emotion keyword information obtained in S200 is filled into the second prompt to generate a prompt word for completely obtaining the specific definition of the emotion keyword. The large language model generates the specific definition of the emotion keyword contained in the text according to the emotion keyword information. For example, sad: feeling very uncomfortable and extremely sad.

[0073] S202: Input the matching result into the sentiment analysis model to obtain the sentiment category corresponding to the text information.

[0074] In the embodiment of the present application, the definition of the emotion keyword contained in the input text is obtained, and the emotion keyword obtained in S201 and its corresponding definition information are combined with the emotion category set in the system predefined knowledge base to fill in the third prompt word to generate a prompt word for obtaining the emotion category. The large language model finally generates the emotion that may be expressed by the user input text based on the prompt word information of the obtained emotion category.

[0075] Further, according to the obtained emotion category, a corresponding digital human audio and video generation template is selected. In the embodiment of the present application, the system selects a template that matches the emotion category determined in step S202 from a preset digital human audio and video template library. For example, if the emotion category is "sad", a digital human template whose expression and voice tone are consistent with the emotion of "sad" is selected.

[0076] S102, identifying the face area in the initial image, and obtaining the gender category corresponding to the digital person in the initial image;

[0077] In practical applications, the system first receives the initial image of the digital human provided by the user. In order to ensure that the subsequent deep learning model can efficiently and accurately process the initial image, the initial image is first preprocessed. The core goal of preprocessing is to convert the image format to match the input requirements of the deep learning model. In order to further improve the image quality, the Gaussian filtering algorithm can be introduced. The Gaussian filtering algorithm is an effective image processing technology. By setting a reasonable Gaussian kernel size and Gaussian parameters, it can effectively reduce the noise in the image, thereby improving the overall clarity and detail expression of the image. The advantage of Gaussian filtering is that it can smooth out unnecessary noise while retaining the edge features of the image, laying a solid foundation for subsequent image processing and feature extraction.

[0078] On the basis of the above embodiment, face detection is performed using the enhanced image. Image semantic segmentation technology, especially the face-parsing method, can be used as a preferred solution for extracting key face areas. The face-parsing method can efficiently identify and extract the face area in the image, providing accurate input for subsequent portrait data feature extraction. The system finally outputs a picture containing key face information, providing strong support for subsequent feature extraction and gender recognition.

[0079] In the embodiment of the present application, after obtaining the face image, the portrait data feature extraction is performed. In order to fully capture the high-dimensional features of the face, a deep learning model can be selected in this embodiment, and the vision transformer model is used as the preferred solution. The vision transformer model is well known for its powerful feature extraction capabilities. It can divide the face image into multiple small areas and use position encoding to accurately identify the order information of each area.

[0080] In specific operations, a face image can be input into the vision transformer model, and the model uses its internal attention mechanism to conduct in-depth analysis of each small area and extract high-dimensional features of the image. In addition, in order to enhance the generalization and adaptability of the model, other deep learning models such as MobileNet, ResNet, and VGG can also be considered, which can also extract high-dimensional features of faces.

[0081] After extracting high-dimensional features, the aggregation and dimensionality reduction capabilities of the deep learning model are used to integrate the above high-dimensional features into a single vector. This vector not only contains the key information of the face, but also has a lower dimension, which is convenient for subsequent classification and recognition tasks.

[0082] In the embodiment of the present application, the output high-dimensional feature vector is used to identify the gender of the portrait. The system can use a multi-layer perceptron (MLP) as a classifier. MLP is known for its powerful nonlinear mapping ability and classification performance, and can accurately determine the gender of the portrait based on the input picture vector.

[0083] In the process of gender recognition, MLP first performs nonlinear transformation on the input image vector, and then outputs two category labels through the softmax function: "male" or "female". These two labels represent the gender information of the portrait and provide important reference for subsequent applications.

[0084] It can be seen that the above embodiment realizes the whole process from the initial image of the digital human to the gender recognition of the digital human. The above embodiment is not only efficient and accurate, but also has strong adaptability and generalization ability.

[0085] In the embodiments of the present application, in the initial stage of text-to-speech synthesis, the problem of overly long input text is usually faced, and the text length exceeds the ability of existing speech synthesis models to directly process long sequences for generation. To solve this problem, a text segmentation method can be adopted.

[0086] S103. Generate audio data corresponding to the digital human based on the emotion category and the gender category;

[0087] In the embodiments of the present application, as Figure 3 shown, according to the punctuation marks of the text, a preset fixed length, or a specific text segmentation algorithm, the initial long-sequence text is reasonably segmented into multiple short-sequence text segments. Utilizing the above segmentation technology not only helps the model process but also maintains the semantic coherence of the text.

[0088] However, even the segmented short-sequence text may contain unconventional symbols and numbers that the speech synthesis model cannot directly process. To further improve the compatibility of the model, further preprocessing of the text is required. Specifically, unconventional symbols in the text, such as special characters, punctuation marks (except for punctuation marks used to separate sentences), etc., can be removed. At the same time, for the numbers in the text, a conversion strategy can be adopted. A single number is converted into the corresponding homophone (e.g., "1" is converted into "一"), while multi-digit numbers are converted into capital Chinese text (e.g., "101" is converted into "一百零一"). Among them, the above conversion not only conforms to the speech expression habits in the Chinese context but also ensures that the speech synthesis model can accurately recognize and synthesize the correct speech.

[0089] After the preprocessing in the above embodiments, multiple optimized and converted short-sequence text segments will be obtained, and the short-sequence text segments will be used as the input for the subsequent speech synthesis steps.

[0090] After obtaining the preprocessed short-sequence text segments, the core stage of speech synthesis will be entered. The main concern is to use the speech synthesis model to segment and generate audio corresponding to the gender and emotion according to the input gender and emotion labels. To achieve this goal, each short-sequence text segment and the corresponding gender and emotion labels are input into the speech synthesis model.

[0091] It can be understood that if the user has the need for voice cloning, one or more short reference audio and the text corresponding to each audio segment also need to be provided. The reference audio and the text will be synchronously input into the speech synthesis module. The speech synthesis module will use voice extraction technology to extract the voice characteristics from the reference audio and generate the corresponding audio vector. This audio vector will be used as a prompt message to guide the model to synthesize speech with the same voice as the reference audio.

[0092] When generating audio clips, the model synthesizes them in the chronological order of short sequence texts to ensure that the final generated audio remains semantically and contextually coherent.

[0093] After obtaining multiple short audio clips, a speech post-processing step is required to splice these clips into a complete synthesized long audio. Specifically, the system can splice these audio clips in sequence according to the order of the short sequence text. At the same time, in order to enhance the naturalness and fluency of the audio, appropriate blank sounds are inserted between clips, and the pause time between clips is adjusted. The above processing process not only helps to eliminate splicing traces, but also makes the final synthesized audio more in line with the natural rhythm and rhythm of human speech.

[0094] It can be seen that based on the above embodiment, the system can obtain a complete, natural and smooth synthesized long audio. The audio not only has the specified gender, emotion and timbre characteristics, but also accurately conveys the information and intention of the original text. It not only solves the problem of long sequence text processing, but also improves the accuracy and naturalness of speech synthesis. At the same time, by introducing timbre cloning technology, it also provides users with a more personalized and customized speech synthesis experience.

[0095] In practical applications, text preprocessing:

[0096] Input: The full text of a novel.

[0097] The processing process includes: dividing the full text into multiple short text segments according to paragraphs or chapters, removing non-conventional symbols, and converting numbers into near-phonetic characters or uppercase Chinese text.

[0098] Speech Synthesis:

[0099] Input: Preprocessed short text segments, and assigned gender (such as female) and emotion (such as gentle) labels.

[0100] Model: Use a pre-trained speech synthesis model.

[0101] Output: Multiple audio clips with specified gender and emotion.

[0102] Tone Cloning:

[0103] Input: one or more reference audio clips (such as a reading audio clip of a well-known host) and their corresponding texts.

[0104] Processing: Extract the timbre features of the reference audio and generate the corresponding audio vector.

[0105] Output: An audio clip with the timbre of the reference audio.

[0106] Speech post-processing:

[0107] Input: Multiple audio clips with specified gender, emotion, and timbre.

[0108] Processing: Splice the audio clips in sequence, insert appropriate blank sounds and adjust pauses to ensure the fluency and naturalness of the speech.

[0109] Output: Complete personalized audiobook audio file.

[0110] It can be seen that through the above technical solution, the personalized audio book audio file produced not only has the specified gender and emotional characteristics, but also can select a specific timbre according to the user's preferences. At the same time, the fluency and naturalness of the voice are also greatly improved, allowing users to enjoy a more immersive reading experience.

[0111] S104, generating an initial digital human video based on the audio data and the initial image;

[0112] In the embodiments of the present application, Figure 4 As shown in the figure, the system aims to accurately locate and identify the eye area of ​​the character through the initial digital human image provided by the user, thereby laying the foundation for subsequent action generation. The system first uses advanced eye area positioning algorithms to carefully analyze the input digital human image and accurately capture the relative position of the eyes in the image.

[0113] In the embodiment of the present application, according to the preset video length (determined by the number of frames and seconds), the system generates a static initial video frame. This video frame provides a basis for the subsequent addition of eye and facial movements. On the basis of the static video, the system traverses each frame and adds blinking action parameters of an indefinite number of frames according to random probability, making the eye movements of the digital human more natural and vivid. At the same time, for non-blinking action frames, the system keeps its original eye state parameters unchanged. Finally, the system outputs a sequence containing eye action parameters to provide data support for subsequent action fusion.

[0114] In order to enrich the facial and head movements of digital humans, the system needs to build a comprehensive action database. The system collects real-person speech videos from multiple perspectives (such as frontal face, left-facing profile, and right-facing profile) to capture rich facial movement data. Video data provides material for subsequent head posture detection and action data extraction.

[0115] Using the head posture detection algorithm, the system conducts a detailed analysis of the collected real-person video and extracts the head's left and right, pitch, and front and back motion trajectory data. The above data is organized into json format and saved in the facial action database to provide data support for subsequent motion capture and fusion. This step aims to capture and generate the facial action parameters of the digital human through facial recognition technology. The system uses the facial recognition algorithm to perform facial recognition on the initial digital human image provided by the user and obtain the relative position data of the head in the image.

[0116] In the embodiment of the present application, according to the facial orientation of the initial image of the digital human, the system selects the motion data of the corresponding orientation from the facial action database. Based on the acquired motion data, the system adds the motion trajectory data of the left and right, pitch, and front and back of the face according to the state data and emotional label of the digital human. At the same time, the system also uses random probability to add facial expression motion parameters of an indefinite number of consecutive frames, making the facial movements of the digital human more natural and rich. Finally, the system outputs a sequence containing facial action parameters.

[0117] This step aims to fuse the eye and facial motion parameters into the static initial video to generate the final digital human video. The system traverses each frame of the static initial video from the beginning, and modifies the facial and eye movements frame by frame according to the eye and facial motion parameter sequence. Based on the motion fusion, the system synthesizes a final digital human initial video without speech based on the generated png image sequence and the set fixed fps (frame rate). This video not only has vivid facial and eye movements, but also maintains the integrity of the initial image of the digital human.

[0118] In summary, the generation of digital human videos is achieved through multiple steps such as eye recognition and motion parameterization, facial and head motion database construction, facial recognition and motion capture, motion fusion and video synthesis. This solution is not only highly flexible and scalable, but also can provide users with a vivid and realistic digital human image experience.

[0119] S105, inputting the initial video of the digital human into a lip alignment model to obtain an output video; the mouth shape of the digital human in the output video is aligned with the audio data.

[0120] In the embodiments of the present application, Figure 5 As shown in the figure, the system first receives the initial digital human video generated by the digital human video generation module. At this stage, the high-precision mouth recognition module is introduced. This module uses a deep learning algorithm to carefully analyze each frame in the video, accurately identify the position of the mouth and its subtle movement changes, and generate a series of fine mask mark sequences. The mask mark sequence details the exact position and shape of the mouth in the video frame.

[0121] Furthermore, the system receives audio data as input. In order to generate a lip-sync video that closely matches the audio content, an intelligent audio-lip sync technology may be used.

[0122] Intelligent audio-lip synchronization technology can automatically generate matching lip movements by analyzing the characteristics of audio signals. By analyzing the frequency, rhythm and intensity of audio signals, it automatically generates corresponding facial area lip movements.

[0123] In the embodiment of the present application, at the fusion stage, the system first accurately segments the mouth area in the video based on the initial video and the mask mark sequence. Using image synthesis technology, the generated lip movement video is seamlessly replaced in the segmented mouth area. In order to ensure that the replaced video remains visually coherent and consistent, an edge fusion algorithm, as well as color and brightness adjustment technology, can be used to eliminate possible splicing traces and achieve a natural transition.

[0124] It is understandable that, considering the possible difference between the length of the initial video and audio of the digital human, as well as the need for system efficiency, a multi-segment smooth splicing process is performed after the fusion step. First, according to the length of the audio and the rhythm of the video, the system intelligently divides the initial video into multiple segments. Then, through the video splicing algorithm, these segments are seamlessly spliced ​​to form a complete digital human video that is completely aligned with the audio length. During the splicing process, attention is paid to maintaining the smoothness and coherence of the video to avoid jumping or freezing.

[0125] like Figure 6 As shown, the present application provides a digital human audio and video generation system, comprising:

[0126] An acquisition module 61 is used to acquire text information and an initial image; the initial image at least includes a static face area used to generate a digital human;

[0127] A sentiment analysis module 62, used to determine the sentiment category corresponding to the text information based on a sentiment analysis model;

[0128] An image gender recognition module 63 is used to recognize the face area in the initial image and obtain the gender category corresponding to the digital person in the initial image;

[0129] An audio synthesis module 64, used to generate audio data corresponding to the digital human based on the emotion category and the gender category;

[0130] A digital human video generation module 65, configured to generate a digital human initial video based on the audio data and the initial image;

[0131] The digital human lip alignment module 66 is used to input the initial digital human video into the lip alignment model to obtain an output video; the mouth shape of the digital human in the output video is aligned with the audio data.

[0132] Optionally, the digital human video generation module includes: an eye recognition module, a facial recognition module, a facial action database, and an action fusion module;

[0133] The eye recognition module is used to recognize the eye region image of the digital human in the initial digital human image;

[0134] The facial recognition module is used to recognize the facial region image of the digital human in the initial digital human image;

[0135] The facial action database is used to store pre-collected facial video data of real people speaking, wherein the facial video data includes video data of real people at multiple angles;

[0136] The action fusion module is used to generate an initial digital human video without audio.

[0137] Optionally, determining the emotion category corresponding to the text information based on a sentiment analysis model includes:

[0138] Identifying emotional keywords in the text information;

[0139] Matching the emotion keywords with a preset emotion vocabulary library to obtain a matching result;

[0140] The matching result is input into the sentiment analysis model to obtain the sentiment category corresponding to the text information.

[0141] Optionally, the identifying a face region in the initial image to obtain a gender category corresponding to the digital person in the initial image includes:

[0142] recognizing the initial image input by the user;

[0143] Extracting the face region in the initial image based on an image segmentation algorithm, and outputting the face region image;

[0144] Extracting a high-dimensional feature vector of the face region image;

[0145] The gender category corresponding to the digital person in the initial image is determined based on the high-dimensional features.

[0146] Optionally, extracting a high-dimensional feature vector of the face region image includes:

[0147] Dividing the face region image into a plurality of sub-regions based on a visual conversion model;

[0148] Each sub-region is sequentially identified based on the position coding to obtain a high-dimensional feature vector corresponding to each sub-region.

[0149] Optionally, the initial image is preprocessed, and the preprocessing process is used to convert the initial image into a format that can be processed by a deep learning model.

[0150] Optionally, the step of inputting the initial digital human video into a lip alignment model to obtain an output video includes:

[0151] Identifying a mouth position in the initial video;

[0152] Generate corresponding lip movement video based on audio data;

[0153] The mouth position of each frame in the initial video is extracted, and the mouth position in the initial video is replaced by the lip movement video corresponding to the current frame to obtain an output video.

[0154] See also Figure 7 , Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. The electronic device 700 includes a memory 710 and a processor 720.

[0155] The present application can be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0156] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or combinations of both.

[0157] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating digital human audio and video, characterized in that: include: Get text information and initial image; The initial image at least includes a static face area used to generate a digital human; Determine the emotion category corresponding to the text information based on the emotion analysis model; Identify the face area in the initial image to obtain the gender category corresponding to the digital person in the initial image; Generate audio data corresponding to the digital human based on the emotion category and the gender category; Generate an initial video of a digital human based on the audio data and the initial image; The initial video of the digital human is input into a lip alignment model to obtain an output video; the mouth shape of the digital human in the output video is aligned with the audio data.

2. The method according to claim 1, characterized in that The determining the emotion category corresponding to the text information based on the sentiment analysis model includes: Identifying emotional keywords in the text information; Matching the emotion keywords with a preset emotion vocabulary library to obtain a matching result; The matching result is input into the sentiment analysis model to obtain the sentiment category corresponding to the text information.

3. The method according to claim 1, characterized in that The step of identifying the face region in the initial image and obtaining the gender category corresponding to the digital person in the initial image includes: recognizing the initial image input by the user; Extracting the face region in the initial image based on an image segmentation algorithm, and outputting the face region image; Extracting a high-dimensional feature vector of the face region image; The gender category corresponding to the digital person in the initial image is determined based on the high-dimensional feature vector.

4. The method according to claim 3, characterized in that The step of extracting a high-dimensional feature vector of the face region image comprises: Dividing the face region image into a plurality of sub-regions based on a visual conversion model; Each sub-region is sequentially identified based on the position coding to obtain a high-dimensional feature vector corresponding to each sub-region.

5. The method according to claim 3, characterized in that: include: The initial image is preprocessed, and the preprocessing process is used to convert the initial image into a format that can be processed by a deep learning model.

6. The method according to claim 1, characterized in that The step of inputting the initial digital human video into the lip alignment model to obtain an output video comprises: Identifying the mouth position of the digital human in the initial video; Generate corresponding lip movement video based on audio data; The mouth position of each frame in the initial digital human video is extracted, and the mouth position in the initial digital human video is replaced by the lip movement video corresponding to the current frame to obtain an output video.

7. A digital human audio and video generation system, characterized in that: include: An acquisition module, used to acquire text information and an initial image; The initial image at least includes a static face area used to generate a digital human; A sentiment analysis module, used to determine the sentiment category corresponding to the text information based on a sentiment analysis model; An image gender recognition module, used to recognize the face area in the initial image and obtain the gender category corresponding to the digital person in the initial image; An audio synthesis module, used for generating audio data corresponding to the digital human based on the emotion category and the gender category; A digital human video generation module, used to generate a digital human initial video based on the audio data and the initial image; The digital human lip alignment module is used to input the initial digital human video into the lip alignment model to obtain an output video; the mouth shape of the digital human in the output video is aligned with the audio data.

8. The system according to claim 7, characterized in that The digital human video generation module includes: an eye recognition module, a facial recognition module, a facial action database, and an action fusion module; The eye recognition module is used to recognize the eye region image of the digital human in the initial digital human image; The facial recognition module is used to recognize the facial region image of the digital human in the initial digital human image; The facial action database is used to store pre-collected facial video data of real people speaking, wherein the facial video data includes video data of real people at multiple angles; The action fusion module is used to generate an initial digital human video without audio.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 6.

Images