Digital human generation method and device, electronic equipment and readable storage medium

Abstract

The invention provides a digital human generation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of digital human. The generation method comprises the steps of obtaining a user question, inputting the user question into a knowledge question-answer large model, and generating an answer corresponding to the user question; obtaining a digital human image, and recognizing a face image of the digital human image; and based on the answer and the face image, performing mouth shape reasoning and fusion, and generating a digital human for answering the user's question. The problems that in the prior art, the real-time requirement cannot be met, the processing efficiency is low, and the processing effect is poor are at least solved. The method is suitable for digital human generation and question-answering scenes.

Description

Technical Field

[0001] The present invention relates to the field of digital human technology, and in particular to a method, device, electronic device and computer-readable storage medium for generating a digital human. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technology, generative digital human technology has become a research hotspot in recent years. Generative digital human technology relies on deep learning models, especially breakthroughs in generative artificial intelligence, to create lifelike human images and behaviors. At the same time, large language models, such as OpenAI's GPT (Generative Pre-Trained Transformer, a deep learning model based on the Internet, trained with available data, and generating text) series, have significantly improved the machine's ability to understand and generate natural language.

[0003] In the field of digital media, live broadcast technology is always pursuing more efficient and interactive solutions. Traditional business knowledge Q&A relies on text interaction, which limits the richness of user experience and the real-time nature of Q&A to a certain extent. In order to break through the limitations of user experience, it is necessary to develop an innovative government business knowledge Q&A technology, namely, a hyper-realistic generative digital human product. The hyper-realistic generative digital human product combines the advantages of generative digital human technology and large language models, aiming to provide users with a more natural, interactive and diverse Q&A experience.

[0004] In the current existing technology, the first method to achieve hyper-realistic generative digital humans is to first perform 3D modeling of the head of the hyper-realistic digital human, and then control its movements through the front-end system, relying on manual adjustment of the various components of the model, and realizing the facial expressions and movements of the digital human through a preset action library or control logic. The second method to achieve hyper-realistic generative digital humans focuses on the optimization processing of lip recognition images and audio data, relying on multi-scale feature extraction and bidirectional long short-term memory networks to generate lip movement videos.

[0005] However, the first method of realizing hyper-realistic generative digital human relies on manual labor, has low processing efficiency and poor processing effect. Although the second method of realizing hyper-realistic generative digital human has significant processing effect, it is limited in real-time and interactivity. Therefore, the existing method of realizing hyper-realistic generative digital human cannot meet the real-time requirements, has low processing efficiency and poor processing effect. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method, device, electronic device and computer-readable storage medium for generating a digital human in view of the above-mentioned deficiencies in the prior art. The method can realize the rapid and intelligent generation of a digital human, improve the efficiency and effect of generating a digital human, meet the real-time and interactive requirements of the user's question-and-answer experience, and improve the richness of the user experience and the real-time nature of the question-and-answer.

[0007] In a first aspect, the present invention provides a method for generating a digital human, comprising: obtaining user questions, and inputting the user questions into a knowledge question and answer model to generate answers corresponding to the user questions; obtaining a digital human image, and identifying a face image of the digital human image; based on the answers and face images, performing mouth shape reasoning and fusion to generate a digital human for answering user questions.

[0008] Preferably, the method of performing lip shape reasoning and fusion based on the answer and the face image to generate a digital human for answering the user's question specifically includes: based on the text-to-speech TTS model and the answer, voice synthesizing the voice answer corresponding to the user's question; performing lip shape reasoning based on the lip shape reasoning model, the voice answer and the face image to obtain a mouth shape change image; and fusing the mouth shape change image, the voice answer and the face image to generate a digital human for answering the user's question.

[0009] Preferably, the mouth shape inference model includes a Wav2Lip model, and the mouth shape inference is performed based on the mouth shape inference model, the voice answer and the face image to obtain the mouth shape change image, which specifically includes: converting the voice answer into a corresponding Mel-spectrogram; dividing the Mel-spectrogram according to a preset frame rate to obtain Mel-spectrogram features; inputting the Mel-spectrogram features and the face image into the Wav2Lip model to infer the mouth shape change image.

[0010] Preferably, the converting the voice answer into a corresponding Mel-spectrogram specifically comprises: converting the voice answer into a corresponding Mel-spectrogram according to formula (1):

[0011] M(f,t)=log(∑ n X(n,t)W(f,n))(1), where M(f,t) represents the Mel spectrum value at time t and frequency f, X(n,t) represents the spectrum intensity of the nth speech signal at time t in the speech response, and W(f,n) represents the Mel filter coefficient of the nth speech signal at frequency f in the speech response.

[0012] Preferably, the step of inputting the Mel-spectrogram features and the face image into the Wav2Lip model to infer the mouth shape change image specifically includes: inferring the mouth shape change image according to formula (2):

[0013] in, Represents the mouth shape change image, x face represents a face image, x mel Represents a mel-spectrogram feature.

[0014] Preferably, the fusion of the mouth shape change image, the voice answer and the face image to generate a digital human for answering user questions specifically includes: extracting the mouth shape image from the face image, and based on the Laplace mixture model, fusing the mouth shape change image and the mouth shape image to obtain the answer mouth shape; linearly weighting the answer mouth shape and the face image to generate a digital human answer image; based on the stream pusher, streaming the digital human answer image and the voice answer to the front end to generate a digital human for answering user questions.

[0015] Preferably, the method of fusing the mouth shape variation image and the mouth shape image based on the Laplace mixture model to obtain the answer mouth shape specifically includes: fusing the mouth shape variation image and the mouth shape image according to formula (3):

[0016] Among them, I output represents the answer mouth shape, α represents the fusion coefficient, Indicates the mouth shape change image, Represents a mouth image.

[0017] In a second aspect, the present invention also provides a device for generating a digital human, including a knowledge question and answer module, an image recognition module and a generation module. The knowledge question and answer module is used to obtain user questions, input the user questions into a knowledge question and answer model, and generate answers corresponding to the user questions. The image recognition module is used to obtain the image of the digital human and recognize the facial image of the digital human image. The generation module is connected to the knowledge question and answer module and the image recognition module respectively, and is used to perform mouth shape reasoning and fusion based on the answers and facial images to generate a digital human for answering user questions.

[0018] Preferably, the generation module adopts the multi-threading mechanism and thread locking mechanism of the unified computing device architecture CUDA and the graphics processing unit GPU.

[0019] In a third aspect, the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the method for generating a digital human provided in the first aspect.

[0020] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a digital human provided in the first aspect above is implemented.

[0021] The present invention provides a method, device, electronic device and computer-readable storage medium for generating digital human. The knowledge question and answer big model has a powerful natural language processing capability, and the knowledge question and answer big model is applied to the generation of digital human. It can process user questions and generate answers in a very short time, provide real-time feedback, improve user experience, and understand complex user questions, and generate smooth and natural answers, so that human-computer interaction is smoother, and the generated digital human can better meet the user's question and answer experience; coordinate the answers and face images generated by the knowledge question and answer big model, perform mouth shape reasoning and fusion, rather than simply combining the answers with the face images, so that the answers can be reasonably integrated into the generation of digital human, reduce the incoherence phenomenon in the process of digital human generation, and improve the coherence of digital human actions and the naturalness of interactive experience. Therefore, the present invention can realize the rapid and intelligent generation of digital human, improve the efficiency and generation effect of digital human generation, meet the real-time and interactive requirements of user question and answer experience, and improve the richness of user experience and the real-time nature of question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a method for generating a digital human according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a flow chart of a method for generating a digital human according to Embodiment 2 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a digital human generation system according to Embodiment 3 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of a digital human generation device according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0027] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0028] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0029] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0030] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0031] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0032] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or by a combination of hardware and computer instructions.

[0033] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0034] Embodiment 1:

[0035] like Figure 1 As shown, this embodiment provides a method for generating a digital human. The method for generating a digital human includes:

[0036] S101, obtaining user questions, and inputting the user questions into the knowledge question and answer model to generate corresponding answers to the user questions.

[0037] In this embodiment, the user inputs a user question in text or voice form through the user interaction interface (i.e., the front end). The front end sends the user question in text form to the knowledge question and answer big model, which matches and generates the corresponding answer to the user question. For the user question in voice form input by the user, speech-to-text technology is used, such as Internet technology to convert the user question in voice form into text form, and then the knowledge question and answer big model generates the corresponding answer to the user question.

[0038] S102, acquiring a digital human image, and identifying a face image of the digital human image.

[0039] In this embodiment, the image includes but is not limited to pictures and videos. The face recognition is performed on the digital human image through the S3DF (Spatio-Temporal 3DFlow, three-dimensional depth flow) model, and the position of the mouth in the digital human image is located to obtain the face image, specifically: ① through OpenCV (open source computer vision library, a BSD (Berkeley Software ① Use the face_detection.FaceAlignment tool (a face detection and key point location library based on deep learning) to detect faces and obtain the bounding box coordinates of the face. The bounding box coordinates of the face are usually a rectangular box that represents the position of the face in the image. ③ Bounding box smoothing is achieved by averaging the coordinates of T consecutive bounding boxes to reduce jitter between frames and avoid sudden changes in the position of the face in the video. ④ Use the facial key point detection model to locate facial feature points, usually based on the bounding box coordinates of the face, output a set of facial key points, such as the coordinates of the mouth shape, and finally generate a face image containing the mouth shape coordinate information for subsequent extraction of the mouth shape image from the face image. This embodiment uses S3DF to combine spatial and temporal information, which helps to improve the detection accuracy of facial feature points (such as mouth position), can effectively distinguish complex facial expressions and dynamic changes, and can process three-dimensional flow information. It can still stably identify and locate the mouth under different head postures, angles and expression changes. It is suitable for digital human images in various environments, and can effectively perform facial recognition and mouth shape positioning regardless of whether it is a static picture or a dynamic video.

[0040] It should be noted that the formula for bounding box smoothing can be expressed as: For each bounding box b i , its smoothed coordinate b i ′ is calculated by taking the mean of itself and its first T-1 boundaries, i.e.

[0041] Commonly used facial key point detection models include Dlib's 68 facial key point model, MediaPipe FaceMesh, and OpenPose.

[0042] S103, based on the answer and the face image, perform mouth shape inference and fusion to generate a digital human for answering the user's question.

[0043] Specifically, S103: based on the answer and the face image, perform mouth shape reasoning and fusion to generate a digital human for answering the user's question, including steps S1031 to S1033:

[0044] S1031, based on the text-to-speech TTS model and the answer, voice synthesize the corresponding voice answer to the user's question.

[0045] In this embodiment, based on the text-to-speech TTS (Text To Speech) model and the answer, the speech synthesis corresponding to the user's question specifically includes: segmenting the answer at the sentence level to obtain a number of text fragments; using the TTS model to perform speech synthesis on the several text fragments in parallel in multiple threads to obtain the speech answer corresponding to the user's question. By segmenting the answer at the sentence level, this embodiment can reasonably divide the long text answer that takes 1-2 minutes to output into text fragments (such as 20 seconds each) that adapt to the digital human generation cycle, so as to facilitate and quickly convert it into a speech answer. The TTS model is used to quickly convert text into speech, ensuring that users can quickly obtain voice answers after asking questions, improving user experience, and generating answers that are closer to human speech, which improves user acceptance; the speed and efficiency of speech synthesis are effectively improved through multi-threaded processing, ensuring that the generated speech is smooth and synchronized with the subsequent video streaming process.

[0046] S1032, performing mouth shape inference based on the mouth shape inference model, the voice answer and the face image to obtain a mouth shape change image.

[0047] In this embodiment, after performing face recognition on the digital human image through the S3DF model and locating the position of the mouth shape in the digital human image to obtain a face image, and synthesizing the corresponding voice answer to the user question based on the text-to-speech TTS model and the answer, the face image and the voice answer are input into the mouth shape inference model to output the mouth shape change image.

[0048] Specifically, the lip shape inference model includes a Wav2Lip model.

[0049] It should be noted that the lip shape inference model is mainly used to synchronize lip shape with sound. In addition to the Wav2Lip model, common models include: SyncNet, a model for lip synchronization in video, which uses deep learning to match the lip shape in audio and video; LipSync, which aims to generate corresponding lip movement through the input audio to enhance the visual effect of speech synthesis applications; LipGAN, a generative adversarial network (GAN), is used to generate lip shape animation based on audio signals and can generate high-quality lip-synced videos; OSM (One-Shot Mouth), quickly generates new lip shape animation based on a set of known lip shape videos, and is suitable for rapid production of animation materials; DeepFake technology, which is mainly used for face replacement, can also be applied to lip shape inference to achieve more natural synthesis by analyzing the correspondence between lip shape movements and sounds.

[0050] Specifically, S1032: Based on the mouth shape inference model, the voice answer and the face image, the mouth shape inference is performed to obtain the mouth shape change image, including: converting the voice answer into the corresponding Mel-spectrogram; according to the preset frame rate, the Mel-spectrogram is segmented to obtain the Mel-spectrogram features; the Mel-spectrogram features and the face image are input into the Wav2Lip model to infer the mouth shape change image.

[0051] Specifically, converting the voice answer into a corresponding Mel-spectrogram includes: converting the voice answer into a corresponding Mel-spectrogram according to formula (1):

[0052] M(f,t)=log(∑ n X(n,t)W(f,n))(1), where M(f,t) represents the Mel spectrum value at time t and frequency f, X(n,t) represents the spectrum intensity of the nth speech signal at time t in the speech response, and W(f,n) represents the Mel filter coefficient of the nth speech signal at frequency f in the speech response.

[0053] Specifically, the Mel-spectrogram features and the face image are input into the Wav2Lip model to infer the mouth shape change image, including: inferring the mouth shape change image according to formula (2):

[0054] in, Represents the mouth shape change image, x face represents a face image, x mel Represents a mel-spectrogram feature.

[0055] In this embodiment, the face image and the voice answer are input into the mouth shape inference model, and the mouth shape change image is output, which specifically includes: ① In this embodiment, the voice signal in the voice answer is processed at a sampling rate of 16000 Hz as an example. According to formula (1), the input voice answer is converted into a Mel-Spectrogram. The generated Mel-Spectrogram contains 80 frequency bands, which are used as the model input features of the Wav2Lip model. ② In this embodiment, the preset frame rate of 25 frames per second is taken as an example. The generated Mel-Spectrogram is divided into multiple mel-chunks at a frame rate of 25 frames per second to ensure that each mel-chunks corresponds to a generated frame sequence as a Mel-Spectrogram feature. The division is based on the formula ③ In this embodiment, the mouth shape inference model is taken as the Wav2Lip model as an example. The Wav2Lip model is a structure based on the combination of convolutional neural networks (CNN) and recurrent neural networks (LSTM). The mel-spectrogram features and the face image are input into the Wav2Lip model. The Wav2Lip model calculates the mouth shape changes according to formula (2), and integrates the mouth shape changes into the face image to generate a mouth shape change image corresponding to the voice answer. In this embodiment, by converting the voice answer into the corresponding mel-spectrogram and mapping the high-frequency signal to the mel scale, the dimension of the audio signal can be reduced, thereby effectively reducing the computational complexity, being suitable for the input of the deep learning model, improving the performance of the model, and having better robustness to noise and deformation. By segmenting the mel-spectrogram, the model can better capture the time and frequency characteristics in the audio, improve the learning ability of the model, further enhance the accuracy of mouth shape inference, and at the same time ensure that the subsequently generated digital human can be stably played at a speed of 25 frames per second or more, reducing the occurrence of freezes in the generated digital human, thereby improving the user's viewing experience and the fluency of interaction. Using the Wav2Lip model for lip shape inference can accurately generate lip movements corresponding to the voice response, ensuring that the generated lip shape is highly consistent with the audio features of the voice response, making the digital human's lip shape and voice perfectly synchronized.

[0056] S1033, fusing the mouth shape change image, the voice answer and the face image to generate a digital human for answering user questions.

[0057] Specifically, S1033: Fusion of mouth shape change images, voice answers and face images to generate a digital human for answering user questions, including: extracting a mouth shape image from the face image, and based on a Laplace mixture model, fusing the mouth shape change image and the mouth shape image to obtain the answer mouth shape; linearly weighting the answer mouth shape and the face image to generate a digital human answer image; based on a stream pusher, streaming the digital human answer image and the voice answer to the front end to generate a digital human for answering user questions.

[0058] Specifically, based on the Laplace mixture model, the mouth shape change image and the mouth shape image are fused to obtain the answer mouth shape, including: according to formula (3), the mouth shape change image and the mouth shape image are fused:

[0059] Among them, I output represents the answer mouth shape, α represents the fusion coefficient, Indicates the mouth shape change image, Represents a mouth image.

[0060] In this embodiment, since the recognized face image contains mouth shape coordinate information, after the mouth shape inference is performed based on the mouth shape inference model, the voice answer and the face image, the mouth shape image is extracted based on the mouth shape coordinate information in the face image, and the mouth shape image and the mouth shape change image are fused according to formula (3) to obtain the answer mouth shape. The answer mouth shape and the face image are linearly weighted using the OpenCV weighting function (cv2.addWeighted) to ensure that the mouth shape change image and the face image are seamlessly connected, and the generated digital human answer image is realistic and natural. In addition, the push stream layer in the prior art includes streaming text decomposition to generate audio, RTSP (Real-Time Streaming Protocol) video and audio push stream and WebRTC (Web Real-Time Communications) protocol pull stream. Before performing lip shape inference based on the lip shape inference model, this embodiment applies for a streamer for the digital human answer image corresponding to the user question; the streamer for the digital human answer image corresponding to the user question is created, and after the digital human answer image is generated, the digital human answer image is pushed to the streaming service based on the streamer; the front end requests to play the address of the digital human answer image corresponding to the user question, and pulls the address of the digital human answer image corresponding to the user question from the streaming service to the front end for display, so as to generate a digital human for answering the user question, and uses the improved audio and video synchronization algorithm in the streamer to ensure that the lip shape movements of the digital human answer image and the voice answer are synchronized with the voice answer with high precision, thereby avoiding semantic incoherence and lip shape mismatch, and breaking through the coordination problem between the voice answer of long text and the generation of digital human.

[0061] It should be noted that in order to ensure high efficiency during concurrent processing, each reasoning task, including speech synthesis, lip shape reasoning and image fusion steps, is accelerated in parallel through the multi-threading mechanism of CUDA (Compute Unified Device Architecture), and during the execution of the reasoning task, CUDA's thread lock mechanism is also used to ensure thread safety. The thread lock mechanism can ensure that when multiple reasoning tasks are executed in parallel, each thread's access to shared resources is safe, preventing data competition between different threads.

[0062] In addition, GPU (graphics processing unit) resources are dynamically allocated according to the task priority and resource usage to ensure that parallel tasks can be executed smoothly. During the GPU parallel reasoning process, the orderly execution of reasoning tasks is ensured by the synchronization mechanism of the GPU stream. After each reasoning task (such as lip shape generation and image fusion) completes its respective steps, it will perform CUDA stream synchronization operations to ensure that other tasks will not be affected by resource competition. After each reasoning task uses the GPU resources, the synchronization mechanism releases the resources to prepare for the next task.

[0063] In the parallel processing of speech synthesis, mouth shape generation and image fusion, batch processing is used to accelerate reasoning tasks. For example, when generating the digital human answer image, the speech answer is divided into multiple small segments, and the mouth shape change image is generated through parallel reasoning. Through concurrent processing and reasoning acceleration, it is possible to handle high-concurrency requests and maintain the real-time generation capability of 25 frames per second, achieving efficient processing of multi-user requests and real-time video generation.

[0064] This embodiment supports multiple users accessing at the same time and performing parallel reasoning on multiple Wav2Lip models. Each user request is assigned to an independent reasoning model instance and reasoning is performed through CUDA's concurrent processing capability. Each Wav2Lip model instance runs in an independent CUDA stream, ensuring that each user request does not interfere with each other, and when multiple users access at the same time, their respective tasks are efficiently executed, ensuring the stability of concurrent tasks.

[0065] In addition to using the Laplace mixture model and the weighted function to fuse the mouth shape change image and the face image, this embodiment can also synthesize the mouth shape change image into the face image through a frame interpolation algorithm based on the Poisson equation, specifically: According to the formula Interpolate between frames to synthesize the mouth shape change image into the face image, ensuring that at least 25 frames are generated per second to avoid video freezes caused by low frame rates. is the Laplace operator, and I is the digital human answer image generated after interpolation.

[0066] Taking into account the inconsistency in the time spent on generating voice answers and digital human answer images, there may be a situation where the voice answers and digital human answer images do not correspond when the stream pusher pushes the voice answers and digital human answer images to the streaming media service. In addition to the improved audio and video synchronization algorithm, the stream pusher also includes a real-time frame algorithm model. The real-time frame algorithm model is used to manage the concurrent streaming of audio and video frames (i.e., voice answers and digital human answer images) to ensure that while the digital human answer images are generated, the digital human answer images and the corresponding voice answers can maintain real-time synchronous transmission.

[0067] The real-time frame algorithm model can not only process voice answers and digital human answer images in parallel, but also allocate GPU resources through dynamic scheduling and priority queue scheduling algorithms to optimize streaming performance when multiple users access at the same time, thereby ensuring the stability of the system in high-concurrency scenarios.

[0068] The priority queue scheduling algorithm is used to handle streaming resource scheduling when multiple users access the service concurrently. It dynamically adjusts the processing order of the digital human's answer images based on the urgency of the user's request and the system resource usage to ensure smoothness under concurrent requests.

[0069] A method for generating a digital human provided in this embodiment utilizes the powerful natural language processing capability of the knowledge question and answer big model, applies the knowledge question and answer big model to the generation of a digital human, can process user questions and generate answers in a very short time, provide real-time feedback, improve user experience, and can understand complex user questions and generate fluent and natural answers, so that human-computer interaction is smoother and the generated digital human better satisfies the user's question and answer experience; coordinates the answers generated by the knowledge question and answer big model with the face image, performs mouth shape reasoning and fusion, rather than simply combining the answers with the face image, so that the answers can be reasonably integrated into the generation of the digital human, reducing the incoherence phenomenon in the digital human generation process, improving the continuity of the digital human's movements and the naturalness of the interactive experience, achieving fast and intelligent generation of the digital human, improving the efficiency and effect of the digital human generation, meeting the real-time and interactivity requirements of the user's question and answer experience, and improving the richness of the user experience and the real-time nature of the question and answer. In addition, the use of S3DF combines spatial and temporal information, which helps to improve the detection accuracy of facial feature points (such as mouth position), can effectively identify complex facial expressions and dynamic changes, and can process three-dimensional flow information. It can still stably identify and locate the mouth under different head postures, angles and expression changes, and is suitable for digital human images in various environments. Whether it is static pictures or dynamic videos, it can effectively perform facial recognition and mouth positioning. By segmenting the answers at the sentence level, the long text answers that take 1-2 minutes to output can be reasonably divided into text segments that adapt to the digital human generation cycle (such as 20 seconds each), which is convenient and fast to convert into voice answers. The TTS model is used to quickly convert text into voice, ensuring that users can quickly get voice answers after asking questions, improving user experience, and the generated answers are closer to human voice, which improves user acceptance; the speed and efficiency of speech synthesis are effectively improved through multi-threaded processing, ensuring that the generated voice is smooth and synchronized with the subsequent video streaming process. By converting the voice answer into the corresponding Mel-spectrogram and mapping the high-frequency signal to the Mel scale, the dimension of the audio signal can be reduced, thereby effectively reducing the computational complexity, making it suitable for the input of the deep learning model, improving the performance of the model, and having better robustness to noise and deformation. By segmenting the Mel-spectrogram, the model can better capture the time and frequency characteristics in the audio, improve the learning ability of the model, further enhance the accuracy of lip shape reasoning, and at the same time ensure that the subsequently generated digital human can be played stably at a speed of 25 frames per second or more, reduce the occurrence of stuttering in the generated digital human, and thus improve the user's viewing experience and the smoothness of interaction. Using the Wav2Lip model for lip shape reasoning can accurately generate the lip shape movement corresponding to the voice answer, ensure that the generated lip shape is highly consistent with the audio characteristics of the voice answer, and make the lip shape of the digital human perfectly synchronized with the voice.By using the improved audio and video synchronization algorithm in the streaming device, the lip movements of the digital human answering the image are synchronized with the voice answers with high precision, thus avoiding semantic incoherence and lip mismatch, and breaking through the coordination problem between the voice answers of long texts and the generation of digital humans.

[0070] Embodiment 2:

[0071] like Figure 2 As shown, this embodiment provides a method for generating a digital human. The method for generating a digital human includes:

[0072] S201, obtaining the answer corresponding to the user's question and sending the answer to the text-to-speech TTS model.

[0073] In this embodiment, the answer is Figure 2 The user inputs the user question in text or voice form through the user interaction interface (i.e., the front end). The front end sends the user question in text form to the knowledge question and answer big model, which matches and generates the corresponding answer to the user question. For the user question in voice form input by the user, speech-to-text technology is used, such as Internet technology to convert the user question in voice form into text form, and then the knowledge question and answer big model generates the corresponding answer to the user question.

[0074] S202, obtaining a digital human image, identifying a face image of the digital human image, and sending the face image to a mouth shape inference model.

[0075] In this embodiment, the digital human image is Figure 2 In the static video, the face image of the digital human image is recognized. Figure 2 The mouth position positioning in the face image is Figure 2The image includes but is not limited to pictures and videos. The digital human image is recognized by the S3DF (Spatio-Temporal 3D Flow, three-dimensional depth flow) model, and the position of the mouth in the digital human image is located to obtain the face image, specifically: ① Through OpenCV (open source computer vision library, a BSD (Berkeley Software ① Use the face_detection.FaceAlignment tool (a face detection and key point location library based on deep learning) to detect faces and obtain the bounding box coordinates of the face. The bounding box coordinates of the face are usually a rectangular box that represents the position of the face in the image. ③ Bounding box smoothing is achieved by averaging the coordinates of T consecutive bounding boxes to reduce jitter between frames and avoid sudden changes in the position of the face in the video. ④ Use the facial key point detection model to locate facial feature points, usually based on the bounding box coordinates of the face, output a set of facial key points, such as the coordinates of the mouth shape, and finally generate a face image containing the mouth shape coordinate information for subsequent extraction of the mouth shape image from the face image. This embodiment uses S3DF to combine spatial and temporal information, which helps to improve the detection accuracy of facial feature points (such as mouth position), can effectively distinguish complex facial expressions and dynamic changes, and can process three-dimensional flow information. It can still stably identify and locate the mouth under different head postures, angles and expression changes. It is suitable for digital human images in various environments, and can effectively perform facial recognition and mouth shape positioning regardless of whether it is a static picture or a dynamic video.

[0076] It should be noted that the formula for bounding box smoothing can be expressed as: For each bounding box b i , its smoothed coordinate b i ′ is calculated by taking the mean of itself and its first T-1 boundaries, i.e.

[0077] Commonly used facial key point detection models include Dlib's 68 facial key point model, MediaPipe FaceMesh, and OpenPose.

[0078] S203, based on the text-to-speech TTS model and the answer, voice synthesize the voice answer corresponding to the user's question, and send the voice answer to the lip shape inference model.

[0079] In this embodiment, based on the text-to-speech TTS (Text To Speech) model and the answer, the speech synthesis corresponding to the user's question specifically includes: segmenting the answer at the sentence level to obtain a number of text fragments; using the TTS model to perform speech synthesis on the several text fragments in parallel in multiple threads to obtain the speech answer corresponding to the user's question. By segmenting the answer at the sentence level, this embodiment can reasonably divide the long text answer that takes 1-2 minutes to output into text fragments (such as 20 seconds each) that adapt to the digital human generation cycle, so as to facilitate and quickly convert it into a speech answer. The TTS model is used to quickly convert text into speech, ensuring that users can quickly obtain voice answers after asking questions, improving user experience, and generating answers that are closer to human speech, which improves user acceptance; the speed and efficiency of speech synthesis are effectively improved through multi-threaded processing, ensuring that the generated speech is smooth and synchronized with the subsequent video streaming process.

[0080] S204, based on the mouth shape inference model, the voice answer and the face image, perform mouth shape inference to obtain a mouth shape change image; extract the mouth shape image from the face image, and based on the Laplace mixture model, fuse the mouth shape change image and the mouth shape image to obtain the answer mouth shape; linearly weight the answer mouth shape and the face image to generate a digital human answer image.

[0081] S205, based on the streaming device, the digital human's answer image and voice answer are streamed to the front end to generate a digital human for answering user questions.

[0082] In this embodiment, after performing face recognition on the digital human image through the S3DF model and locating the position of the mouth shape in the digital human image to obtain a face image, and synthesizing the corresponding voice answer to the user question based on the text-to-speech TTS model and the answer, the face image and the voice answer are input into the mouth shape inference model to output the mouth shape change image.

[0083] It should be noted that the lip shape inference model is mainly used to synchronize lip shape with sound. In addition to the Wav2Lip model, common models include: SyncNet, a model for lip synchronization in video, which uses deep learning to match the lip shape in audio and video; LipSync, which aims to generate corresponding lip movement through the input audio to enhance the visual effect of speech synthesis applications; LipGAN, a generative adversarial network (GAN), is used to generate lip shape animation based on audio signals and can generate high-quality lip-synced videos; OSM (One-Shot Mouth), quickly generates new lip shape animation based on a set of known lip shape videos, and is suitable for rapid production of animation materials; DeepFake technology, which is mainly used for face replacement, can also be applied to lip shape inference to achieve more natural synthesis by analyzing the correspondence between lip shape movements and sounds.

[0084] The face image and voice answer are input into the mouth shape inference model, and the mouth shape change image is output, which specifically includes: ① This embodiment takes the sampling rate of 16000Hz to process the voice signal in the voice answer as an example. According to formula (1), the input voice answer is converted into a Mel-Spectrogram. The generated Mel-Spectrogram contains 80 frequency bands as the model input feature of the Wav2Lip model. ② This embodiment takes the preset frame rate of 25 frames per second as an example. According to the frame rate of 25 frames per second, the generated Mel-Spectrogram is divided into multiple mel-chunks small fragments to ensure that each mel-chunks small fragment corresponds to the generated frame sequence as the Mel-Spectrogram feature. The division is based on the formula ③ In this embodiment, the mouth shape inference model is taken as the Wav2Lip model as an example. The Wav2Lip model is a structure based on the combination of convolutional neural networks (CNN) and recurrent neural networks (LSTM). The mel-spectrogram features and the face image are input into the Wav2Lip model. The Wav2Lip model calculates the mouth shape changes according to formula (2), and integrates the mouth shape changes into the face image to generate a mouth shape change image corresponding to the voice answer. In this embodiment, by converting the voice answer into the corresponding mel-spectrogram and mapping the high-frequency signal to the mel scale, the dimension of the audio signal can be reduced, thereby effectively reducing the computational complexity, being suitable for the input of the deep learning model, improving the performance of the model, and having better robustness to noise and deformation. By segmenting the mel-spectrogram, the model can better capture the time and frequency characteristics in the audio, improve the learning ability of the model, further enhance the accuracy of mouth shape inference, and at the same time ensure that the subsequently generated digital human can be stably played at a speed of 25 frames per second or more, reducing the occurrence of freezes in the generated digital human, thereby improving the user's viewing experience and the fluency of interaction. Using the Wav2Lip model for lip shape inference can accurately generate lip movements corresponding to the voice response, ensuring that the generated lip shape is highly consistent with the audio features of the voice response, making the digital human's lip shape and voice perfectly synchronized.

[0085] Since the recognized face image contains mouth shape coordinate information, after the mouth shape inference model, voice answer and face image are used to perform mouth shape inference and obtain the mouth shape change image, the mouth shape image is extracted based on the mouth shape coordinate information in the face image, and the mouth shape image and the mouth shape change image are fused according to formula (3) to obtain the answer mouth shape. The answer mouth shape and the face image are linearly weighted using the OpenCV weighting function (cv2.addWeighted) to ensure that the mouth shape change image and the face image are seamlessly connected, and the generated digital human answer image is realistic and natural. In addition, the push stream layer in the prior art includes streaming text decomposition to generate audio, RTSP (Real-Time Streaming Protocol) video and audio push stream and WebRTC (Web Real-Time Communications) protocol pull stream. Before performing lip shape inference based on the lip shape inference model, this embodiment applies for a streamer for the digital human answer image corresponding to the user question; the streamer for the digital human answer image corresponding to the user question is created, and after the digital human answer image is generated, the digital human answer image is pushed to the streaming service based on the streamer; the front end requests to play the address of the digital human answer image corresponding to the user question, and pulls the address of the digital human answer image corresponding to the user question from the streaming service to the front end for display, so as to generate a digital human for answering the user question, and uses the improved audio and video synchronization algorithm in the streamer to ensure that the lip shape movements of the digital human answer image and the voice answer are synchronized with the voice answer with high precision, thereby avoiding semantic incoherence and lip shape mismatch, and breaking through the coordination problem between the voice answer of long text and the generation of digital human.

[0086] It should be noted that in order to ensure high efficiency during concurrent processing, each reasoning task, including speech synthesis, lip shape reasoning and image fusion steps, is accelerated in parallel through CUDA's multi-threading mechanism, and during the execution of the reasoning task, CUDA's thread lock mechanism is also used to ensure thread safety. The thread lock mechanism can ensure that when multiple reasoning tasks are executed in parallel, each thread's access to shared resources is safe, preventing data competition between different threads.

[0087] In addition, GPU resources are dynamically allocated based on task priority and resource usage to ensure that parallel tasks can be executed smoothly. During GPU parallel reasoning, the orderly execution of reasoning tasks is ensured through the synchronization mechanism of the GPU stream. After each reasoning task (such as lip shape generation and image fusion) completes its respective steps, it will perform CUDA stream synchronization operations to ensure that other tasks will not be affected by resource competition. After each reasoning task uses GPU resources, it releases resources through the synchronization mechanism to prepare for the next task.

[0088] In the parallel processing of speech synthesis, mouth shape generation and image fusion, batch processing is used to accelerate reasoning tasks. For example, when generating the digital human answer image, the speech answer is divided into multiple small segments, and the mouth shape change image is generated through parallel reasoning. Through concurrent processing and reasoning acceleration, it is possible to handle high-concurrency requests and maintain the real-time generation capability of 25 frames per second, achieving efficient processing of multi-user requests and real-time video generation.

[0089] This embodiment supports multiple users accessing at the same time and performing parallel reasoning on multiple Wav2Lip models. Each user request is assigned to an independent reasoning model instance and reasoning is performed through CUDA's concurrent processing capability. Each Wav2Lip model instance runs in an independent CUDA stream, ensuring that each user request does not interfere with each other, and when multiple users access at the same time, their respective tasks are efficiently executed, ensuring the stability of concurrent tasks.

[0090] In addition to using the Laplace mixture model and the weighted function to fuse the mouth shape change image and the face image, this embodiment can also synthesize the mouth shape change image into the face image through a frame interpolation algorithm based on the Poisson equation, specifically: According to the formula Interpolate between frames to synthesize the mouth shape change image into the face image, ensuring that at least 25 frames are generated per second to avoid video freezes caused by low frame rates. is the Laplace operator, and I is the digital human answer image generated after interpolation.

[0091] Taking into account the inconsistency in the time spent on generating voice answers and digital human answer images, there may be a situation where the voice answers and digital human answer images do not correspond when the stream pusher pushes the voice answers and digital human answer images to the streaming media service. In addition to the improved audio and video synchronization algorithm, the stream pusher also includes a real-time frame algorithm model. The real-time frame algorithm model is used to manage the concurrent streaming of audio and video frames (i.e., voice answers and digital human answer images) to ensure that while the digital human answer images are generated, the digital human answer images and the corresponding voice answers can maintain real-time synchronous transmission.

[0092] The real-time frame algorithm model can not only process voice answers and digital human answer images in parallel, but also allocate GPU resources through dynamic scheduling and priority queue scheduling algorithms to optimize streaming performance when multiple users access at the same time, thereby ensuring the stability of the system in high-concurrency scenarios.

[0093] The priority queue scheduling algorithm is used to handle streaming resource scheduling when multiple users access the service concurrently. It dynamically adjusts the processing order of the digital human's answer images based on the urgency of the user's request and the system resource usage to ensure smoothness under concurrent requests.

[0094] A method for generating a digital human provided in this embodiment utilizes the powerful natural language processing capability of the knowledge question and answer big model, applies the knowledge question and answer big model to the generation of a digital human, can process user questions and generate answers in a very short time, provide real-time feedback, improve user experience, and can understand complex user questions and generate fluent and natural answers, so that human-computer interaction is smoother and the generated digital human better satisfies the user's question and answer experience; coordinates the answers generated by the knowledge question and answer big model with the face image, performs mouth shape reasoning and fusion, rather than simply combining the answers with the face image, so that the answers can be reasonably integrated into the generation of the digital human, reducing the incoherence phenomenon in the digital human generation process, improving the continuity of the digital human's movements and the naturalness of the interactive experience, achieving fast and intelligent generation of the digital human, improving the efficiency and effect of the digital human generation, meeting the real-time and interactivity requirements of the user's question and answer experience, and improving the richness of the user experience and the real-time nature of the question and answer.

[0095] Embodiment 3:

[0096] like Figure 3 As shown, this embodiment also provides a digital human generation system. The digital human generation system includes a front end 31, a model layer 32, a streaming layer 33 and a digital human scene application layer 34.

[0097] The front end 31 is used to obtain answers to user questions and to visualize the digital human to answer user questions.

[0098] The model layer 32 is used to synthesize the speech answer corresponding to the user's question based on the text-to-speech TTS model and the answer, and is also used to obtain the digital human image and recognize the face image of the digital human image. It is also used to perform mouth shape inference based on the mouth shape inference model, the speech answer and the face image to obtain the mouth shape change image. It is also used to extract the mouth shape image from the face image, fuse the mouth shape change image and the mouth shape image based on the Laplace mixture model to obtain the answer mouth shape, and linearly weight the answer mouth shape and the face image to generate the digital human answer image.

[0099] In this embodiment, the model layer includes a speech acquisition synthesis model, a facial recognition model, a mouth action generation model (i.e., a mouth shape inference model), a video clarity optimization model, and a real-time frame algorithm model. The speech acquisition synthesis model includes but is not limited to the Bert-VITS2 model. The facial recognition model includes but is not limited to the S3FD model. The mouth action generation model includes but is not limited to the Wav2Lip model, the difference image algorithm, and the Poisson algorithm optimization model. The video clarity optimization model includes but is not limited to the GfpGAN model. The real-time frame algorithm model includes but is not limited to the concurrent model.

[0100] The streaming layer 33 is used to stream the digital human's answer images and voice answers to the front end based on the streaming device, and generate a digital human to answer user questions.

[0101] In this embodiment, the streaming layer includes a traditional video stream push service and a streaming breakpoint resume video stream push service. The traditional video stream push service includes real-time video stream processing, RTSP (Real-Time Streaming Protocol) service (audio and video push), WebRTC (Web Real-Time Communications) connection, and streaming media server ZLMediaKit. The streaming breakpoint resume video stream push service includes a streaming text video generation service (CoreTech), RTSP service (audio and video push), WebRTC connection, and streaming media server ZLMediaKit.

[0102] The digital human scene application layer 34 is used to connect to the digital human application scene.

[0103] In this embodiment, the application scenarios of digital humans include digital human capability opening scenarios and touch point opening scenarios. The digital human capability opening scenarios include government and enterprise knowledge Q&A, rules and regulations systems and other business systems. The touch point opening scenarios include government and enterprise product digital humans, bid evaluation digital humans and digital humans in other scenarios.

[0104] A method for generating a digital human provided in this embodiment utilizes the powerful natural language processing capability of the knowledge question and answer big model, applies the knowledge question and answer big model to the generation of a digital human, can process user questions and generate answers in a very short time, provide real-time feedback, improve user experience, and can understand complex user questions and generate fluent and natural answers, so that human-computer interaction is smoother and the generated digital human better satisfies the user's question and answer experience; coordinates the answers generated by the knowledge question and answer big model with the face image, performs mouth shape reasoning and fusion, rather than simply combining the answers with the face image, so that the answers can be reasonably integrated into the generation of the digital human, reducing the incoherence phenomenon in the digital human generation process, improving the continuity of the digital human's movements and the naturalness of the interactive experience, achieving fast and intelligent generation of the digital human, improving the efficiency and effect of the digital human generation, meeting the real-time and interactivity requirements of the user's question and answer experience, and improving the richness of the user experience and the real-time nature of the question and answer.

[0105] Embodiment 4:

[0106] like Figure 4As shown, this embodiment also provides a digital human generation device, including a knowledge question and answer module 41, an image recognition module 42 and a generation module 43. The knowledge question and answer module 41 is used to obtain user questions, and input the user questions into the knowledge question and answer model to generate answers corresponding to the user questions. The image recognition module 42 is used to obtain the digital human image and recognize the face image of the digital human image. The generation module 43 is connected to the knowledge question and answer module 41 and the image recognition module 42 respectively, and is used to perform mouth shape reasoning and fusion based on the answers and face images to generate a digital human for answering user questions.

[0107] Specifically, the generation module 43 includes: a speech synthesis unit 431, a mouth shape inference unit 432 and a fusion unit 433. The speech synthesis unit 431 is used to synthesize a speech answer corresponding to the user question based on the text-to-speech TTS model and the answer. The mouth shape inference unit 432 is used to perform mouth shape inference based on the mouth shape inference model, the voice answer and the face image to obtain a mouth shape change image. The fusion unit 433 is used to fuse the mouth shape change image, the voice answer and the face image to generate a digital human for answering user questions.

[0108] Specifically, the mouth shape inference unit 432 includes: a conversion subunit, a segmentation subunit and an inference subunit. The conversion subunit is used to convert the voice response into a corresponding Mel-spectrogram. The segmentation subunit is used to segment the Mel-spectrogram according to a preset frame rate to obtain Mel-spectrogram features. The inference subunit is used to input the Mel-spectrogram features and the face image into the Wav2Lip model to infer the mouth shape change image.

[0109] Specifically, the conversion subunit includes: a minimum conversion unit, which is used to convert the speech answer into a corresponding Mel-spectrogram according to formula (1):

[0110] M(f,t)=log(∑ n X(n,t)W(f,n))(1), where M(f,t) represents the Mel spectrum value at time t and frequency f, X(n,t) represents the spectrum intensity of the nth speech signal at time t in the speech response, and W(f,n) represents the Mel filter coefficient of the nth speech signal at frequency f in the speech response.

[0111] Specifically, the inference subunit includes: an inference minimum unit, which is used to infer the mouth shape change image according to formula (2):

[0112] in, Represents the mouth shape change image, x face represents a face image, x mel Represents a mel-spectrogram feature.

[0113] Specifically, the fusion unit 433 includes: a fusion subunit, a linear weighting subunit and a streaming subunit. The fusion subunit is used to extract the mouth shape image from the face image, and based on the Laplace mixture model, fuse the mouth shape change image and the mouth shape image to obtain the answer mouth shape. The linear weighting subunit is used to linearly weight the answer mouth shape and the face image to generate a digital human answer image. The streaming subunit is used to stream the digital human answer image and voice answer to the front end based on the streaming device to generate a digital human for answering user questions.

[0114] Specifically, the fusion subunit includes: a fusion minimum unit, which is used to fuse the mouth shape change image and the mouth shape image according to formula (3):

[0115] Among them, I output represents the answer mouth shape, α represents the fusion coefficient, Indicates the mouth shape change image, Represents a mouth image.

[0116] It can be understood that the digital human generation device provided above executes the digital human generation method corresponding to the embodiment 1 provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the scheme corresponding to the digital human generation method of the embodiment 1 above, and will not be repeated here.

[0117] Embodiment 5:

[0118] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the method for generating a digital human in the above-mentioned embodiment 1.

[0119] Embodiment 6:

[0120] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for generating a digital human in the above-mentioned embodiment 1 is implemented.

[0121] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating a digital human, characterized in that: include: Obtain user questions, input them into the knowledge question-answering model, and generate answers to the user questions; Acquire a digital human image and recognize a face image of the digital human image; Based on the answer and the face image, mouth shape inference and fusion are performed to generate a digital human for answering user questions.

2. The method for generating a digital human according to claim 1, characterized in that: The method of performing mouth shape reasoning and fusion based on the answer and the face image to generate a digital human for answering the user's question specifically includes: Based on the text-to-speech TTS model and the answer, synthesize a speech answer corresponding to the user's question; Based on the mouth shape inference model, voice answer and face image, mouth shape inference is performed to obtain the mouth shape change image; The mouth shape change images, voice answers and face images are combined to generate a digital human to answer user questions.

3. The method for generating a digital human according to claim 2, characterized in that: The lip shape inference model includes a Wav2Lip model, The method of performing mouth shape reasoning based on the mouth shape reasoning model, the voice answer and the face image to obtain the mouth shape change image specifically includes: Convert the speech response into the corresponding Mel-spectrogram; According to the preset frame rate, the Mel-spectrogram is segmented to obtain Mel-spectrogram features; The Mel-spectrogram features and face images are input into the Wav2Lip model to infer the mouth shape change image.

4. The method for generating a digital human according to claim 3, characterized in that: The converting of the speech response into the corresponding Mel-spectrogram specifically includes: According to formula (1), the speech answer is converted into the corresponding Mel spectrogram: M(f,t)=log(∑ n X(n,t)W(f,n))(1), where M(f,t) represents the Mel spectrum value at time t and frequency f, X(n,t) represents the spectrum intensity of the nth speech signal at time t in the speech response, and W(f,n) represents the Mel filter coefficient of the nth speech signal at frequency f in the speech response.

5. The method for generating a digital human according to claim 3, characterized in that: The mel-spectrogram features and the face image are input into the Wav2Lip model to infer the mouth shape change image, specifically including: According to formula (2), the mouth shape change image is inferred: in, Represents the mouth shape change image, x face represents a face image, x mel Represents a mel-spectrogram feature.

6. The method for generating a digital human according to claim 2, characterized in that: The method of fusing the mouth shape change image, the voice answer and the face image to generate a digital human for answering user questions specifically includes: Extract the mouth shape image from the face image, and fuse the mouth shape change image and the mouth shape image based on the Laplace mixture model to obtain the answer mouth shape; Linearly weight the answering mouth shape and face image to generate the digital human answering image; Based on the streaming device, the digital human's answer images and voice answers are streamed to the front end, generating a digital human to answer user questions.

7. The method for generating a digital human according to claim 6, characterized in that: The method of fusing the mouth shape change image and the mouth shape image based on the Laplace mixture model to obtain the answer mouth shape specifically includes: According to formula (3), the mouth shape change image and the mouth shape image are fused: Among them, I output represents the answer mouth shape, α represents the fusion coefficient, Indicates the mouth shape change image, Represents a mouth image.

8. A digital human generation device, characterized in that: It includes knowledge question-answering module, image recognition module and generation module. The knowledge question answering module is used to obtain user questions, input the user questions into the knowledge question answering model, and generate the corresponding answers to the user questions. The image recognition module is used to obtain the digital human image and recognize the face image of the digital human image. The generation module is connected to the knowledge question and answer module and the image recognition module respectively, and is used to perform mouth shape reasoning and fusion based on the answers and face images to generate a digital human for answering user questions.

9. The digital human generation device according to claim 8, characterized in that: The generation module adopts the multi-thread mechanism and thread lock mechanism of the unified computing device architecture CUDA and the graphics processing unit GPU.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement a method for generating a digital human as claimed in any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a method for generating a digital human as claimed in any one of claims 1 to 7 is implemented.

Images