Digital human real-time question and answer method, system and device for intelligent emotion interaction and medium

Abstract

The invention provides a digital human real-time question-answering method, system and device for intelligent emotion interaction and a medium, and the method comprises the steps: capturing identity information, facial expressions and the like of different figures through face recognition and face attribute analysis, and obtaining an instruction and emotion of a user based on voice recognition and voice emotion analysis; information such as instructions, identities and emotions of a user is fused through a large language model to output appropriate answers, multi-round question answering can be conducted on associated question association contexts, meanwhile, appropriate emotions are given to the answers of the large model in combination with context analysis and emotion analysis, and what expressions and actions are inserted at a certain time are calculated; performing speech synthesis by using proper emotion, and realizing matching of audio and digital population types by using a mouth shape generation algorithm; and finally, generating expressions and actions at proper opportunities in combination with contexts and emotions while broadcasting the voice of the digital person. According to the method, high-simulation digital human real-time question answering with high intelligent emotion interaction can be realized.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal technology of deep learning, and in particular relates to a real-time question-answering method, system, device and medium for a digital human with intelligent emotional interaction. Background Art

[0002] With the continuous development of artificial intelligence technology and the widespread application of large models, real-time question-and-answer digital human technology has also ushered in more innovations and breakthroughs, and has been widely used in the fields of medical care, education, finance, tourism, customer service, entertainment, etc. In the field of live broadcasting, digital humans can act as virtual anchors to interact with the audience, increasing the personalization and fun of live broadcasts. In the service field, digital humans can become intelligent assistants to provide users with detailed answers to questions and reduce labor costs.

[0003] Digital human refers to a virtual character with a digital appearance, which has image ability, expression ability, perception ability, and interaction ability. Digital human can be divided into 2D digital human and 3D digital human by type. 2D digital human has a lower cost and a shorter production cycle, so it is more widely used because it does not require modeling, motion capture, etc. At the same time, 2D high-fidelity digital human is more in line with the image of real people. The intelligent emotional interactive digital human system proposed in the present invention analyzes the user's name, gender, age, question content and emotional state through face recognition, voice recognition, emotion recognition and other technologies, conducts multiple rounds of text dialogue and emotional output based on a large language model, and uses speech synthesis, character expression drive, action drive and other technologies to provide real-time voice replies to users in combination with actions and expressions.

[0004] The existing question-and-answer digital human systems lack the ability to analyze the attributes and emotions of users, and the answers to questions also lack emotional expression, and are insufficient in the ability to interact with actions or expressions. For example, Patent Document 1 (Application Number: 202310964008.9) provides a real-time question-and-answer virtual human video generation method, electronic device, and storage medium. This solution achieves consistency between audio and character lip shape based on a face-driven algorithm, and can perform real-time streaming questions and answers, but the digital human lacks the ability to understand and express emotions, and cannot interact with actions and expressions. For example, Patent Document 2 (Application Number: 202410235564.7) provides a 2D digital human interaction system and a 2D digital human interaction method. This solution can realize the interaction of 2D digital humans with expressions and emotions, but lacks the ability to analyze the attributes of the characters and interact with actions. That is, the current 2D real-time question-and-answer digital human is not comprehensive enough in character attribute analysis and emotional understanding, and has deficiencies in the interactive dimension. The problems include: 1) lack of analysis of character attributes such as name, gender, age, expression, and emotion, and inability to adopt different intelligent interaction strategies for different users in different states; 2) no emotional and contextual analysis of the reply text, the broadcast audio lacks emotional expression, the digital human image lacks action, expression interaction, or the timing of the interaction does not match the context. Summary of the invention

[0005] In view of the above technical problems, the technical solution adopted by the present invention is:

[0006] According to a first aspect of the present invention, a real-time question-answering method for a digital human with intelligent emotional interaction is provided, the method comprising the following steps:

[0007] S100, in response to detecting an interactive user, waking up the digital human and broadcasting a greeting.

[0008] S200, in response to currently receiving an interactive voice signal, obtaining an interactive user corresponding to the interactive voice signal as a target user of the current interaction, and performing multimodal information recognition on the target user to obtain multimodal information of the target user, wherein the multimodal information includes identity information, attribute information, interactive instructions, facial expressions, and voice emotions.

[0009] S300, instructing the large language model to generate reply content with emotional understanding based on the multimodal information of the target user; the large language model generates the corresponding reply content based on the streaming generation method.

[0010] S400, if it is detected that the large language model currently generates a reply sentence, instruct the large language model to perform context and sentiment analysis on the currently generated reply sentence to obtain a corresponding analysis result, wherein the analysis result includes the emotion, facial expression and action used by the digital human when broadcasting the reply sentence.

[0011] S500, driving the facial expression of the current digital human material based on the facial expression used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the facial expression, and driving the action of the digital human material driven by the facial expression based on the action used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the action, as the intermediate digital human material.

[0012] S600, generating the current audio to be broadcast based on the currently generated reply sentence, the specified timbre and the emotion used by the digital human when broadcasting the reply sentence, and generating a digital human material whose lip shape matches the current audio to be broadcast based on the current audio to be broadcast and the face image in the intermediate digital human material as the target digital human material.

[0013] S700, using the target digital human material to broadcast the current audio to be broadcasted, and executing S400.

[0014] According to a second aspect of the present invention, a digital human real-time question-answering system with intelligent emotional interaction is provided, the system comprising:

[0015] The wake-up module is used to wake up the digital human and broadcast a greeting in response to detecting an interactive user.

[0016] The multimodal information recognition module, in response to currently receiving an interactive voice signal, obtains the interactive user corresponding to the interactive voice signal as the target user of the current interaction, and performs multimodal information recognition on the target user to obtain the multimodal information of the target user, wherein the multimodal information includes identity information, attribute information, interactive instructions, facial expressions and voice emotions.

[0017] The reply content generation module is used to instruct the large language model to generate reply content with emotional understanding based on the multimodal information of the target user; the large language model generates the corresponding reply content based on the streaming generation method.

[0018] The analysis module is used to instruct the large language model to perform context and sentiment analysis on the currently generated reply sentence if it is detected that the large language model currently generates a reply sentence, and obtain the corresponding analysis result, wherein the analysis result includes the emotion, facial expression and action used by the digital human when broadcasting the reply sentence.

[0019] The driving module is used to drive the facial expression of the current digital human material based on the facial expression used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the facial expression, and to drive the action of the digital human material driven by the facial expression based on the action used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the action as the intermediate digital human material.

[0020] The speech synthesis and lip shape generation module is used to generate the current audio to be broadcast based on the currently generated reply sentence, the specified timbre and the emotion used by the digital human when broadcasting the reply sentence, and to generate a digital human material whose lip shape matches the current audio to be broadcast based on the current audio to be broadcast and the face image in the intermediate digital human material, as the target digital human material.

[0021] The broadcasting module is used to broadcast the current audio to be broadcasted using the target digital human material.

[0022] According to a third aspect of the present invention, there is provided an electronic device comprising a processor and a memory; the processor is used to execute the steps of the method described in the first aspect of the present invention by calling a program or instruction stored in the memory.

[0023] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method described in the first aspect of the present invention.

[0024] The present invention has at least the following beneficial effects:

[0025] The digital human real-time question-and-answer method with intelligent emotional interaction provided by the embodiment of the present invention has at least the following advantages: it integrates multimodal information recognition algorithms and has a deeper understanding of multiple dimensional information such as the user's identity, attributes, emotions, and instructions; the large language model question-and-answer method integrates the identity and emotional information of the character and generates replies with emotional expression, which is more natural; it has the ability to understand emotions and scenes, and can be reflected through interactive forms such as voice emotions, digital human expressions, and actions.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flow chart of a real-time question-and-answer method for a digital human with intelligent emotional interaction provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0031] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0032] Aiming at the situation that the emotional interaction of digital human in the background technology is not comprehensive and intelligent enough, the present invention proposes a real-time question-and-answer method for digital human with intelligent emotional interaction. The present invention utilizes multiple artificial intelligence technologies such as face recognition, face attribute recognition, speech recognition, speech emotion analysis, large language model question-and-answer and context analysis, speech synthesis, lip generation, expression generation, and action generation. First, the identity information and facial expressions of different characters are captured through face recognition and face attribute analysis, and the user's instructions and emotions are obtained based on speech recognition and speech emotion analysis; the large language model is used to integrate the user's instructions, identity, emotions and other information to output appropriate answers, and multiple rounds of questions and answers can be conducted for related questions and contexts, and at the same time, the context analysis and emotion analysis are combined to give appropriate emotions to the answers of the large model and calculate what kind of expressions and actions to insert at a certain time; speech synthesis is performed using appropriate emotions, and the lip generation algorithm is used to achieve the matching of audio and digital human mouth shape; finally, while the digital human voice is broadcasting, the expression and action are generated at the appropriate time in combination with the context and emotion, thereby realizing the real-time question-and-answer of highly realistic digital human with highly intelligent emotional interaction.

[0033] Furthermore, the embodiment of the present invention provides a real-time question-answering method for a digital human with intelligent emotional interaction, which may include: Figure 1 The following steps are shown:

[0034] S100, in response to detecting an interactive user, waking up the digital human and broadcasting a greeting.

[0035] In the embodiment of the present invention, the digital human real-time question-and-answer system will use a camera to shoot the surrounding environment in real time. If a user is photographed staying in front of the system for a time longer than the set time, it is determined that an interactive user is detected. At this time, the digital human will be awakened and a greeting will be played. If there are multiple interactive users detected, the greeting can be "Hello, how can I help you?" If the interactive user detected is one person, the user's face recognition can be performed to confirm the identity information and attribute information. For example, through face recognition, the current interactive user is Xiao A, and the greeting can be "Xiao A, hello, how can I help you?" If the current interactive user is not in the database, a greeting can be given based on the gender and age of the face attribute recognition result.

[0036] In the embodiment of the present invention, the identity information may include name, and the attribute information may include gender, age, and coordinates of key points of the face, etc. The identity information and attribute information of the interactive user will be introduced later.

[0037] S200, in response to currently receiving an interactive voice signal, obtaining an interactive user corresponding to the interactive voice signal as a target user of the current interaction, and performing multimodal information recognition on the target user to obtain multimodal information of the target user, wherein the multimodal information includes identity information, attribute information, interactive instructions, facial expressions, and voice emotions.

[0038] In the embodiment of the present invention, the target user can be obtained through the following steps:

[0039] S201, if the number of interactive users p=1, take the interactive user as the target user, if p>1, execute S202.

[0040] S202: Acquire a face region of each interactive user based on a preset face detection algorithm.

[0041] In an embodiment of the present invention, the preset facial feature extraction algorithm is a YOLOv5 face detection algorithm.

[0042] S203, obtaining the key point coordinates of each face region based on the set face key point detection model.

[0043] In an embodiment of the present invention, the preset face feature extraction algorithm is an ArcFace face feature extraction algorithm. The feature vector of the face region may be a 512-dimensional feature vector, which may include 68 key feature points.

[0044] S204: Based on the acquired coordinates of the lip key points in each face region, the corresponding lip opening and closing degree is acquired.

[0045] In the embodiment of the present invention, among the 68 contour feature points of the face, the coordinates of points 63, 67, 61, and 65 represent the upper and lower and left and right boundaries of the lips, and the degree of lip opening and closing. mouth It can be expressed as: mouth =normalize((P y 67 -P y 63 ) / (P x 65 -P x 61 )); where normalize() is the normalization function, P y 67 and P y 63 denote the y coordinates of contour points 67 and 63, respectively,x 65 and P x 61 Represent the x-coordinates of contour points 65 and 61 respectively.

[0046] In an embodiment of the present invention, the facial key point detection model may be set to an existing facial key point detection model, such as the InsightFace model.

[0047] S205: The interactive user corresponding to the face area whose lip opening and closing degree changes with the time sequence of the interactive voice signal is taken as the target user.

[0048] Generally, when the mouth is closed, the degree of lip opening and closing tends to 0, and when the mouth is spoken, the degree of lip opening and closing continuously changes between 0 and 1. When identifying interactive voice signals, the target user can be determined by continuously judging the changes in the degree of mouth opening and closing of multiple characters in the picture, that is, the interactive user corresponding to the face area whose lip opening and closing degree has been changing during the time period corresponding to the interactive voice signal is taken as the target user. In this way, the target user who is currently interacting with the digital human system can be accurately identified in a multi-person scene.

[0049] Furthermore, the identity information of the target user can be obtained based on the following steps:

[0050] S110: Acquire a face region of a target user based on a preset face detection algorithm.

[0051] S111, obtaining a feature vector of the face region based on a preset face feature extraction algorithm.

[0052] In an embodiment of the present invention, the preset facial feature extraction algorithm is the ArcFace facial feature extraction algorithm.

[0053] S112, obtaining the similarity between the facial feature vector and the i-th facial feature vector in the preset facial feature library. If the similarity is greater than a set similarity threshold, the person identity information corresponding to the i-th facial region is used as the identity information of the target user. The value of i ranges from 1 to m, the initial value of i is 1, and m is the number of facial feature vectors in the preset facial feature library.

[0054] In the embodiment of the present invention, the similarity may be cosine similarity. The specific algorithm of cosine similarity may be prior art. In the embodiment of the present invention, the similarity threshold may be set as an empirical value, for example, 0.5.

[0055] In the embodiment of the present invention, attribute information can be obtained through existing facial attribute analysis algorithms. For example, gender can be obtained through gender recognition algorithms based on eigenfaces, gender recognition methods based on Fisher criteria, and facial gender classification algorithms based on Adaboost+SVM. Age can be obtained by predicting the age in a facial image using a deep learning model trained based on a large amount of facial data of different age groups. Key point coordinates can be obtained through existing facial key point detection models, such as InsightFace model detection.

[0056] In the embodiment of the present invention, the target user's interactive instruction is obtained by performing voice recognition on the interactive voice signal. Specifically, the audio, i.e., the interactive voice signal, can be converted into the corresponding text content, i.e., the interactive instruction, through a streaming voice recognition algorithm.

[0057] In an embodiment of the present invention, the face image of the target user may be input into a trained expression recognition algorithm to obtain the facial expression of the target user, which may include happiness, sadness, anger, surprise, fear, disgust, and neutrality.

[0058] In an embodiment of the present invention, a trained deep learning speech emotion recognition algorithm can be used to perform emotion analysis on the audio based on the time domain features and spectral features of the audio, and analyze the speech emotions of the target user, such as neutrality, joy, sadness, anger, etc.

[0059] S300, instructing the large language model to generate reply content with emotional understanding based on the multimodal information of the target user; the large language model generates the corresponding reply content based on the streaming generation method.

[0060] In an embodiment of the present invention, the large language model may be an existing large language model. Conventional generative dialogue models respond based on user question texts, and the responses appear mechanical and lack emotional expression. Based on the dialogue model, the present invention clarifies the identity information and attribute information of the current user, and further adds an emotional adjustment mechanism. The large language model can generate a more natural response that is more in line with the user's emotional state based on the results of the above multimodal information recognition.

[0061] In an embodiment of the present invention, by constructing a question text instruction of a large language model, the large language model can be combined with the identity information, attribute information and expression of the character to make a response with emotional understanding based on the user's interactive instructions. The large language model response generation adopts streaming generation, and there is no need to wait for all the text to be generated before responding. It can respond sentence by sentence, which can improve the response speed. When generating an emotional response, the large voice model will combine the historical data of the target user, including the previous questions of the same user and the replies of the large language model, to ensure the coherence of the reply content, and it is also necessary to maintain emotional coherence in multiple rounds of dialogue. Since subsequent questions may have logical correlations with previous questions, the large language model can automatically generate relevant replies in combination with previous historical data, and maintain emotional coherence in combination with historical data.

[0062] In an exemplary embodiment, the question text instruction of the constructed large language model may be: Character name: Zhang San; Gender: Male; Age: 10; Emotion: Sad; Question: What are the fun attractions in Beijing? Please combine the above character information and question to make a reply that is consistent with the identity and emotional state.

[0063] Based on the received text instructions, the large language model will generate the following response content:

[0064] "Little Zhang San, I know you may feel a little sad right now, but I hope you can remember that there are many wonderful things in life waiting for us to discover. Beijing is a city full of history and culture, and there are many interesting places to make you happy. Here are some suggestions that may help you find some fun:

[0065] Tiananmen Square: This is one of the largest city center squares in the world! Although you are in a bad mood now, standing in such a large square, watching the people coming and going, and feeling the vitality of the city may make you feel better.

[0066] The Palace Museum: It used to be where the emperor lived, but now it has become a museum. There are many ancient buildings and beautiful cultural relics inside. You can imagine yourself as a little explorer, exploring stories from the past. Maybe this will become a special adventure that will make you forget your troubles for a while.

[0067] Summer Palace: This is a very beautiful royal garden with lakes, bridges and corridors. In spring, the flowers here are particularly beautiful. You can ask your parents to take you for a walk, breathe fresh air, and see the beautiful scenery, which will gradually improve your mood.

[0068] Zoo: If you like animals, there are all kinds of cute animals waiting for you to meet them! "

[0069] S400, if it is detected that the large language model currently generates a reply sentence, instruct the large language model to perform context and sentiment analysis on the currently generated reply sentence to obtain a corresponding analysis result, wherein the analysis result includes the emotion, facial expression and action used by the digital human when broadcasting the reply sentence.

[0070] In the embodiment of the present invention, the context and sentiment analysis text instructions can be constructed to instruct the large language model to perform context and sentiment analysis on the currently generated reply sentence. The large language model analyzes the sentiment and scene of the reply content to analyze what kind of sentiment is used in the voice broadcast and what expressions and actions the digital human uses. For example, when expressing welcome, the digital human uses a cheerful tone to broadcast, with a happy expression and a waving gesture; when introducing with pictures, a neutral tone and expression can be used, with the palm open and pointing to the picture.

[0071] With all the reply content generated above, the constructed text instructions can be as follows:

[0072] "Little friend Zhang San, I know you may feel a little sad right now, but I hope you can remember that there are many wonderful things in life waiting for us to discover. Beijing is a city full of history and culture, and there are many interesting places to cheer you up. Here are some suggestions that may help you find some fun: Tiananmen Square: This is one of the largest city center squares in the world! Although you may not be in a good mood now, standing in such a large square, watching the people coming and going, and feeling the vitality of the city may make you feel better. The Palace Museum: It used to be where the emperor lived, but now it has become a museum. There are many ancient buildings and beautiful cultural relics in it. You can imagine yourself as a little explorer, exploring stories from the past here. Who knows? It will be a special adventure that will make you forget your worries temporarily. Summer Palace: This is a very beautiful royal garden with lakes, bridges and corridors. In spring, the flowers here are particularly beautiful. You can ask your parents to take you for a walk, breathe fresh air, and see the beautiful scenery, so that your mood will gradually improve. Zoo: If you like animals, there are all kinds of cute little animals waiting for you to meet! You are an intelligent assistant. Please analyze the context and emotions of the above text, and analyze what actions or expressions should be used for each sentence to be reported, and use the form of [action, expression] at the end, which can be empty. Actions are limited to [waving, clapping, spreading palms, thumbs up, none], and expressions are limited to [neutral, joy, sadness, anger, none]".

[0073] The large language model generates analysis results based on the received text instructions, which can be shown as follows:

[0074] “Little Zhang San, I know you may feel sad right now, but I hope you can remember that there are many beautiful things in life waiting for us to discover [palms open, sad].

[0075] Beijing is a city full of history and culture, with many interesting places to keep you entertained. Here are some suggestions that may help you find some fun: [None, Neutral]

[0076] Tiananmen Square: This is one of the largest city center squares in the world! Although you are not in a good mood now, standing in such a large square, watching the people coming and going, and feeling the vitality of the city may make you feel better. [Open palm, neutral]

[0077] The Palace Museum: It used to be where the emperor lived, but now it has become a museum. There are many ancient buildings and beautiful cultural relics inside. You can imagine yourself as a little explorer, exploring the stories of the past here. Maybe this will become a special adventure that will make you forget your troubles for a while. [Waving, joy]

[0078] Summer Palace: This is a very beautiful royal garden with lakes, bridges and corridors. In spring, the flowers here are particularly beautiful. You can ask your parents to take you for a walk, breathe fresh air, and see the beautiful scenery, so that your mood will gradually improve. [Like, joy]

[0079] Zoo: If you like animals, there are all kinds of cute little animals waiting for you to meet! [Applause, joy]".

[0080] In an embodiment of the present invention, if the large language model is not detected to generate a reply statement at present, it means that the generation of the reply statement of the large voice model for the current interactive instruction is completed, and then S200 is executed, that is, continuing to detect whether there is a next interactive voice signal, that is, whether there is a next round of inquiry from the user.

[0081] S500, driving the facial expression of the current digital human material based on the facial expression used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the facial expression, and driving the action of the digital human material driven by the facial expression based on the action used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the action, as the intermediate digital human material.

[0082] The digital human real-time question-and-answer system has built-in reference images or videos of digital human faces with different expressions. Based on the MCNET face driving algorithm, the digital human is driven to use the corresponding expression to broadcast at a specific moment, and the expression video of character B can be used to drive the face image of character A to make the corresponding expression. Specifically, in S500, the facial expression of the current digital human material is driven based on the facial expression used by the digital human when broadcasting the reply sentence, and the digital human material after the facial expression drive is obtained, which specifically includes:

[0083] S501, based on the facial expression used by the digital human when broadcasting the reply sentence, obtain the corresponding expression driving video from the preset facial expression material library, and obtain the facial image set corresponding to the expression driving video as the driving facial image set.

[0084] S502: Establish a key point correspondence relationship between the face image of the current digital human material and the first frame face image in the driving face image set.

[0085] S503, using the depth prediction encoder and decoder to estimate the depth of the first face image img1 in the driving face image set, to obtain the corresponding first depth image Depth1, and based on img1 and Depth1, using the set face key point detection model to predict the corresponding first face key point parameter kp1, and use it as the driven face key parameter kps.

[0086] In order to ensure a natural face driving effect, the posture of the facial key points of the first frame driving image must be consistent with the driven image (digital human face image), that is, the degree of opening and closing of the eyes and mouth, etc. are consistent. The embodiment of the present invention adopts a self-driving method, and the driven image is also used as the driving image of the first frame.

[0087] Those skilled in the art know that using a depth prediction encoder and a decoder to estimate the depth of a face image to obtain a corresponding depth image is an existing technology.

[0088] In an embodiment of the present invention, the facial key point parameters may include the coordinates of 15 key points.

[0089] S504, set the image number counting variable j=2.

[0090] S505, using the depth prediction encoder and decoder to estimate the depth of the j-th target face image imgj in the driving face image set, to obtain the corresponding j-th depth image Depthj, and based on imgj and Depthj, using the set face key point detection model to predict the corresponding j-th face key point parameter kpj.

[0091] S506, based on kp1, kps and kpj, obtain the normalization parameter kpnormj=Normalize(kp1, kps, kpj), where Normalize() is a normalization function;

[0092] S507, input img1, kp1, kpnormj, Depth1 and Depthj into the feature deformation module and cross-modal attention module of the face synthesis network to distort the facial features and correct them, and obtain the self-driven high-definition image imgcj corresponding to imgj.

[0093] In the embodiment of the present invention, imgcj=Generator(img 1 , kp 1 , kp norm j , Depth 1 , Depth j ). Generator is the feature deformation module and cross-modal attention module of the face synthesis network.

[0094] Those skilled in the art know that 1 、kp 1 、kp norm j 、Depth 1 and Depth j The face features are distorted and corrected in the feature deformation module and cross-modal attention module of the input face synthesis network to obtain img j The corresponding self-driven high-definition image can be the prior art.

[0095] S508, set j=j+1, if j≤m, execute S505, otherwise, exit the current control program to obtain the digital human material after facial expression driving; m is the number of images in the driving face image set.

[0096] Since the current mainstream action generation algorithm is slow, cannot achieve real-time performance, and has defects in the generated details, the present invention adopts the solution of atomic action modeling and frame insertion. When recording the digital human image, the basic material of the character standing still is recorded, the character posture remains unchanged, and each atomic action, the starting and ending posture of the atomic action are consistent with the basic material.

[0097] When the digital human is broadcasting, when an action is triggered at a specific moment, the corresponding atomic action is obtained from the preset action material library and inserted into the basic material, and the digital human can trigger the corresponding action. In order to optimize the sudden change of the picture when switching between static material and action material, the interpolation algorithm is used to smooth the adjacent frames in the switching gap, so that the action switching of the digital human is more natural.

[0098] Further, in S500, the action of the digital human material driven by the facial expression is driven based on the action used by the digital human when broadcasting the reply sentence, to obtain the action-driven digital human material, which specifically includes:

[0099] S510, based on the action used by the digital human when broadcasting the reply sentence, the corresponding atomic action is obtained from the preset action material library as the driving action.

[0100] S511, using an interpolation algorithm to switch the current action of the digital human material to the driving action, to obtain the action-driven digital human material.

[0101] In the embodiment of the present invention, when the digital human material switches from the current action to the driving action, a set frame of images may be inserted, for example, 15 frames of images may be inserted for smooth transition.

[0102] Those skilled in the art know that any method of switching the current action of the digital human material to the driving action using an interpolation algorithm falls within the protection scope of the present invention.

[0103] S600, generating the current audio to be broadcast based on the currently generated reply sentence, the specified timbre and the emotion used by the digital human when broadcasting the reply sentence, and generating a digital human material whose lip shape matches the current audio to be broadcast based on the current audio to be broadcast and the face image in the intermediate digital human material as the target digital human material.

[0104] In the embodiment of the present invention, voice cloning can be performed using the voices of the same character with different emotions based on the CosyVoice algorithm.

[0105] Algorithm training: Input a preset audio length, such as 10 seconds, and the corresponding text of the audio, and the algorithm can clone the sound, simulate the character's timbre and emotions, thereby training the timbre of neutral, joyful, sad, angry and other emotions.

[0106] In the embodiment of the present invention, the designated timbre is the timbre of the designated speaker, that is, the timbre of the speaker used by the digital human. By inputting the currently generated reply sentence, the designated timbre, and the emotion used by the digital human when broadcasting the reply sentence into the trained CosyVoice algorithm, an audio with the corresponding emotion can be obtained.

[0107] In the embodiment of the present invention, the hubert audio feature encoder can be used to extract the phoneme features of the audio to be broadcast, and 25 feature vectors can be extracted from one second of audio, corresponding to the generation of 25 frames. The current frame face image and audio feature vector are input, and the SyncTalk lip shape generation algorithm is used to generate the image of the corresponding lip shape.

[0108] Model training: Use a preset length of, for example, 5 minutes of synchronized audio and video video of a real person speaking to train the dedicated SyncTalk lip shape generation model to learn the lip shape and pronunciation habits of different people.

[0109] Model reasoning: Based on the trained SyncTalk lip generation model, a video image sequence of a digital human speaking can be generated for other audio, so that the lip shape and audio are synchronized and consistent. That is, the current audio to be broadcast and the face image in the intermediate digital human material are input into the trained SyncTalk lip generation model to obtain a digital human material with a lip shape that matches the current audio to be broadcast.

[0110] S700, using the target digital human material to broadcast the current audio to be broadcasted, and executing S400.

[0111] In the embodiment of the present invention, rtmp streaming technology is used to achieve a video streaming delay of <200ms. In addition, the digital human question and answer screen is generated in a streaming manner, and the corresponding expressions, movements and lip shapes are generated frame by frame, that is, the expressions, movements and lip shapes of the digital human material are generated frame by frame, the server pushes the screen and audio in real time, and the client pulls the video stream in real time and broadcasts it instantly. In this way, a quick reply can be made within 2 seconds for user questions.

[0112] Based on the same inventive concept, an embodiment of the present invention provides a digital human real-time question-answering system with intelligent emotional interaction, wherein the system comprises:

[0113] The wake-up module is used to wake up the digital human and broadcast a greeting in response to detecting an interactive user.

[0114] The multimodal information recognition module, in response to receiving the interactive voice signal, obtains the interactive user corresponding to the interactive voice signal as the target user of the current interaction, and performs multimodal information recognition on the target user to obtain the multimodal information of the target user, wherein the multimodal information includes identity information, attribute information, interactive instructions, facial expressions and voice emotions;

[0115] The reply content generation module is used to instruct the large language model to generate reply content with emotional understanding based on the multimodal information of the target user; the large language model generates the corresponding reply content based on the streaming generation method.

[0116] The analysis module is used to instruct the large language model to perform context and sentiment analysis on the currently generated reply sentence if it is detected that the large language model currently generates a reply sentence, and obtain the corresponding analysis result, wherein the analysis result includes the emotion, facial expression and action used by the digital human when broadcasting the reply sentence.

[0117] The driving module is used to drive the facial expression of the current digital human material based on the facial expression used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the facial expression, and to drive the action of the digital human material driven by the facial expression based on the action used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the action as the intermediate digital human material.

[0118] The speech synthesis and lip shape generation module is used to generate the current audio to be broadcast based on the currently generated reply sentence, the specified timbre and the emotion used by the digital human when broadcasting the reply sentence, and to generate a digital human material whose lip shape matches the current audio to be broadcast based on the current audio to be broadcast and the face image in the intermediate digital human material, as the target digital human material.

[0119] The broadcasting module is used to broadcast the current audio to be broadcasted using the target digital human material.

[0120] The device can be used to perform Figure 1 Therefore, for the functions that can be realized by each functional module of the device, please refer to Figure 1 The description of the illustrated embodiment is not repeated in detail.

[0121] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0122] The embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time question-answering method for digital human with intelligent emotional interaction, characterized in that: The method comprises the following steps: S100, in response to detecting an interactive user, waking up the digital human and broadcasting a greeting; S200, in response to currently receiving an interactive voice signal, obtaining an interactive user corresponding to the interactive voice signal as a target user of the current interaction, and performing multimodal information recognition on the target user to obtain multimodal information of the target user, wherein the multimodal information includes identity information, attribute information, interactive instructions, facial expressions, and voice emotions; S300, instructing the large language model to generate response content with emotional understanding based on the multimodal information of the target user; The large language model generates corresponding reply content based on a streaming generation method; S400, if it is detected that the large language model currently generates a reply sentence, instruct the large language model to perform context and sentiment analysis on the currently generated reply sentence to obtain a corresponding analysis result, wherein the analysis result includes the emotion, facial expression and action used by the digital human when broadcasting the reply sentence; S500, driving the facial expression of the current digital human material based on the facial expression used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the facial expression, and driving the action of the digital human material driven by the facial expression based on the action used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the action, as the intermediate digital human material; S600, generating the current audio to be broadcast based on the currently generated reply sentence, the specified timbre and the emotion used by the digital human when broadcasting the reply sentence, and generating a digital human material whose lip shape matches the current audio to be broadcast based on the current audio to be broadcast and the face image in the intermediate digital human material as the target digital human material; S700, using the target digital human material to broadcast the current audio to be broadcasted, and executing S400.

2. The method according to claim 1, characterized in that The identity information includes name, and the attribute information includes gender, age and coordinates of key points of the face; In S200, the identity information of the target user is obtained based on the following steps: S210, acquiring a face region of a target user based on a preset face detection algorithm; S211, obtaining a feature vector of the face region based on a preset face feature extraction algorithm; S212, obtaining the similarity between the face feature vector and the i-th face feature vector in the preset face feature library, and if the similarity is greater than a set similarity threshold, using the person identity information corresponding to the i-th face region as the identity information of the target user, where i ranges from 1 to m, the initial value of i is 1, and m is the number of face feature vectors in the preset face feature library; In S200, the target user's interactive instruction is obtained by performing voice recognition on the interactive voice signal.

3. The method according to claim 2, characterized in that The preset facial feature extraction algorithm is the YOLOv5 face detection algorithm, and the preset facial feature extraction algorithm is the ArcFace facial feature extraction algorithm.

4. The method according to claim 1, characterized in that: In S500, the facial expression of the current digital human material is driven based on the facial expression used by the digital human when broadcasting the reply sentence, and the digital human material driven by the facial expression is obtained, which specifically includes: S501, based on the facial expression used by the digital human when broadcasting the reply sentence, obtain a corresponding expression driving video from a preset facial expression material library, and obtain a facial image set corresponding to the expression driving video as a driving facial image set; S502, establishing a key point correspondence relationship between the face image of the current digital human material and the first frame face image in the driving face image set; S503, using a depth prediction encoder and a decoder to estimate the depth of the first face image img1 in the driving face image set, to obtain a corresponding first depth image Depth1, and based on img1 and Depth1, using a set face key point detection model to predict the corresponding first face key point parameter kp1, and use it as the driven face key parameter kps; S504, set the image number counting variable j=2; S505, using a depth prediction encoder and a decoder to estimate the depth of the j-th target face image imgj in the driving face image set, to obtain the corresponding j-th depth image Depthj, and based on imgj and Depthj, using a set face key point detection model to predict the corresponding j-th face key point parameter kpj; S506, based on kp1, kps and kpj, obtain the normalization parameter kpnormj=Normalize(kp1, kps, kpj), where Normalize() is a normalization function; S507, input img1, kp1, kpnormj, Depth1 and Depthj into the feature deformation module and cross-modal attention module of the face synthesis network to distort the facial features and correct them, and obtain the self-driven high-definition image imgcj corresponding to imgj; S508, set j=j+1, if j≤m, execute S505, otherwise, exit the current control program to obtain the digital human material after facial expression driving; m is the number of images in the driving face image set.

5. The method according to claim 1, characterized in that: In S500, the action of the digital human material driven by the facial expression is driven based on the action used by the digital human when broadcasting the reply sentence, and the action-driven digital human material is obtained, which specifically includes: S510, based on the action used by the digital human when broadcasting the reply sentence, obtain the corresponding atomic action from the preset action material library as the driving action; S511, using an interpolation algorithm to switch the current action of the digital human material to the driving action, to obtain the action-driven digital human material.

6. The method according to claim 1, characterized in that in, The expressions, movements and lip shapes of the digital human material are generated frame by frame.

7. The method according to claim 1, characterized in that In S200, the target user is obtained through the following steps: S201, if the number of interactive users p=1, take the interactive user as the target user, if p>1, execute S202; S202, obtaining a face region of each interactive user based on a preset face detection algorithm; S203, obtaining the key point coordinates of each face region based on a set face key point detection model; S204, based on the coordinates of the lip key points in each face region, obtaining the corresponding lip opening and closing degree; S205: The interactive user corresponding to the face area whose lip opening and closing degree changes with the time sequence of the interactive voice signal is taken as the target user.

8. A real-time question-answering system for digital humans with intelligent emotional interaction, characterized in that: The system comprises: A wake-up module, used to wake up the digital human and broadcast a greeting in response to detecting an interactive user; The multimodal information recognition module, in response to receiving the interactive voice signal, obtains the interactive user corresponding to the interactive voice signal as the target user of the current interaction, and performs multimodal information recognition on the target user to obtain the multimodal information of the target user, wherein the multimodal information includes identity information, attribute information, interactive instructions, facial expressions and voice emotions; A reply content generation module, used to instruct the large language model to generate reply content with emotional understanding based on the multimodal information of the target user; the large language model generates the corresponding reply content based on the streaming generation method; The analysis module is used to instruct the large language model to perform context and sentiment analysis on the currently generated reply sentence if it is detected that the large language model currently generates a reply sentence, and obtain a corresponding analysis result, wherein the analysis result includes the emotion, facial expression and action used by the digital human when broadcasting the reply sentence; A driving module is used to drive the facial expression of the current digital human material based on the facial expression used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the facial expression, and to drive the action of the digital human material driven by the facial expression based on the action used by the digital human when broadcasting the reply sentence, to obtain the digital human material driven by the action as the intermediate digital human material; The speech synthesis and lip shape generation module is used to generate the current audio to be broadcast based on the currently generated reply sentence, the specified timbre and the emotion used by the digital human when broadcasting the reply sentence, and to generate a digital human material whose lip shape matches the current audio to be broadcast based on the current audio to be broadcast and the face image in the intermediate digital human material as the target digital human material; The broadcasting module is used to broadcast the current audio to be broadcasted using the target digital human material.

9. An electronic device, characterized in that: including a processor and a memory; The processor is used to execute the steps of the method according to any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a program or an instruction, wherein the program or the instruction enables a computer to execute the steps of the method according to any one of claims 1 to 7.

Images