Real-time dialogue human simulation system, real-time dialogue human simulation method and computer program product thereof

Abstract

This invention discloses a real-time dialogue virtual humanoid system, a real-time dialogue virtual humanoid method, and a computer program product thereof. It includes analyzing received images and voice to generate reply text and reply emotion based on the voice analysis results and image analysis results; then using a real human voice model to generate virtual humanoid voice based on the reply text and reply emotion; next, using a real human audio-visual model to generate virtual humanoid images based on the reply emotion and virtual humanoid voice; generating virtual humanoid audio-visual in a question-and-answer state based on the virtual humanoid voice and virtual humanoid images; and further generating virtual humanoid images in a waiting state based on the image analysis results, so as to provide virtual humanoid images in the waiting state and virtual humanoid audio-visual in the question-and-answer state in the audio-visual stream.

Claims

1. A real-time dialogue-based humanoid system, comprising: An image analysis module analyzes received user images to generate image analysis results. This module performs feature attribute recognition on individuals in the user images to identify their personal attribute information and generate a corresponding response style. A speech analysis module analyzes received user speech to generate speech analysis results. This module performs speech-to-text processing on the user speech to generate user questions. A question-and-answer module generates response text and response sentiment based on the image and speech analysis results. This module includes text units and sentiment units. The text unit performs contextual understanding on the user question to generate a complete question, searches a question-and-answer knowledge base based on the complete question, and combines the response style to generate the response text. The sentiment unit generates the response sentiment with sentiment parameters based on the semantic content of the response text. A speech synthesis module uses a real human speech model to... The reply text and the reply emotion generate a lifelike voice. The speech synthesis module preprocesses and converts the reply text to speech, and then generates the lifelike voice based on the processed reply text and the corresponding reply emotion. The lifelike voice model takes real acoustic features as input and outputs a real voice signal. The video generation module uses a real video model to generate a lifelike image based on the reply emotion and the lifelike voice, so as to generate a question-and-answer state lifelike video based on the lifelike image and the lifelike voice. The video generation module renders the lifelike image with lip-sync and emotional expression through the real video model based on the emotional features extracted from the reply emotion and the audio features extracted from the lifelike voice. The real video model takes audio features and facial control parameters as input and outputs a real image signal, and then integrates the lifelike image and the lifelike voice synchronously to generate the question-and-answer state lifelike video.

2. The real-time dialogue simulation system as described in claim 1, wherein, The features include facial attributes, human shape, and clothing. The personal attribute information includes gender, age group, and clothing category. The image analysis results include the answer style. The voice analysis results include the user's question.

3. The real-time dialogue simulation system as described in claim 1, wherein, The preprocessing includes normalization, handling of broken sounds, tone modulation, word segmentation, and / or part-of-speech tagging, and then uses the real human speech model to generate the pseudo-real human speech based on the emotion of the reply and the processed reply text.

4. The real-time dialogue simulation system as described in claim 1, wherein the audio-visual generation module further includes: The emotion mapping unit extracts the emotion features from the response emotion and converts them into facial control parameters according to the emotion mapping rules; the lip shape and expression unit extracts the audio features from the simulated human voice and uses the facial control parameters and audio features as input conditions for the real-life audio-visual model to render facial expressions and lip shapes for the simulated human image; the gesture or action unit uses the real-life audio-visual model to generate gestures or actions based on the response emotion and the simulated human voice; and the audio-visual output unit splices the facial expressions, lip shapes, and gestures or actions to generate the simulated human image, which is then integrated with the simulated human voice to generate the question-and-answer state simulated human audio-visual.

5. The real-time dialogue simulation system as described in claim 1, wherein, The video generation module generates a simulated real-life image of the waiting state based on the image analysis results, and provides the simulated real-life image of the waiting state and the simulated real-life video of the question-and-answer state in the video streaming.

6. A real-time dialogue method for simulating human life, comprising: The system analyzes received user images to generate image analysis results. This includes feature attribute recognition of individuals in the user images to identify their personal attribute information and generate a corresponding response style. It also analyzes received user speech to generate speech analysis results. This involves speech-to-text processing to generate user questions. Based on the image and speech analysis results, it generates response text and response sentiment. This includes contextual understanding of the user questions to generate complete questions, searching a question-and-answer knowledge base based on the complete questions, generating response text based on the response style, and generating response sentiment with emotional parameters based on the semantic content of the response text. Finally, it uses a human voice model to generate a near-human voice based on the response text and response sentiment. The text undergoes preprocessing and text-to-speech processing. Then, based on the processed reply text and the corresponding reply emotion, a simulated human voice is generated. This simulated human voice model takes real-person acoustic features as input and outputs a real-person sound signal. A simulated human video model is used to generate a simulated human video based on the reply emotion and the simulated human voice. Based on the emotional features extracted from the reply emotion and the audio features extracted from the simulated human voice, the simulated human video model renders a lip-synced and emotionally expressive simulated human video. This simulated human video model takes audio features and facial control parameters as input and outputs a real-person video signal. Finally, based on the simulated human video and the simulated human voice, a question-and-answer state simulated human video is generated. The simulated human video and the simulated human voice are synchronously integrated to generate the question-and-answer state simulated human video.

7. The real-time dialogue simulation method as described in claim 6, wherein, The features include facial attributes, human shape, and clothing. The personal attribute information includes gender, age group, and clothing category. The image analysis results include the question-and-answer style. The voice analysis results include the user's question.

8. The real-time dialogue simulation method as described in claim 6, wherein, The steps of generating a pseudo-human voice using a real human voice model based on the reply text and the reply emotion include: normalizing the reply text, removing broken sounds, transposing the text, segmenting words, and / or tagging parts of speech; and generating the pseudo-human voice using the real human voice model based on the reply emotion and the processed reply text.

9. The real-time dialogue simulation method as described in claim 6, wherein, The steps of generating a simulated real-life image based on the response emotion and the simulated real-life voice using a real-life audio-visual model, and generating a question-and-answer state simulated real-life audio-visual based on the simulated real-life image and the simulated real-life voice, include: extracting the emotion feature from the response emotion, and converting the emotion feature into the facial control parameter according to the emotion mapping rule; extracting the audio feature from the simulated real-life voice, and using the facial control parameter and the audio feature as input conditions for the real-life audio-visual model, thereby rendering facial expressions and mouth shapes for the simulated real-life image; generating gestures or actions using the real-life audio-visual model based on the response emotion and the simulated real-life voice; and splicing the facial expressions and mouth shapes with the gestures or actions to generate the simulated real-life image, and then integrating it with the simulated real-life voice to generate the question-and-answer state simulated real-life audio-visual.

10. The real-time dialogue simulation method as described in claim 6 further includes: Based on the image analysis results, a simulated real-life image of the waiting state is generated, and the simulated real-life image of the waiting state and the simulated real-life audio and video of the question-and-answer state are provided in the audio and video stream.

11. A computer program product, downloaded to a computer to perform the real-time conversational simulated human method as described in any one of claims 6 to 10.

Images