Emotion understanding and feedback method for full-duplex interactive question and answer digital human

Abstract

The invention discloses an emotional understanding and feedback method for a full-duplex interactive question-answering digital person, and the method comprises the steps: S1, arousing interaction: enabling a digital person of a digital person system to enter an interaction state in a human body detection arousing or voice instruction arousing manner by a user; s2, state analysis: the digital human system judges the state of the user in real time by collecting image data and audio data of the user, and identifies and analyzes the direction, emotion and speech content of the user in real time; and S3, feedback output: the digital human system drives the digital human in voice, expression and body movement through a screen in real time according to the analysis result of the direction, emotion and speech content of the user, and feeds back the result to the user in a manner of adding auxiliary characters, images and videos. The method has the advantages that the emotional state of the user is accurately judged through an advanced algorithm, digital human expressions, body actions and voice feedback are driven in real time according to the emotion and speech content of the user, response is made in time, and a real-person-like real-time interaction scene is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion understanding and feedback method for a full-duplex interactive question-and-answer digital human. Background Art

[0002] Existing technologies may be affected by factors such as environmental noise, unclear speech, and ambiguous text semantics during sentiment analysis, resulting in low sentiment recognition accuracy. Existing digital human feedback is often mechanical, lacking natural expressions, body movements, and personalized responses, and some digital human responses may be delayed, especially when dealing with complex sentiment analysis and multimodal interactions, which cannot meet the requirements of real-time interaction. Summary of the invention

[0003] The purpose of the present invention is to provide a full-duplex interactive question-answering digital human emotion understanding and feedback method, thereby solving the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A full-duplex interactive question-answering digital human emotion understanding and feedback method, comprising the following steps:

[0006] S1. Invoke interaction: The user uses human body detection or voice command to make the digital human of the digital human system enter an interactive state;

[0007] S2. State analysis: The digital human system collects the user's image data and audio data to determine the user's state in real time, and conducts real-time recognition and analysis of the user's direction, emotions, and speech content;

[0008] S3. Feedback output: The digital human system analyzes the user's direction, emotions and speech content, drives the digital human in real time through the screen with voice, expression, and body movements, and provides feedback to the user in the form of auxiliary text, images, and videos.

[0009] Preferably, the human body detection arousal is specifically:

[0010] When a user enters a specific recognition area of ​​the camera of the digital human system, the camera detects the user's presence through image recognition methods, and uses the trained target detection algorithm model to identify the human body in the image and determine whether the human body is in the specific recognition area. When it is detected that the human body is in the specific recognition area, the model outputs a trigger signal, and the digital human system receives the trigger signal, causing the digital human to enter an interactive state.

[0011] Preferably, the voice command evokes specifically:

[0012] The user clearly speaks a specific wake-up word or command sentence into the microphone of the digital human system. After receiving the audio signal, the microphone transmits it to the speech recognition system. The speech recognition system first processes the audio signal to remove noise and interference, and then converts the audio signal into text format. After that, the converted text format is matched with the preset wake-up command. When the match is successful, a trigger signal is sent to the digital human system to put the digital human into an interactive state.

[0013] Preferably, step S2 specifically includes the following contents:

[0014] S21, the camera of the Digital Human System continuously collects the image data of the user and pre-processes the image;

[0015] S22, the digital human system uses a human posture estimation algorithm to determine the orientation of the user's body and head in the image to determine the user's direction;

[0016] S23, the microphone of the digital human system collects the audio data of the user, and converts the audio signal into text using a speech recognition method;

[0017] S24, the digital human system uses a sentiment analysis algorithm to analyze the converted text to determine the user's emotional state;

[0018] S25. The digital human system uses a pre-trained natural language processing model to perform semantic understanding on the converted text to determine the content of the user's speech.

[0019] Preferably, in step S24, the sentiment analysis algorithm is specifically:

[0020] The sentiment analysis model based on machine learning or deep learning is trained through a large amount of text data annotated with sentiment tags, and the trained sentiment analysis model is used to identify the emotional tendencies in the text.

[0021] Preferably, in step S3, the digital human gives corresponding facial expressions and body movements according to the emotions expressed by the user; the facial expressions and body movement animations of the digital human are realized through 3D modeling and animation production, and when the digital human is running, the corresponding animation clips are selected for playback according to the input emotions and situation parameters to realize the dynamic expression of the digital human.

[0022] Preferably, in step S3, when the user asks a question, the digital human will answer in voice and display auxiliary text, images or videos on the screen;

[0023] The digital human's voice is synthesized through a speech synthesis method. The speech synthesis system converts text into a natural-flowing speech signal and adapts to different scenarios and user needs by selecting different voice styles and timbres.

[0024] Preferably, in step S3, a large amount of text and / or images and / or videos related to the user's questions and scenarios are stored in the database, and based on the user's questions and scenarios, keyword matching or voice search is used to search the database for text and / or images and / or videos related to the current user's questions and scenarios, and display them to the customer on the screen.

[0025] Preferably, after step S3, the method further includes:

[0026] S4, real-time response: After receiving the analysis results fed back by the digital human system, the user can interrupt the digital human's output through specific voice commands or gesture commands. When the digital human system detects these interruption signals, it will immediately stop the digital human's current output and re-enter step S2 to start a new round of interaction cycle.

[0027] Preferably, step S4 specifically comprises:

[0028] During the digital human's feedback process, the digital human system continuously collects the user's image and audio data. The camera constantly monitors the user's position, direction and posture changes, and the microphone receives the user's voice signal in real time. If the user issues a new instruction, changes his mood or direction, the digital human system will immediately interrupt the current feedback process and re-analyze and respond.

[0029] The beneficial effects of the present invention are: 1. The present invention accurately determines the user status by real-time acquisition of camera and audio data, and provides feedback with natural voice, vivid expressions and appropriate body movements combined with auxiliary text, images, videos and other multimodal information, bringing users a natural and smooth experience as if communicating with a real person. 2. With advanced sentiment analysis algorithms and deep learning models, the present invention can accurately identify user emotions and give personalized feedback to establish a deep emotional connection; efficient algorithms and data capabilities ensure rapid detection of user status changes and timely response, and users can re-interact at any time without obvious delays. 3. The present invention improves the performance of digital humans through continuously optimized algorithms and models, promotes the advancement of related technologies, and brings new possibilities for digital human applications. 4. The present invention performs well in user experience, emotional communication, interactive real-time and technological innovation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of a method in an embodiment of the present invention;

[0031] Figure 2 4 is a flow chart of state analysis in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] like Figure 1 As shown, in this embodiment, a full-duplex interactive question-answering digital human emotion understanding and feedback method is provided, including the following parts:

[0034] 1. Arouse interaction

[0035] The user can make the digital human of the digital human system enter an interactive state by awakening it through human body detection or voice command.

[0036] (1) Human body detection arousal

[0037] When a user enters a specific recognition area of ​​the camera of the digital human system, the camera detects the user's presence through image recognition technology. Using the target detection algorithm model in computer vision technology, the trained model can recognize the human body in the image and determine whether it is within the specific recognition area. When the user is detected to enter the recognition area, the model will output a trigger signal to start the digital human interaction.

[0038] (2) Voice command wakeup

[0039] The user can clearly speak a specific wake-up word or command sentence to the microphone of the digital human system. After the microphone receives the sound signal, it transmits it to the speech recognition system. The speech recognition system first pre-processes the sound to remove noise and interference, and then converts the audio signal into text format. Then, the system will match the converted text with the preset wake-up command. If the match is successful, a signal will be sent to the digital human system to make it enter an interactive state.

[0040] 2. Status Analysis

[0041] The digital human system collects the user's image and audio data to determine the user's status in real time, and conducts real-time recognition and analysis of the user's direction, emotions, and speech content. Figure 2 As shown, it specifically includes the following contents:

[0042] 2.1. The camera of the Digital Human System continuously collects the user's image data and pre-processes the image through image pre-processing technology, such as denoising and contrast enhancement, to improve image quality.

[0043] 2.2 The digital human system uses the human posture estimation algorithm in computer vision to determine the orientation of the user's body and head in the image to determine the user's direction. For example, the user's facial orientation can be inferred by detecting key feature points on the user's face, such as eyes, nose, mouth, etc., to determine the user's direction.

[0044] 2.3. The microphone of the Digital Human system collects the user's audio data and converts the audio signal into text using speech recognition technology.

[0045] 2.4. The digital human system uses sentiment analysis algorithms to analyze the converted text to determine the user's emotional state. Sentiment analysis algorithms are usually based on machine learning or deep learning technology and are trained using a large amount of text data annotated with sentiment labels. The trained model can identify the emotional tendencies in the text, such as happiness, sadness, anger, etc., and thus determine the user's emotional state.

[0046] 2.5. The Digital Human system uses pre-trained natural language processing technology to perform semantic understanding on the converted text to determine the content of the user's speech. Semantic understanding specifically includes steps such as part-of-speech tagging, named body recognition, and syntactic analysis to determine the meaning and structure of the text. For example, it can identify different types of speech content such as questions, requests, and statements from users.

[0047] 3. Feedback Output

[0048] The results of the user's direction, emotions and speech content analysis are used to drive the digital human in real time through the screen with voice, expressions, and body movements, and are fed back to the user in the form of auxiliary text, images, and videos.

[0049] (1) If the user shows a happy emotion, the digital human's expression will become more cheerful, such as smiling, with bright eyes, etc. The digital human's expression and body movement animation are realized through 3D modeling and animation production technology. At runtime, according to the input emotion and situation parameters, the corresponding animation clips are selected for playback to realize the dynamic performance of the digital human, making the digital human look more vivid and lifelike.

[0050] (2) If the user asks a question, the digital human will answer it with voice and display auxiliary text, images or videos on the screen. For example, if the user asks about information about a tourist attraction, the digital human can display pictures, video introductions or related text descriptions of the attraction on the screen while answering the question.

[0051] (3) The voice of the digital human is generated through speech synthesis technology. The speech synthesis system converts text into natural and fluent speech signals. Different voice styles and timbres can be selected to adapt to different scenarios and user needs.

[0052] (4) Auxiliary text, images and videos are retrieved and displayed from the database based on the user's questions and context. The database stores a large amount of relevant information. Through technologies such as keyword matching or semantic search, content related to the user's question can be quickly found and displayed to the user on the screen.

[0053] 4. Real-time response

[0054] After receiving the analysis results fed back by the digital human system, the user can interrupt the digital human's output through specific voice commands or actions. When these interruption signals are detected, the digital human's current output will be stopped immediately, and the second step will be re-entered to start a new round of interaction cycle.

[0055] (1) During the digital human’s feedback process, the system continuously collects camera and audio data. The camera constantly monitors the user’s position, direction, and posture changes, and the microphone receives the user’s voice signals in real time. If the user issues a new instruction, changes his or her mood or direction, the system will immediately interrupt the current feedback process and re-analyze and respond.

[0056] (2) The user can interrupt the output of the digital human through specific voice commands or gesture commands. For example, the user can say keywords such as "stop" or "pause". Or make specific gestures such as waving or shaking the head. When the system detects these interruption signals, it will immediately stop the current output of the digital human and re-enter the second step to start a new round of interaction cycle.

[0057] The recognition of voice commands can use speech recognition technology to convert the user's voice signal into text and then match it with the preset interruption keywords. The recognition of gesture commands can use computer vision technology to collect the user's image data through the camera and detect the user's gesture movements.

[0058] 5. Circular Interaction

[0059] Through the above four-step cyclic interaction, a real-time interaction scenario similar to that of real people can be achieved.

[0060] Repeat steps 1 to 4 continuously to ensure that the interaction between the user and the digital human can continue. During the interaction, the performance of the digital human will become closer and closer to that of a real person. By continuously optimizing algorithms and models, the accuracy and expressiveness of the digital human can be improved. For example, by collecting more user data, the sentiment analysis model and the semantic understanding model can be further trained to improve their recognition accuracy of user emotions and speech content. At the same time, the animation effect and speech synthesis quality of the digital human can be continuously improved to make it look more natural and realistic and speak more fluently and vividly.

[0061] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0062] The present invention provides a full-duplex interactive question-and-answer digital human emotion understanding and feedback method. The present invention collects camera and audio data in real time to accurately judge the user's state, and uses natural voice, vivid expressions and appropriate body movements combined with auxiliary text, images, videos and other multimodal information for feedback, bringing users a natural and smooth experience as if communicating with a real person. With advanced emotion analysis algorithms and deep learning models, the present invention can accurately identify user emotions and give personalized feedback, and establish a deep emotional connection; efficient algorithms and data capabilities ensure rapid detection of user status changes and timely response, and users can also re-interact at any time without obvious delay. The present invention improves the performance of digital humans through continuously optimized algorithms and models, promotes the advancement of related technologies, and brings new possibilities for digital human applications. The present invention performs well in user experience, emotional communication, interactive real-time and technological innovation.

[0063] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A full-duplex interactive question-answering digital human emotion understanding and feedback method, characterized by: The following steps are included: S1. Invoke interaction: The user uses human body detection or voice command to make the digital human of the digital human system enter an interactive state; S2. State analysis: The digital human system collects the user's image data and audio data to determine the user's state in real time, and conducts real-time recognition and analysis of the user's direction, emotions, and speech content; S3. Feedback output: The digital human system analyzes the user's direction, emotions and speech content, drives the digital human in real time through the screen with voice, expression, and body movements, and provides feedback to the user in the form of auxiliary text, images, and videos.

2. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: The human body detection arousal is specifically: When a user enters a specific recognition area of ​​the camera of the digital human system, the camera detects the user's presence through image recognition methods, and uses the trained target detection algorithm model to identify the human body in the image and determine whether the human body is in the specific recognition area. When it is detected that the human body is in the specific recognition area, the model outputs a trigger signal, and the digital human system receives the trigger signal, causing the digital human to enter an interactive state.

3. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: The voice command evokes specifically: The user clearly speaks a specific wake-up word or command sentence into the microphone of the digital human system. After receiving the audio signal, the microphone transmits it to the speech recognition system. The speech recognition system first processes the audio signal to remove noise and interference, and then converts the audio signal into text format. After that, the converted text format is matched with the preset wake-up command. When the match is successful, a trigger signal is sent to the digital human system to put the digital human into an interactive state.

4. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: Step S2 specifically includes the following contents: S21, the camera of the Digital Human System continuously collects the image data of the user and pre-processes the image; S22, the digital human system uses a human posture estimation algorithm to determine the orientation of the user's body and head in the image to determine the user's direction; S23, the microphone of the digital human system collects the audio data of the user, and converts the audio signal into text using a speech recognition method; S24, the digital human system uses a sentiment analysis algorithm to analyze the converted text to determine the user's emotional state; S25. The digital human system uses a pre-trained natural language processing model to perform semantic understanding on the converted text to determine the content of the user's speech.

5. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: In step S24, the sentiment analysis algorithm is specifically: The sentiment analysis model based on machine learning or deep learning is trained through a large amount of text data annotated with sentiment tags, and the trained sentiment analysis model is used to identify the emotional tendencies in the text.

6. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: In step S3, the digital human gives corresponding facial expressions and body movements according to the emotions expressed by the user; the facial expressions and body movement animations of the digital human are realized through 3D modeling and animation production. When the digital human is running, the corresponding animation clips are selected for playback according to the input emotions and situation parameters to realize the dynamic performance of the digital human.

7. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: In step S3, when the user asks a question, the digital human will answer it in voice and display auxiliary text, images or videos on the screen; The digital human's voice is synthesized through a speech synthesis method. The speech synthesis system converts text into a natural-flowing speech signal and adapts to different scenarios and user needs by selecting different voice styles and timbres.

8. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: In step S3, a large amount of text and / or images and / or videos related to the user's questions and scenarios are stored in the database. According to the user's questions and scenarios, keyword matching or voice search is used to search the database for text and / or images and / or videos related to the current user's questions and scenarios, and the text and / or images and / or videos are displayed to the customer on the screen.

9. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 1 is characterized by: Step S3 also includes: S4, real-time response: After receiving the analysis results fed back by the digital human system, the user can interrupt the digital human's output through specific voice commands or gesture commands. When the digital human system detects these interruption signals, it will immediately stop the digital human's current output and re-enter step S2 to start a new round of interaction cycle.

10. The full-duplex interactive question-answering digital human emotion understanding and feedback method according to claim 9 is characterized by: Step S4 specifically includes: During the digital human's feedback process, the digital human system continuously collects the user's image and audio data. The camera constantly monitors the user's position, direction and posture changes, and the microphone receives the user's voice signal in real time. If the user issues a new instruction, changes his mood or direction, the digital human system will immediately interrupt the current feedback process and re-analyze and respond.

Images