Virtual digital human interactive question and answer method, system and equipment and storage medium

Abstract

The invention discloses a virtual digital human interactive question and answer method, system and device and a storage medium, and relates to the field of virtual reality. The method comprises the following steps: carrying out first processing on dialogue content of a user, and carrying out second processing on a facial image of the user to obtain a first emotional state; performing third processing on the text after the first processing to obtain a second emotional state of the user, and determining a target emotional state in combination with the first emotional state and the second emotional state; performing fourth processing on the text after the first processing based on a deep learning model to generate a first answer, and adjusting an expression mode of the first answer according to the target emotional state and the dialogue context to generate a second answer; and constructing a virtual digital person by using a virtual reality technology, and determining the facial expression, the body movement and the voice intonation of the virtual digital person according to the emotional tendency in the second answer to display the second answer. By implementing the technical scheme provided by the invention, the emotional state of the user can be identified more accurately, and then the answer more conforming to the emotion of the user is generated.

Description

Technical Field

[0001] The present application relates to the technical field of virtual reality, and in particular to a method, system, device and storage medium for interactive question and answer of a virtual digital human. Background Art

[0002] With the development of science and technology, human-computer interaction technology is changing with each passing day. Traditional question-answering systems mainly communicate with users through text or voice. This monotonous communication method can no longer meet users' needs for rich interactive experience. In recent years, virtual reality technology has emerged, and building virtual digital humans for interaction has become a new research direction.

[0003] At present, there are some interactive question-and-answer systems based on virtual reality on the market, but these systems have made certain innovations in visual presentation, and there are still deficiencies in user emotion recognition and feedback, and they cannot provide personalized and humanized interactive experience.

[0004] Therefore, how to incorporate more accurate user emotion recognition into the virtual reality interactive question-answering system, and how to make virtual digital humans express answers more naturally, have become issues that need further research and improvement. Summary of the invention

[0005] The present application provides a method, system, device and storage medium for interactive question and answer of a virtual digital human, which can more accurately identify the emotional state of a user, thereby generating an answer that is more in line with the user's emotions, and enable the virtual digital human to express the answer more naturally.

[0006] In a first aspect of the present application, a method for interactive question and answer of a virtual digital human is provided, which is applied to an interactive question and answer platform, and the method comprises: Collecting a user's facial image and conversation content, performing a first process on the conversation content using a natural language processing technology, and performing a second process on the facial image using a preset sentiment analysis model to obtain a first sentiment state, wherein the first process includes word segmentation and semantic role labeling, and the second process includes facial expression feature extraction and sentiment recognition; Performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user, and determining a target sentiment state by combining the first sentiment state and the second sentiment state; Performing a fourth processing on the first processed text based on the deep learning model to generate a first answer, and adjusting the expression of the first answer according to the target emotional state and the conversation context to generate a second answer; A virtual digital person is constructed using virtual reality technology, and facial expressions, body movements, and voice intonation of the virtual digital person are determined according to the emotional tendency in the second answer to display the second answer. The step of performing a second processing on the facial image using a preset emotion analysis model to obtain a first emotion state includes: Extracting feature points from the facial image, matching the feature points with preset expression categories, converting the matched expression categories into corresponding first emotional states, and generating corresponding first emotional values ​​according to the first emotional states, wherein the first emotional states include positive emotions, negative emotions, and neutral emotions, The performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user comprises: Extracting preset features related to sentiment analysis from the text after the first processing, wherein the preset features include sentiment words, negative words, and degree words; Inputting the preset features into a preset sentiment analysis model to obtain an output result; According to the output result, the emotional tendency of the text is mapped to a preset emotional category, a second emotional state is determined according to the mapped emotional category, and a corresponding second emotional value is generated according to the second emotional state, wherein the second emotional state includes positive emotion, negative emotion and neutral emotion.

[0007] By adopting the above technical solution, collecting the user's facial image and conversation content, and combining natural language processing and sentiment analysis technology, the system can more accurately understand the user's intention and emotional state. This multi-dimensional information input makes the interaction process closer to natural communication between humans, enhancing the realism and immersion of the user experience. By combining the first emotional state (based on facial images) and the second emotional state (based on text content) to determine the target emotional state, the system can more comprehensively grasp the user's emotional changes and adjust the expression of the answer accordingly. This personalized feedback mechanism helps to establish a deeper emotional connection and enhance the user's trust and satisfaction with the system. The answers generated based on the deep learning model are already highly accurate and targeted, but the expression method is further adjusted according to the target emotional state and the context of the conversation, so that the answers are more in line with the current situation and user needs. This dynamic adjustment capability enables the system to cope with more complex interactive scenarios and provide more effective help and suggestions. The virtual digital human built using virtual reality technology not only has vivid facial expressions and body movements, but also can adjust the voice tone according to the emotional tendency of the answer. This multimodal expression method makes the virtual digital human more lifelike and enhances the emotional communication and interactive experience with the user. In the process of interacting with virtual digital people, users can feel more real and natural emotional feedback. By accurately extracting feature points (such as the position and shape changes of key parts such as eyes, eyebrows, and mouth) from facial images, the system can capture subtle changes in facial expressions. These feature points, as the key basis for emotion recognition, help improve the accuracy and reliability of emotion recognition. The rapid matching process of feature points with preset expression categories enables the system to capture and identify the user's emotional state in real time. This instant feedback mechanism helps the system quickly adjust the interaction strategy to better fit the user's current emotional state. Converting the matched expression category into the corresponding first emotional state (positive emotion, negative emotion, and neutral emotion) not only realizes the basic classification of emotions, but also provides users with more detailed emotional feedback. This detailed emotional classification helps the system better understand the user's emotional changes, thereby providing more intimate and personalized services. The corresponding first emotional value is generated according to the first emotional state, so that the emotional state can be quantified. This quantification process not only facilitates subsequent data analysis and processing, but also provides an important reference for the system to adjust the answer expression method and select appropriate interaction strategies. Accurately identifying the user's emotional state and adjusting the interaction method according to the emotional state helps promote the naturalness and fluency of human-computer interaction. Users can feel the system's sensitivity and attention to their emotional changes, thereby enhancing their trust and satisfaction with the system. By adopting the above technical solution and extracting preset features closely related to sentiment analysis (such as sentiment words, negative words, and degree words), the system can focus on the key parts of the text that express emotions, reduce the interference of irrelevant information, and thus improve the accuracy of sentiment analysis.The selection of preset features not only considers the vocabulary (emotional words) that directly express emotions, but also covers negative words and degree words that affect the intensity of emotions. This comprehensive feature extraction method enables the system to more accurately grasp the overall emotional tendency of the text and avoid the one-sidedness that may be caused by a single feature. The emotional tendency of the text is mapped to the preset emotional categories (positive emotions, negative emotions, and neutral emotions), and the corresponding second emotional values ​​are generated according to the mapping results, so that the emotional state can be quantified. This quantitative evaluation method not only facilitates subsequent data processing and analysis, but also provides strong support for the system to adjust the interaction strategy and optimize the user experience. The entire process from feature extraction to emotional analysis to the mapping of emotional categories and the generation of values ​​is automatically completed based on the preset emotional analysis model. This highly automated processing method greatly improves the real-time performance of emotional analysis and enables the system to quickly respond to the emotional changes of users. Accurately identifying the emotional tendency in the text and adjusting the interaction method according to the emotional state will help improve the intelligent level of human-computer interaction. The system can more sensitively capture the emotional needs of users and provide more intimate and personalized services, thereby enhancing user satisfaction and loyalty.

[0008] Optionally, the determining a target emotional state by combining the first emotional state and the second emotional state includes: When the first emotional state and the second emotional state are consistent, determining the first emotional state as a target emotional state; When the first emotional state and the second emotional state are inconsistent, the first emotional value and the second emotional value are weightedly summed to obtain a third emotional value, and the third emotional state corresponding to the third emotional value is determined through a preset value mapping table, and the third emotional state is determined as the target emotional state.

[0009] By adopting the above technical solution, when the first emotional state (based on facial expression) and the second emotional state (based on text content) are consistent, the emotional state is directly adopted as the target emotional state, which simplifies the decision-making process and ensures the accuracy of emotion recognition. When the two are inconsistent, the contribution of the two emotional states is comprehensively considered by weighted summation, so that the system can more comprehensively evaluate the user's emotional tendency, thereby improving the robustness of emotion recognition. In the case of inconsistency, the third emotional value is obtained by weighted summation, and the third emotional state is determined as the target emotional state with the help of a preset numerical mapping table. This method not only considers the reliability differences of different emotional sources (for example, facial expressions and text content may have different weights when expressing emotions), but also realizes the precise quantification of emotional states through a numerical mapping table, further enhancing the accuracy of emotion recognition. Comprehensively considering facial expressions and text content to determine the target emotional state enables the system to more accurately grasp the user's real emotional needs. This emotion recognition method based on multi-source information helps the system provide more intimate and personalized services, thereby optimizing the user experience. For example, in a chatbot or customer service system, adjusting the tone and wording of the reply according to the user's emotional state can improve user satisfaction and trust. By combining multiple emotion recognition technologies (such as facial expression recognition and text sentiment analysis) and adopting strategies such as weighted summation to determine the target emotional state, the system can handle complex emotional interaction scenarios more intelligently. This intelligent processing method not only improves the system's response speed and accuracy, but also enables the system to better adapt to the emotional expression and habits of different users.

[0010] Optionally, the determining a target emotional state by combining the first emotional state and the second emotional state includes: Use time series analysis methods to analyze the changing trend of users' emotional states over time to predict future emotional states; The third emotional state is adjusted according to the future emotional state, and the adjusted third emotional state is determined as the target emotional state.

[0011] By adopting the above technical solution, the system can capture the changing trend of the user's emotional state over time and predict the future emotional state through the time series analysis method. This forward-looking ability enables the system to prepare and adjust the interaction strategy in advance to better cope with the possible emotional changes of the user, thereby providing more intimate and timely services. The user's emotional state changes dynamically, while traditional emotion recognition methods can often only process static emotional data. By introducing time series analysis, the system can track the changes in the user's emotional state in real time and dynamically adjust the target emotional state according to the changing trend. This dynamic adaptability enables the system to respond to complex emotional interaction scenarios more flexibly and improve the accuracy and effectiveness of emotion recognition. Adjusting the target emotional state based on the prediction of future emotional states helps the system provide an interactive experience that is more in line with user expectations. For example, when it is predicted that the user may have negative emotions, the system can adjust the tone and wording of the reply in advance to alleviate the user's negative emotions; when it is predicted that the user may have positive emotions, the system can further enhance the fun and affinity of the interaction to deepen the user's positive experience. The introduction of time series analysis methods and emotion prediction marks that the system's intelligence level in emotion recognition has been further improved. By learning and understanding the changing patterns of the user's emotional state, the system can gradually build up an emotional model of the user and provide more intelligent and personalized services accordingly. This improvement in the level of intelligence not only improves the response speed and accuracy of the system, but also enables the system to better meet the emotional needs of users. The embodiments of this application demonstrate the latest application results of time series analysis and emotion prediction in the field of emotion computing. By continuously exploring and optimizing emotion recognition technologies and strategies, the development of the field of emotion computing can be promoted, providing powerful emotion analysis capability support for more intelligent applications. This technological innovation not only helps to improve user experience and satisfaction, but also brings more commercial value and social benefits to enterprises and society.

[0012] Optionally, adjusting the expression of the first answer according to the target emotional state and the conversation context to generate the second answer includes: Traversing the preset rule library, matching the first target rule according to the current target emotional state and the conversation context, wherein the conversation context includes the user's previous questions, the system's answers, and background information, and the first target rule includes adding preset positive words when the target emotional state is positive and the conversation context does not contain preset negative words, or adding preset comforting words when the target emotional state is negative and the conversation context contains preset negative words; adjusting the expression of the first answer according to the first target rule; When multiple first target rules are matched, the second target rule to be finally executed is selected according to the priorities of the multiple first target rules or a preset conflict resolution mechanism.

[0013] By adopting the above technical solution, the expression of the answer is dynamically adjusted according to the user's current emotional state and the conversation context, so that the answer is closer to the user's emotional needs. For example, adding preset positive words in a positive emotional state can enhance the user's pleasure; adding preset comforting words in a negative emotional state can alleviate the user's negative emotions. This personalized answer method helps to improve user satisfaction and loyalty. Through the analysis of the conversation context, the system can more accurately understand the user's intentions and emotional state, thereby generating more natural and fluent answers. This context-based understanding ability makes the conversation process more coherent and reduces the occurrence of misunderstandings and ambiguities. The rules in the preset rule library are summarized based on a large amount of data and experience, and have high accuracy and pertinence. By matching the target rules and adjusting the answer expression, the system can generate answers that are more in line with user expectations and needs. When multiple first target rules are matched, the system can select the second target rule to be finally executed according to the priority of the rules or the preset conflict resolution mechanism. This flexible conflict resolution mechanism ensures that the system can make reasonable decisions when facing complex situations and avoid conflicts and contradictions between rules.

[0014] Optionally, determining the facial expression, body movement and voice intonation of the virtual digital person according to the emotional tendency in the second answer to display the second answer includes: When the emotional tendency in the second answer is a positive emotion, the facial expression of the virtual digital person is set to smile, blink or raise eyebrows, the body movement of the virtual digital person is set to wave, jump or applaud, and the voice tone of the virtual digital person is set to a speed greater than a first speed threshold, a volume greater than a first volume threshold and an upward tone; When the emotional tendency in the second answer is a negative emotion, the facial expression of the virtual digital person is set to frown, lower the head or pout, the body movement of the virtual digital person is set to lower the head, cross the arms or shake the head slowly, and the voice tone of the virtual digital person is set to a speaking speed less than a second speaking speed threshold, a volume less than a second volume threshold, and a descending tone, the second speaking speed threshold is less than the first speaking speed threshold, and the second volume threshold is less than the first volume threshold.

[0015] By adopting the above technical solution, the facial expressions, body movements and voice intonation of the virtual digital person are closely associated with the emotional tendency in the second answer, which can ensure the accuracy of emotional expression. Whether it is positive or negative emotion, it can be accurately conveyed through corresponding non-verbal signals, so that users can more intuitively feel the emotional color behind the answer. The emotional expression makes the interaction of the virtual digital person more vivid and interesting, which can attract the attention of users and enhance their sense of participation. When the virtual digital person presents a positive answer with positive gestures such as smiling and waving, it can enhance the user's pleasure and satisfaction; when the negative answer is presented with negative gestures such as frowning and bowing, it can resonate with and pay attention to the user, thereby deepening their understanding and memory of the answer. Through the refined design of facial expressions, body movements and voice intonation, the virtual digital person can present a more realistic emotional response, making the user feel as if he is in a real dialogue scene. This sense of immersion and substitution helps to enhance the user's trust and dependence on the virtual digital person, thereby promoting deeper interaction and cooperation. The embodiment of the present application is not limited to the expression of positive and negative emotional tendencies, but can also be expanded and customized according to specific needs. For example, more combinations of facial expressions, body movements, and voice intonation can be set to express different emotional levels and nuances, thereby meeting the needs of richer and more complex emotional expression.

[0016] In a second aspect of the present application, a virtual digital human interactive question-answering system is provided, comprising a collection module, an emotion module, an answer module and a virtual module, wherein: A collection module configured to collect a user's facial image and conversation content, perform a first process on the conversation content using a natural language processing technology, and perform a second process on the facial image using a preset sentiment analysis model to obtain a first sentiment state, wherein the first process includes word segmentation and semantic role labeling, and the second process includes facial expression feature extraction and sentiment recognition; an emotion module, configured to perform a third processing on the first processed text using a preset emotion analysis model to obtain a second emotion state of the user, and determine a target emotion state by combining the first emotion state and the second emotion state; an answer module, configured to perform a fourth processing on the first processed text based on a deep learning model to generate a first answer, and adjust an expression of the first answer according to the target emotional state and the conversation context to generate a second answer; a virtual module configured to construct a virtual digital person using virtual reality technology, and determine the facial expression, body movement and voice intonation of the virtual digital person according to the emotional tendency in the second answer to display the second answer, The step of performing a second processing on the facial image using a preset emotion analysis model to obtain a first emotion state includes: Extracting feature points from the facial image, matching the feature points with preset expression categories, converting the matched expression categories into corresponding first emotional states, and generating corresponding first emotional values ​​according to the first emotional states, wherein the first emotional states include positive emotions, negative emotions, and neutral emotions, The performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user comprises: Extracting preset features related to sentiment analysis from the text after the first processing, wherein the preset features include sentiment words, negative words, and degree words; Inputting the preset features into a preset sentiment analysis model to obtain an output result; According to the output result, the emotional tendency of the text is mapped to a preset emotional category, a second emotional state is determined according to the mapped emotional category, and a corresponding second emotional value is generated according to the second emotional state, wherein the second emotional state includes positive emotion, negative emotion and neutral emotion.

[0017] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0018] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0019] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By collecting the user's facial image and conversation content and performing sentiment analysis respectively, the user's emotional state can be understood more accurately. This emotion recognition method based on multimodal information enables the system to more naturally integrate into the user's emotional communication and improve the naturalness and fluency of the interaction; 2. Combining emotion analysis of facial images and emotion analysis of text content, the system can capture the user's emotional tendencies from multiple dimensions, thereby more accurately determining the target emotional state. This multimodal emotion recognition technology significantly improves the accuracy and reliability of emotion recognition; 3. The first answer generated by the deep learning model is adjusted according to the target emotional state and the conversation context, so that the expression of the answer is more in line with the user's emotional needs and context. This personalized answer generation method can improve user satisfaction and trust; 4. The virtual digital human built using virtual reality technology can automatically adjust facial expressions, body movements, and voice intonation according to the emotional tendency in the second answer. This emotional expression method enables the virtual digital human to convey information more vividly and vividly, enhancing the user's sense of immersion and involvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of the method for interactive question and answer of a virtual digital human disclosed in an embodiment of the present application; Figure 2 It is a module diagram of a system for interactive question and answer of a virtual digital human disclosed in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0021] Explanation of the accompanying drawings: 201, acquisition module; 202, emotion module; 203, answer module; 204, virtual module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0022] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0023] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0024] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0025] This embodiment discloses a method for interactive question and answer of a virtual digital human, which is applied to an interactive question and answer platform. Figure 1 is a flow chart of the method for interactive question and answer of a virtual digital human disclosed in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: S110, collecting a facial image and conversation content of a user, performing a first processing on the conversation content by using a natural language processing technology, and performing a second processing on the facial image by using a preset sentiment analysis model to obtain a first sentiment state, wherein the first processing includes word segmentation and semantic role labeling, and the second processing includes facial expression feature extraction and sentiment recognition; Use a camera or other image capture device to obtain a user's facial image in real time or in advance. These images contain rich non-verbal information, such as facial expressions and eye contact, which are crucial for sentiment analysis. Capture the user's voice through an audio input device such as a microphone and convert it into text form. This usually involves speech recognition technology to ensure accurate acquisition of the conversation content.

[0026] Natural language processing of the conversation content (i.e., the first processing): (1) Split the conversation text into individual words or phrases. This is the basic step of natural language processing and helps with subsequent more complex analysis. For example, in English, the sentence "I am happy" will be segmented into "I", "am", and "happy"; (2) Semantic role labeling (SRL): Based on the segmentation, further analyze the semantic relationship between the various components in the sentence. Semantic role labeling can identify the arguments of the verb, such as the agent (the initiator of the action) and the patient (the object of the action), so as to more accurately understand the deeper meaning of the sentence. For example, in the sentence "I saw the cat with binoculars", SRL can identify that "I" is the agent of "saw", "the cat" is the patient, and "with binoculars" is an instrumental adverbial.

[0027] Emotion analysis of facial images (i.e., the second process): (1) Facial expression feature extraction: Using computer vision technology to extract key features from facial images, such as the shape and position of eyes, eyebrows, and mouth, these features are crucial for identifying different emotional states (such as happiness, sadness, anger, etc.); (2) Emotion recognition: Compare the extracted facial expression features with the patterns in the preset emotion analysis model to determine the user's current emotional state. Emotion analysis models are usually trained based on a large number of annotated data sets and can identify multiple emotion categories and their corresponding expression features.

[0028] Combining the natural language processing results of the conversation content with the sentiment analysis results of the facial images can provide users with a more comprehensive and in-depth understanding. For example, in a customer service system, this can help customer service staff better understand the user's emotions and needs, thereby providing more considerate and effective services. In human-computer interaction scenarios, this multimodal sentiment analysis technology can also be used to enhance the emotional intelligence of robots, making them more natural and smooth in their interactions with users.

[0029] The performing a second processing on the facial image using a preset emotion analysis model to obtain a first emotion state comprises: Feature points are extracted from the facial image, the feature points are matched with preset expression categories, the matched expression categories are converted into corresponding first emotional states, and a corresponding first emotional value is generated according to the first emotional state, wherein the first emotional state includes positive emotion, negative emotion and neutral emotion.

[0030] Facial recognition technology is used to extract key feature points from captured facial images, such as the outline, position, and shape of the eyes, eyebrows, and mouth. The extracted feature points are encoded into numerical vectors, which can describe the relative position and shape changes of various parts of the face and are the basis for identifying different expressions. The interactive question-answering platform has a built-in sentiment analysis model trained with a large amount of labeled data. The model associates different facial feature vectors with preset expression categories (such as happiness, sadness, anger, surprise, disgust, fear, and neutrality). These expression categories are further summarized into three emotional states: positive emotions (such as happiness), negative emotions (such as sadness, anger, disgust, fear), and neutral emotions. The encoded facial feature vectors are input into the sentiment analysis model, and the model finds the most matching expression category through comparison and calculation. Then, the matched expression category is converted into the corresponding first emotional state (positive, negative, or neutral). In order to further quantify the emotional state, the interactive question-answering platform can assign an emotional numerical range to each emotional state. For example, positive emotions can be assigned a value from 1.0 to 0.5 (1.0 means very positive, 0.5 means slightly positive but close to neutral), negative emotions are -0.5 to -1.0 (-0.5 means slightly negative but close to neutral, -1.0 means very negative), and neutral emotions are 0.0. Based on the matched emotional state, the system generates the corresponding emotional value. For example, if the user's facial image is identified as "happy", the emotional state is positive, and the platform assigns it an emotional value of 0.8.

[0031] By extracting fine feature points (such as the position and shape changes of the corners of the eyes and mouth) from facial images and matching them with a preset sentiment analysis model trained with a large amount of data, the accuracy of emotion recognition can be significantly improved. This method is more reliable than methods that rely solely on the overall facial contour or a single feature point. The recognized expression category is converted into the corresponding emotional state (positive, negative, neutral), and the emotional value is further generated, so that the emotional state can be quantified. This quantitative representation is not only convenient for computer processing, but also convenient for humans to understand and compare the emotional changes of different users or the same user at different time points. In human-computer interaction scenarios, accurately identifying the user's emotional state and adjusting the interaction method accordingly (such as adjusting the voice tone, changing the reply content, etc.) can enhance the user experience and make users feel more intimate and personalized services. For example, in online education platforms, adjusting teaching strategies according to students' emotional states can stimulate students' interest and enthusiasm in learning. In some decision-making scenarios that require sentiment analysis (such as market research, customer feedback analysis, etc.), it can help decision makers more accurately understand the emotional tendencies of the target group, so as to formulate more reasonable and effective decision-making plans.

[0032] S120, performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user, and determining a target sentiment state by combining the first sentiment state and the second sentiment state; Identify sentiment-related words from the text, which may include adjectives, adverbs, emotional verbs, etc., which play a key role in expressing sentiment. Based on the preset sentiment dictionary or sentiment analysis model, determine the polarity (positive, negative, or neutral) of each sentiment word. Sentiment dictionaries usually contain a large number of words and their corresponding sentiment polarity annotations. In addition to polarity, it is also necessary to evaluate the strength of sentiment words, that is, how strongly they express sentiment. This can be achieved through the weight or score of sentiment words. Based on the polarity and strength of all sentiment words in the text, an aggregation algorithm (such as weighted average) is used to determine the overall sentiment state of the entire text, that is, the second sentiment state. Ensure that the first sentiment state (based on facial images) and the second sentiment state (based on text content) are consistent in representation, such as using three categories of positive, negative, and neutral. Assign different weights to the first and second sentiment states according to the needs of the application scenario. For example, in some scenarios, facial expressions may more directly reflect the user's immediate emotions, so the first sentiment state should be given a higher weight; while in other scenarios, the text content may contain more contextual information, making the second sentiment state more important. According to the assigned weights, the first emotional state and the second emotional state are weighted averaged or otherwise comprehensively calculated to determine the target emotional state. The target emotional state is a more comprehensive emotional representation that combines both text and image modal information.

[0033] The performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user comprises: Extracting preset features related to sentiment analysis from the text after the first processing, wherein the preset features include sentiment words, negative words, and degree words; Inputting the preset features into a preset sentiment analysis model to obtain an output result; According to the output result, the emotional tendency of the text is mapped to a preset emotional category, a second emotional state is determined according to the mapped emotional category, and a corresponding second emotional value is generated according to the second emotional state, wherein the second emotional state includes positive emotion, negative emotion and neutral emotion.

[0034] Sentiment words are words that directly express emotions in a text, such as "happy" and "sad". These words usually have clear sentiment polarity (positive or negative) and are an important basis for judging the sentiment tendency of a text. Negative words such as "no" and "no" can change the sentiment polarity of sentiment words. For example, "unhappy" and "happy" are opposite in sentiment tendency, and "no" here is a negative word. Degree words such as "very" and "slightly" are used to modify the intensity of sentiment words and affect the degree of sentiment expression in the text. For example, "very happy" is stronger in sentiment intensity than "happy". When extracting these preset features, predefined sentiment dictionaries, negation word dictionaries, and degree word dictionaries are usually used to assist in identification. In addition, machine learning or deep learning methods can also be used to automatically identify these features by training models. The extracted preset features (sentiment words, negation words, degree words, and related information) are used as input and sent to a pre-trained sentiment analysis model. This model may be a rule-based system or a model based on machine learning or deep learning. A series of rules are defined to match the features in the text, and the sentiment tendency of the text is inferred based on these rules. This method is simple and intuitive, but the design and maintenance of rules are relatively complex. A large amount of labeled data is used to train the model so that it can automatically learn the relationship between features and sentiment in the text. This method has good generalization ability when dealing with complex text and unknown vocabulary.

[0035] After the model processes the input features, it will output one or more numerical values ​​or vectors representing the sentiment tendency of the text. These output results need to be mapped according to the preset sentiment categories to determine the second sentiment state of the text. The results of the model output are mapped to the preset sentiment categories, such as positive sentiment, negative sentiment, and neutral sentiment. This is usually achieved by comparing the output results with the sentiment category threshold or performing multi-classification. According to the mapping results, the second sentiment state of the text is determined. For example, if the results of the model output indicate that the text has a positive sentiment tendency, the second sentiment state is positive sentiment. In order to further quantify the sentiment state of the text, an emotional value range can be assigned to each sentiment category. For example, positive sentiment can be assigned a value from 1.0 to 0.5 (1.0 means very positive, 0.5 means slightly positive but close to neutral), negative sentiment is a value from -0.5 to -1.0 (-0.5 means slightly negative but close to neutral, -1.0 means very negative), and neutral sentiment is 0.0. According to the second sentiment state, the corresponding sentiment value is generated.

[0036] By carefully extracting preset features related to sentiment analysis from the text (such as sentiment words, negative words, and degree words), the sentiment information in the text can be captured more comprehensively. These features play a key role in sentiment expression, and their accurate identification helps improve the accuracy of sentiment analysis. The preset sentiment analysis model is usually trained with a large amount of annotated data and can handle various complex text situations. Inputting the preset features into such a model can make full use of the learning results of the model and improve the robustness of sentiment analysis in different text types and contexts. The sentiment tendency of the text is mapped to the preset sentiment categories (positive sentiment, negative sentiment, and neutral sentiment), and the corresponding sentiment values ​​are generated, so that the results of sentiment analysis can be quantitatively represented. This quantitative representation is not only convenient for computer processing, but also convenient for humans to understand and compare the sentiment changes of different texts or the same text at different time points.

[0037] Optionally, the determining a target emotional state by combining the first emotional state and the second emotional state includes: When the first emotional state and the second emotional state are consistent, determining the first emotional state as a target emotional state; When the first emotional state and the second emotional state are inconsistent, the first emotional value and the second emotional value are weightedly summed to obtain a third emotional value, and the third emotional state corresponding to the third emotional value is determined through a preset value mapping table, and the third emotional state is determined as the target emotional state.

[0038] When the first emotional state and the second emotional state are consistent, it means that the emotional tendency analyzed from the facial image and the text content is the same. In this case, the first emotional state (or the second emotional state, because they are the same) can be directly determined as the target emotional state. This is because the information of the two different modalities (image and text) points to the same emotional direction, which increases the credibility of the emotional judgment.

[0039] The situation becomes complicated when the first emotional state and the second emotional state are inconsistent. This may be due to a variety of reasons, such as the user's contradiction in expressing emotions (such as saying one thing and thinking another), or the facial image and text content capture the user's emotional state at different times or in different situations. To deal with this situation, the process adopts the following strategy: First, the first emotional value and the second emotional value are weighted and summed to obtain a comprehensive emotional value (third emotional value). The distribution of weights can be adjusted according to actual conditions, such as setting according to the importance of facial images and text in specific application scenarios; then, the third emotional state corresponding to the third emotional value is determined by a preset value mapping table. This mapping table may be a range partitioning table that maps different emotional value ranges to specific emotional categories (such as positive, negative, neutral or more detailed categories). By looking up this mapping table, the third emotional value can be converted into the corresponding emotional state; finally, the third emotional state is determined as the target emotional state. This state is the result of combining the two modal information of facial images and text, and more comprehensively and accurately reflects the user's emotional tendency.

[0040] When the first and second emotional states are consistent, one of them is directly used as the target emotional state. This strategy is based on the principle of consistency, simplifies the decision-making process, and retains high-accuracy emotional judgment. When the two are inconsistent, the target emotional state is determined by weighted summation and preset numerical mapping table. This method comprehensively considers multiple emotional information sources, reduces the deviation that may be caused by a single information source, and thus improves the accuracy of emotional analysis. Multimodal emotional analysis (combining facial images and text) itself can improve the robustness of the system, because different information sources may have different advantages and reliability in different situations. By integrating these information sources, the system can better cope with complex and changeable emotional expressions. The use of weighted summation and preset numerical mapping table further enhances the robustness of the system, enabling the system to make reasonable judgments when facing inconsistent emotional information. In scenarios where user interaction based on emotional analysis is required (such as intelligent customer service, online education, etc.), accurately judging the user's emotional state is crucial to improving user experience. By combining multiple emotional information sources to determine the target emotional state, the system can more accurately understand the user's emotional needs, thereby providing more intimate and personalized services. This technical process is not only applicable to sentiment analysis of facial images and texts, but can also be extended to the combination of other multimodal emotion information sources (such as speech, physiological signals, etc.). This enables the technology to be applied to a wider range of scenarios, such as affective computing, mental health monitoring, human-computer interaction, etc.

[0041] Optionally, the determining a target emotional state by combining the first emotional state and the second emotional state includes: Use time series analysis methods to analyze the changing trend of users' emotional states over time to predict future emotional states; The third emotional state is adjusted according to the future emotional state, and the adjusted third emotional state is determined as the target emotional state.

[0042] Use time series analysis methods (such as ARIMA model, LSTM network, etc.) to analyze the changing trend of users' emotional state over time in the past period of time. This includes collecting the emotional values ​​of users at each time point in the past period of time (which can be the first emotional value, the second emotional value or the third emotional value, depending on the actual situation and needs) and constructing time series data. Through the time series model, the emotional state of the user at a certain point in the future, that is, the future emotional state, can be predicted. According to the predicted future emotional state, it may be found that it is different from the third emotional state currently obtained through multimodal analysis. At this time, the third emotional state can be fine-tuned according to the predicted future emotional state to better reflect the user's future emotional changes. The adjusted third emotional state is the final target emotional state. It not only takes into account the current multimodal emotional information, but also incorporates the prediction of the change of emotional state over time, so it is more comprehensive and accurate.

[0043] The third emotional state can be adjusted according to the future emotional state as follows: (1) Combining the change trend of the future emotional state with the current third emotional state to form a new emotional state that combines the current state and the future trend. Specifically, first analyze the change trend of the future emotional state (such as rising, falling, stable, etc.), and then fine-tune the third emotional state according to the strength and direction of the trend. For example, if the prediction shows that the future emotional state will rise significantly, a certain positive emotional component can be added to the current third emotional state. (2) Setting thresholds for the emotional state. When the predicted future emotional state exceeds or falls below these thresholds, the adjustment of the third emotional state is triggered. Specifically, different emotional thresholds (such as positive emotional thresholds, negative emotional thresholds, etc.) are set according to the needs of the application scenario. When the predicted future emotional state exceeds or falls below these thresholds, the third emotional state is automatically adjusted to reflect this change. For example, if the predicted future emotional state is below the negative emotional threshold, the third emotional state may need to be adjusted to a more negative state. (3) Considering the user's current scenario and context information, the third emotional state is adaptively adjusted. Specifically, the user's current scene and context are analyzed through multiple data sources such as user behavior data, location information, time information, etc. Then, the third emotional state is adjusted in a targeted manner based on this information and the predicted future emotional state. For example, if the user is currently in a work scene and the future emotional state is predicted to become negative, the third emotional state may need to be adjusted to reduce its impact on work.

[0044] Through the time series analysis method, the system can capture the dynamic trend of the user's emotional state over time, so as to more accurately predict the future emotional state. This prediction is not only based on the current emotional state, but also takes into account the changing rules of historical emotional data, so it has higher accuracy. Traditional sentiment analysis often only focuses on the emotional state at a certain moment or a period of time, while the time series analysis method can cross the time axis and regard the user's emotional state as a continuous changing process. This continuity makes sentiment analysis more comprehensive and in-depth, and helps to discover the laws and patterns of emotional changes. The prediction of future emotional state provides a strong basis for adjusting the third emotional state. By fine-tuning the third emotional state according to the prediction results, the system can ensure that the target emotional state is closer to the user's real emotional state in the future. This optimization not only improves the accuracy of sentiment analysis, but also enhances the adaptability and flexibility of the system. Accurately predicting and adjusting the target emotional state helps to provide users with more personalized and considerate services. For example, on social media platforms, the system can recommend appropriate content or activities based on the user's future emotional state to alleviate negative emotions or enhance positive experiences. This service based on emotion prediction can significantly improve user satisfaction and loyalty.

[0045] S130, performing a fourth processing on the first processed text based on the deep learning model to generate a first answer, and adjusting the expression of the first answer according to the target emotional state and the conversation context to generate a second answer; A deep learning model is a neural network model that has been trained to understand and process natural language text. Such models usually include but are not limited to recurrent neural networks (RNNs), long short-term memory networks (LSTMs), transformers, etc., which can capture semantic information, contextual relationships, etc. in text. The deep learning model is used to further process and analyze the text after the first processing to generate a preliminary answer (i.e., the first answer). This process may include multiple steps such as text understanding, intent recognition, and information extraction, depending on the training objectives and task requirements of the model. Based on the fourth processing result of the deep learning model, the system can generate a preliminary answer. This answer may be a simple reply, an information summary, or a complex sentence, depending on the complexity of the question and the training effect of the model. The conversation context refers to the history of the current conversation, including the user's previous questions, the system's responses, and the interactive relationship between them. The conversation context is crucial for understanding user intent, predicting user behavior, and generating appropriate answers. After obtaining the first answer, the system will adjust the way the answer is expressed according to the target emotional state and the conversation context. This adjustment may include changing the tone, adding or removing emotional words, adjusting the sentence structure, etc., in order to make the answer more in line with the user's emotional needs and conversation context. After the above adjustments, the system finally generates an answer (the second answer) that is more accurate, considerate, and in line with user expectations. This answer not only answers the user's question, but also takes into account the user's emotional state and conversation context, thereby improving user experience and satisfaction.

[0046] Optionally, adjusting the expression of the first answer according to the target emotional state and the conversation context to generate the second answer includes: Traversing the preset rule library, matching the first target rule according to the current target emotional state and the conversation context, wherein the conversation context includes the user's previous questions, the system's answers, and background information, and the first target rule includes adding preset positive words when the target emotional state is positive and the conversation context does not contain preset negative words, or adding preset comforting words when the target emotional state is negative and the conversation context contains preset negative words; adjusting the expression of the first answer according to the first target rule; When multiple first target rules are matched, the second target rule to be finally executed is selected according to the priorities of the multiple first target rules or a preset conflict resolution mechanism.

[0047] The preset rule base is a database or collection that stores a variety of adjustment rules. These rules are used to guide how to adjust the expression of answers based on emotional state and conversation context. Each rule in the rule base defines specific conditions (such as emotional state, key words in the conversation context, etc.) and corresponding operations (such as adding specific words, changing the tone, etc.). The system will first traverse all the rules in the preset rule base and check whether they are applicable to the current target emotional state and conversation context. By traversing, the system can find all possible applicable rules and prepare for subsequent matching and adjustment.

[0048] The system will match the most appropriate rule, i.e., the first target rule, based on the current target emotional state and the conversation context (including the user's previous questions, the system's answers, and background information). When the target emotional state is positive and there are no preset negative words in the conversation context, the system may choose to add preset positive words; when the target emotional state is negative and the conversation context contains preset negative words, the system may choose to add preset comforting words. When the first target rule is found, the system will adjust the expression of the first answer according to the requirements of the rule. This may include adding, deleting, or replacing specific words, and changing the strength of the tone. The adjusted answer (i.e., the second answer) will be more in line with the user's emotional needs and conversation context. In some cases, the system may match multiple first target rules, which may conflict or overlap with each other. To handle this situation, the system usually selects the second target rule to be executed based on the priority of the rule or the preset conflict resolution mechanism. The priority can be determined based on the importance of the rule, the scope of application, or other preset criteria; and the conflict resolution mechanism may include strategies such as rule merging, rule substitution, or rule ignoring.

[0049] By considering the target emotional state and the conversation context, the system can generate more personalized answers. These answers not only answer the user's questions, but also incorporate the user's emotional needs and conversation context, thereby enhancing the user's satisfaction and sense of belonging. When the system detects the user's positive emotions, by adding preset positive words and phrases, the user's positive emotions can be further stimulated and the emotional resonance with the user can be enhanced. Similarly, when the user's negative emotions are detected, adding preset comforting words and phrases can help alleviate the user's negative emotions, reflecting the system's care and support. By traversing the preset rule base and matching the most applicable rules, the system can ensure that the generated answers are both accurate and relevant. This rule-based adjustment method avoids the arbitrariness and uncertainty of the answers and improves the reliability and stability of the system. The present application can handle situations involving multiple emotional states and complex conversation contexts. Through the preset conflict resolution mechanism, the system can select the most appropriate rule to execute when multiple rules are matched, thereby ensuring that the generated answers meet the user's expectations and avoid internal contradictions. By continuously updating and optimizing the preset rule base, the system can continuously learn and adapt to new conversation situations and emotional states. This intelligence and adaptability enable the system to better meet the needs of users and improve user experience. Since the rules in the rule base are pre-defined, the system can quickly traverse and match the most applicable rules, thereby quickly generating a second answer. This efficient processing method improves the system's response speed and efficiency, allowing users to get satisfactory answers more quickly.

[0050] S140: construct a virtual digital human using virtual reality technology, and determine the facial expression, body movement, and voice intonation of the virtual digital human according to the emotional tendency in the second answer to display the second answer.

[0051] Based on VR (virtual reality) technology, highly realistic virtual digital humans can be built. These digital humans not only have realistic appearance, but also can perform complex interactions and movements. Through 3D modeling, texture mapping, skeletal animation and other technologies, virtual digital humans with different shapes and smooth movements can be created. The second answer is generated based on the user's question and the context of the conversation, which contains specific emotional tendencies, such as positive, negative, neutral, etc. This emotional tendency is a direct reflection of the user's emotional state and an important basis for adjusting the performance of the virtual digital human. Using natural language processing (NLP) and sentiment analysis technology, the emotional tendency of the text in the second answer can be identified. By analyzing the vocabulary, syntax and semantic information in the text, the type of emotion expressed by the answer can be determined. According to the emotional tendency in the second answer, the facial expression of the virtual digital human can be dynamically adjusted. For example, when the answer expresses positive emotions, the digital human can be made to smile or show a happy expression. In addition to facial expressions, the body movements of the virtual digital human can also be adjusted according to the emotional tendency. Appropriate body movements can enhance the expression of emotions and make the performance of the digital human more vivid and realistic. For example, when expressing positive emotions, the digital human can be made to wave or jump. Voice is one of the important ways to express emotions. By adjusting the voice and intonation of the virtual digital person, the expression of emotions can be further enhanced. For example, when expressing positive emotions, the voice of the digital person can be made brighter and more cheerful. Users can not only see the text content of the second answer, but also feel the emotional tendency expressed by the answer through the facial expressions, body movements and voice intonation of the virtual digital person.

[0052] Optionally, determining the facial expression, body movement and voice intonation of the virtual digital person according to the emotional tendency in the second answer to display the second answer includes: When the emotional tendency in the second answer is a positive emotion, the facial expression of the virtual digital person is set to smile, blink or raise eyebrows, the body movement of the virtual digital person is set to wave, jump or applaud, and the voice tone of the virtual digital person is set to a speed greater than a first speed threshold, a volume greater than a first volume threshold and an upward tone; When the emotional tendency in the second answer is a negative emotion, the facial expression of the virtual digital person is set to frown, lower the head or pout, the body movement of the virtual digital person is set to lower the head, cross the arms or shake the head slowly, and the voice tone of the virtual digital person is set to a speaking speed less than a second speaking speed threshold, a volume less than a second volume threshold, and a descending tone, the second speaking speed threshold is less than the first speaking speed threshold, and the second volume threshold is less than the first volume threshold.

[0053] When the emotional tendency in the second answer is positive, the facial expression of the virtual digital person will be set to smile, blink or raise eyebrows to express positive emotions. These expressions can intuitively convey positive emotions such as joy, happiness or satisfaction, and enhance the user's positive feelings. At the same time, the body movements of the virtual digital person will be adjusted accordingly, such as waving, jumping or clapping. These actions further strengthen the expression of positive emotions, allowing users to feel the excitement and vitality of the digital person. In terms of voice, the speed of the virtual digital person will be greater than the first speed threshold, indicating that his speech is fluent and full of vitality; the volume will be greater than the first volume threshold to ensure the clear communication of information; the tone will rise, presenting a relaxed and pleasant atmosphere.

[0054] On the contrary, when the emotional tendency in the second answer is negative, the facial expression of the virtual digital person will change to frowning, lowering the head or pouting, etc. to express negative emotions. These expressions accurately convey negative emotions such as sadness, frustration or dissatisfaction, allowing users to feel the emotional resonance of the digital person. In terms of body movements, the virtual digital person will show lowering the head, crossing arms or shaking the head slowly. These actions reflect the heaviness and uneasiness in his heart, further deepening the expression of negative emotions. In terms of voice, the speed of the virtual digital person will be less than the second speed threshold (this threshold is less than the first speed threshold when the emotion is positive), indicating that his speech is slow and heavy; the volume will be less than the second volume threshold (this threshold is less than the first volume threshold when the emotion is positive) to reflect his inner depression; the tone will drop, creating a sad or depressing atmosphere.

[0055] By accurately matching emotional tendencies with corresponding facial expressions, body movements, and voice intonations, the embodiments of the present application can ensure that the virtual digital human accurately conveys the emotional information in the second answer. Whether it is positive or negative emotion, users can intuitively feel the emotional color contained in the answer through the performance of the virtual digital human, thereby improving the accuracy and effectiveness of information transmission. As a medium for interacting with users, the vivid and realistic performance of the virtual digital human can greatly enhance the user experience. When the virtual digital human can respond accordingly according to the emotional tendency, the user will feel more natural and friendly, as if communicating with a real individual. This immersive experience can enhance the user's sense of participation and satisfaction, and improve the user's trust and dependence on the system. By dynamically adjusting the facial expressions, body movements, and voice intonations of the virtual digital human, the embodiments of the present application make the interaction process more vivid and interesting. Users no longer simply receive text or voice information, but can feel the ups and downs of emotions by observing the performance of the virtual digital human, thereby increasing the fun and attractiveness of the interaction. The embodiments of the present application are not limited to the expression of positive and negative emotional tendencies, but can also be expanded to more types of emotional expressions according to actual needs. By presetting or learning more emotional rules and expressions, virtual digital humans can cope with more complex and changeable emotional communication scenarios and provide users with more comprehensive and personalized services.

[0056] This embodiment also discloses a virtual digital human interactive question and answer system. Figure 2 is a module diagram of a virtual digital human interactive question-answering system disclosed in an embodiment of the present application, such as Figure 2 As shown, the system includes a collection module 201, an emotion module 202, an answer module 203 and a virtual module 204, wherein: The acquisition module 201 is configured to acquire a facial image and a conversation content of a user, perform a first processing on the conversation content by using a natural language processing technology, and perform a second processing on the facial image by using a preset sentiment analysis model to obtain a first sentiment state, wherein the first processing includes word segmentation and semantic role labeling, and the second processing includes facial expression feature extraction and sentiment recognition; The emotion module 202 is configured to perform a third process on the first processed text using a preset emotion analysis model to obtain a second emotion state of the user, and determine a target emotion state by combining the first emotion state and the second emotion state; An answer module 203 is configured to perform a fourth processing on the first processed text based on a deep learning model to generate a first answer, and adjust the expression of the first answer according to the target emotional state and the conversation context to generate a second answer; The virtual module 204 is configured to construct a virtual digital person using virtual reality technology, and determine the facial expression, body movement and voice intonation of the virtual digital person according to the emotional tendency in the second answer to display the second answer. The step of performing a second processing on the facial image using a preset emotion analysis model to obtain a first emotion state includes: Extracting feature points from the facial image, matching the feature points with preset expression categories, converting the matched expression categories into corresponding first emotional states, and generating corresponding first emotional values ​​according to the first emotional states, wherein the first emotional states include positive emotions, negative emotions, and neutral emotions, The performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user comprises: Extracting preset features related to sentiment analysis from the text after the first processing, wherein the preset features include sentiment words, negative words, and degree words; Inputting the preset features into a preset sentiment analysis model to obtain an output result; According to the output result, the emotional tendency of the text is mapped to a preset emotional category, a second emotional state is determined according to the mapped emotional category, and a corresponding second emotional value is generated according to the second emotional state, wherein the second emotional state includes positive emotion, negative emotion and neutral emotion.

[0057] Optionally, the emotion module 202 is configured to: When the first emotional state and the second emotional state are consistent, determining the first emotional state as a target emotional state; When the first emotional state and the second emotional state are inconsistent, the first emotional value and the second emotional value are weightedly summed to obtain a third emotional value, and the third emotional state corresponding to the third emotional value is determined through a preset value mapping table, and the third emotional state is determined as the target emotional state.

[0058] Optionally, the emotion module 202 is configured to: Use time series analysis methods to analyze the changing trend of users' emotional states over time to predict future emotional states; The third emotional state is adjusted according to the future emotional state, and the adjusted third emotional state is determined as the target emotional state.

[0059] Optionally, the answer module 203 is configured to: Traversing the preset rule library, matching the first target rule according to the current target emotional state and the conversation context, wherein the conversation context includes the user's previous questions, the system's answers, and background information, and the first target rule includes adding preset positive words when the target emotional state is positive and the conversation context does not contain preset negative words, or adding preset comforting words when the target emotional state is negative and the conversation context contains preset negative words; adjusting the expression of the first answer according to the first target rule; When multiple first target rules are matched, the second target rule to be finally executed is selected according to the priorities of the multiple first target rules or a preset conflict resolution mechanism.

[0060] Optionally, the virtual module 204 is configured to: When the emotional tendency in the second answer is a positive emotion, the facial expression of the virtual digital person is set to smile, blink or raise eyebrows, the body movement of the virtual digital person is set to wave, jump or applaud, and the voice tone of the virtual digital person is set to a speed greater than a first speed threshold, a volume greater than a first volume threshold and an upward tone; When the emotional tendency in the second answer is a negative emotion, the facial expression of the virtual digital person is set to frown, lower the head or pout, the body movement of the virtual digital person is set to lower the head, cross the arms or shake the head slowly, and the voice tone of the virtual digital person is set to a speaking speed less than a second speaking speed threshold, a volume less than a second volume threshold, and a descending tone, the second speaking speed threshold is less than the first speaking speed threshold, and the second volume threshold is less than the first volume threshold.

[0061] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0062] This embodiment also discloses an electronic device, referring to Figure 3 The electronic device may include: at least one processor 301 , at least one communication bus 302 , a user interface 303 , a network interface 304 , and at least one memory 305 .

[0063] The communication bus 302 is used to realize the connection and communication between these components.

[0064] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0065] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0066] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0067] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may also be optionally at least one storage device located away from the aforementioned processor 301. As Figure 3As shown, the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of the method for interactive question and answer of a virtual digital human.

[0068] exist Figure 3 In the electronic device shown, the user interface 303 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 301 can be used to call the application program of the method of interactive question and answer of virtual digital human stored in the memory 305. When executed by one or more processors 301, the electronic device executes one or more methods in the above-mentioned embodiments.

[0069] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0070] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0074] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory 305 and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory 305 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0075] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for interactive question and answering of a virtual digital human, characterized in that: Applied to an interactive question-answering platform, the method comprises: Collecting a user's facial image and conversation content, performing a first process on the conversation content using a natural language processing technology, and performing a second process on the facial image using a preset sentiment analysis model to obtain a first sentiment state, wherein the first process includes word segmentation and semantic role labeling, and the second process includes facial expression feature extraction and sentiment recognition; Performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user, and determining a target sentiment state by combining the first sentiment state and the second sentiment state; Performing a fourth processing on the first processed text based on the deep learning model to generate a first answer, and adjusting the expression of the first answer according to the target emotional state and the conversation context to generate a second answer; A virtual digital person is constructed using virtual reality technology, and facial expressions, body movements, and voice intonation of the virtual digital person are determined according to the emotional tendency in the second answer to display the second answer. The step of performing a second processing on the facial image using a preset emotion analysis model to obtain a first emotion state includes: Extracting feature points from the facial image, matching the feature points with preset expression categories, converting the matched expression categories into corresponding first emotional states, and generating corresponding first emotional values ​​according to the first emotional states, wherein the first emotional states include positive emotions, negative emotions, and neutral emotions, The performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user comprises: Extracting preset features related to sentiment analysis from the text after the first processing, wherein the preset features include sentiment words, negative words, and degree words; Inputting the preset features into a preset sentiment analysis model to obtain an output result; According to the output result, the emotional tendency of the text is mapped to a preset emotional category, a second emotional state is determined according to the mapped emotional category, and a corresponding second emotional value is generated according to the second emotional state, wherein the second emotional state includes positive emotion, negative emotion and neutral emotion.

2. The method for interactive question and answer of a virtual digital human according to claim 1, characterized in that: The determining a target emotional state by combining the first emotional state and the second emotional state comprises: When the first emotional state and the second emotional state are consistent, determining the first emotional state as a target emotional state; When the first emotional state and the second emotional state are inconsistent, the first emotional value and the second emotional value are weightedly summed to obtain a third emotional value, and the third emotional state corresponding to the third emotional value is determined through a preset value mapping table, and the third emotional state is determined as the target emotional state.

3. The method for interactive question and answer of a virtual digital human according to claim 2, characterized in that: The determining a target emotional state by combining the first emotional state and the second emotional state comprises: Use time series analysis methods to analyze the changing trend of users' emotional states over time to predict future emotional states; The third emotional state is adjusted according to the future emotional state, and the adjusted third emotional state is determined as the target emotional state.

4. The method for interactive question and answer of a virtual digital human according to claim 1, characterized in that: The step of adjusting the expression of the first answer according to the target emotional state and the conversation context to generate a second answer includes: Traversing the preset rule library, matching the first target rule according to the current target emotional state and the conversation context, wherein the conversation context includes the user's previous questions, the system's answers, and background information, and the first target rule includes adding preset positive words when the target emotional state is positive and the conversation context does not contain preset negative words, or adding preset comforting words when the target emotional state is negative and the conversation context contains preset negative words; adjusting the expression of the first answer according to the first target rule; When multiple first target rules are matched, the second target rule to be finally executed is selected according to the priorities of the multiple first target rules or a preset conflict resolution mechanism.

5. The method for interactive question and answer of virtual digital human according to claim 1, characterized in that: Determining the facial expression, body movement and voice tone of the virtual digital person according to the emotional tendency in the second answer to display the second answer includes: When the emotional tendency in the second answer is a positive emotion, the facial expression of the virtual digital person is set to smile, blink or raise eyebrows, the body movement of the virtual digital person is set to wave, jump or applaud, and the voice tone of the virtual digital person is set to a speed greater than a first speed threshold, a volume greater than a first volume threshold and an upward tone; When the emotional tendency in the second answer is a negative emotion, the facial expression of the virtual digital person is set to frown, lower the head or pout, the body movement of the virtual digital person is set to lower the head, cross the arms or shake the head slowly, and the voice tone of the virtual digital person is set to a speaking speed less than a second speaking speed threshold, a volume less than a second volume threshold, and a descending tone, the second speaking speed threshold is less than the first speaking speed threshold, and the second volume threshold is less than the first volume threshold.

6. A virtual digital human interactive question-answering system, characterized in that: It includes acquisition module, emotion module, answer module and virtual module, among which: A collection module configured to collect a user's facial image and conversation content, perform a first process on the conversation content using a natural language processing technology, and perform a second process on the facial image using a preset sentiment analysis model to obtain a first sentiment state, wherein the first process includes word segmentation and semantic role labeling, and the second process includes facial expression feature extraction and sentiment recognition; an emotion module configured to perform a third processing on the first processed text using a preset emotion analysis model to obtain a second emotion state of the user, and determine a target emotion state by combining the first emotion state and the second emotion state; an answer module, configured to perform a fourth processing on the first processed text based on a deep learning model to generate a first answer, and adjust an expression of the first answer according to the target emotional state and the conversation context to generate a second answer; a virtual module configured to construct a virtual digital person using virtual reality technology, and determine the facial expression, body movement and voice intonation of the virtual digital person according to the emotional tendency in the second answer to display the second answer, The step of performing a second processing on the facial image using a preset emotion analysis model to obtain a first emotion state includes: Extracting feature points from the facial image, matching the feature points with preset expression categories, converting the matched expression categories into corresponding first emotional states, and generating corresponding first emotional values ​​according to the first emotional states, wherein the first emotional states include positive emotions, negative emotions, and neutral emotions, The performing a third processing on the first processed text using a preset sentiment analysis model to obtain a second sentiment state of the user comprises: Extracting preset features related to sentiment analysis from the text after the first processing, wherein the preset features include sentiment words, negative words, and degree words; Inputting the preset features into a preset sentiment analysis model to obtain an output result; According to the output result, the emotional tendency of the text is mapped to a preset emotional category, a second emotional state is determined according to the mapped emotional category, and a corresponding second emotional value is generated according to the second emotional state, wherein the second emotional state includes positive emotion, negative emotion and neutral emotion.

7. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 5 is performed.

Images