An AI virtual human chat content analysis and user emotion recognition method and system

Abstract

This invention relates to a method and system for AI virtual human chat content analysis and user emotion recognition, belonging to the field of AI virtual human emotion recognition technology. It includes acquiring initial text, initial voice, and historical dialogue data; obtaining text data and voice data; extracting global context information from historical dialogue data; extracting event keywords, searching for external documents online to obtain external emotion data; performing emotion polarity analysis to obtain text emotion tags and a first confidence level, voice emotion tags and a second confidence level, and external emotion tags; if the text emotion tags and voice emotion tags are inconsistent, using the external emotion tags as a benchmark, calculating a first similarity and a second similarity; correcting to obtain a third confidence level and a fourth confidence level; comparing to obtain the final confidence level with the higher value, and outputting the emotion tag corresponding to the final confidence level and the corresponding confidence level. This invention has the effect of improving the targeting of judging users' real emotions.

Description

Technical Field

[0001] This invention relates to the field of AI virtual human emotion recognition technology, and in particular to a method and system for AI virtual human chat content analysis and user emotion recognition. Background Technology

[0002] With the continuous development of human-computer interaction technology, AI virtual humans with real-time dialogue and interaction capabilities have been widely used in scenarios such as companionship, psychological counseling, and intelligent customer service.

[0003] Currently, there are two main types of methods for AI virtual humans to perceive and recognize user emotions. One type is single-modal recognition, which includes text recognition and speech recognition: text recognition collects the user's dialogue text and extracts emotion-related information based on the semantics of the text; speech recognition collects the user's audio voice and completes emotion judgment based on the sound signal. The other type can simultaneously collect both text and speech information to achieve multimodal recognition, then extracts a fixed length of historical dialogue sentences to splice and model them, and uses preset fixed weights to weight and fuse the emotion judgment results from the text and speech channels, finally outputting the corresponding emotion label.

[0004] Regarding the aforementioned technologies, when the semantic features of the text and the acoustic features of the speech represent contradictory emotional tendencies, relying solely on weighted fusion with fixed weights is insufficient to specifically distinguish the user's true emotions, and there is still room for improvement. Summary of the Invention

[0005] To improve the accuracy of judging users' true emotions, this invention provides a method and system for AI virtual human chat content analysis and user emotion recognition.

[0006] In a first aspect, the present invention provides a method for AI virtual human chat content analysis and user emotion recognition, employing the following technical solution: A method for AI virtual human chat content analysis and user emotion recognition, comprising: Step 1: Obtain the initial text, initial speech, and historical dialogue data of the current conversation input by the user; Step 2: Perform text preprocessing on the initial text to obtain text data, and extract speech-text and tone features from the initial speech to obtain speech data; Step 3: Retrieve global context information from historical dialogue data based on text and voice data; Step 4: Extract event keywords based on text data, voice data, and global context information; search for external documents online based on event keywords; and perform sentiment analysis on the external documents to obtain external sentiment data. Step 5: Based on the preset sentiment discrimination rule base, perform sentiment polarity analysis on the text data, speech data and external sentiment data respectively to obtain the text sentiment label and the corresponding first confidence level, the speech sentiment label and the corresponding second confidence level, and the external sentiment label; Step 6: If the text sentiment label and the voice sentiment label are inconsistent, use the external sentiment label as a benchmark to calculate the first similarity between the text sentiment label and the external sentiment label, and calculate the second similarity between the voice sentiment label and the external sentiment label. Step 7: Correct the first confidence level based on the first similarity to obtain the third confidence level, and correct the second confidence level based on the second similarity to obtain the fourth confidence level; Step 8: Compare the third confidence level and the fourth confidence level to obtain the final confidence level with the higher value, and output the sentiment label and the corresponding confidence level of the final confidence level.

[0007] Optionally, methods for correcting the first confidence level based on the first similarity to obtain the third confidence level, and correcting the second confidence level based on the second similarity to obtain the fourth confidence level, include: Step 70: Perform semantic analysis on the text and voice data to obtain logical keywords and the corresponding number of keywords; Step 71: Perform a coherence judgment on adjacent logical keywords to extract non-coherent keywords and their corresponding number of non-coherent keywords. Step 72: Compare the number of incoherent words with the number of keywords to obtain the incoherence ratio; Step 73: If the incoherence ratio is not greater than the preset incoherence threshold, then the first confidence level is corrected according to the preset correction weight and the first similarity to obtain the third confidence level, and the second confidence level is corrected according to the preset correction weight and the second similarity to obtain the fourth confidence level. Step 74: If the incoherence ratio is greater than the preset incoherence threshold, then increase the preset correction weight to the preset incoherence weight. Step 75: Correct the first confidence level according to the non-continuous weight and the first similarity to obtain the third confidence level, and correct the second confidence level according to the non-continuous weight and the second similarity to obtain the fourth confidence level.

[0008] Optionally, it also includes a correction method for obtaining a third confidence level by correcting the first confidence level based on the first similarity and obtaining a fourth confidence level by correcting the second confidence level based on the second similarity, the method comprising: Step 700: Obtain the user's historical input speed and the current input speed of the current conversation; Step 701: Compare the current input speed with the historical input speed, and calculate the speed change and the direction of speed change; Step 702: If the change in speed is greater than the preset speed threshold, search for the corresponding emotional fluctuation tag set in the preset emotional fluctuation library according to the direction of the speed change; Step 7020: If the text sentiment tag falls into the fluctuating sentiment tag set, then the third confidence level corresponding to the text sentiment tag is corrected according to the preset fluctuation correction range to obtain the text fluctuation confidence level, so as to update the third confidence level and output it. Step 7021: If the voice emotion label falls into the fluctuating emotion label set, then the fourth confidence level corresponding to the voice emotion label is corrected according to the preset fluctuation correction range to obtain the voice fluctuation confidence level, so as to update the fourth confidence level and output it.

[0009] Optionally, methods for obtaining external sentiment data by performing sentiment analysis on external documents include: Step 40: Perform sentiment analysis on external documents to extract individual sentiment data from each party. Step 41: Extract keywords from personal emotional data to obtain personal keywords; Step 42: Perform semantic analysis on personal keywords and classify them to obtain multiple sets of keywords of the same type; Step 43: Count the number of personal keywords in each set of similar keywords, filter out the set of similar keywords with the largest number of personal keywords, and output the personal sentiment data corresponding to the set of similar keywords with the largest number of personal keywords as external sentiment data.

[0010] Optionally, methods for obtaining external sentiment data by performing sentiment analysis on external documents also include: Step 400: Perform semantic analysis on historical dialogue data to extract historical modal particles, and perform semantic analysis on external documents to extract external modal particles; Step 401: Statistically determine the historical tone characteristics based on historical modal particles, and statistically determine the external tone characteristics of each party based on external modal particles; Step 402: Calculate the tone similarity with the external tone features of each party based on historical tone features; Step 403: Select the external tone features with the highest tone similarity and define them as similar tone features. Perform sentiment analysis on the external documents corresponding to the similar tone features to obtain external sentiment data.

[0011] Optionally, it also includes a correction method for obtaining a first confidence level and a second confidence level by performing sentiment polarity analysis on text data and speech data based on a preset sentiment discrimination rule base, respectively. This method includes: Step 50: Perform semantic integrity analysis on the text data to obtain text integrity, and perform speech intelligibility analysis on the speech data to obtain speech intelligibility; Step 510: If the text completeness is not lower than the preset standard completeness, then output the first confidence level; Step 511: If the text completeness is lower than the preset standard completeness, calculate the completeness ratio by comparing the text completeness with the standard completeness, calculate the text completeness confidence based on the completeness ratio and the first confidence, update the first confidence and output it; Step 520: When the speech clarity is not lower than the preset standard clarity, output the second confidence level; Step 521: When the speech clarity is lower than the preset standard clarity, the speech clarity is compared with the standard clarity to calculate the clarity ratio. The speech clarity confidence is calculated based on the clarity ratio and the second confidence to update the second confidence and output it.

[0012] Optionally, it also includes a correction method for obtaining a first confidence level and a second confidence level by performing sentiment polarity analysis on text data and speech data based on a preset sentiment discrimination rule base, and the method further includes: Step 500: Extract subjective and objective keywords from the text data, and extract subjective and objective keywords from the speech data; Step 501: Count the number of subjective keywords in the text to obtain the subjective quantity of the text, and count the number of objective keywords in the text to obtain the objective quantity of the text; Step 502: Count the number of subjective keywords in the speech to obtain the subjective speech count, and count the number of objective keywords in the speech to obtain the objective speech count; Step 503: Calculate the first subjective ratio based on the subjective quantity and the objective quantity of the text; Step 504: Calculate the second subjective ratio based on the subjective and objective quantities of speech. Step 505: If the first subjective ratio is higher than the preset subjective threshold, calculate the text subjective confidence based on the first subjective ratio and the first confidence, update the first confidence and output it; Step 506: If the second subjective ratio is higher than the preset subjective threshold, calculate the speech subjective confidence based on the second subjective ratio and the second confidence, update the second confidence and output it.

[0013] Optional, also includes: Step 53: Extract behavior description information based on global context information, wherein the behavior description information includes the agent and the recipient; Step 54: Determine the user's identity and stance based on the agent, the recipient, and the preset stance determination rules; Step 55: Based on the user's identity and stance, and the preset stance sentiment database, find the corresponding directional sentiment tag; Step 56: Add the directional emotion tags to the text emotion tag and the voice emotion tag respectively to update the text emotion tag and the voice emotion tag.

[0014] Optionally, the feature is that it further includes a method for obtaining external sentiment tags, the method comprising: Step 410: Identify the people around the user from the historical dialogue data and extract the surrounding historical data of each person; Step 411: Based on the preset sentiment discrimination rule base, perform sentiment polarity analysis on the surrounding historical data to obtain the surrounding sentiment labels; Step 412: Filter out the most frequently occurring surrounding sentiment tags from the surrounding sentiment tags and define them as the mainstream sentiment tags; Step 413: Output mainstream sentiment tags as external sentiment tags.

[0015] Secondly, this invention provides an AI virtual human chat content analysis and user emotion recognition system, which adopts the following technical solution: An AI virtual human chat content analysis and user emotion recognition system includes: The acquisition module is used to acquire initial text, initial speech, historical dialogue data, historical input speed, and current input speed; The memory is used to store the program for the AI ​​virtual human chat content analysis and user emotion recognition method described above; The processor loads and executes programs from memory.

[0016] In summary, the present invention has at least one of the following beneficial technical effects: It solves the problem of being unable to accurately judge user emotions when the semantics of the text and the acoustic features of the speech represent contradictory emotional tendencies, and improves the pertinence of judging users' true emotions. It achieves the effect of adjusting confidence based on logical coherence. Attached Figure Description

[0017] Figure 1 This is a flowchart of an AI virtual human chat content analysis and user emotion recognition method according to an embodiment of this application; Figure 2 This is a flowchart of a method in this application embodiment for correcting a first confidence level based on a first similarity to obtain a third confidence level, and correcting a second confidence level based on a second similarity to obtain a fourth confidence level; Figure 3 This is a system block diagram of an AI virtual human chat content analysis and user emotion recognition system according to an embodiment of this application. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0019] This invention discloses a method for analyzing chat content and recognizing user emotions in an AI virtual human. (Refer to...) Figure 1 A method for AI virtual human chat content analysis and user emotion recognition includes: Step 1: Obtain the initial text, initial speech, and historical dialogue data of the current conversation input by the user.

[0020] Initial text refers to the information entered by the user in text form during the current conversation. The text content submitted directly by the user through the text input box in the AI ​​virtual human interaction interface constitutes the initial text. If the user has not entered any text, they should be prompted to add it.

[0021] Initial speech refers to the information input by the user in the current conversation in voice form. The audio signal recorded and submitted in real time by the user through the voice input button in the AI ​​virtual human interaction interface is the initial speech. If the user does not input any voice, they should be prompted to add voice input.

[0022] Historical dialogue data refers to all or part of the past dialogue records between the user and the AI ​​virtual human prior to the current dialogue, including contextual information related to the events described in the current dialogue. Historical dialogue data is retrieved from the AI ​​virtual human's dialogue storage.

[0023] Step 2: Perform text preprocessing on the initial text to obtain text data, and extract speech-text and tone features from the initial speech to obtain speech data.

[0024] Text data refers to the simplified text obtained after preprocessing the initial text. First, the initial text is segmented into sentences and words, resulting in sentence sequences and word sequences. Each word in the word sequences is then matched against a pre-defined stop word list. Matched stop words are removed, and unmatched words are retained. The retained words are then combined in their original order to form the simplified text, which serves as the text data. The stop word list is pre-set by those skilled in the art, collecting frequently occurring words in natural language that contribute little to semantics as the initial stop word set, including structural particles, modal particles, and interjections. Further filtering is then performed to extract words that may carry semantic information, ensuring that useful words are not mistakenly deleted.

[0025] Speech data refers to the character recognition features and tonal features in the speech-to-text recognition results. Speech-to-text recognition employs publicly available speech recognition methods. Initial speech is input into an acoustic model, and its acoustic features are mapped to a phoneme sequence. This phoneme sequence is then input into a language model, decoded into a text sequence, and the speech-to-text recognition result is output. The acoustic model uses a Hidden Markov Model (HMM) to describe the temporal changes of phonemes and a Gaussian Mixture Model (GMM) to describe the acoustic feature distribution corresponding to each hidden state. The language model uses an N-gram model to predict the probability of the next word based on the recognized word sequence, thus assisting the decoding process. Both the HMM-Gaussian Mixture Model and the N-gram model are publicly available technologies in the field of speech recognition. Following the same acquisition method as text data, the speech-to-text recognition results are segmented into sentences and words to obtain sentence and word sequences, ultimately yielding the character recognition features.

[0026] Pitch features refer to acoustic parameters extracted from initial speech that represent the prosody and emotional expression of a user's speech, implemented using publicly available speech signal processing methods. Pitch features include at least one of fundamental frequency trajectory or short-time energy. The fundamental frequency trajectory is a sequence of vocal cord vibration frequencies changing over time in a speech signal. It is extracted by segmenting the initial speech into frames with a frame length of 20 to 30 milliseconds and a frame shift of 10 milliseconds. The fundamental frequency value of each frame is calculated using the autocorrelation method, and all fundamental frequency values ​​are arranged in chronological order to obtain the fundamental frequency trajectory. A higher fundamental frequency is associated with emotions such as excitement and anger, while a lower fundamental frequency is associated with emotions such as sadness and fatigue.

[0027] Short-time energy refers to the energy level of a speech signal within a short frame. The initial speech is segmented into frames, and the short-time energy value of each frame is calculated. Increased short-time energy is associated with emotions such as excitement and joy, while decreased short-time energy is associated with emotions such as sadness and depression.

[0028] Step 3: Retrieve global context information from historical dialogue data based on text and voice data.

[0029] Global contextual information refers to the set of dialogue fragments retrieved from historical dialogue data that are related to the events involved in the current dialogue. Using both text data and speech-to-text recognition results from the speech data as query conditions, core semantic units are extracted from the query conditions as search keywords. These core semantic units include at least one of entity words, event words, and action words. The semantic similarity between the core semantic units and each dialogue fragment in the historical dialogue data is calculated. Semantic similarity is calculated using a combination of TF-IDF vectors and cosine similarity. The core semantic units of the current dialogue are used as keyword sequences. The frequency of each word in the keyword sequence is calculated. The frequency of each word in all historical dialogue segments is also calculated, and the reciprocal of this frequency is used as the inverse document frequency (IVF). The TF-IDF weight value is obtained by multiplying the word frequency by the IVF, and this weight value is then used to fill TF-IDF vectors according to the word's position in the candidate word set. The candidate word set consists of all non-repeating words that have appeared in all dialogue segments of the historical dialogue data. TF-IDF vectors are generated for the search keywords and each historical dialogue segment. The dot product of the two vectors is divided by the product of their magnitudes to obtain the cosine similarity, which ranges from 0 to 1, with higher values ​​indicating closer semantics. Historical dialogue segments with cosine similarities higher than a preset similarity threshold are selected as global context information. This similarity threshold is preset by those skilled in the art, for example, 0.65.

[0030] Step 4: Extract event keywords based on text data, voice data, and global context information. Search for external documents online based on the event keywords and perform sentiment analysis on the external documents to obtain external sentiment data.

[0031] Event keywords refer to a set of words or phrases that represent the core elements of the current dialogue describing the event. Core elements include at least one of the following: the subject of the event, actions, involved parties, and the time or place of occurrence. The text data of the current dialogue is combined with the speech-to-text recognition results as the extraction source. If the global context information is not empty, it is also included in the extraction source. The frequency of each word in the merged, abbreviated text is counted, and words with frequencies higher than a preset frequency threshold are selected as event keywords. This preset frequency threshold is set by those skilled in the art based on actual needs, for example, a preset frequency threshold of 2%. If the number of extracted keywords is zero, the word with the highest frequency is selected as the event keyword. The words retained in the abbreviated text include personal pronouns, sentiment words, negation words, degree adverbs, opinion verbs, and nouns and verbs with semantic value. The nouns and verbs include the subject words, action words, and object words required for extracting event keywords.

[0032] External documents refer to online information carriers related to the current event, obtained through internet search engines using event keywords as the query statement. Examples include web pages, news articles, and publicly available content on social media.

[0033] The query statement is formed by concatenating event keywords using spaces or logical operators to create a string that conforms to search engine query specifications. The query is submitted to a public internet search engine via a standardized application programming interface (API) to retrieve search results related to the event keywords. Standardized APIs include, but are not limited to, search engine APIs provided by search engines, such as Bing Search or Google Custom Search. Search results must contain at least the following information: document title, document summary or full text, document source URL, and document publication time. The top-ranked search results, representing a preset number of searches, are selected, and their corresponding full text or summaries are treated as external documents. The preset number of searches is set by those skilled in the art based on actual needs, for example, 10 documents. If the number of search results returned is less than the preset number, the actual number of documents retrieved is used, and this is noted in subsequent processing as insufficient external documents. If the search engine returns no search results, subsequent processes do not rely on external sentiment data.

[0034] External sentiment data refers to the collection of raw information related to the current event extracted from external documents, including sentiment words, sentiment word event descriptions, and behavioral descriptions. Usernames, author names, or document source domains in external documents are used to distinguish different parties. All text content corresponding to the same source identifier is aggregated into the personal document data of that party. Sentiment words, sentiment event descriptions, and behavioral descriptions are extracted from the personal document data. Sentiment words are extracted by matching a pre-defined sentiment dictionary; sentiment event descriptions are extracted by semantic role labeling to extract event sentence fragments associated with sentiment words; and behavioral descriptions are extracted by dependency parsing and event extraction to extract sentence fragments involving subjects, actions, and objects. If the same party appears in multiple external documents, the sentiment-related information extracted from each document is aggregated into the same party's record. If no party can be identified in the external documents, each external document is treated as an independent source of party data.

[0035] Step 5: Based on the preset sentiment discrimination rule base, perform sentiment polarity analysis on the text data, speech data and external sentiment data respectively to obtain the text sentiment label and the corresponding first confidence level, the speech sentiment label and the corresponding second confidence level, and the external sentiment label.

[0036] The sentiment discrimination rule library contains two parts: text sentiment discrimination rules and speech sentiment discrimination rules. Text sentiment discrimination rules refer to the set of rules that can perform sentiment polarity analysis on text data. The text sentiment discrimination rules include a sentiment lexicon, which stores several sentiment words and their corresponding sentiment categories and sentiment intensity values. The text sentiment discrimination rules also include a negation word table and a degree adverb table. The negation word table stores several negation words, and the degree adverb table stores several degree adverbs and their corresponding intensity modification coefficients.

[0037] First, a set of basic emotion words representing six emotion categories—anger, sadness, joy, fear, disgust, and neutrality—is collected. Then, annotators label the corresponding emotion category of each word according to its usual meaning in natural language and assign an emotion intensity value between 0 and 1 based on the intensity of the emotion, thus forming a basic emotion dictionary. Based on the basic emotion dictionary, the emotion dictionary is further expanded through synonyms, antonyms, and word form changes to obtain a new emotion dictionary.

[0038] Collect words in natural language that express negation, including no, not, no, not, and absolutely not, and store these words to form a negation word list.

[0039] This study collects words in natural language that modulate the intensity of emotion, including adverbs of degree such as extremely, very, extremely, and very. Each adverb of degree is assigned a corresponding intensity modification coefficient to form a table of adverbs of degree. The intensity modification coefficient refers to the extent to which the adverb of degree modulates the intensity of emotion, and is set by those skilled in the art based on linguistic common sense.

[0040] Speech emotion discrimination rules refer to a set of rules that can perform emotional polarity analysis on the tonal features in speech data, and are pre-set by those skilled in the art. The speech emotion discrimination rules store the fundamental frequency trajectory range and short-time energy range corresponding to each emotion category. The standard ranges corresponding to each emotion category are established based on common knowledge in the field of speech emotion recognition; anger and joy correspond to increased fundamental frequency and increased short-time energy, while sadness and fear correspond to decreased fundamental frequency and decreased short-time energy.

[0041] Text sentiment tags refer to the sentiment categories expressed by text data after sentiment polarity analysis. The text data is segmented into words, and the segmentation results are matched with a sentiment dictionary to extract matched sentiment words and their corresponding sentiment categories and sentiment intensity values. The presence of negation words is detected in the text data; if a sentiment word falls within the modification range of a negation word, its sentiment polarity is reversed. The presence of degree adverbs is also detected; if a sentiment word falls within the modification range of a degree adverb, its sentiment intensity value is multiplied by the intensity modification coefficient corresponding to the degree adverb. The sentiment categories and sentiment intensity values ​​of all sentiment words after the above adjustments are combined to obtain the overall sentiment category and confidence level of the text data. The sum of the sentiment intensity values ​​of each sentiment category is calculated, and the sentiment category with the highest sum of intensity values ​​is selected as the overall sentiment tag. The ratio of the highest sum of intensity values ​​to the sum of all sentiment intensity values ​​is used as the first confidence level. If no sentiment words are matched in the text data, the sentiment category is labeled as neutral, and the first confidence level is the preset minimum confidence level. The minimum confidence level is preset by those skilled in the art, for example, 0.1.

[0042] First confidence level refers to the quantitative value of the credibility of a text sentiment tag. The method for obtaining first confidence level is the same as described in the section on obtaining text sentiment tags.

[0043] Voice emotion labels refer to the emotion categories expressed by voice data after performing emotion polarity analysis. The process involves comparing the tonal features to be identified with the standard tonal feature ranges of various emotions in a mapping table, calculating the distance between the tonal feature vector and the standard range intervals of each emotion, and selecting the emotion category with the closest distance as the voice emotion label. The distance is then normalized to the 0-1 interval, and 1 minus the normalized value is used as the second confidence score. Alternatively, a preferred approach is to perform a weighted fusion of the tonal feature analysis results and the text recognition analysis results to obtain the voice emotion label and the second confidence score.

[0044] The second confidence level refers to the quantitative value of the credibility of the voice emotion tag. The method for obtaining the second confidence level is the same as that described in the section on voice emotion tag acquisition.

[0045] External sentiment labels refer to the sentiment categories expressed by external sentiment data after performing sentiment polarity analysis. Sentiment words, sentiment event descriptions, and behavioral descriptions from the external sentiment data are used as the text to be analyzed. Sentiment polarity analysis is performed using the same text sentiment discrimination rules as for text sentiment labels, and finally, the sentiment category with the highest sum of intensity values ​​is selected as the external sentiment label output.

[0046] Step 6: If the text sentiment label and the voice sentiment label are inconsistent, use the external sentiment label as a benchmark to calculate the first similarity between the text sentiment label and the external sentiment label, and calculate the second similarity between the voice sentiment label and the external sentiment label.

[0047] The first similarity refers to the quantitative value of the semantic proximity between text sentiment tags and external sentiment tags. The second similarity refers to the quantitative value of the semantic proximity between voice sentiment tags and external sentiment tags.

[0048] The calculation of the first and second similarities employs a one-hot vector representation based on emotion categories. Those skilled in the art first predefine an emotion space, which is a vector space composed of a standard set of emotion categories. This standard set of emotion categories includes at least six categories: anger, sadness, joy, fear, disgust, and neutrality. Each emotion category corresponds to a one-hot vector of length six, where the dimension corresponding to the emotion category has a value of 1, and the other dimensions have values ​​of 0. After mapping text emotion tags and external emotion tags to one-hot vectors in the emotion space, the cosine similarity between the two vectors in the emotion space is calculated as the first similarity. Similarly, after mapping voice emotion tags and external emotion tags to one-hot vectors in the emotion space, the cosine similarity between the two vectors in the emotion space is calculated as the second similarity. Since there is only one non-zero dimension in the one-hot vector in the emotion space, the cosine similarity is calculated as 1 if the two emotion tags are the same and 0 if they are different. The values ​​of the first and second similarities range from 0 to 1; higher values ​​indicate that the two emotion tags are closer in the emotion space. For example, in the emotion space, anger is mapped to the vector (1, 0, 0, 0, 0, 0), and sadness is mapped to the vector (0, 1, 0, 0, 0, 0). The cosine similarity between anger and sadness is 0, and the cosine similarity between anger and anger is 1.

[0049] If the text sentiment label and the voice sentiment label are inconsistent, it means that the emotional direction expressed by the user's spoken text content is different from that expressed by the tone, intonation and rhythm of their speech. At this time, it is necessary to use external sentiment labels as a reference to determine which of the text and voice content is closer to the user's true emotions. Therefore, the first similarity between the text sentiment label and the external sentiment label is calculated, and the second similarity between the voice sentiment label and the external sentiment label is calculated.

[0050] If the text sentiment label and the voice sentiment label are consistent, it means that the emotions expressed by the user's spoken text content and the tone, intonation and rhythm of their speech are consistent, and the consistent sentiment label and the corresponding confidence level are directly output.

[0051] Step 7: Correct the first confidence level based on the first similarity to obtain the third confidence level, and correct the second confidence level based on the second similarity to obtain the fourth confidence level.

[0052] The third confidence level refers to the updated text confidence value obtained after correcting the first confidence level using the first similarity score. The first confidence level and the first similarity score are weighted and summed; that is, the third confidence level equals the product of the correction weight and the first confidence level, plus 1, minus the product of the difference between the correction weight and the first similarity score. For example, when the correction weight is 0.6, the third confidence level equals 0.6 multiplied by the first confidence level plus 0.4 multiplied by the first similarity score. The correction weight refers to the proportion of the uncorrected confidence level in the calculation of the corrected confidence level, which is preset by those skilled in the art, for example, 0.6. The correction weight ranges from 0 to 1; a higher correction weight indicates a greater proportion of the first confidence level in the third confidence level, and a weaker corrective effect of the first similarity score on the first confidence level.

[0053] The fourth confidence level refers to the updated speech confidence value obtained after the second confidence level is corrected by the second similarity level. The second confidence level and the second similarity level are weighted and summed, that is, the fourth confidence level is equal to the product of the correction weight and the second confidence level plus 1 minus the product of the difference of the correction weight and the second similarity level.

[0054] When external sentiment labels are unavailable, in order to directly compare the first and second confidence levels, the sentiment label corresponding to the higher confidence level and its corresponding confidence level are selected for output.

[0055] Step 8: Compare the third confidence level and the fourth confidence level to obtain the final confidence level with the higher value, and output the sentiment label and the corresponding confidence level of the final confidence level.

[0056] The final confidence level refers to the higher value selected from the third and fourth confidence levels as the quantification of the reliability of the final output result. If the third and fourth confidence levels are equal, the third confidence level is selected first.

[0057] The sentiment label corresponding to the final confidence level and the value of the final confidence level are output as the sentiment recognition result to the AI ​​virtual human, and the sentiment recognition result is used as the basis for responding to users.

[0058] Reference Figure 2 The method for obtaining a third confidence level by correcting the first confidence level based on the first similarity and obtaining a fourth confidence level by correcting the second confidence level based on the second similarity includes: Step 70: Perform semantic analysis on the text and voice data to obtain logical keywords and the corresponding number of keywords.

[0059] Logical keywords refer to words that represent the core semantics within text and speech, including at least one of nouns, verbs, adjectives, and adverbs. Semantic analysis is performed on the speech-to-text recognition results of both text and speech data to extract logical keywords. The text data is a simplified version after stop word removal, retaining only content words; therefore, each word in the text data is directly used as a candidate logical keyword. The candidate logical keywords are then filtered, removing words without actual semantic meaning and retaining nouns, verbs, adjectives, and adverbs as logical keywords. The speech-to-text recognition results in the speech data undergo the same stop word removal process, and logical keywords are extracted using the same filtering method.

[0060] Keyword count refers to the total number of logical keywords extracted from the text and voice data of the current dialogue. The keyword count is obtained by counting the extracted logical keywords.

[0061] Step 71: Perform a coherence judgment on adjacent logical keywords to extract non-coherent keywords and their corresponding number of non-coherent keywords.

[0062] Non-coherent keywords refer to keywords that lack a reasonable semantic logical relationship with their adjacent keywords. Logical keywords are arranged in the order they appear in the original text to obtain a logical keyword sequence. For each pair of adjacent logical keywords in the sequence, a coherence assessment is performed. Using existing technologies such as the Word2Vec model, the word vectors corresponding to the two adjacent logical keywords are obtained, and cosine similarity is calculated. When the cosine similarity is less than a preset coherence threshold, the latter keyword is marked as a non-coherent keyword. When the cosine similarity is greater than or equal to the preset coherence threshold, a reasonable semantic logical relationship is determined between the two adjacent logical keywords, and no marking is performed. The preset coherence threshold is pre-set by those skilled in the art, preferably 0.2.

[0063] The number of non-coherent keywords refers to the total number of keywords marked as non-coherent keywords in a logical keyword sequence, which is obtained by counting all non-coherent keywords in the logical keyword sequence.

[0064] Step 72: Compare the number of incoherent words with the number of keywords to obtain the incoherence ratio.

[0065] The incoherence ratio is the ratio of the number of incoherent keywords to the number of keywords. The incoherence ratio equals the number of incoherent keywords divided by the number of keywords. For example, when there are 10 keywords and 3 incoherent keywords, the incoherence ratio is 0.3.

[0066] Step 73: If the incoherence ratio is not greater than the preset incoherence threshold, then the first confidence level is corrected according to the preset correction weight and the first similarity to obtain the third confidence level, and the second confidence level is corrected according to the preset correction weight and the second similarity to obtain the fourth confidence level.

[0067] The incoherence threshold is a critical ratio used to determine whether a user's expression contains logical inconsistencies. It is preset by someone skilled in the art, for example, 0.2. Those skilled in the art test a large number of text samples with varying degrees of coherence, statistically analyze the correspondence between the incoherence ratio of each sample and the manually annotated logical coherence score, and select the ratio that best distinguishes logically correct text from logically inconsistent text as the incoherence threshold.

[0068] Refer to the detailed instructions in step 7 for adjusting the weights.

[0069] If the incoherence ratio is not greater than the preset incoherence threshold, it indicates that the user's expression logic is normal; at this time, the third and fourth confidence levels are obtained by adjusting the weights.

[0070] Step 74: If the discontinuity ratio is greater than the preset discontinuity threshold, then increase the preset correction weight to the preset discontinuity weight.

[0071] Incoherence weight refers to the corrective weight applied when there is logical inconsistency in the text expression. The incoherence weight is pre-set by those skilled in the art and is lower than the conventional corrective weight. When a user's expression is logically incoherent, the credibility of the user's own text and speech signals decreases. Therefore, the weight of the first and second confidence scores in the calculation of the third and fourth confidence scores is reduced, and the weight of the first and second similarity scores is correspondingly increased, allowing external sentiment labels to play a greater role in confidence correction.

[0072] If the incoherence ratio is greater than the preset incoherence threshold, it indicates that the user's expression is logically confused. In this case, the correction weight will be increased to the incoherence weight to increase the proportion of the first similarity and the second similarity.

[0073] Step 75: Correct the first confidence level according to the non-continuous weight and the first similarity to obtain the third confidence level, and correct the second confidence level according to the non-continuous weight and the second similarity to obtain the fourth confidence level.

[0074] The third confidence level in this step is equal to the difference between the incoherent weights multiplied by the first confidence level plus 1 and the incoherent weights minus the first similarity level. The fourth confidence level in this step is equal to the difference between the incoherent weights multiplied by the second confidence level plus 1 and the incoherent weights minus the second similarity level.

[0075] This includes a correction method for obtaining a third confidence level by correcting the first confidence level based on the first similarity, and obtaining a fourth confidence level by correcting the second confidence level based on the second similarity. This method includes: Step 700: Obtain the user's historical input speed and the current input speed of the current conversation.

[0076] Historical input speed refers to the average speed at which a user types text and speaks in historical conversations, including both typing and speaking speeds. Text data is extracted from historical conversation data, recording the number of characters in each segment and the time taken to type it. The typing speed for each segment is calculated and averaged to obtain the historical typing speed. Similarly, speech data is extracted from historical conversation data, and speech-to-text recognition is performed to obtain the text content. The number of characters in each speech segment and the duration of that segment are recorded. The speaking speed for each segment is calculated and averaged to obtain the historical speaking speed.

[0077] Current input speed refers to the user's real-time typing speed when entering text in the current conversation and their real-time speaking speed when using voice input, including both current typing and speaking speeds. Referring to the method used to obtain historical input speeds, the number of characters in each segment of text data and the time spent typing that segment are recorded. The typing speed is calculated and averaged to obtain the current typing speed. Similarly, the number of characters in each segment of speech data and the duration of that segment are recorded. The speaking speed is calculated and averaged to obtain the current speaking input speed.

[0078] Step 701: Compare the current input speed with the historical input speed to calculate the speed change and the direction of speed change.

[0079] Speed ​​change refers to the magnitude of change in current input speed compared to historical input speed, including changes in typing speed and speaking speed. The speed change is calculated by subtracting the historical input speed from the current input speed, dividing by the historical input speed, and taking the absolute value.

[0080] The direction of speed change refers to the trend of the current input speed relative to the historical input speed, including positive and negative changes. A positive change indicates an increase in speed, and a negative change indicates a decrease in speed. The direction of speed change is determined by subtracting the historical input speed from the current input speed, based on the sign of the difference. A positive difference indicates a positive speed change; a negative difference indicates a negative speed change; and a difference of 0 indicates no change in speed.

[0081] Step 702: If the change in speed is greater than the preset speed threshold, search for the corresponding emotional fluctuation tag set in the preset emotional fluctuation library according to the direction of the speed change.

[0082] The speed threshold is a critical value used to determine whether the change in input speed is significant. It is preset by those skilled in the art based on experience, and the value ranges from 0.1 to 0.5, preferably 0.2, to determine that the speed change is significant.

[0083] The emotion fluctuation database contains a mapping relationship between the direction of speed changes and emotion labels, primarily corresponding to positive and negative changes. When a user is emotionally agitated, their input speed and speech rate typically increase; conversely, when a user is emotionally depressed or frustrated, their input speed and speech rate typically decrease. Based on this common knowledge, a mapping relationship between the direction of speed changes and emotion labels is pre-established—this is the emotion fluctuation database. Positive changes correspond to at least one of anger, anxiety, excitement, and tension; negative changes correspond to at least one of frustration, sadness, hesitation, and repression.

[0084] A fluctuating emotion tag set refers to a collection of emotion tags found in an emotion fluctuation database that match the current direction of velocity change. At least one corresponding emotion tag is found in the emotion fluctuation database based on the direction of velocity change, and these emotion tags are then integrated to form the fluctuating emotion tag set.

[0085] If the change in speed is greater than the preset speed threshold, it indicates that the user may be in a state of emotional fluctuation and it is necessary to further determine the specific emotional type. At this time, the corresponding emotional fluctuation tag set should be found first.

[0086] If the change in speed is not greater than the preset speed threshold, it means that the user's current input behavior is within the normal range, and no additional operation will be performed.

[0087] Step 7020: If the text sentiment label falls into the fluctuating sentiment label set, the third confidence level corresponding to the text sentiment label is corrected according to the preset fluctuation correction range to obtain the text fluctuation confidence level, so as to update the third confidence level and output it.

[0088] The fluctuation correction amplitude refers to the increment by which the confidence level of the sentiment label is adjusted when the user's input speed is abnormal and the sentiment label matches the fluctuation sentiment label set. It is preset by those skilled in the art according to the actual application scenario, and preferably 0.1.

[0089] The text volatility confidence level refers to the updated value obtained after applying volatility correction to the third confidence level. The text volatility confidence level equals the third confidence level plus the volatility correction. When the value of the third confidence level plus the volatility correction exceeds 1, the text volatility confidence level is set to 1.

[0090] If the text sentiment label falls into the fluctuating sentiment label set, it means that the emotion expressed by the user in the current text is consistent with the emotional state reflected by the speed change. At this time, the third confidence score is double-validated and should be increased. The third confidence score is then corrected according to the fluctuation correction range to obtain the text fluctuation confidence score, and the third confidence score is updated.

[0091] If the text sentiment tag does not fall into the fluctuating sentiment tag set, it means that the emotion expressed by the user's current text is inconsistent with the emotional state reflected by the speed change, and no additional action will be taken in this case.

[0092] Step 7021: If the voice emotion label falls into the fluctuating emotion label set, then the fourth confidence level corresponding to the voice emotion label is corrected according to the preset fluctuation correction range to obtain the voice fluctuation confidence level, so as to update the fourth confidence level and output it.

[0093] Speech fluctuation confidence level refers to the updated value obtained after applying a fluctuation correction margin to the fourth confidence level. Speech fluctuation confidence level equals the fourth confidence level plus the fluctuation correction margin. When the value of the fourth confidence level plus the fluctuation correction margin exceeds 1, the speech fluctuation confidence level is set to 1.

[0094] If the voice emotion label falls into the fluctuating emotion label set, it means that the emotion expressed by the user's current voice is consistent with the emotional state reflected by the speed change. At this time, the fourth confidence score is double-validated and should be increased. The fourth confidence score is then corrected according to the fluctuation correction range to obtain the voice fluctuation confidence score, and the fourth confidence score is updated.

[0095] If the voice emotion tag does not fall into the fluctuating emotion tag set, it means that the emotion expressed by the user's current voice is inconsistent with the emotional state reflected by the speed change, and no additional action will be taken.

[0096] Methods for obtaining external sentiment data by performing sentiment analysis on external documents include: Step 40: Perform sentiment analysis on external documents to extract the personal sentiment data of each party.

[0097] Personal sentiment data refers to a collection of raw information extracted from external documents related to the emotions of a specific individual. This includes sentiment words, descriptions of the emotional event, and behavioral descriptions made by the individual related to the current event. The method for obtaining personal sentiment data is the same as that for obtaining external sentiment data in step 4, except that personal sentiment data only covers a single individual.

[0098] Step 41: Extract keywords from personal emotional data to obtain personal keywords.

[0099] Personal keywords refer to words that represent the core elements of the individual's emotional expression. The method for obtaining personal keywords is the same as that for obtaining logical keywords in step 70, the difference being that the source of extraction is personal emotional data.

[0100] Step 42: Perform semantic analysis on personal keywords and classify them to obtain multiple sets of keywords of the same type.

[0101] A set of keywords of the same type refers to a set of keywords belonging to the same semantic category after being categorized according to semantic similarity among multiple parties. Using the personal keywords of each party as input, a word vector-based clustering algorithm is employed to group semantically similar or identical personal keywords into the same set; each set is thus a set of keywords of the same type. The clustering algorithm used is K-means clustering, and the number of clusters is preset by those skilled in the art.

[0102] Step 43: Count the number of personal keywords in each set of similar keywords, filter out the set of similar keywords with the largest number of personal keywords, and output the personal sentiment data corresponding to the set of similar keywords with the largest number of personal keywords as external sentiment data.

[0103] The number of personal keywords refers to the total number of personal keywords contained in each set of keywords of the same type. It is obtained by counting the number of personal keywords contained in each set of keywords of the same type.

[0104] The personal sentiment data corresponding to the set of similar keywords with the largest number of personal keywords represents the emotional tendency of most individuals and is used as external sentiment data output.

[0105] Methods for obtaining external sentiment data by performing sentiment analysis on external documents also include: Step 400: Perform semantic analysis on historical dialogue data to extract historical modal particles, and perform semantic analysis on external documents to extract external modal particles.

[0106] Historical modal particles refer to a set of modal particles that reflect the user's habitual tone of voice. The extraction method for historical modal particles is the same as the matching method for the modal particle list in step 2. The words in the historical dialogue text after word segmentation are matched sequentially with the modal particle list to obtain the historical modal particles.

[0107] External modal particles refer to a set of modal particles that reflect the tone and expression habits of various parties involved. The extraction method for external modal particles is the same as that for historical modal particles.

[0108] Step 401: Calculate the historical tone features based on historical modal particles, and calculate the external tone features of each party based on external modal particles.

[0109] Historical tone features refer to the feature vector of users' tone expression habits. The total frequency of occurrence of each of the three types of tone particles—sentence-final, sentence-intermediate, and exclamatory—is statistically analyzed. These three statistical values ​​are combined into a three-dimensional feature vector, which serves as the historical tone feature.

[0110] External tone features refer to the feature vectors of each party's tone expression habits. The method for obtaining external tone features is the same as that for obtaining historical tone features; external tone features are obtained by statistically analyzing the external tone particles of each party in the external document.

[0111] Step 402: Calculate the tone similarity with the external tone features of each party based on the historical tone features.

[0112] Tone similarity refers to the quantitative value of the closeness between historical tone features and external tone features. It is calculated by performing cosine similarity calculation on the three-dimensional feature vectors of historical tone features and external tone features, and then dividing the dot product of the two vectors by the product of their magnitudes. The result is the tone similarity.

[0113] Step 403: Select the external tone features with the highest tone similarity and define them as similar tone features. Perform sentiment analysis on the external documents corresponding to the similar tone features to obtain external sentiment data.

[0114] By comparing the similarity of different tones, the external tone feature with the highest tone similarity is selected, and the external document corresponding to the external tone feature is subjected to sentiment analysis to obtain external sentiment data.

[0115] This includes a correction method for obtaining a first confidence level and a second confidence level by performing sentiment polarity analysis on text data and speech data based on a preset sentiment discrimination rule base. This method includes: Step 50: Perform semantic integrity analysis on the text data to obtain text integrity, and perform speech intelligibility analysis on the speech data to obtain speech intelligibility.

[0116] Text completeness is a quantitative metric that measures whether the semantic information contained in user-input text fully expresses an event. It involves extracting event elements from text data to identify key elements describing the event. These key elements include at least one of the following: subject, action, object, time, location, and cause. The number of identified key elements is compared to the pre-defined total number of key elements a complete event should contain. The ratio of the identified elements to the pre-defined total number of elements is the text completeness. The pre-defined total number of elements is preset by those skilled in the art based on the event type; for example, a complete event might be defined as including subject, action, and object. When only subject and action elements are identified in the text, the text completeness is 2 divided by 3, which equals 0.67.

[0117] Speech proficiency is a quantitative indicator that measures the clarity of a user's input speech signal. In the process of speech-to-text recognition, the acoustic model and language model calculate a recognition confidence score while decoding the output text sequence. This confidence score represents the degree to which the speech-to-text recognition result is correctly converted, and is directly used as the speech proficiency score.

[0118] Step 510: If the text completeness is not lower than the preset standard completeness, then output the first confidence level.

[0119] Standard completeness refers to the standard value for judging whether the semantic information of text is complete. It is preset by those skilled in the art according to the actual application scenario, and is preferably 0.6.

[0120] If the text completeness is not lower than the preset standard completeness, it means that the semantic information of the text input by the current user is complete, and the first confidence level is output directly.

[0121] Step 511: If the text completeness is lower than the preset standard completeness, calculate the completeness ratio by comparing the text completeness with the standard completeness, calculate the text completeness confidence based on the completeness ratio and the first confidence, update the first confidence and output it.

[0122] The completeness ratio is the ratio of text completeness to standard completeness, obtained by dividing text completeness by standard completeness.

[0123] The text completeness confidence score is the confidence score adjusted by the completeness ratio from the initial confidence score. The text completeness confidence score equals the initial confidence score multiplied by the completeness ratio.

[0124] If the text completeness is lower than the preset standard completeness, it means that the semantic information of the text input by the current user is incomplete. At this time, the first confidence score needs to be corrected. Therefore, the text completeness confidence score is calculated based on the completeness ratio and the first confidence score, and is output instead of the first confidence score.

[0125] Step 520: When the speech clarity is not lower than the preset standard clarity, the second confidence level is output.

[0126] Standard clarity refers to the standard value for judging whether a speech signal is clear and usable. It is preset by those skilled in the art based on the actual application scenario, for example, 0.5.

[0127] When the voice clarity is not lower than the preset standard clarity, it means that the current user's voice signal is clear and usable, and the second confidence level is output directly.

[0128] Step 521: When the speech clarity is lower than the preset standard clarity, the speech clarity is compared with the standard clarity to calculate the clarity ratio. The speech clarity confidence is calculated based on the clarity ratio and the second confidence to update the second confidence and output it.

[0129] The intelligibility ratio is the ratio of speech intelligibility to standard intelligibility, obtained by dividing speech intelligibility by standard intelligibility.

[0130] Speech clarity confidence level refers to the confidence level after adjusting the second confidence level by the clarity ratio. Speech clarity confidence level equals the second confidence level multiplied by the clarity ratio.

[0131] When the speech clarity is lower than the preset standard clarity, it indicates that the confidence of the speech-to-text recognition result is low. In this case, the second confidence level needs to be corrected. Therefore, the speech clarity confidence level is calculated based on the clarity ratio and the second confidence level and is output instead of the second confidence level.

[0132] This includes a correction method for obtaining a first confidence level and a second confidence level by performing sentiment polarity analysis on text data and speech data based on a preset sentiment discrimination rule base. This method further includes: Step 500: Extract subjective and objective keywords from the text data, and extract subjective and objective keywords from the speech data.

[0133] Subjective keywords in text refer to words in text data that express a user's subjective viewpoint and personal feelings. These can include first-person pronouns, subjective evaluation words, sentiment words, and opinion verbs. The text data is segmented, and the segmentation results are sequentially matched against a pre-defined subjective keyword list. The matched words are then used as the text's subjective keywords. The subjective keyword list is pre-set by those skilled in the art and stores first-person pronouns, subjective evaluation words, sentiment words, and opinion verbs. First-person pronouns include words like "I" and "we"; subjective evaluation words include words like "I think," "I believe," and "I feel"; sentiment words include words like "like," "dislike," and "happy"; and opinion verbs include words like "hope," "suggest," and "believe."

[0134] Objective keywords in text refer to words that express objective facts and statements, which may include factual element words, third-person pronouns, and objective declarative words. The text data is segmented and named entity recognition is performed. Specific numbers, dates, times, locations, names of people, and names of organizations identified are extracted as factual element words. The segmentation results are matched against pre-defined lists of third-person pronouns and objective declarative words, and the matched words are extracted as third-person pronouns and objective declarative words, respectively. The extracted results are then combined to form the objective keywords in the text. Both the third-person pronoun and objective declarative word lists are pre-set by those skilled in the art. Third-person pronouns include words such as "he," "she," and "they"; objective declarative words include words such as "according to," "shows," and "indicates."

[0135] Voice subjective keywords refer to words in the speech-to-text recognition results that express the user's subjective opinions and personal feelings. The method for obtaining voice subjective keywords is the same as for text subjective keywords; they can be extracted from the speech-to-text recognition results.

[0136] Objective keywords in speech data refer to words that represent objective facts and statements in the speech-to-text recognition results. The types and acquisition methods of objective keywords in speech data are the same as those for objective keywords in text; they can be extracted from the speech-to-text recognition results.

[0137] Step 501: Count the number of subjective keywords in the text to obtain the subjective quantity of the text, and count the number of objective keywords in the text to obtain the objective quantity of the text.

[0138] The subjective count refers to the total number of subjective keywords in the text, while the objective count refers to the total number of objective keywords. Both are obtained by counting the corresponding keywords. Both the subjective and objective counts are integers greater than or equal to zero.

[0139] Step 502: Count the number of subjective keywords in the speech to obtain the subjective speech quantity, and count the number of objective keywords in the speech to obtain the objective speech quantity.

[0140] The subjective quantity of speech refers to the total number of subjective keywords in speech, while the objective quantity of speech refers to the total number of objective keywords in speech, obtained by counting the corresponding keywords. Both the subjective and objective quantities of speech are integers greater than or equal to zero.

[0141] Step 503: Calculate the first subjective ratio based on the subjective quantity and the objective quantity of the text.

[0142] The first subjective ratio refers to the result obtained by dividing the subjective quantity of the text by the sum of the subjective quantity and the objective quantity of the text.

[0143] Step 504: Calculate the second subjective ratio based on the subjective quantity and the objective quantity of speech.

[0144] The second subjective ratio refers to the result obtained by dividing the subjective quantity of speech by the sum of the subjective quantity of speech and the objective quantity of speech.

[0145] Step 505: If the first subjective ratio is higher than the preset subjective threshold, the text subjective confidence is calculated based on the first subjective ratio and the first confidence, and the first confidence is updated and output.

[0146] The subjective threshold refers to the critical ratio for judging whether the text expression has a significant subjective bias. It is preset by those skilled in the art based on the actual application scenario, and is preferably 0.6.

[0147] The subjective confidence level of a text refers to the confidence level after adjusting the first confidence level using the first subjective ratio. The subjective confidence level of a text is equal to the first confidence level multiplied by the first subjective ratio.

[0148] If the first subjective ratio is higher than the preset subjective threshold, it means that the proportion of subjective keywords in the current text data is significantly higher than the proportion of objective keywords, and the user's expression is mostly subjective. The credibility of such expression is relatively low compared with the statement of objective facts. Therefore, the first confidence score is reduced by the first subjective ratio to obtain the text subjective confidence score, so as to update the first confidence score and output it.

[0149] If the first subjective ratio is not higher than the preset subjective threshold, it means that the ratio of subjective keywords to objective keywords in the current text data is within the normal range, and the user's expression is normal. No additional operation will be performed at this time.

[0150] Step 506: If the second subjective ratio is higher than the preset subjective threshold, calculate the speech subjective confidence based on the second subjective ratio and the second confidence, update the second confidence and output it.

[0151] Speech subjective confidence refers to the confidence level after adjusting the second confidence level using a second subjective ratio. Speech subjective confidence is equal to the second confidence level multiplied by the second subjective ratio.

[0152] If the second subjective ratio is higher than the preset subjective threshold, it indicates that the proportion of subjective keywords in the speech-to-text recognition results of the current speech data is significantly higher than the proportion of objective keywords, and the user's expression is mostly subjective. The credibility of such expressions is relatively low compared to the objective factual statements. Therefore, the second confidence is reduced by the second subjective ratio to obtain the speech subjective confidence, so as to update the second confidence and output it.

[0153] If the second subjective ratio is not higher than the preset subjective threshold, it means that the ratio of subjective keywords to objective keywords in the speech-to-text recognition results of the current speech data is within the normal range, and the user's expression is normal. No additional operation is required at this time.

[0154] This also includes: Step 53: Extract behavior description information based on global context information. Behavior description information includes the agent and the recipient.

[0155] Behavioral description information refers to a structured collection of information representing the behavioral interactions between various entities within a global context, including two elements: the actor and the receiver. The actor is the entity that performs the action in the event. The receiver is the entity that is affected by the action or is the object of the action.

[0156] Sentences within the global context are labeled with their semantic roles to identify statements or sentence fragments describing events or actions. Semantic roles include agent, receiver, and action. Information labeled as agent and receiver is extracted from the labeling results, and the extracted results are combined into action description information. For example, in the sentence "The company fired me," the company is the agent, and I am the receiver. Semantic role labeling is an existing technology in the field of natural language processing and can be implemented using methods based on pre-trained language models.

[0157] Step 54: Determine the user's identity and stance based on the agent, the recipient, and the preset stance determination rules.

[0158] Position determination rules refer to a set of rules for determining a user's role in an event based on the actor and the recipient, and are pre-defined by those skilled in the art. Specifically, they include four rules: when the actor is a user and the recipient is another party, the user's position is determined to be that of the active party; when the actor is another party and the recipient is a user, the user's position is determined to be that of the passive party; when both the actor and the recipient are users, the user's position is determined to be that of the self-referential party; and when the user is neither the actor nor the recipient, the user's position is determined to be that of the bystander.

[0159] Identity stance refers to the behavioral role a user plays in the described event, including active party, passive party, self-referential party, and bystander. The agent and the recipient are compared with the user's identity, and the user's identity stance is determined according to the four determination rules mentioned above.

[0160] Step 55: Find the corresponding directional emotion tag based on the user's identity and stance and the preset stance emotion database.

[0161] The positional emotion database contains a mapping relationship between identity positions and directional emotion tags, pre-defined by those skilled in the art based on common knowledge in psychology and linguistics. Directional emotion tags are emotion category tags that indicate the direction in which a user's emotion is directed, including reflexive emotion tags, externally referential emotion tags, introspective emotion tags, and neutral emotion tags. Specifically, the active party corresponds to reflexive emotion tags; the passive party corresponds to externally referential emotion tags; the self-referential party corresponds to introspective emotion tags; and the bystander corresponds to neutral emotion tags. Reflexive emotion tags indicate emotions directed inward and with a tendency for self-reflection or self-attribution, including self-blame, guilt, and shame; externally referential emotion tags indicate emotions directed towards others or external factors, including anger, resentment, and dissatisfaction; introspective emotion tags indicate emotions directed inward and with a tendency for self-reflection, including thinking and introspection; and neutral emotion tags indicate emotions without a clear target, including calmness, objectivity, and rationality. Searching the positional emotion database based on a user's identity position will yield the corresponding directional emotion tag.

[0162] Step 56: Add the directional emotion tags to the text emotion tag and the voice emotion tag respectively to update the text emotion tag and the voice emotion tag.

[0163] The directional emotion tags are combined with the text emotion tags and voice emotion tags as additional dimensions to update the text emotion tags and voice emotion tags. For example, when the text emotion tag is anger and the directional emotion tag is outward-pointing, the updated text emotion tag is anger (outward-pointing), indicating that the user's anger is directed towards an external object, rather than towards themselves. This update adds an emotion-direction dimension to the original emotion category, making emotion recognition more accurate.

[0164] This also includes a method for obtaining external sentiment tags, which includes: Step 410: Identify the people around the user from the historical dialogue data and extract the surrounding historical data of each person.

[0165] "People in the surrounding area" refers to other individuals with whom the user has a social relationship, including family, friends, colleagues, and neighbors, mentioned in historical conversations. Semantic analysis is performed on historical conversation data to identify interpersonal relationship expressions, including kinship terms, social relationship titles, names / nicknames, and referential expressions. Word segmentation and named entity recognition are then performed on the historical conversation data, and the objects corresponding to the identified interpersonal relationship expressions are designated as "people in the surrounding area." The same person in the surrounding area may appear in different forms in different conversation rounds; referential resolution techniques are used to merge different expressions into the same person in the surrounding area.

[0166] Surrounding historical data refers to the collection of all dialogue content involving a particular person in the surrounding area within historical dialogue data. Using the entity name corresponding to the person in the surrounding area as a keyword, a matching search is performed in the historical dialogue data to extract statements or paragraphs containing that keyword, which are then aggregated into the surrounding historical data of that person.

[0167] Step 411: Based on the preset sentiment discrimination rule base, perform sentiment polarity analysis on the surrounding historical data to obtain the surrounding sentiment labels.

[0168] Peripheral sentiment labels refer to the sentiment category labels of a person's emotional inclination towards a current event. The historical data of each person's surroundings are input into the sentiment discrimination rule base. Using the same sentiment polarity analysis method as in step 5 (text sentiment labels), sentiment polarity analysis is performed on the historical data of each person's surroundings, outputting the corresponding sentiment category label as the peripheral sentiment label. Each person in the surroundings corresponds to one peripheral sentiment label.

[0169] Step 412: Filter out the most frequently occurring surrounding sentiment tags from the surrounding sentiment tags and define them as the mainstream sentiment tags.

[0170] The mainstream sentiment tag refers to the sentiment category tag that appears most frequently among all surrounding sentiment tags. It is obtained by statistically analyzing the frequency of occurrence of all surrounding sentiment tags.

[0171] Step 413: Output mainstream sentiment tags as external sentiment tags.

[0172] Mainstream sentiment tags can be used as external reference benchmarks to calculate similarity with text sentiment tags and voice sentiment tags, and the user's true emotional state can be determined by the general emotional reactions of people around them.

[0173] Based on the same inventive concept, embodiments of the present invention provide an AI virtual human chat content analysis and user emotion recognition system.

[0174] Reference Figure 3 An AI virtual human chat content analysis and user emotion recognition system includes: The acquisition module is used to acquire initial text, initial speech, historical dialogue data, historical input speed, and current input speed; The memory stores a computer program that can be loaded and executed by a processor to perform an AI virtual human chat content analysis and user emotion recognition method. The processor loads and executes programs from memory.

[0175] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for analyzing chat content and recognizing user emotions in an AI virtual human, characterized in that, include: Step 1: Obtain the initial text, initial speech, and historical dialogue data of the current conversation input by the user; Step 2: Perform text preprocessing on the initial text to obtain text data, and extract speech-text and tone features from the initial speech to obtain speech data; Step 3: Retrieve global context information from historical dialogue data based on text and voice data; Step 4: Extract event keywords based on text data, voice data, and global context information; search for external documents online based on event keywords; and perform sentiment analysis on the external documents to obtain external sentiment data. Step 5: Based on the preset sentiment discrimination rule base, perform sentiment polarity analysis on the text data, speech data and external sentiment data respectively to obtain the text sentiment label and the corresponding first confidence level, the speech sentiment label and the corresponding second confidence level, and the external sentiment label; Step 6: If the text sentiment label and the voice sentiment label are inconsistent, use the external sentiment label as a benchmark to calculate the first similarity between the text sentiment label and the external sentiment label, and calculate the second similarity between the voice sentiment label and the external sentiment label. Step 7: Correct the first confidence level based on the first similarity to obtain the third confidence level, and correct the second confidence level based on the second similarity to obtain the fourth confidence level; Step 8: Compare the third confidence level and the fourth confidence level to obtain the final confidence level with the higher value, and output the sentiment label and the corresponding confidence level of the final confidence level.

2. The method for AI virtual human chat content analysis and user emotion recognition according to claim 1, characterized in that, The methods for obtaining a third confidence level by correcting the first confidence level based on the first similarity level, and obtaining a fourth confidence level by correcting the second confidence level based on the second similarity level, include: Step 70: Perform semantic analysis on the text and voice data to obtain logical keywords and the corresponding number of keywords; Step 71: Perform a coherence judgment on adjacent logical keywords to extract non-coherent keywords and their corresponding number of non-coherent keywords. Step 72: Compare the number of incoherent words with the number of keywords to obtain the incoherence ratio; Step 73: If the incoherence ratio is not greater than the preset incoherence threshold, then the first confidence level is corrected according to the preset correction weight and the first similarity to obtain the third confidence level, and the second confidence level is corrected according to the preset correction weight and the second similarity to obtain the fourth confidence level. Step 74: If the incoherence ratio is greater than the preset incoherence threshold, then increase the preset correction weight to the preset incoherence weight. Step 75: Correct the first confidence level according to the non-continuous weight and the first similarity to obtain the third confidence level, and correct the second confidence level according to the non-continuous weight and the second similarity to obtain the fourth confidence level.

3. The method for AI virtual human chat content analysis and user emotion recognition according to claim 2, characterized in that, It also includes a correction method for obtaining a third confidence level by correcting the first confidence level based on the first similarity, and obtaining a fourth confidence level by correcting the second confidence level based on the second similarity. This method includes: Step 700: Obtain the user's historical input speed and the current input speed of the current conversation; Step 701: Compare the current input speed with the historical input speed, and calculate the speed change and the direction of speed change; Step 702: If the change in speed is greater than the preset speed threshold, search for the corresponding emotional fluctuation tag set in the preset emotional fluctuation library according to the direction of the speed change; Step 7020: If the text sentiment tag falls into the fluctuating sentiment tag set, then the third confidence level corresponding to the text sentiment tag is corrected according to the preset fluctuation correction range to obtain the text fluctuation confidence level, so as to update the third confidence level and output it. Step 7021: If the voice emotion label falls into the fluctuating emotion label set, then the fourth confidence level corresponding to the voice emotion label is corrected according to the preset fluctuation correction range to obtain the voice fluctuation confidence level, so as to update the fourth confidence level and output it.

4. The method for AI virtual human chat content analysis and user emotion recognition according to claim 1, characterized in that, Methods for obtaining external sentiment data by performing sentiment analysis on external documents include: Step 40: Perform sentiment analysis on external documents to extract individual sentiment data from each party. Step 41: Extract keywords from personal emotional data to obtain personal keywords; Step 42: Perform semantic analysis on personal keywords and classify them to obtain multiple sets of keywords of the same type; Step 43: Count the number of personal keywords in each set of similar keywords, filter out the set of similar keywords with the largest number of personal keywords, and output the personal sentiment data corresponding to the set of similar keywords with the largest number of personal keywords as external sentiment data.

5. The method for AI virtual human chat content analysis and user emotion recognition according to claim 4, characterized in that, Methods for obtaining external sentiment data by performing sentiment analysis on external documents also include: Step 400: Perform semantic analysis on historical dialogue data to extract historical modal particles, and perform semantic analysis on external documents to extract external modal particles; Step 401: Statistically determine the historical tone characteristics based on historical modal particles, and statistically determine the external tone characteristics of each party based on external modal particles; Step 402: Calculate the tone similarity with the external tone features of each party based on historical tone features; Step 403: Select the external tone features with the highest tone similarity and define them as similar tone features. Perform sentiment analysis on the external documents corresponding to the similar tone features to obtain external sentiment data.

6. The method for AI virtual human chat content analysis and user emotion recognition according to claim 3, characterized in that, It also includes a correction method for obtaining a first confidence level and a second confidence level by performing sentiment polarity analysis on text data and speech data based on a preset sentiment discrimination rule base, respectively. This method includes: Step 50: Perform semantic integrity analysis on the text data to obtain text integrity, and perform speech intelligibility analysis on the speech data to obtain speech intelligibility; Step 510: If the text completeness is not lower than the preset standard completeness, then output the first confidence level; Step 511: If the text completeness is lower than the preset standard completeness, calculate the completeness ratio by comparing the text completeness with the standard completeness, calculate the text completeness confidence based on the completeness ratio and the first confidence, update the first confidence and output it; Step 520: When the speech clarity is not lower than the preset standard clarity, output the second confidence level; Step 521: When the speech clarity is lower than the preset standard clarity, the speech clarity is compared with the standard clarity to calculate the clarity ratio. The speech clarity confidence is calculated based on the clarity ratio and the second confidence to update the second confidence and output it.

7. The method for AI virtual human chat content analysis and user emotion recognition according to claim 6, characterized in that, It also includes a correction method for obtaining a first confidence level and a second confidence level by performing sentiment polarity analysis on text data and speech data based on a preset sentiment discrimination rule base, and this method further includes: Step 500: Extract subjective and objective keywords from the text data, and extract subjective and objective keywords from the speech data; Step 501: Count the number of subjective keywords in the text to obtain the subjective quantity of the text, and count the number of objective keywords in the text to obtain the objective quantity of the text; Step 502: Count the number of subjective keywords in the speech to obtain the subjective speech count, and count the number of objective keywords in the speech to obtain the objective speech count; Step 503: Calculate the first subjective ratio based on the subjective quantity and the objective quantity of the text; Step 504: Calculate the second subjective ratio based on the subjective and objective quantities of speech. Step 505: If the first subjective ratio is higher than the preset subjective threshold, calculate the text subjective confidence based on the first subjective ratio and the first confidence, update the first confidence and output it; Step 506: If the second subjective ratio is higher than the preset subjective threshold, calculate the speech subjective confidence based on the second subjective ratio and the second confidence, update the second confidence and output it.

8. The method for AI virtual human chat content analysis and user emotion recognition according to claim 1, characterized in that, Also includes: Step 53: Extract behavior description information based on global context information, wherein the behavior description information includes the agent and the recipient; Step 54: Determine the user's identity and stance based on the agent, the recipient, and the preset stance determination rules; Step 55: Based on the user's identity and stance, and the preset stance sentiment database, find the corresponding directional sentiment tag; Step 56: Add the directional emotion tags to the text emotion tag and the voice emotion tag respectively to update the text emotion tag and the voice emotion tag.

9. The method for AI virtual human chat content analysis and user emotion recognition according to claim 5, characterized in that, It also includes methods for obtaining external sentiment tags, which include: Step 410: Identify the people around the user from the historical dialogue data and extract the surrounding historical data of each person; Step 411: Based on the preset sentiment discrimination rule base, perform sentiment polarity analysis on the surrounding historical data to obtain the surrounding sentiment labels; Step 412: Filter out the most frequently occurring surrounding sentiment tags from the surrounding sentiment tags and define them as the mainstream sentiment tags; Step 413: Output mainstream sentiment tags as external sentiment tags.

10. An AI virtual human chat content analysis and user emotion recognition system, characterized in that, include: The acquisition module is used to acquire initial text, initial speech, historical dialogue data, historical input speed, and current input speed; A memory for storing a program for an AI virtual human chat content analysis and user emotion recognition method as described in any one of claims 1 to 9; The processor loads and executes programs from memory.

Images