Robot multi-turn dialogue control method based on emotional large language model

Abstract

The present application relates to the technical field of speech processing, in particular to a robot multi-round dialogue control method based on an emotion large language model, comprising: determining text semantic similarity and emotion consistency features of each round of dialogue, fusing the two aspects of data to obtain the matching degree of each round of dialogue, dividing all rounds of dialogue into multiple emotion segments according to the emotion similarity, obtaining user emotion reliable features of the emotion segment from the matching degree contained by the emotion segment; obtaining emotion transition fluency features of the emotion segment according to the intention consistency features and emotion mutation features of the emotion segment; fusing the user emotion reliable features and emotion transition fluency features of all emotion segments to obtain robot multi-round dialogue quality results. The present application simultaneously performs microscopic and macroscopic analysis, improves the accuracy of obtaining the dialogue quality of the current multi-round dialogue, so as to be able to determine abnormal conditions occurring in the multi-round dialogue according to the dialogue quality, and finally improve the user experience.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, specifically to a method for controlling multi-turn dialogue in robots based on an emotional large language model. Background Technology

[0002] In scenarios requiring emotional resonance and expression, such as customer service, emotional support, and psychological counseling, users typically engage in multi-turn dialogues with the chatbot, involving multiple rounds of interaction and prolonged contextual dependence, rather than single-turn question-and-answer sessions. Traditional dialogue systems based on large-scale emotional language models can only achieve single-turn emotion recognition and response generation, lacking multi-turn contextual emotion tracking. This makes it impossible to accurately assess the dialogue quality of each round, thus failing to identify issues such as poor emotion matching and unnatural communication, ultimately degrading the user experience. Summary of the Invention

[0003] To address the technical problem of poor accuracy in acquiring the quality of multi-turn dialogue in robots, the present invention aims to provide a robot multi-turn dialogue control method based on an emotional large language model. The specific technical solution adopted is as follows: In a first aspect of the present invention, a method for controlling multi-turn dialogue in a robot based on an emotion-based large language model is provided, comprising: Determine the textual semantic similarity and emotional consistency features of the speech pairs in each round of dialogue; the speech pairs consist of user-input speech and robot-output speech; By integrating the text semantic similarity and emotion consistency features, the matching degree of each round of dialogue is obtained; Based on similar emotional states, all rounds of dialogue are divided into multiple emotional segments; The reliable user emotion characteristics of the emotion segment are obtained from the degree of matching contained in the emotion segment; According to the meaning of the emotional segment Figure 1 Based on the characteristics of emotional transition and the characteristics of abrupt changes in emotion, the smoothness of the emotional transition in the emotional segment is obtained; the meaning... Figure 1 The emotional signature is obtained from the similarity of keywords in two adjacent rounds of dialogue in the emotional segment, and the emotional change feature is obtained from the difference in the emotional consistency feature of two adjacent emotional segments; By integrating reliable user emotion features and smooth emotion transition features across all emotion segments, the quality results of multi-turn dialogues with the robot are obtained.

[0004] In an exemplary embodiment, the process of obtaining the emotion consistency feature includes: Determine the number of various emotion words contained in the user input speech and the robot output speech, respectively; Based on the difference in the proportion of the same emotion word in the user's input speech and the robot's output speech, an expression consistency index is determined for various emotion words; the expression consistency index is inversely correlated with the difference in the proportion. The dominant emotion is determined by the maximum number of various emotion words. The emotion consistency feature is obtained based on the proportion of emotion words corresponding to the main emotion and the expression consistency index corresponding to the main emotion.

[0005] In an exemplary embodiment, dividing all rounds of dialogue into multiple emotion segments based on emotional similarity includes: each round of dialogue with the same main emotion in consecutive adjacent rounds constitutes an emotion segment.

[0006] In an exemplary embodiment, the process of obtaining the reliable characteristics of user emotions includes: Determine the overall level and fluctuation of the degree of matching contained in the emotional segment; The reliable characteristics of user sentiment are obtained from the overall level and the degree of fluctuation; the reliable characteristics of user sentiment are positively correlated with the overall level and negatively correlated with the degree of fluctuation.

[0007] In one exemplary embodiment, the intention Figure 1 The process of obtaining the features includes: Keyword vectors are determined for each keyword in the first candidate round dialogue and the second candidate round dialogue respectively; the first candidate round dialogue and the second candidate round dialogue constitute the two adjacent round dialogues. The set of matching keywords for the two adjacent rounds of dialogue is determined by the keyword vector similarity between each keyword in the first candidate round of dialogue and each keyword in the second candidate round of dialogue. The meaning of an emotional segment is obtained by analyzing the number of matching keywords in the matching keyword set of each two adjacent rounds of dialogue within that segment. Figure 1 Characteristics.

[0008] In an exemplary embodiment, the process of obtaining the matching keyword set includes: Compare the keyword vector similarity between each keyword in the first round of dialogue and each keyword in the second round of dialogue with the preset vector similarity threshold. Two keywords whose keyword vector similarity exceeds the preset vector similarity threshold are identified to obtain the matching keyword set.

[0009] In an exemplary embodiment, the meaning of the emotional segment is obtained from the number of matching keywords in the matching keyword set of each adjacent two rounds of dialogue within the emotional segment. Figure 1 Characteristics, including: Calculate the average percentage of matching keywords in each of two adjacent rounds of dialogue to obtain the intended meaning. Figure 1 Characteristics.

[0010] In an exemplary embodiment, the process of obtaining the emotion mutation feature includes: The absolute value of the difference between the emotional consistency feature of the emotional segment and its adjacent previous emotional segment is determined, and the emotional abrupt change feature of the emotional segment is obtained from the absolute value of the difference.

[0011] In an exemplary embodiment, the fusion of reliable user emotion features and smooth emotion transition features across all emotion segments to obtain the robot's multi-turn dialogue quality results includes: Based on the user's emotional reliability characteristics and emotional transition fluency characteristics for each emotional segment, the dialogue quality characteristics for each emotional segment are obtained; the dialogue quality characteristics are positively correlated with both the user's emotional reliability characteristics and emotional transition fluency characteristics. By integrating the dialogue quality features of all emotional segments, a multi-turn dialogue quality index for the robot is obtained, which characterizes the quality result of the robot's multi-turn dialogue.

[0012] In an exemplary embodiment, the robot multi-turn dialogue control method based on an emotion-based large language model further includes: Compare the robot's multi-turn dialogue quality index with a preset quality threshold: If the robot's multi-turn dialogue quality index is greater than or equal to the preset quality threshold, a robot multi-turn dialogue quality qualified instruction signal is output; if the robot's multi-turn dialogue quality index is less than the preset quality threshold, a robot multi-turn dialogue quality unqualified instruction signal is output.

[0013] In a second aspect of the present invention, a robot multi-turn dialogue control system based on an emotional big language model is provided, comprising: a memory and a processor; the memory is connected to the processor; the memory is used to store program instructions; the processor is used to implement the above-described robot multi-turn dialogue control method based on an emotional big language model when the program instructions are executed.

[0014] This invention offers the following advantages: By simultaneously considering the textual and emotional features of each round of dialogue, the matching degree of each round is obtained, achieving a comprehensive quantitative evaluation of single-round dialogues in both content and emotion dimensions; by dividing multi-round dialogues into multiple emotion segments according to the law of emotional evolution, structured processing of long-term contextual dialogues is achieved, thus facilitating more targeted data processing subsequently; by evaluating the reliability of user emotions within an emotion segment through the matching degree of all rounds of dialogue within that segment, the invention accurately reflects whether the robot's response to user emotions is consistently reliable during a relatively stable emotional exchange process; Figure 1 The semantic features ensure the coherence of the dialogue, while the emotion abrupt change features characterize drastic changes in emotion during the dialogue. This is crucial for recognizing sudden shifts in user emotions. Then, the meaning of the emotion segments is analyzed in a fusion manner. Figure 1 By combining reliable features and emotion transition features, we can obtain the smoothness features of emotion transitions in different emotion segments, enabling a comprehensive and in-depth assessment of the smoothness of transitions between different emotion stages in the dialogue. Finally, by integrating and analyzing the reliable features of user emotions and the smoothness features of emotion transitions in all emotion segments, we can obtain the quality of the robot's multi-turn dialogue, achieving simultaneous micro and macro analysis. This improves the accuracy of obtaining the dialogue quality of the current multi-turn dialogue, allowing us to identify anomalies in multi-turn dialogues based on the dialogue quality, and ultimately enhance the user experience. Attached Figure Description

[0015] Figure 1 This is a flowchart of a robot multi-turn dialogue control method based on an emotional large language model provided by the present invention; Figure 2 This is a flowchart of the process for obtaining emotion consistency features provided by the present invention. Detailed Implementation

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. All data and information collected in this application have been obtained with full consent.

[0017] This embodiment provides a robot multi-turn dialogue control method based on an emotional large language model. During multi-turn dialogues between a user and the robot, the quality of the current multi-turn dialogue is determined based on the previous rounds. This robot multi-turn dialogue control method is applicable to scenarios such as customer service, emotional support, and psychological counseling. In customer service, emotional support, and especially psychological counseling scenarios, users need to express their inner stress, anxiety, and depression through multiple rounds of continuous, slow, and deliberate expression. The user's expression may be intermittent, with long pauses, emotional choking, and gaps in thought. Furthermore, the entire interaction process relies on multi-turn context to accurately understand the user's psychological state. Therefore, continuous data collection of the user-robot multi-turn dialogues is necessary to provide an accurate, reliable, and time-series-clear data foundation for dialogue quality evaluation. Since multi-turn dialogues involve not only information transmission but also emotional exchange, intent understanding, and the continuity of user experience, evaluating the quality of multi-turn dialogues directly affects the robot's ability to understand users and the accuracy of its emotional responses.

[0018] Existing psychological counseling robots are all equipped with standardized audio acquisition modules. During multi-turn dialogues, the robot uses well-known voice activation technologies (such as wake-up words and automatic voice detection) to trigger the voice acquisition function in real time. When the user begins to speak, the acquisition module automatically starts, acquiring the user's input voice with low latency and simultaneously recording the robot's output voice. After each round of dialogue ends (e.g., determined by user voice pauses or semantic end detection technologies), acquisition automatically stops, ensuring a one-to-one correspondence between the user's input voice and the robot's output voice in each round of dialogue, forming a complete multi-turn voice dialogue dataset. Therefore, each round of dialogue consists of two parts: user input voice and robot output voice. Thus, the user's input voice and the robot's output voice in each round of dialogue constitute a voice pair for each round of dialogue.

[0019] like Figure 1 As shown, the multi-turn dialogue control method for this robot includes the following steps: Step S1: Determine the textual semantic similarity and emotional consistency features of the speech pairs in each round of dialogue; Step S2: Integrate text semantic similarity and sentiment consistency features to obtain the matching degree of each round of dialogue; Step S3: Based on the similarity of emotions, divide all rounds of dialogue into multiple emotional segments; Step S4: Based on the degree of matching contained in the emotion segment, obtain the reliable characteristics of user emotions in the emotion segment; Step S5: Based on the meaning of the emotional segment Figure 1 By analyzing the characteristics of emotional transition and the characteristics of abrupt changes in emotion, we can obtain the characteristics of smooth emotional transition in the emotional segment. Step S6: Integrate the reliable features of user emotions and the smoothness of emotion transitions across all emotion segments to obtain the quality results of the robot's multi-turn dialogue.

[0020] The following is a detailed explanation of each step.

[0021] Step S1: Determine the textual semantic similarity and emotional consistency features of the speech pairs in each round of dialogue.

[0022] While traditional dialogue systems can understand users' voice or text input to some extent, they often fail to effectively handle emotional fluctuations. Robots often simply respond to questions, ignoring emotional changes and causing emotional inconsistencies. For example, when a user expresses anxiety, the robot may still respond in a flat tone, failing to convey the necessary care and support. Therefore, this embodiment considers both the textual and emotional features of each round of dialogue to obtain the textual semantic similarity and emotional consistency features of the voice pairs in each round of dialogue.

[0023] In this embodiment, the c-th round of dialogue is taken as an arbitrary round. Text recognition is performed on both the user's input speech and the robot's output speech in the c-th round of dialogue to obtain the text of the user's input speech and the robot's output speech. Then, the semantic similarity between the text of the user's input speech and the robot's output speech is obtained. It should be understood that the semantic similarity of the text between two speech signals is obtained using existing technologies, such as: using pre-trained models (e.g., SentenceBERT, UniversalSentence Encoder, BERTflow) to encode the text of the user's input speech and the robot's output speech into fixed-dimensional semantic vectors, and then calculating the cosine similarity between these two semantic vectors. For ease of subsequent processing, the cosine similarity needs to be normalized as follows: (cosine similarity + 1) / 2. All cosine similarities mentioned below are normalized results. Alternatively, a cross-encoder (e.g., BERT's [CLS] with a classification head) can be used to directly concatenate the text of the user's input speech and the robot's output speech, outputting a similarity score. Another approach is to call a general-purpose large language model (e.g., GPT-4), inputting the text of the user's input speech and the robot's output speech, along with constructed prompts such as "Please evaluate the semantic similarity between the text of the user's input speech and the robot's output speech," into the general-purpose large language model, outputting the semantic similarity result. This yields the text semantic similarity of the speech pair in the c-th round of dialogue.

[0024] Keywords are extracted from the user input and robot output voice in the c-th round of dialogue. In this embodiment, the TF-IDF algorithm can be used for extraction. It should be understood that, in extreme cases, if the duration of the user input or robot output voice in the c-th round of dialogue is too short to obtain keywords, the c-th round of dialogue will be removed from the multi-round dialogue, and the order of each round of dialogue will be re-determined according to the time sequence.

[0025] Based on the keywords in the user's input speech and the robot's output speech in the c-th round of dialogue, the emotion consistency features of the c-th round of dialogue are obtained, such as... Figure 2 As shown, the following is a specific process for obtaining emotion consistency features: Step S11: Determine the number of various emotion words contained in the user's input speech and the robot's output speech, respectively.

[0026] Various emotion words are identified from the keywords in the user's input speech during the c-th round of dialogue. In this embodiment, emotion words are categorized into three types: positive emotion words, negative emotion words, and neutral emotion words. The keywords in the user's input speech during the c-th round of dialogue can be input into the HowNet emotion dictionary for emotion word recognition. Then, the number of each emotion word in the user's input speech during the c-th round of dialogue is determined. Similarly, various emotion words are identified from the keywords in the robot's output speech during the c-th round of dialogue. Then, the number of each emotion word in the robot's output speech during the c-th round of dialogue is determined.

[0027] Step S12: Based on the difference in the proportion of the same emotion word in the user's input speech and the robot's output speech, determine the consistency index for the expression of various emotion words.

[0028] For any given emotion word, taking the nth emotion word as an example: Calculate the proportion of the nth emotion word in the user's input speech during the c-th round of dialogue. Specifically: determine the number of keywords in the user's input speech during the c-th round of dialogue, and calculate the ratio of the number of the nth emotion word in the user's input speech during the c-th round of dialogue to the number of those keywords. This ratio is taken as the proportion of the nth emotion word in the user's input speech during the c-th round of dialogue. Similarly, calculate the proportion of the nth emotion word in the robot's output speech during the c-th round of dialogue. Specifically: determine the number of keywords in the robot's output speech during the c-th round of dialogue, and calculate the ratio of the number of the nth emotion word in the robot's output speech during the c-th round of dialogue to the number of those keywords. This ratio is taken as the proportion of the nth emotion word in the robot's output speech during the c-th round of dialogue.

[0029] The difference between the proportion of the nth emotion word in the user's input speech and its proportion in the robot's output speech in the c-th round of dialogue is determined. The smaller the difference, the more similar the emotional features of the user's input speech and the robot's output speech in the c-th round of dialogue for the nth emotion word, and the higher the expression consistency index. Therefore, the expression consistency index is inversely correlated with this difference in proportion. Based on this logical analysis, the following is a specific calculation method for the expression consistency index in the c-th round of dialogue for the nth emotion word: ; in, This represents the consistency index of expression in the c-th round of dialogue for the nth emotion word. This represents the number of the nth emotion word in the user's input speech during the c-th round of the conversation. This indicates the number of keywords in the user's voice input during the c-th round of the conversation. This represents the percentage of the nth emotion word in the user's input speech during the c-th round of the conversation. This represents the number of the nth emotion word in the robot's output speech during the cth round of dialogue. This represents the number of keywords in the robot's output speech during the c-th round of dialogue. This represents the percentage of the nth emotion word in the robot's output speech during the cth round of dialogue.

[0030] Specifically, if the number of keywords in the robot's output speech in the c-th round of dialogue is 0, that is, the denominator of the expression consistency index calculation formula is 0, then the expression consistency index is directly set to 1.

[0031] As can be seen from the above calculation method, if a certain emotion word does not appear in either the user's input voice or the robot's output voice, then the result of the expression consistency index for the c-th round of dialogue for that emotion word is 1, indicating that there is no difference in emotional expression for that emotion word.

[0032] Step S13: Determine the dominant emotion from the largest number of various emotion words.

[0033] In psychological counseling scenarios, although the emotional expression of users in each round of dialogue may contain multiple emotional words, the core is a certain dominant emotion (i.e., the primary emotion). If the user's primary emotion cannot be accurately captured, it will lead to biased emotional responses. For example, if the user's primary emotion is anxiety, but a bland response is received, it is necessary to further identify the user's primary emotion in each round of dialogue to avoid the ambiguity of emotional judgment caused by the mixing of multiple emotions.

[0034] Calculate the sum of the number of the nth emotion word in the user's input speech and the number of the nth emotion word in the robot's output speech during the c-th round of dialogue. This sum is used as the count of the nth emotion word in the c-th round of dialogue. This yields the count of all emotion words in the c-th round of dialogue.

[0035] Determine the maximum number of various emotion words in round c of the dialogue, and use the emotion type of the emotion word corresponding to the maximum number as the dominant emotion of round c. For example, if the emotion word corresponding to the maximum number is a positive emotion word, then the dominant emotion of round c is a positive emotion. If the number of various emotion words is equal and reaches the maximum value, determine the dominant emotion according to the priority of "negative-positive-neutral". If the number of all emotion words is 0, the dominant emotion defaults to "neutral".

[0036] After obtaining the dominant emotion of the c-th round of dialogue, determine the percentage of emotion words corresponding to that dominant emotion. Specifically: calculate the number of emotion words corresponding to the dominant emotion in the c-th round of dialogue; calculate the sum of the number of keywords in the user's input and the robot's output in the c-th round of dialogue (defined as the total number of keywords in the c-th round of dialogue); then calculate the ratio of the number of emotion words corresponding to the dominant emotion in the c-th round of dialogue to the total number of keywords in the c-th round of dialogue, and use this ratio as the percentage of emotion words corresponding to the dominant emotion in the c-th round of dialogue. For example: if the dominant emotion of the c-th round of dialogue is positive, then determine the sum of the number of positive emotion words in the user's input and the number of positive emotion words in the c-th round of dialogue, and use this sum as the number of positive emotion words in the c-th round of dialogue; then calculate the ratio of the number of positive emotion words in the c-th round of dialogue to the total number of keywords in the c-th round of dialogue, and use this ratio as the percentage of positive emotion words in the c-th round of dialogue.

[0037] From the consistency index of expression for various emotion words in the c-th round of dialogue obtained above, find the consistency index of expression corresponding to the main emotion. For example, if the main emotion of the c-th round of dialogue is a positive emotion, then determine the consistency index of expression for the c-th round of dialogue for positive emotion words.

[0038] Step S14: Obtain the emotion consistency feature based on the proportion of emotion words corresponding to the main emotion and the expression consistency index corresponding to the main emotion.

[0039] The higher the proportion of emotion words corresponding to the main emotion in the c-th round of dialogue, the stronger the emotional consistency feature of the c-th round of dialogue; the two are positively correlated. Similarly, the higher the expression consistency index corresponding to the main emotion in the c-th round of dialogue, the stronger the emotional consistency feature of the c-th round of dialogue; the two are also positively correlated. Therefore, based on the proportion of emotion words corresponding to the main emotion in the c-th round of dialogue and the expression consistency index corresponding to the main emotion in the c-th round of dialogue, the emotional consistency feature of the c-th round of dialogue can be obtained. Based on the above logical analysis, a specific calculation method for the emotional consistency feature of the c-th round of dialogue is given below: calculate the average of the proportion of emotion words corresponding to the main emotion in the c-th round of dialogue and the expression consistency index corresponding to the main emotion in the c-th round of dialogue, and use this average as the emotional consistency feature of the c-th round of dialogue. This yields the emotional consistency features of each round of dialogue.

[0040] Step S2: Integrate text semantic similarity and emotional consistency features to obtain the matching degree of each round of dialogue.

[0041] For the c-th round of dialogue, the text semantic similarity of the c-th round of dialogue is analyzed from the text content level, and the emotional consistency feature of the c-th round of dialogue is analyzed from the sentiment level. Therefore, by integrating the text semantic similarity and emotional consistency features of the c-th round of dialogue, the matching degree of the c-th round of dialogue is obtained, and a comprehensive quantitative evaluation of the c-th round of dialogue in both content and sentiment dimensions is achieved.

[0042] The higher the text semantic similarity of the c-th round of dialogue, the better the match between the user's input and the robot's output, resulting in a higher degree of matching for the c-th round of dialogue; these two are positively correlated. Similarly, the higher the emotional consistency feature of the c-th round of dialogue, the better the match between the user's input and the robot's output, resulting in a higher degree of matching for the c-th round of dialogue; these two are also positively correlated. Based on the above logical analysis, a specific method for calculating the degree of matching for the c-th round of dialogue is as follows: calculate the average of the text semantic similarity and the emotional consistency feature of the c-th round of dialogue; this average value is the degree of matching for the c-th round of dialogue. The degree of matching for each round of dialogue can then be obtained in this way.

[0043] Step S3: Based on the similarity of emotions, divide all rounds of dialogue into multiple emotional segments.

[0044] In psychological counseling scenarios, a user's primary emotion cannot be fully expressed in a single round of dialogue. They often express the same primary emotion through multiple rounds of dialogue. For example, the anxiety caused by insomnia is usually mentioned repeatedly in multiple rounds of dialogue. In order to reflect this continuous emotional state of the user, it is necessary to integrate multiple rounds of dialogue containing the same primary emotion into an emotional segment with the same emotional characteristics for subsequent analysis.

[0045] In an exemplary embodiment, step S2 obtains the dominant emotion of each round of dialogue and arranges the rounds of dialogue chronologically. Then, at least two consecutive rounds of dialogue with the same dominant emotion constitute an emotion segment, thereby dividing the currently conducted multi-round dialogue into several emotion segments. It should be understood that if the dominant emotion of a round of dialogue differs from the dominant emotions of its preceding and following rounds, then that round of dialogue is treated as a separate emotion segment. For example: With a total of 7 rounds of dialogue, and the dominant emotions being: positive emotion, positive emotion, positive emotion, negative emotion, neutral emotion, neutral emotion, and negative emotion, the first three rounds of dialogue constitute one emotion segment, the fourth round constitutes a separate emotion segment, the fifth and sixth rounds constitute another emotion segment, and the seventh round constitutes a separate emotion segment, resulting in a total of four emotion segments.

[0046] Step S4: Based on the degree of matching contained in the emotion segment, obtain the reliable characteristics of the user's emotion in the emotion segment.

[0047] For any given emotional segment, let's take the d-th emotional segment as an example. We obtain the dialogues contained in the d-th emotional segment, thus obtaining the matching degree of each dialogue. We then construct the matching degree sequence of the d-th emotional segment by combining the matching degrees of each dialogue contained in the d-th emotional segment.

[0048] Based on the matching degree sequence of the d-th emotion segment, the reliable characteristics of the user's emotion in the d-th emotion segment are obtained. In an exemplary embodiment, the overall level and volatility of the matching degree sequence of the d-th emotion segment are first obtained. The overall level characterizes the overall numerical variation of the matching degree sequence of the d-th emotion segment; in this embodiment, the average matching degree of the matching degree sequence of the d-th emotion segment is calculated as the overall level. The volatility characterizes the numerical fluctuation of the matching degree sequence of the d-th emotion segment; in this embodiment, the standard deviation of the matching degree of the matching degree sequence of the d-th emotion segment is calculated as the volatility.

[0049] The higher the overall level of the matching degree sequence of the d-th emotion segment, the better the matching between the user's input voice and the robot's output voice in each round of dialogue, based on the stability of the main emotion. This indicates that the user's emotion is more reliable in the d-th emotion segment, and the higher the reliability feature of the user's emotion in the d-th emotion segment. Therefore, the reliability feature of the user's emotion is positively correlated with the overall level of the matching degree sequence.

[0050] The lower the fluctuation of the matching degree sequence of the d-th emotion segment, the more stable the matching between the user's input voice and the robot's output voice in each round of dialogue, based on the stability of the main emotion. This indicates that the user's emotion is more reliable in the d-th emotion segment, and the higher the reliability feature of the user's emotion in the d-th emotion segment. Therefore, the reliability feature of the user's emotion is inversely correlated with the fluctuation of the matching degree sequence.

[0051] Based on the above logic, the following is a specific method for calculating the reliable feature of user emotion in the d-th emotion segment: ; in, This represents the reliable characteristics of user emotions in the d-th emotion segment. This represents the normalized value of the fluctuation in the matching degree sequence of the d-th emotion segment. This represents the normalized value of the overall level of the matching degree sequence of the d-th emotion segment.

[0052] When normalizing the degree of fluctuation and the overall level, the maximum and minimum value normalization method is used. The maximum and minimum value normalization method is an existing technology and will not be elaborated on here.

[0053] It should be understood that for an emotion segment containing only one round of dialogue, the reliable feature of the user's emotion in the emotion segment is set as the degree of matching of that round of dialogue.

[0054] Step S5: Based on the meaning of the emotional segment Figure 1 By analyzing the characteristics of emotional transition and the characteristics of abrupt changes in emotion, we can obtain the characteristics of smooth emotional transition in the emotional segment.

[0055] During multiple rounds of dialogue, a user's emotions may shift. This is usually due to a change in the user's genuine feelings, rather than a result of the robot's inconsistent tone or emotions. Failing to distinguish between emotional continuity within the same emotional range and natural shifts across emotional ranges may misinterpret normal emotional changes as a failure of the robot's emotional matching, thus failing to accurately reflect the quality of the robot's dialogue.

[0056] Because the same emotional state implies that the user often maintains the same dominant emotion, such as persistent anxiety or persistent pleasure, they will correspondingly engage in multiple rounds of expression around the same intention (such as the same topic of conversation or core demand). Normally, the higher the quality of the user's dialogue with the robot, the more meaningful the multiple rounds of dialogue within the same emotional state will be. Figure 1 The higher the emotional intensity, the more important it is to analyze the meaning of multiple rounds of dialogue within an emotional segment. Figure 1 The characteristics were analyzed.

[0057] Taking the d-th emotional segment as an example, the meaning of the d-th emotional segment is obtained based on the similarity of keywords between two adjacent rounds of dialogue in the d-th emotional segment. Figure 1 Characteristics.

[0058] First, as described above, determine the keywords for each round of dialogue in the d-th emotion segment, including keywords from the user's input voice and the robot's output voice. For any two adjacent rounds of dialogue in the d-th emotion segment, the two rounds of dialogue are respectively designated as the first candidate round dialogue and the second candidate round dialogue. For ease of explanation, the first candidate round dialogue is taken as the c-th round dialogue, and the second candidate round dialogue is taken as the (c+1)-th round dialogue.

[0059] Using word vector models (such as Word2Vec, GloVe, FastText, etc.), keyword vectors for each keyword in the c-th and c+1-th rounds of dialogue are determined respectively. It should be understood that when the same word vector model is used, the keyword vectors of each keyword have the same dimension, which makes it easier to obtain the similarity between two keyword vectors.

[0060] Starting from the c-th round of dialogue, select one keyword, and then select another keyword from the (c+1)-th round of dialogue. Obtain the keyword vector similarity between these two keywords. In an exemplary embodiment, the keyword vector similarity is specifically the cosine similarity of the keyword vectors. For ease of subsequent processing, the cosine similarity needs to be normalized as follows: (cosine similarity + 1) / 2. All cosine similarities mentioned below are normalized results. Then, iterate through all keywords in the c-th round and all keywords in the (c+1)-th round of dialogue to obtain the keyword vector similarity between each keyword in the c-th round and each keyword in the (c+1)-th round. Based on the keyword vector similarity between each keyword in the c-th round and each keyword in the (c+1)-th round, determine the matching keyword set for the c-th and (c+1)-th rounds of dialogue. Keywords in the matching keyword set are defined as matching keywords, representing keywords with high keyword vector similarity. In an exemplary embodiment, a vector similarity threshold is preset. This vector similarity threshold is used as a benchmark for matching keywords to determine whether the similarity of keyword vectors is high. The numerical range of the vector similarity threshold is 0 to 1. The specific value is set according to the actual judgment needs, or it can be set as an empirical value. In this embodiment, 0.7 is used as an example.

[0061] The similarity of keyword vectors between each keyword in round c and each keyword in round c+1 is compared to a similarity threshold. Keyword vector similarities greater than the threshold are obtained, resulting in the two keywords corresponding to each similarity value exceeding the threshold. These keywords from both round c and round c+1 are then combined to form a keyword set, which is the matching keyword set for both round c and round c+1. ​​It should be understood that during the construction of the matching keyword set, if at least two keywords in the set are the same, other duplicate keywords are removed, retaining only one keyword. This ensures that no two keywords are identical, resulting in a final matching keyword set composed of distinct matching keywords.

[0062] Obtain the number of matching keywords in the matching keyword sets of the c-th and c+1-th rounds of dialogue, and thus obtain the number of matching keywords in the matching keyword sets of each pair of adjacent rounds of dialogue in the d-th emotion segment. Based on the number of matching keywords in the matching keyword sets of each pair of adjacent rounds of dialogue in the d-th emotion segment, obtain the meaning of the d-th emotion segment. Figure 1 Consistent features. Among them, the more matching keywords in the matching keyword set, the more... Figure 1 The more obvious the characteristic, therefore, the more... Figure 1The characteristics are positively correlated with the number of matching keywords. In an exemplary embodiment, the proportion of matching keywords in the matching keyword sets of each adjacent two rounds of dialogue in the d-th emotion segment is obtained. Taking the c-th and c+1-th rounds of dialogue as an example, the intersection of the keywords of the c-th and c+1-th rounds of dialogue is obtained, and the number of keywords in the intersection is determined. Then, the ratio of the number of matching keywords in the matching keyword sets of the c-th and c+1-th rounds of dialogue to the number of keywords in the intersection is calculated. The result is the proportion of matching keywords in the matching keyword sets of the c-th and c+1-th rounds of dialogue. Therefore, the proportion of matching keywords in the matching keyword sets of the c-th and c+1-th rounds of dialogue essentially represents the coverage of the matching keywords in the matching keyword sets of the c-th and c+1-th rounds of dialogue relative to all keywords in the c-th and c+1-th rounds of dialogue. The larger the coverage, the higher the degree of overlap of intent between the c-th and c+1-th rounds of dialogue, indicating that the core needs and dialogue intent remain continuous, stable, and uninterrupted as the user's emotions change.

[0063] Using the above process, the proportion of matching keywords in the matching keyword sets of each adjacent pair of dialogues in the d-th emotion segment is obtained. Then, the average proportion of matching keywords in all adjacent pairs of dialogues in the d-th emotion segment is calculated, and this average is taken as the meaning of the d-th emotion segment. Figure 1 To define the meaning of an emotion segment, it should be understood that for an emotion segment containing only two rounds of dialogue, the proportion of matching keywords corresponding to the two rounds of dialogue within the emotion segment is used as the meaning of the emotion segment. Figure 1 Characteristics; for emotional segments containing only one round of dialogue, the meaning of the emotional segment will be... Figure 1 The characteristic is directly set to 1. This yields the meaning of each emotional segment. Figure 1 Characteristics.

[0064] During multi-turn conversations, a user's emotions may gradually shift from anxiety to relief. Therefore, in addition to analyzing stable expressions within the same emotional segment, it is also necessary to consider whether the emotional transitions between different emotional segments are natural. The emotional abrupt change characteristic is obtained by analyzing the difference in emotional consistency features between two adjacent emotional segments. Taking the d-th emotional segment as an example, the preceding emotional segment adjacent to the d-th emotional segment is the (d-1)-th emotional segment. The average of the emotional consistency features of all turns included in the d-th emotional segment is calculated as the overall emotional consistency feature of the d-th emotional segment; the average of the emotional consistency features of all turns included in the (d-1)-th emotional segment is also calculated as the overall emotional consistency feature of the (d-1)-th emotional segment. Then, the absolute value of the difference between the overall emotional consistency feature of the d-th emotional segment and the overall emotional consistency feature of the (d-1)-th emotional segment is calculated, and the emotional abrupt change characteristic of the d-th emotional segment is obtained from this absolute value.

[0065] In an exemplary embodiment, when determining the emotional abrupt change characteristics of the d-th emotional segment, the difference between the primary emotions of the d-th emotional segment and the (d-1)-th emotional segment can also be considered. The greater the difference between the primary emotions, the more positively it affects the emotional abrupt change characteristics. Specifically, this embodiment assigns three different emotional label values ​​to positive emotions, neutral emotions, and negative emotions. Generally speaking, positive and negative emotions are relative. Therefore, the emotional label values ​​assigned to positive and negative emotions differ significantly, while the emotional label value assigned to neutral emotions is in the middle. Based on meeting the above requirements and facilitating subsequent data processing, an example of positive, neutral, and negative emotions is given below: the emotional label value assigned to positive emotions is 0, the emotional label value assigned to neutral emotions is 0.5, and the emotional label value assigned to negative emotions is 1. This determines the emotion label value of the dominant emotion in the d-th emotion segment and the emotion label value of the dominant emotion in the (d-1)-th emotion segment. The absolute value of the difference between the emotion label values ​​of the d-th and (d-1)-th emotion segments is then calculated as the emotion label fluctuation value of the d-th emotion segment. Therefore, when the dominant emotion changes from positive to negative, or from negative to positive, the emotion fluctuation is larger, and the emotion label fluctuation value is also larger. Conversely, when the dominant emotion changes from positive or negative to neutral, or from neutral to positive or negative, the emotion fluctuation is relatively smaller, and the emotion label fluctuation value is smaller. Since the dominant emotions of adjacent emotion segments are necessarily different, the emotion label fluctuation value of each emotion segment is necessarily not 0. Based on the numerical examples above, the emotion label fluctuation value has two possibilities: 0.5 and 1.

[0066] The emotional label fluctuation value of the d-th emotional segment is used as the coefficient weight of the absolute value of the difference between the overall emotional consistency feature of the d-th emotional segment and the overall emotional consistency feature of the (d-1)-th emotional segment. The two are multiplied together to obtain the emotional abrupt change feature of the d-th emotional segment. The calculation formula is as follows: ; in, This represents the emotional abrupt change characteristic of the d-th emotional segment. This represents the overall consistent emotional characteristic of the d-th emotional segment. This represents the overall consistent emotional characteristic of the (d-1)th emotional segment. This represents the emotional label fluctuation value of the d-th emotional segment. The greater the difference in emotional consistency between the d-th emotional segment and the (d-1)-th emotional segment, and the larger the emotional fluctuation label value, the more unnatural and volatile the emotional transition between the d-th and (d-1)-th emotional segments is, and the higher the corresponding emotional abrupt change characteristic is, indicating that it is more likely to be a sudden and emotionally disjointed response from a robot.

[0067] In another implementation, this embodiment may disregard the differences between the main emotions of adjacent emotional segments and directly use the absolute value of the difference between the overall emotional consistency feature of the d-th emotional segment and the overall emotional consistency feature of the (d-1)-th emotional segment as the emotional mutation feature of the d-th emotional segment.

[0068] It should be understood that for the first emotional segment in the time sequence, since there are no preceding emotional segments, its emotional abrupt change characteristic is directly set to 0. This yields the emotional abrupt change characteristics for each emotional segment.

[0069] In obtaining the meaning of the dth emotional segment Figure 1 After determining the emotional transition characteristics and the emotional abrupt change characteristics, the smoothness of the emotional transition in the d-th emotional segment is obtained based on these two parameters. Among them, the meaning of the d-th emotional segment... Figure 1 The stronger the emotional transition feature, the smoother the emotional transition in each round of dialogue within the d-th emotional segment, and the stronger the smoothness feature of the emotional transition in the d-th emotional segment; the two are positively correlated. Conversely, the stronger the emotional abrupt change feature of the d-th emotional segment, the more severe the emotional abrupt changes in each round of dialogue within the d-th emotional segment, the stronger the emotional switching fluctuations, the less smooth the emotional transition in each round of dialogue within the d-th emotional segment, and the weaker the smoothness feature of the emotional transition in the d-th emotional segment; the two are inversely correlated. Therefore, based on this logical analysis, the following is a specific calculation method for the smoothness feature of the emotional transition in the d-th emotional segment: ; in, This indicates the smoothness of the emotional transition in the d-th emotional segment. This indicates the meaning of the d-th emotional segment. Figure 1 Characteristics were identified. This allowed for the deriving of smooth emotional transition characteristics across different emotional segments.

[0070] Step S6: Integrate the reliable features of user emotions and the smoothness of emotional transitions across all emotion segments to obtain the quality results of the robot's multi-turn dialogue.

[0071] Step S5 analyzes the emotional consistency and smooth emotional transitions between the user and the robot during multi-turn dialogues. This helps determine the robot's dialogue quality in multi-turn conversations, enabling optimization of the robot's performance in handling different user emotions and needs, improving its ability to cope with complex dialogue situations and enhancing the user experience. Taking the d-th emotional segment as an example, the stronger the user's emotional reliability and emotional transition fluency in the d-th emotional segment, the higher the robot's dialogue quality. Therefore, the dialogue quality features for each emotional segment are first obtained based on the user's emotional reliability and emotional transition fluency features. The dialogue quality features are positively correlated with both the user's emotional reliability and emotional transition fluency features. Then, the dialogue quality features of all emotional segments are merged to obtain the robot's multi-turn dialogue quality index. Specifically, taking the d-th emotional segment as an example, the following is a specific calculation method for the dialogue quality features of the d-th emotional segment: ; in, This represents the dialogue quality characteristics of the d-th emotional segment.

[0072] Then, the average of the dialogue quality features across all emotional segments is calculated, and this average is used as the current robot multi-turn dialogue quality index. The obtained robot multi-turn dialogue quality index characterizes the quality result of the robot's multi-turn dialogue. It should be understood that the higher the value of the robot multi-turn dialogue quality index, the higher the robot's dialogue quality, and the better it indicates the current robot's emotional adaptation, emotional transition, and intention... Figure 1 It meets users' requirements in terms of consistency, etc.

[0073] In an exemplary embodiment, after obtaining the robot's multi-turn dialogue quality index, the robot multi-turn dialogue control method provided in this embodiment further includes a judgment process for whether the current robot multi-turn dialogue quality is qualified, as follows: This embodiment presets a quality threshold, which serves as a judgment benchmark to determine whether the robot's multi-turn dialogue quality index is high, thereby determining whether the current robot's multi-turn dialogue quality is acceptable. The quality threshold ranges from 0 to 1, and the specific value is set according to actual needs. For example, if a more stringent judgment logic is required, the quality threshold can be set slightly higher. Alternatively, the quality threshold can also be manually set by robot administrators based on experience. As an example, the quality threshold is 0.7.

[0074] Compare the robot's multi-turn dialogue quality index with the quality threshold: if the robot's multi-turn dialogue quality index is greater than or equal to the quality threshold, the current robot's multi-turn dialogue quality is deemed qualified, and a robot multi-turn dialogue quality qualified instruction signal is output; if the robot's multi-turn dialogue quality index is less than the quality threshold, the current robot multi-turn dialogue quality is deemed unqualified, and a robot multi-turn dialogue quality unqualified instruction signal is output.

[0075] In addition, after the robot manager receives the instruction signal that the robot's multi-turn dialogue quality is qualified, the response strategy, tone style, and emotional intensity of the current emotional big language model in the robot can remain unchanged.

[0076] After the robot manager receives a signal indicating that the robot's multi-turn dialogue quality is substandard, it is determined that the robot has at least one of the following problems: poor emotion matching, abrupt emotional transition, or unnaturalness. At this time, the robot manager can manually adjust and correct the generation strategy of the emotional big language model in the robot, and improve the above problems through manual verification.

[0077] This embodiment also provides a robot multi-turn dialogue control system based on an emotional big language model, including: a memory and a processor; the memory is connected to the processor, and the memory is used to store program instructions; the processor is used to implement the steps in the above embodiment of the robot multi-turn dialogue control method based on an emotional big language model when the program instructions are executed.

[0078] In one exemplary embodiment, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the embodiment of the robot multi-turn dialogue control method based on an emotion-based large language model.

[0079] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A multi-turn dialogue control method for robots based on an emotional large language model, characterized in that, include: Determine the textual semantic similarity and emotional consistency features of the speech pairs in each round of dialogue; the speech pairs consist of user-input speech and robot-output speech; By integrating the text semantic similarity and emotion consistency features, the matching degree of each round of dialogue is obtained; Based on similar emotional states, all rounds of dialogue are divided into multiple emotional segments; The reliable user emotion characteristics of the emotion segment are obtained from the degree of matching contained in the emotion segment; Based on the intention consistency feature and emotion change feature of the emotion segment, the smoothness feature of the emotion transition of the emotion segment is obtained; the intention consistency feature is obtained by the similarity of keywords in two adjacent rounds of dialogue in the emotion segment, and the emotion change feature is obtained by the difference in the emotion consistency feature of two adjacent emotion segments. By integrating reliable user emotion features and smooth emotion transition features across all emotion segments, the quality results of multi-turn dialogues with the robot are obtained.

2. The robot multi-turn dialogue control method based on an emotion-based large language model as described in claim 1, characterized in that, The process of obtaining the emotion consistency feature includes: Determine the number of various emotion words contained in the user input speech and the robot output speech, respectively; Based on the difference in the proportion of the same emotion word in the user's input speech and the robot's output speech, an expression consistency index is determined for various emotion words; the expression consistency index is inversely correlated with the difference in the proportion. The dominant emotion is determined by the maximum number of various emotion words. The emotion consistency feature is obtained based on the proportion of emotion words corresponding to the main emotion and the expression consistency index corresponding to the main emotion.

3. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 2, characterized in that, Based on the similarity of emotions, all rounds of dialogue are divided into multiple emotional segments, including: each round of dialogue with the same main emotion is composed of consecutive adjacent rounds of dialogue.

4. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 1, characterized in that, The process of obtaining reliable user emotion features includes: Determine the overall level and fluctuation of the degree of matching contained in the emotional segment; The reliable characteristics of user sentiment are obtained from the overall level and the degree of fluctuation; the reliable characteristics of user sentiment are positively correlated with the overall level and negatively correlated with the degree of fluctuation.

5. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 1, characterized in that, The process of obtaining the intent consistency feature includes: Keyword vectors are determined for each keyword in the first candidate round dialogue and the second candidate round dialogue respectively; the first candidate round dialogue and the second candidate round dialogue constitute the two adjacent round dialogues. The set of matching keywords for the two adjacent rounds of dialogue is determined by the keyword vector similarity between each keyword in the first candidate round of dialogue and each keyword in the second candidate round of dialogue. The consistency of intent in an emotion segment is determined by the number of matching keywords in the matching keyword set of each two adjacent rounds of dialogue.

6. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 5, characterized in that, The process of obtaining the matching keyword set includes: Compare the keyword vector similarity between each keyword in the first round of dialogue and each keyword in the second round of dialogue with the preset vector similarity threshold. Two keywords whose keyword vector similarity exceeds the preset vector similarity threshold are identified to obtain the matching keyword set.

7. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 5, characterized in that, The intention consistency feature of the emotional segment is obtained by calculating the number of matching keywords in the matching keyword set of each adjacent two rounds of dialogue within the emotional segment, including: The average percentage of matching keywords in each of two adjacent rounds of dialogue is calculated to obtain the intent consistency feature.

8. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 1, characterized in that, The process of obtaining the emotional mutation features includes: The absolute value of the difference between the emotional consistency feature of the emotional segment and its adjacent previous emotional segment is determined, and the emotional abrupt change feature of the emotional segment is obtained from the absolute value of the difference.

9. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 1, characterized in that, The fusion of reliable user emotion features and smooth emotion transition features across all emotion segments yields the robot's multi-turn dialogue quality results, including: Based on the user's emotional reliability characteristics and emotional transition fluency characteristics for each emotional segment, the dialogue quality characteristics for each emotional segment are obtained; the dialogue quality characteristics are positively correlated with both the user's emotional reliability characteristics and emotional transition fluency characteristics. By integrating the dialogue quality features of all emotional segments, a multi-turn dialogue quality index for the robot is obtained, which characterizes the quality result of the robot's multi-turn dialogue.

10. The robot multi-turn dialogue control method based on an emotional large language model as described in claim 9, characterized in that, The robot multi-turn dialogue control method based on the emotion-based large language model also includes: Compare the robot's multi-turn dialogue quality index with a preset quality threshold: If the robot's multi-turn dialogue quality index is greater than or equal to the preset quality threshold, a robot multi-turn dialogue quality qualified instruction signal is output; if the robot's multi-turn dialogue quality index is less than the preset quality threshold, a robot multi-turn dialogue quality unqualified instruction signal is output.

Images