Multi-modal based companion robot dialogue quality evaluation method and computer device

Abstract

This application provides a multimodal dialogue quality assessment method and computer device for companion robots, relating to the field of robot control technology. The method includes: collecting multimodal raw data of the current dialogue turn during the interaction process; extracting semantic features, speech prosody features, facial expression features, and physiological emotion features based on dialogue speech, facial expression images, and user physiological signals; obtaining semantic relevance scores, emotional consistency scores, and contextual coherence scores based on facial expression images, semantic features, speech prosody features, facial expression features, physiological emotion features, response text, and historical text scores; and obtaining the current dialogue quality assessment result of the companion robot based on the semantic relevance score, emotional consistency score, and contextual coherence score. This application improves the accuracy of the current dialogue quality assessment result of the companion robot, thereby improving the service quality of the companion robot.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a method and computer device for evaluating the dialogue quality of a multimodal companion robot. Background Technology

[0002] With the accelerating aging of the population and the increasing strain on family care resources, companion robots with emotional interaction capabilities have been widely deployed in scenarios such as home-based elderly care, community-based health and wellness, and companionship for children with special needs. These robots not only need to accurately understand user commands and generate appropriate responses, but also need to continuously perceive the user's emotional state and maintain the coherence of the conversation in multiple rounds of dialogue, while adapting to the individualized needs of different cognitive levels and expression habits.

[0003] Currently, dialogue quality assessment relies heavily on text-level similarity calculations. For example, comparing the robot's response with preset reference answers by word overlap or vector distance, and introducing simple facial expression recognition or speech pause analysis as auxiliary judgment criteria; or using historical dialogue texts concatenated and input into a language model to roughly estimate whether the current response is a reasonable continuation.

[0004] However, existing technologies cannot capture users' micro-expressions, recognize the deeper meaning of voice, or sense emotional changes through heart rate in companionship scenarios. They process each round of dialogue in a fragmented manner and use the same strategy for different users, resulting in evaluation results that are out of touch with the real interaction context. This makes it impossible to reflect the actual companionship effect or drive the continuous evolution of the service. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a multimodal method and computer equipment for evaluating the dialogue quality of companion robots, thereby improving the accuracy of the current dialogue quality evaluation results of companion robots and thus improving the service quality of companion robots.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a method for evaluating the dialogue quality of a companion robot based on multimodality, the method comprising: The system acquires multimodal raw data of the current dialogue round collected by the companion robot during its interaction with the user. The multimodal raw data includes: dialogue voice, facial expression images, and user physiological signals. Semantic features and prosodic features are extracted from the dialogue speech; The facial expression image is subjected to expression recognition to obtain expression features; Emotion extraction is performed on the user's physiological signals to obtain physiological emotional features; A semantic relevance score is obtained by scoring the semantic relevance based on the facial expression features and the semantic features. An emotional consistency score is obtained by scoring the facial expression image, the physiological emotional features, and the speech prosody features. The context coherence score is obtained by scoring the context coherence based on the response text corresponding to the response voice in the dialogue, the historical dialogue text of the historical dialogue rounds, and the facial expression features. The current dialogue quality assessment result of the companion robot is obtained based on the semantic relevance score, the emotional consistency score, and the contextual coherence score.

[0007] Optionally, the step of scoring emotional consistency based on the facial expression image, the physiological emotional features, and the speech prosody features to obtain an emotional consistency score includes: Emotional features are extracted from the facial expression images to obtain the probability distribution of facial expression emotions; Emotional feature extraction is performed on the aforementioned physiological and emotional features to obtain the physiological and emotional probability distribution; Emotional features are extracted from the prosodic features of the speech to obtain the speech emotion probability distribution; Based on the probability distribution of facial expressions, the probability distribution of physiological emotions, and the probability distribution of speech emotions, fused emotional features are obtained; Based on the dialogue context of the current dialogue round, obtain the expected user emotional characteristics; The emotional consistency score is obtained by performing an emotional consistency score based on the fused emotional features and the expected user emotional features.

[0008] Optionally, the step of extracting emotional features from the facial expression image to obtain an expression emotion probability distribution includes: Spatial features are extracted from each frame of the facial expression image to obtain local texture features of key facial regions; Temporal dynamic modeling is performed on multiple consecutive frames of the facial expression images to obtain expression change features; Facial motion recognition is performed on the facial expression image to obtain facial motion features; The probability distribution of facial expression is obtained based on the local texture features, the expression change features, and the facial movement features.

[0009] Optionally, obtaining the fused emotional features based on the facial expression emotion probability distribution, the physiological emotion probability distribution, and the speech emotion probability distribution includes: Based on the current environmental state and the user's historical physical state, calculate the confidence scores corresponding to the emotional probability distributions of the facial expressions, physiological emotions, and voice emotions, as well as the emotional probability distributions of each modality. The weights of each modality are determined based on the confidence scores. Based on the weights of each modality, the probability distributions of facial expressions, physiological emotions, and speech emotions are weighted to obtain the fused emotional features.

[0010] Optionally, the dialogue speech further includes: a questioning speech; the extraction of semantic features and prosodic features from the dialogue speech includes: Speech recognition is performed on the question and answer voices to obtain the question text and answer text; Semantic features are extracted from the question and answer texts to obtain the semantic features; Extract the prosodic features from the questioning speech.

[0011] Optionally, the semantic features include: question semantic features and answer semantic features; the step of scoring semantic relevance based on the facial expression features and the semantic features to obtain a semantic relevance score includes: Based on the facial expression features, attention enhancement is performed on the question semantic features to obtain enhanced question semantic features; The semantic relevance score is obtained by scoring the semantic relevance based on the enhanced question semantic features and the answer semantic features.

[0012] Optionally, obtaining the current dialogue quality assessment result of the companion robot based on the semantic relevance score, the sentiment consistency score, and the contextual coherence score includes: Based on the user profile, determine the user's personalized correction coefficient; Based on the personalized correction coefficient, the preset weights of multiple evaluation dimensions are adjusted to obtain the evaluation weights of each evaluation dimension. Based on the evaluation weights of each evaluation dimension, the semantic relevance score, the emotional consistency score, and the contextual coherence score are weighted to obtain the current dialogue quality evaluation result of the companion robot.

[0013] Optionally, adjusting the preset weights of multiple evaluation dimensions according to the personalized correction coefficient to obtain the evaluation weights of each evaluation dimension includes: The adjustment range is calculated based on the personalized correction coefficient. Based on the adjustment range, obtain the correction coefficient range for each of the evaluation dimensions; Based on the correction coefficient range of each evaluation dimension, the preset weights of the multiple evaluation dimensions are adjusted to obtain the evaluation weights of each evaluation dimension.

[0014] Optionally, the step of extracting emotions from the user's physiological signals to obtain physiological emotional features includes: Local pattern extraction is performed on the user's physiological signals to obtain short-time local pattern features; Periodic pattern extraction is performed on the user's physiological signals to obtain rhythmic pattern features; The user's physiological signals are captured in time series to obtain trend characteristics. Emotion extraction is performed based on the short-term local pattern features, the rhythmic pattern features, and the trend features to obtain the physiological emotion features.

[0015] Secondly, another embodiment of this application provides a multimodal companion robot dialogue quality assessment device, the device comprising: The acquisition module is used to acquire multimodal raw data of the current dialogue round collected by the companion robot during its interaction with the user. The multimodal raw data includes: dialogue voice, facial expression images, and user physiological signals. The first extraction module is used to extract semantic features and prosodic features from the dialogue speech; The recognition module is used to perform facial expression recognition on the facial expression image to obtain facial expression features; The second extraction module is used to extract emotions from the user's physiological signals to obtain physiological and emotional features; The first scoring module is used to score semantic relevance based on the facial features and the semantic features, and obtain a semantic relevance score. The second scoring module is used to score emotional consistency based on the facial expression image, the physiological emotional features and the speech prosody features, and to obtain an emotional consistency score. The third scoring module is used to score the contextual coherence based on the response text corresponding to the response voice in the dialogue, the historical dialogue text of the historical dialogue rounds, and the facial expression features, and to obtain a contextual coherence score. The determination module is used to obtain the current dialogue quality assessment result of the companion robot based on the semantic relevance score, the emotional consistency score, and the contextual coherence score.

[0016] Thirdly, another embodiment of this application provides a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the multimodal companion robot dialogue quality assessment method as described in any of the first aspects above.

[0017] Fourthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the multimodal companion robot dialogue quality assessment method described in any of the first aspects above.

[0018] The beneficial effects of this application are: This application provides a multimodal method and computer device for evaluating the dialogue quality of a companion robot. First, it acquires multimodal raw data from the current dialogue round collected during the interaction between the companion robot and the user. This multimodal raw data includes: dialogue speech, facial expression images, and user physiological signals. Semantic features and prosodic features are extracted from the dialogue speech; facial expression recognition is performed on the facial expression images to obtain expression features; and emotion extraction is performed on the user's physiological signals to obtain physiological emotion features. This feature extraction based on multimodal raw data overcomes the blind spot of traditional single-modal evaluation in perceiving implicit user feedback, thereby improving the completeness and authenticity of the dialogue quality evaluation.

[0019] Semantic relevance scoring is performed based on facial expression and semantic features to obtain a semantic relevance score. This score is then fused with facial expression and emotional state to enhance semantic intent alignment. Emotional consistency scoring is performed based on facial expression images, physiological emotional features, and speech prosody features to obtain an emotional consistency score. This score is dynamically weighted and fused from multi-source heterogeneous signals and mapped to the expected emotional space. Contextual coherence scoring is performed based on the response text corresponding to the response in the dialogue, historical dialogue text from previous dialogue rounds, and facial expression features to obtain a contextual coherence score. This score is then combined with historical dialogue text and the current facial expression for hierarchical modeling. By using semantic relevance, emotional consistency, and contextual coherence scores, the system collaboratively overcomes the problems of dimensional fragmentation and static isolation in existing technologies, enabling the evaluation results to more accurately reflect the user's subjective experience and cognitive load in real-world caregiving scenarios.

[0020] Finally, the current dialogue quality assessment result of the companion robot is obtained based on the semantic relevance score, sentiment consistency score, and contextual coherence score. By organically integrating the semantic relevance score, sentiment consistency score, and contextual coherence score, the final current dialogue quality assessment result can comprehensively reflect the robot's information transmission, providing a unified, stable, and interpretable quantitative basis for service quality monitoring, user experience analysis, and model iteration optimization.

[0021] This allows the present application to avoid the limitations of traditional dialogue evaluation that relies solely on textual semantics, making the evaluation results truly close to the user's subjective experience, thus making the evaluation results of dialogue quality more accurate, thereby driving the continuous optimization of the robot's dialogue capabilities and realizing an integrated closed loop of evaluation and evolution. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a method for assessing the dialogue quality of a companion robot based on multimodal communication, provided as an embodiment of this application. Figure 2 A flowchart illustrating the process of determining emotional consistency scores in a multimodal-based method for evaluating the dialogue quality of a companion robot, as provided in an embodiment of this application. Figure 3 A flowchart illustrating the process of determining the probability distribution of facial expressions and emotions in a multimodal companion robot dialogue quality assessment method provided in this application embodiment; Figure 4 A schematic diagram illustrating the process of determining fused emotional features in a multimodal companion robot dialogue quality assessment method provided in this application embodiment; Figure 5 A schematic diagram illustrating the feature extraction process in a multimodal companion robot dialogue quality assessment method provided in this application embodiment; Figure 6 A schematic diagram of the feature extraction process in a multimodal companion robot dialogue quality assessment method provided in this application embodiment; Figure 7 A schematic diagram illustrating the process of determining the current dialogue quality assessment result in a multimodal companion robot dialogue quality assessment method provided in this application embodiment; Figure 8A flowchart illustrating the process of determining evaluation weights in a multimodal-based companion robot dialogue quality assessment method provided in this application embodiment; Figure 9 A flowchart illustrating the determination of physiological and emotional characteristics in a multimodal-based method for assessing the dialogue quality of a companion robot, as provided in an embodiment of this application. Figure 10 A schematic diagram of a multimodal companion robot dialogue quality assessment device provided in this application embodiment; Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] Currently, companion robots are widely used in elderly care, healthcare, child development, and special needs care. In elderly care, they can monitor the elderly's health data in real time, remind them to take medication on time, assist with remote family communication via voice and video, and have fall detection and automatic alarm functions. For patients with chronic diseases or post-operative recovery, robots can guide rehabilitation training, record medication responses, and provide basic medical support in conjunction with family doctors. For children, they can assist early education through interactive games and learning content, and also serve as a temporary monitoring tool, detecting dangerous behaviors and sending alarms when children are alone. In addition, in the lives of people with disabilities or young people living alone, they can improve convenience and safety through emotional companionship and smart home control.

[0025] To improve the companionship quality of companion robots, it is necessary to evaluate the dialogue quality of the companion robots and optimize their dialogue models based on the evaluation results, thereby improving the companionship quality. Currently, existing methods for evaluating dialogue quality are typically limited to assessing the semantic accuracy of the dialogue content, neglecting the emotional feedback generated by users during interaction with the robot. This results in evaluation results that fail to accurately reflect the user's emotional experience and make it difficult to measure the robot's emotional companionship effect. Therefore, this application provides a multimodal-based method for evaluating the dialogue quality of companion robots. Through the companion robot's data acquisition module, multimodal raw data from the current dialogue round is collected during the interaction between the companion robot and the user. Based on this multimodal raw data, semantic relevance score, emotional consistency score, and contextual coherence score are determined. Thus, the current dialogue quality evaluation result of the companion robot is obtained based on these scores. This application incorporates a multimodal evaluation index generation module that integrates the three dimensions of semantic relevance, emotional consistency, and contextual coherence, making the evaluation results more accurate and truly reflecting the user experience. It is worth noting that the companion robot in this application can be a humanoid robot, an animal-like robot, a wheeled robot, a desktop fixed robot, etc., and the embodiments of this application do not limit this.

[0026] To clearly describe the method provided in the embodiments of this application, the following description, in conjunction with several accompanying drawings, illustrates the multimodal-based dialogue quality assessment method for companion robots provided in this application. Figure 1 A flowchart illustrating a multimodal-based dialogue quality assessment method for companion robots, as provided in this application embodiment, is shown below. Figure 1 As shown, the method includes: Step 101: Obtain the multimodal raw data of the current dialogue round collected by the companion robot during its interaction with the user.

[0027] The multimodal raw data includes: dialogue voice, facial expression images, and user physiological signals.

[0028] The companion robot is equipped with corresponding voice acquisition units, image acquisition units, and physiological signal acquisition units. The voice acquisition unit collects the user's dialogue during interaction with the companion robot. It can be integrated into the companion robot's multi-channel far-field voice sensing system and can be a microphone array. The image acquisition unit captures images of the user's facial expressions during interaction. This unit can be mounted on the companion robot's eye / forehead area multispectral visual sensing subsystem, including an RGB (red, green, blue) camera and a depth sensor, for continuously acquiring video streams of the user's facial micro-expressions, limb movement sequences, and ambient lighting / posture information. The physiological signal acquisition unit refers to a hybrid physiological monitoring subsystem that primarily uses non-contact optical sensing, supplemented by wearable technology, for non-invasively collecting indicators of the user's autonomic nervous system activity.

[0029] Optionally, the companion robot can collect dialogue voice, facial expression images, and user physiological signals during the current dialogue round of interaction with the user through the voice acquisition unit, image acquisition unit, and physiological signal acquisition unit mounted on the companion robot.

[0030] Step 102: Extract semantic features and prosodic features from the dialogue speech.

[0031] The dialogue voice refers to the two-way voice stream collected during the interaction with the companion robot, which may include the user's question voice, the robot's answer voice, and the user's real-time feedback voice to the answer. The user's real-time feedback voice to the answer may be "Ah?", "Understood", or "Say it again", etc., and this application embodiment does not limit this. Semantic features are the attributes at the meaning level expressed by words and sentences, which may include lexical concepts, syntactic relations, referents, emotional tendencies, intentions, etc. Speech prosodic features refer to the suprasegmental features in speech that go beyond a single phonological segment, used to reflect the rhythm, intonation, stress, emotion, etc. of speech.

[0032] Optionally, semantic features are obtained by extracting semantic features from the text in the dialogue speech, and prosodic features are obtained by extracting prosodic features from the speech speech.

[0033] Step 103: Perform facial expression recognition on the facial expression image to obtain expression features.

[0034] Among them, the facial expression images are a sequence of consecutive frames of the user's facial expressions captured continuously by the image acquisition unit, and the sequence of consecutive frames of facial expressions is ordered according to time. Facial expression features are used to determine the emotional state when asking questions.

[0035] Optionally, the user's current emotional features can be obtained by inputting a sequence of multiple facial expression images into a preset convolutional neural network or based on a preset expression recognition model. .

[0036] Step 104: Extract emotions from user physiological signals to obtain physiological and emotional features.

[0037] The user's physiological signals may include data related to the user's physiological signals, such as electrocardiogram, skin conductance response, respiratory rate, and photoplethysmography pulse wave; this application embodiment does not limit this. Physiological emotional features are quantitative indicators extracted from physiological signals that can characterize or predict emotional states.

[0038] Optionally, the user's physiological signals are preprocessed, and time-domain, frequency-domain, and nonlinear features are extracted from the processed signals, which are then mapped to corresponding physiological and emotional features.

[0039] Step 105: Score the semantic relevance based on facial expression features and semantic features to obtain the semantic relevance score.

[0040] The semantic relevance score is a quantitative score of the semantic matching degree between the user's question text and the companion robot's answer text.

[0041] Optionally, the user's feedback to the companion robot's answer can be determined based on facial expression features, thereby determining the similarity between the user's question text and the companion robot's answer text as a semantic relevance score.

[0042] Step 106: Based on facial expression images, physiological emotional characteristics, and speech prosody characteristics, emotional consistency is scored to obtain an emotional consistency score.

[0043] Among them, the emotional consistency score is used to determine whether the actual emotions of the user corresponding to the companion robot's response are consistent with the expected user emotions.

[0044] Optionally, the user's current actual emotion is determined based on facial expression images, physiological emotional characteristics, and speech prosody characteristics. An emotion consistency score is obtained by scoring the user's actual emotion against their expected emotion.

[0045] Step 107: Analyze the contextual coherence score based on the response text corresponding to the response voice in the dialogue, the historical dialogue text of the historical dialogue rounds, and facial expression features.

[0046] The contextual coherence score is a quantitative score indicating whether the companion robot's current response is logically, semantically, and emotionally consistent with the historical dialogue context. It assesses whether the conversation between the user and the companion robot is off-topic, contradictory, or contains unreasonable jumps.

[0047] Optionally, a historical state vector is determined by splicing historical dialogue text in chronological order, a current response state vector is determined by the text of the robot's response in this round, and user feedback on the current response is determined based on facial expression features. Thus, a contextual coherence score is generated based on the historical state vector, the current response state vector, and user feedback.

[0048] Optionally, based on the historical dialogue text from multiple rounds. A pre-set bidirectional LSTM encoder is used to obtain the historical state vector. , , For the state dimension, a preset bidirectional LSTM encoder is used to obtain the current state vector. , Determine the historical state vector. and the current state vector The cosine similarity is used as the similarity value. Based on the historical state vector and the current state vector The contradiction value is determined by using a pre-defined binary classifier. Based on facial features Determine the confused expression indicator function Therefore, based on similarity values Contradiction value and confused expression indicator function Determine the context coherence score .in, , as well as The weighting coefficient is set according to user needs and the current scenario; this application embodiment does not impose any restrictions on it.

[0049] Step 108: Obtain the current dialogue quality assessment results of the companion robot based on the semantic relevance score, sentiment consistency score, and contextual coherence score.

[0050] The current dialogue quality assessment result is a comprehensive evaluation of the dialogue quality of the companion robot in the current round. The current dialogue quality assessment result can be rated as excellent, good, average, or poor, or a percentage score. The current dialogue quality assessment result is used to guide the robot to adjust its subsequent dialogue strategies.

[0051] Optionally, the current dialogue quality assessment result of the companion robot can be obtained by comprehensively summarizing the semantic relevance score, sentiment consistency score, and contextual coherence score.

[0052] Optionally, it can be determined whether the overall quality score is lower than a preset threshold. If the overall quality score is greater than the preset threshold, the quality of the dialogue response is considered acceptable, and the interaction data is stored in the normal database for subsequent user profile updates. If the overall quality score is less than or equal to the preset threshold, the complete multimodal data of the dialogue is marked as a negative sample and packaged and uploaded to a cloud training server or local training cluster. These negative samples will be used to trigger reinforcement learning or fine-tuning training of the dialogue generation model, including, for example: extracting user questions and robot answers from negative samples, constructing answer pairs, generating pairs as positive samples through manual annotation or automatic comparative learning; and using positive and negative samples to incrementally train the robot's current dialogue model and optimize model parameters.

[0053] In this embodiment, the system acquires multimodal raw data of the current dialogue round collected by the companion robot during its interaction with the user. This multimodal raw data includes: dialogue speech, facial expression images, and user physiological signals. Semantic features and prosodic features are extracted from the dialogue speech. Expression recognition is performed on the facial expression images to obtain expression features. Emotional extraction is performed on the user's physiological signals to obtain physiological emotional features. Semantic relevance is scored based on the expression features and semantic features to obtain a semantic relevance score. Emotional consistency is scored based on the facial expression images, physiological emotional features, and prosodic features to obtain an emotional consistency score. Contextual coherence is scored based on the response text corresponding to the response speech in the dialogue speech, the historical dialogue text of previous dialogue rounds, and the expression features to obtain a contextual coherence score. Finally, the current dialogue quality assessment result of the companion robot is obtained based on the semantic relevance score, emotional consistency score, and contextual coherence score. This application avoids the limitations of traditional dialogue evaluation that relies solely on text semantics. By simultaneously collecting multimodal data such as speech, facial expressions, and physiological signals, it starts from user reactions and focuses on both the accuracy and coherence of the robot's responses. By capturing users' micro-expressions and implicit physiological changes, and combining this with dialogue history to identify topic jumps and logical contradictions, the evaluation results truly reflect the user's subjective experience. This makes the evaluation of dialogue quality more accurate, thereby driving the continuous optimization of the robot's dialogue capabilities and achieving an integrated closed loop of evaluation and evolution.

[0054] Based on the above embodiments, this application also provides a process for determining emotional consistency scores in a multimodal companion robot dialogue quality assessment method. Figure 2 A flowchart illustrating the process of determining emotional consistency scores in a multimodal-based companion robot dialogue quality assessment method provided in this application embodiment is shown below. Figure 2 As shown, in step 106 above, emotional consistency is scored based on facial expression images, physiological emotional features, and speech prosody features to obtain an emotional consistency score, including: Step 201: Extract emotional features from facial expression images to obtain the probability distribution of facial expression emotions.

[0055] Among them, emotion feature extraction refers to the process of automatically identifying and quantifying the characteristics of emotional states from the original images, and is used to reflect the intermediate representation of emotion intensity, dimension and dynamic evolution. The facial expression emotion probability distribution refers to the confidence distribution of several basic emotion categories predicted for each image frame in a series of facial expression images. It is in the form of a vector, for example, happy 0.12, sad 0.05, confused 0.76, neutral 0.07, and the sum of each component is 1.

[0056] Optionally, a pre-defined deep convolutional neural network model is used to process the sequence of facial expression images. Emotional features were extracted to obtain the probability distribution of facial expressions. The default deep convolutional neural network model can be the MobileNet-V2 model. , Used to represent The set of all real numbers represents that The expression categories are categorized into several dimensions. Step 202: Extract emotional features from physiological and emotional characteristics to obtain the physiological and emotional probability distribution.

[0057] Among them, the physiological emotion probability distribution is used to represent the probability distribution of multiple emotion categories.

[0058] Optionally, a pre-defined multilayer perceptron is used to assess physiological and emotional characteristics. Emotional features were extracted to obtain the probability distribution of facial expressions. The preset multilayer perceptron can be a multilayer perceptron (MLP). , Used to represent The set of all real numbers represents that The expression categories are categorized into several dimensions. Step 203: Extract emotional features from the speech prosody features to obtain the speech emotion probability distribution.

[0059] The speech emotion probability distribution is used to represent the probability distribution of multiple emotion categories. Speech prosody in conversational speech includes: fundamental frequency, short-time energy, and Mel-frequency cepstral coefficients. The fundamental frequency reflects pitch changes; short-time energy reflects volume changes and is related to emotion intensity; Mel-frequency cepstral coefficients describe vocal tract shape and assist in identifying emotion categories. Speech prosody is non-semantic information in conversational speech; for example, rising intonation indicates questioning or surprise, while slowed speech indicates confusion or thinking. Speech prosodic features can include biological features such as pitch features, energy features, speech rate features, and pause features.

[0060] Optionally, for phonological prosodic features Emotional feature extraction was performed using a Long Short-Term Memory (LSTM) network to obtain the probability distribution of speech emotion. . , Used to represent The set of all real numbers represents that The expression categories are categorized into several dimensions. Step 204: Based on the probability distribution of facial expressions, physiological emotions, and speech emotions, obtain the fused emotional features.

[0061] Optionally, a comprehensive emotion probability distribution is obtained by fusing the probability distributions of facial expressions, physiological emotions, and speech emotions. This comprehensive emotion probability distribution is then normalized to obtain the fused emotion features. .

[0062] Step 205: Obtain the expected user emotional characteristics based on the dialogue context of the current dialogue round.

[0063] The current dialogue context refers to the text content that the companion robot has seen and recorded before generating this answer. This can include: the text of the user's current question; the text of the user's questions in the previous three rounds; and the text of the robot's answers to these questions. The expected user emotional characteristics refer to the emotional reaction that the user should have after the robot provides an answer, inferred from the dialogue context of the current dialogue round.

[0064] Optionally, based on the dialogue context of the current dialogue turn. Using a pre-defined sentiment mapping network, the expected user sentiment characteristics are obtained. . , Used to represent The set of all real numbers represents that The expression categories are categorized into several dimensions. The preset emotion mapping network can be a neural network model, a large language model, etc., and this application embodiment does not impose any limitations on this.

[0065] Step 206: Calculate the emotional consistency score based on the fused emotional features and the expected user emotional features.

[0066] Optionally, based on the fusion of emotional characteristics and expected user emotional characteristics Emotional consistency scores are obtained by calculating pre-similarity. .in, .

[0067] In the embodiments of this application, by extracting the probability distribution of emotions from three independent channels—facial expressions, physiological signals, and speech prosody—it is possible to make full use of information from three complementary modalities—visual, physiological, and acoustic—overcoming the limitations of a single modality in specific scenarios, improving the robustness and reliability of emotion recognition, and thus achieving more natural and intelligent human-computer emotional interaction.

[0068] Based on the above embodiments, this application also provides a process for determining the probability distribution of facial expressions and emotions in a multimodal companion robot dialogue quality assessment method. Figure 3 This application provides a flowchart illustrating the process of determining the probability distribution of facial expressions and emotions in a multimodal-based method for assessing the dialogue quality of a companion robot. Figure 3 As shown, in step 201 above, emotional features are extracted from the facial expression image to obtain the probability distribution of facial expression emotion, including: Step 301: Extract spatial features from each frame of facial expression image to obtain local texture features of key facial regions.

[0069] Spatial feature extraction involves identifying and quantifying the texture details of key anatomical regions such as the eyebrows, eyes, corners of the mouth, and nostrils from each frame of facial expression images. Spatial features can include wrinkle direction and muscle prominence. Correspondingly, key facial regions can be the eyebrows, eyes, corners of the mouth, and nostrils. Local texture features can include pixel intensity distribution, edge direction, and contrast variations.

[0070] Optionally, a pre-defined convolutional network is used to extract spatial features from each frame of facial expression images to obtain local texture features of key facial regions. The pre-defined convolutional network can be a ConvNeXtV2 network.

[0071] Step 302: Perform temporal dynamic modeling on multiple consecutive frames of facial expression images to obtain expression change features.

[0072] Temporal dynamic modeling involves determining the temporal variation patterns of facial expression images across multiple consecutive frames. Examples include the process of eyebrows gradually narrowing and the trend of the corners of the mouth drooping. Expression change features are used to describe the temporal attributes of an expression, such as its speed, amplitude, and duration from its inception to development.

[0073] Optionally, a temporal dynamic model of multiple consecutive frames of facial expression images is performed using a pre-defined neural network to obtain expression change features. The pre-defined neural network can be a lightweight Transformer network.

[0074] Step 303: Perform facial motion recognition on the facial expression image to obtain facial motion features.

[0075] Facial motion recognition refers to identifying the facial expression corresponding to a facial expression image based on a preset facial motion coding system. The preset facial motion coding system can be an internationally recognized facial motion coding system. Facial motion features are the features corresponding to facial expression images, such as frowning or turning up the corners of the mouth.

[0076] Optionally, facial expression images can be matched with facial movements using a preset facial movement coding system to obtain facial movement features corresponding to the facial expression images.

[0077] Step 304: Obtain the probability distribution of facial expressions based on local texture features, facial expression change features, and facial movement features.

[0078] Optionally, local texture features, facial expression change features, and facial motion features are concatenated into a joint feature vector, which is then normalized using a pre-defined deep convolutional neural network model to output the probability distribution of facial expression.

[0079] In this embodiment, three types of features are extracted: local texture of a single frame image, facial expression dynamics of a multi-frame sequence, and anatomically defined action units. Each of these features independently represents different physical attributes of the facial expression. The three features are then fused at the feature level to avoid information aliasing, thereby improving the accuracy of the generated facial expression probability distribution.

[0080] Based on the above embodiments, this application also provides a process for determining the fused emotional features in a multimodal companion robot dialogue quality assessment method. Figure 4 This application provides a flowchart illustrating the process of determining fused emotional features in a multimodal-based method for assessing the dialogue quality of a companion robot. Figure 4 As shown, in step 206 above, based on the probability distributions of facial expressions, physiological emotions, and speech emotions, fused emotional features are obtained, including: Step 401: Based on the current environmental state and the user's historical physical state, calculate the confidence score corresponding to the emotional probability distribution of each modality in the facial expression emotional probability distribution, physiological emotional probability distribution, and voice emotional probability distribution.

[0081] The current environmental conditions can include real-time physical conditions affecting sensor data quality, such as camera illumination intensity, background noise levels, device operating temperature, and network connection stability. The user's historical physical state can be individualized reference data such as baseline values ​​of physiological signals, common facial expression ranges, and speech clarity fluctuations recorded from past interactions. The confidence score is a quantitative assessment of the reliability of the probability distributions for facial expressions, physiological states, and speech, with values ​​ranging from 0 to 1, reflecting the reliability of the current output of that modality.

[0082] Optionally, based on factors such as whether the current lighting is sufficient, whether the background noise exceeds the threshold, whether the user's recent physiological signals are stable, and whether the speech is often interrupted due to weak breath, the confidence scores of the facial expression and physiological and speech emotion probability distributions are determined respectively.

[0083] Step 402: Determine the weight of each modality based on the confidence score.

[0084] The weight of each mode is an adjustment coefficient for that mode, which is used to amplify the influence of the reliable mode during fusion. Optionally, based on the confidence score of the probability distribution of facial expression emotion. Confidence score of physiological emotion probability distribution and the confidence score of the probability distribution of speech emotion The weights of the probability distribution of facial expressions and emotions are determined by gating weights. Weights of the probability distribution of physiological emotions and the weights of the probability distribution of voice emotion . ,in, and These are learnable parameters.

[0085] Step 403: Based on the weights of each modality, the probability distributions of facial expression emotion, physiological emotion, and speech emotion are weighted to obtain the fused emotion features.

[0086] Optionally, based on the weights of the probability distribution of facial expressions and emotions. Weights of the probability distribution of physiological emotions and the weights of the probability distribution of voice emotion and the probability distribution of facial expressions and emotions Probability distribution of facial expressions and emotions and the probability distribution of voice emotion Weighting is performed to obtain integrated emotional characteristics. .in, .

[0087] In this embodiment, the confidence score of each modality is calculated by introducing the current environmental state and the user's historical physical state. This makes the weight allocation no longer a fixed empirical value, but an adaptive adjustment based on the real-time scenario. This avoids the error of a single modality dominating the emotion judgment and improves the practicality and stability of the companion robot in real home or medical scenarios.

[0088] Based on the above embodiments, the dialogue voice also includes: questioning voice. This application also provides a feature extraction process in a multimodal companion robot dialogue quality assessment method. Figure 5 This application provides a schematic diagram of the feature extraction process in a multimodal companion robot dialogue quality assessment method, as illustrated in the embodiments of this application. Figure 5 As shown, in step 102 above, semantic features and prosodic features are extracted from the dialogue speech, including: Step 501: Perform speech recognition on the question and answer voices to obtain the question text and answer text.

[0089] The question voice is the original sound signal emitted by the user when asking a question to the companion robot. The answer voice is the voice content generated and played by the robot in response to the user's question. The question text is the Chinese text result obtained after speech recognition of the question voice. The answer text is the Chinese text result obtained after speech recognition of the answer voice.

[0090] Optionally, the question can be asked in voice. Answer voice The text is fed into a preset speech recognition model, and the corresponding question text is output. and answer text .

[0091] Step 502: Extract semantic features from the question and answer texts to obtain semantic features.

[0092] The semantic features include question semantic features and answer semantic features.

[0093] Optionally, the question text With the answer text The data is concatenated sequentially, input into a pre-trained language model, and after encoding, outputs two fixed-length semantic vectors, representing the question semantic features. and answer semantic features .

[0094] Step 503: Extract speech prosodic features from the question speech.

[0095] Among them, the prosodic features are non-linguistic acoustic information extracted from the question speech, which may include pitch fluctuations, loudness changes, speech rate, pause positions and durations.

[0096] Optionally, the question speech waveform is processed by frame segmentation, and the fundamental frequency, short-time energy, Mel frequency cepstral coefficients and inter-frame pause features of each frame are extracted as prosodic feature sequences.

[0097] In this embodiment, by performing speech recognition on the question and answer voices separately, the text content of the dialogue can be completely preserved, providing structured multi-turn dialogue data for subsequent semantic relevance analysis and contextual coherence assessment. After obtaining the question and answer texts, semantic features are extracted independently, accurately capturing the semantic mapping relationship between the user's question and the robot's answer, and avoiding the direct transmission of speech recognition errors to the semantic understanding stage.

[0098] Based on the above embodiments, the semantic features include: question semantic features and answer semantic features. This application also provides a feature extraction process in a multimodal companion robot dialogue quality assessment method. Figure 6 A schematic diagram of the feature extraction process in a multimodal companion robot dialogue quality assessment method provided in this application embodiment is shown below. Figure 6 As shown, in step 105 above, semantic relevance is scored based on facial expression features and semantic features to obtain a semantic relevance score, including: Step 601: Based on the facial expression features, perform attention enhancement on the question semantic features to obtain the enhanced question semantic features.

[0099] Among them, facial expression features are quantitative indicators that reflect the user's emotional state, extracted from facial images collected during questioning, including the intensity of micro-expressions and the degree of muscle activity in key areas.

[0100] Optionally, based on facial features semantic features of the question Weighting is applied to enhance attention, resulting in enhanced semantic features of the question. .

[0101] Step 602: Based on the enhanced semantic features of the question and the semantic features of the answer, perform semantic relevance scoring to obtain the semantic relevance score.

[0102] Optionally, based on the enhanced question semantic features and answer semantic features Cosine similarity is calculated to achieve semantic relevance scoring, thus obtaining a semantic relevance score.

[0103] In this embodiment, by introducing facial expression features as the source of attention weights, the user's non-verbal visual emotional state can be integrated into the semantic understanding process. This allows the model to dynamically focus on the semantic dimension most relevant to the current expression when processing text, effectively alleviating the ambiguity caused by the lack of emotional context in pure text semantic analysis. It also enhances the model's ability to perceive the user's implicit needs, making the semantic relevance score more reflective of whether the robot's answer truly solves the user's emotional problem.

[0104] Based on the above embodiments, this application also provides a process for determining the current dialogue quality assessment result in a multimodal companion robot dialogue quality assessment method. Figure 7 This application provides a flowchart illustrating the process of determining the current dialogue quality assessment result in a multimodal companion robot dialogue quality assessment method, as shown in the embodiments of this application. Figure 7 As shown, in step 108 above, obtaining the current dialogue quality assessment result of the companion robot based on the semantic relevance score, the emotional consistency score, and the contextual coherence score includes: Step 701: Determine the user's personalization correction coefficient based on the user profile.

[0105] The user profile includes the user's age, gender, occupation, average speaking speed, common vocabulary complexity, frequency of rhetorical questions, memory retention rate, comprehension time, emotional fluctuation range, response strength to comforting statements, preference for brief answers or detailed explanations, and preference for humor or seriousness. Age, gender, and occupation can be determined from the user's registration information; average speaking speed, common vocabulary complexity, and frequency of rhetorical questions can be determined based on the user's historical voice-text statistics; memory retention rate and comprehension time are determined based on historical interaction analysis, with memory retention rate indicating whether the user can remember the content of the previous round of dialogue, and comprehension time being the delay from question to answer; emotional fluctuation range and response strength to comforting statements are determined through historical physiological or facial expression data; preference for brief answers or detailed explanations, and preference for humor or seriousness, are based on historical satisfaction feedback.

[0106] Optionally, static attribute characteristics can be determined based on age, gender, and occupation; language habit characteristics can be determined based on average speech rate, complexity of commonly used vocabulary, and frequency of rhetorical questions; cognitive level characteristics can be determined based on memory retention rate and comprehension time; emotional sensitivity characteristics can be determined based on the amplitude of emotional fluctuations and the intensity of response to comforting statements; and interaction preference characteristics can be determined based on whether users prefer short answers or detailed explanations, and whether they prefer humor or a serious style. Thus, user characteristics can be determined based on static attribute characteristics, language habit characteristics, cognitive level characteristics, emotional sensitivity characteristics, and interaction preference characteristics. ,in, Used to indicate the dimension of a feature, that is, the existence of Features in 100 dimensions.

[0107] Based on user characteristics The correction coefficients are determined using a pre-set multilayer perceptron. .in, , 、 as well as All of these are trainable parameters pre-defined in the multilayer perceptron. This is the Sigmoid function, which restricts the output to the interval [0,1].

[0108] Step 702: Adjust the preset weights of multiple evaluation dimensions according to the personalized correction coefficients to obtain the evaluation weights of each evaluation dimension.

[0109] Optionally, based on the personalized correction coefficient, the correction coefficients corresponding to the preset weights of multiple evaluation dimensions are determined, and the preset weights of multiple evaluation dimensions are adjusted according to the correction coefficients corresponding to the preset weights of multiple evaluation dimensions to obtain the evaluation weights of each evaluation dimension.

[0110] Step 703: Based on the evaluation weights of each evaluation dimension, the semantic relevance score, emotional consistency score, and contextual coherence score are weighted to obtain the current dialogue quality evaluation result of the companion robot.

[0111] Optionally, the preset weights of multiple evaluation dimensions are adjusted according to personalized correction coefficients to obtain the evaluation weights of each evaluation dimension. The semantic relevance score, emotional consistency score, and contextual coherence score are then weighted according to the evaluation weights of each evaluation dimension to obtain the comprehensive quality score of the companion robot. .in, , For correction factor, Preset weights for semantic relevance scores, Score the semantic relevance. Preset weights for affective consistency scores. For emotional consistency score, Preset weights for contextual coherence scores, The score is based on contextual coherence.

[0112] Optionally, the current dialogue quality assessment result can be determined based on the overall quality score. The corresponding level can be determined based on the score as the current dialogue quality assessment result, or the score can be directly used as the current dialogue quality assessment result.

[0113] In this embodiment of the application, by deeply integrating user profiles into the evaluation process, the quality evaluation can be dynamically adapted to the individual characteristics of users, thereby making the evaluation results truly close to the cognitive abilities, emotional expression habits and interaction preferences of different users, avoiding the evaluation distortion caused by a one-size-fits-all approach, enhancing the evaluation system's ability to depict the real companionship effect, and improving the subjective consistency of human-computer interaction quality evaluation.

[0114] Based on the above embodiments, this application also provides a process for determining evaluation weights in a multimodal companion robot dialogue quality evaluation method. Figure 8This application provides a flowchart illustrating the process of determining evaluation weights in a multimodal-based method for evaluating the dialogue quality of a companion robot, as shown in the embodiments below. Figure 8 As shown, in step 702 above, the preset weights of multiple evaluation dimensions are adjusted according to the personalized correction coefficients to obtain the evaluation weights of each evaluation dimension, including: Step 801: Calculate the adjustment range based on the personalized correction coefficient.

[0115] Optionally, a preset multilayer perceptron is used to adjust the values ​​based on personalized coefficients. Calculate the adjustment range .in, .

[0116] Step 802: Based on the adjustment range, obtain the correction coefficient range for each evaluation dimension.

[0117] Optionally, based on the adjustment range Obtain the range of correction coefficients for each evaluation dimension. The correction factor ranges for the adjustment coefficients of each dimension are all within the correction factor range. Within the range, the corresponding adjustment range for each dimension They are all the same. That is, the correction coefficient for the semantic relevance score. Correction coefficient for affective consistency score Correction coefficient for context coherence score .

[0118] Step 803: Adjust the preset weights of multiple evaluation dimensions according to the correction coefficient range of each evaluation dimension to obtain the evaluation weight of each evaluation dimension.

[0119] Optionally, the preset weights of multiple evaluation dimensions are adjusted according to the correction coefficient range of each evaluation dimension and the weight adjustment formula of each evaluation dimension to obtain the evaluation weight of each evaluation dimension.

[0120] The weight adjustment formula for each evaluation dimension is as follows:

[0121]

[0122]

[0123]

[0124] Preset weights for semantic relevance scores, The weights for evaluating semantic relevance scores, This is the correction coefficient for the semantic relevance score. .

[0125] Preset weights for affective consistency scores. The weighting for the emotional consistency score. This is the correction factor for the emotional consistency score. .

[0126] Preset weights for contextual coherence scores, As the evaluation weight for contextual coherence score, This is a correction factor for the context coherence score. Among them, the adjustment range corresponding to each dimension They are all the same.

[0127] Correspondingly, the overall quality score .

[0128] For example, the companion robot needs to serve two users: a 78-year-old senior living alone (User A) and an 8-year-old child (User B). The preset weights are α=0.4, β=0.3, and γ=0.3.

[0129] User A's profile was extracted: Age: 78 years old; Average speaking speed: slow; Vocabulary complexity: low; Memory retention rate: low; Comprehension time: long; Emotional fluctuation: stable; Interaction preference: prefers short and repetitive answers. Based on the user profile, personalized adjustment coefficients were determined for each evaluation dimension. 1.2 0.9 and It is 1.0.

[0130] Based on the preset weights α=0.4, β=0.3, γ=0.3 and the correction coefficients for each evaluation dimension. 1.2 0.9 The weight is 1.0. This determines the evaluation weight of the semantic relevance score. Preset weights for affective consistency scores And the evaluation weight of contextual coherence score .

[0131] Suppose that in a certain interaction, the semantic relevance score of the companion robot's response is... =0.8, emotional consistency score is =0.6, context coherence score is =0.7.

[0132] So, what is User A's overall rating? =0.457×0.8+0.257×0.6+0.286×0.7=0.3656+0.1542+0.2002=0.72.

[0133] User profiles were extracted for User B: Age: 8 years old; Average speaking speed: fast; Vocabulary complexity: low; Memory retention rate: high; Comprehension time: short; Emotional fluctuation: active; Interaction preference: desires interesting and story-like responses. Personalization correction coefficients were determined based on the user profile, resulting in correction coefficients for each evaluation dimension. 0.8 For 1.3 and It is 1.2.

[0134] Based on the preset weights α=0.4, β=0.3, γ=0.3 and the correction coefficients for each evaluation dimension. 0.8 1.3 The weight is 1.2. This yields the evaluation weight for the semantic relevance score. Preset weights for affective consistency scores And the evaluation weight of contextual coherence score .

[0135] Suppose that in a certain interaction, the semantic relevance score of the companion robot's response is... =0.8, emotional consistency score is =0.6, context coherence score is =0.7.

[0136] So, user B's overall rating =0.299×0.8+0.364×0.6+337×0.7=0.2392+0.2184+0.2359=0.6935.

[0137] Results Analysis: For elderly person A, although the emotional score was low, the overall score still reached 0.72 due to the increased semantic weight, indicating that the answer basically met the elderly person's need for information accuracy. For child B, the low emotional score was more severely penalized, and the overall score was only 0.6935, lower than the elderly person's score, suggesting that the answer was not interesting enough for the child and needs optimization. If the traditional non-personalized scoring method (base weights 0.4, 0.3, 0.3) is used, the scores for both individuals would be 0.4×0.8+0.3×0.6+0.3×0.7=0.32+0.18+0.21=0.71, making it impossible to distinguish the true satisfaction of different users.

[0138] In this embodiment, by introducing a dual constraint mechanism of adjustment range and correction coefficient range, the personalized weight adjustment maintains sufficient flexibility to respond to user differences, while also possessing inherent stability and reasonable boundaries, preventing the distortion of evaluation weights due to fluctuations in profile features or data noise; thus, in the process of dynamic adaptation, it takes into account both personalized expression and system robustness, ensuring that the evaluation results are sensitive to changes in real needs, while not deviating from the logical consistency and quality benchmark of the overall evaluation system.

[0139] Based on the above embodiments, this application also provides a process for determining physiological and emotional characteristics in a multimodal companion robot dialogue quality assessment method. Figure 9 This application provides a flowchart illustrating the process of determining physiological and emotional features in a multimodal-based method for assessing the dialogue quality of a companion robot, as illustrated in the embodiments of this application. Figure 9 As shown, in step 104 above, emotion extraction is performed on the user's physiological signals to obtain physiological emotion features, including: Step 901: Extract local patterns from the user's physiological signals to obtain short-term local pattern features.

[0140] The user's physiological signals can include electrocardiogram (ECG), skin conductance response, respiratory rate, and photoplethysmography (PPG). ECG reflects heart rate variability and is related to emotion; skin conductance response reflects the degree of sympathetic nerve activation and is related to emotional intensity; respiratory rate reflects relaxation or tension; and PPG helps determine heart rate changes. Local pattern extraction refers to transient changes in physiological signals that have emotional indicative significance within a very short period, such as sudden sharp peaks, steep rising edges, or brief plateaus in the signal waveform. Short-term local pattern features can include transient changes such as sudden increases in heart rate and skin conductance spikes.

[0141] Optionally, a 1D convolutional layer is used to segment the continuously acquired raw signal into time segments, and morphological recognition and anomaly detection are performed independently on each segment to obtain short-term local pattern features.

[0142] Step 902: Extract the periodic rhythm pattern from the user's physiological signal to obtain rhythmic pattern features.

[0143] Among them, the periodic rhythm pattern refers to the recurring, stable, and continuous fluctuation pattern in physiological signals, reflecting the rhythmic regulatory state maintained by the human autonomic nervous system under certain emotional states. A characteristic of the rhythmic pattern can be the periodic variation in respiratory rate, reflecting the sustained state of emotion.

[0144] Optionally, a temporal convolutional network can be used to extract periodic patterns from the user's physiological signals to obtain rhythmic pattern features.

[0145] Step 903: Capture the user's physiological signals over time to obtain trend characteristics.

[0146] Among them, the trend characteristics refer to the directional evolution path of physiological signals over time during the entire interaction process, reflecting the overall dynamic trend of emotional state from the beginning to the end.

[0147] Optionally, a convolutional self-attention mechanism can be used to model the overall evolution trend of physiological signals throughout the entire dialogue cycle, distinguishing between transient fluctuations and genuine emotional changes, thereby obtaining trend features.

[0148] Step 904: Extract emotions based on short-term local pattern features, rhythmic pattern features, and trend features to obtain physiological emotional features.

[0149] Optionally, short-term local pattern features, rhythmic pattern features, and trend features are co-modeled to obtain the user's emotional features in multiple dimensions. These emotional features are then mapped to determine the user's physiological emotional features. Alternatively, short-term local pattern features, rhythmic pattern features, and trend features are co-modeled to obtain the user's emotional features in multiple dimensions. A pre-defined classifier model is then used to derive the physiological emotional features based on these multiple emotional features.

[0150] In this embodiment, local pattern extraction captures the user's emotional triggering response at the moment of dialogue, enabling the evaluation system to be sensitive to sudden emotional changes. Secondly, periodic pattern extraction is used to characterize the stability features of the emotion maintenance phase, ensuring accurate identification of persistent emotional states. Thirdly, temporal capture reflects the development direction and evolution rhythm of emotions throughout the interaction process, giving the evaluation a sense of temporal depth and dynamic continuity. The three are organically integrated so that the generated physiological emotional features neither rely solely on instantaneous signals nor ignore long-term trends, nor sever the rhythmic basis. This achieves the unity of micro-response, meso-endurance, and macro-evolution in the time dimension, improving the completeness of physiological signals for emotion recognition.

[0151] Based on the same inventive concept, this application also provides a multimodal companion robot dialogue quality assessment device corresponding to the multimodal companion robot dialogue quality assessment method. Since the principle of the device in this application is similar to the multimodal companion robot dialogue quality assessment method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0152] Figure 10 A schematic diagram of a multimodal companion robot dialogue quality assessment device provided in this application embodiment is shown below. Figure 10As shown, the device includes: a data acquisition module 1001, used to acquire multimodal raw data of the current dialogue round collected by the companion robot during its interaction with the user. The multimodal raw data includes: dialogue voice, facial expression images and user physiological signals. The first extraction module 1002 is used to extract semantic features and prosodic features from the dialogue speech; The recognition module 1003 is used to perform facial expression recognition on facial expression images to obtain expression features; The second extraction module 1004 is used to extract emotions from the user's physiological signals to obtain physiological and emotional features; The first scoring module 1005 is used to score semantic relevance based on facial expression features and semantic features, and obtain a semantic relevance score. The second scoring module 1006 is used to score emotional consistency based on facial expression images, physiological and emotional characteristics and speech prosody characteristics, and to obtain an emotional consistency score. The third scoring module 1007 is used to score the contextual coherence based on the answer text corresponding to the answer voice in the dialogue, the historical dialogue text of the historical dialogue rounds, and facial expression features, and to obtain a contextual coherence score. The determination module 1008 is used to obtain the current dialogue quality assessment result of the companion robot based on the semantic relevance score, the sentiment consistency score, and the contextual coherence score.

[0153] In one possible implementation, the second scoring module 1006 is specifically used to: extract emotional features from facial expression images to obtain an expression emotion probability distribution; Emotional features are extracted from physiological and emotional characteristics to obtain the physiological and emotional probability distribution; Emotional features are extracted from the prosodic features of speech to obtain the probability distribution of speech emotion; Based on the probability distributions of facial expressions, physiological emotions, and speech emotions, fused emotional features are obtained. Based on the dialogue context of the current dialogue round, obtain the expected user emotional characteristics; An emotional consistency score is obtained by combining the emotional characteristics of the user with the expected emotional characteristics of the user.

[0154] In one possible implementation, the second scoring module 1006 is specifically used to: extract spatial features from each frame of facial expression image to obtain local texture features of key facial regions; Temporal dynamic modeling is performed on multiple consecutive frames of facial expression images to obtain facial expression change features; Facial motion recognition is performed on facial expression images to obtain facial motion features; Based on local texture features, facial expression change features, and facial movement features, the probability distribution of facial expression emotion is obtained.

[0155] In one possible implementation, the second scoring module 1006 is specifically used to: calculate the confidence score corresponding to the emotional probability distribution of each modality in the facial expression emotional probability distribution, physiological emotional probability distribution, and voice emotional probability distribution based on the current environmental state and the user's historical physical state. The weights of each modality are determined based on the confidence scores; Based on the weights of each modality, the probability distributions of facial expression emotion, physiological emotion, and speech emotion are weighted to obtain fused emotion features.

[0156] In one possible implementation, the dialogue speech further includes: a question speech; the first extraction module 1002 is specifically used to: perform speech recognition on the question speech and the answer speech to obtain the question text and the answer text; Semantic features are extracted from the question and answer texts to obtain semantic features; Extracting prosodic features from the spoken questions.

[0157] In one possible implementation, the first extraction module 1002 is specifically used to: perform attention enhancement on the question semantic features based on facial expression features to obtain enhanced question semantic features; The semantic relevance score is obtained by scoring the semantic relevance based on the enhanced semantic features of the question and the semantic features of the answer.

[0158] In one possible implementation, the determining module 1008 is specifically used to: determine the user's personalized correction coefficient based on the user's user profile; Based on the personalized correction coefficient, the preset weights of multiple evaluation dimensions are adjusted to obtain the evaluation weights of each evaluation dimension. Based on the evaluation weights of each evaluation dimension, the semantic relevance score, emotional consistency score, and contextual coherence score are weighted to obtain the current dialogue quality evaluation result of the companion robot.

[0159] In one possible implementation, the determining module 1008 is specifically used to: calculate the adjustment range based on the personalized correction coefficient; Based on the adjustment range, obtain the correction coefficient range for each evaluation dimension; Based on the correction coefficient range of each evaluation dimension, the preset weights of multiple evaluation dimensions are adjusted to obtain the evaluation weights of each evaluation dimension.

[0160] In one possible implementation, the second extraction module 1004 is specifically used to: extract local patterns from the user's physiological signals to obtain short-term local pattern features; Periodic pattern extraction is performed on user physiological signals to obtain rhythmic pattern features; By capturing users' physiological signals over time, we can obtain their trend characteristics. Emotional features are extracted based on short-term local pattern features, rhythmic pattern features, and trend features to obtain physiological emotional features.

[0161] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0162] This application also provides a computer device. Figure 11 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 11 As shown, the system includes a processor 1101, a memory 1102, and optionally, a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101. When the computer device is running, the processor 1101 and the memory 1102 communicate via the bus 1103. When the machine-readable instructions are executed by the processor 1101, the steps of the aforementioned multimodal-based companion robot dialogue quality assessment method are performed. The computer device can be the control device for the companion robot or an external computer device. When the computer device is the control device for the companion robot, the current dialogue quality assessment result of the companion robot is determined by acquiring the multimodal raw data of the current dialogue round collected during the interaction between the companion robot and the user. When the computer device is an external device, the companion robot's control device collects multimodal raw data of the current dialogue round during the interaction between the companion robot and the user, and then sends the multimodal raw data to the computer device via a wired or wireless network. The computer device determines the current dialogue quality assessment result of the companion robot based on the multimodal raw data, and sends the current dialogue quality assessment result to the robot's control device via a wired or wireless network.

[0163] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method for evaluating the dialogue quality of a multimodal companion robot.

[0164] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0165] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0166] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for evaluating the dialogue quality of a companion robot based on multimodal communication, characterized in that, The method includes: The system acquires multimodal raw data of the current dialogue round collected by the companion robot during its interaction with the user. The multimodal raw data includes: dialogue voice, facial expression images, and user physiological signals. Semantic features and prosodic features are extracted from the dialogue speech; The facial expression image is subjected to expression recognition to obtain expression features; Emotion extraction is performed on the user's physiological signals to obtain physiological emotional features; A semantic relevance score is obtained by scoring the semantic relevance based on the facial expression features and the semantic features. An emotional consistency score is obtained by scoring the facial expression image, the physiological emotional features, and the speech prosody features. The context coherence score is obtained by scoring the context coherence based on the response text corresponding to the response voice in the dialogue, the historical dialogue text of the historical dialogue rounds, and the facial expression features. The current dialogue quality assessment result of the companion robot is obtained based on the semantic relevance score, the emotional consistency score, and the contextual coherence score.

2. The method according to claim 1, characterized in that, The process of scoring emotional consistency based on the facial expression image, the physiological emotional features, and the speech prosody features to obtain an emotional consistency score includes: Emotional features are extracted from the facial expression images to obtain the probability distribution of facial expression emotions; Emotional feature extraction is performed on the aforementioned physiological and emotional features to obtain the physiological and emotional probability distribution; Emotional features are extracted from the prosodic features of the speech to obtain the speech emotion probability distribution; Based on the probability distribution of facial expressions, the probability distribution of physiological emotions, and the probability distribution of speech emotions, fused emotional features are obtained; Based on the dialogue context of the current dialogue round, obtain the expected user emotional characteristics; The emotional consistency score is obtained by performing an emotional consistency score based on the fused emotional features and the expected user emotional features.

3. The method according to claim 2, characterized in that, The step of extracting emotional features from the facial expression image to obtain an expression emotion probability distribution includes: Spatial features are extracted from each frame of the facial expression image to obtain local texture features of key facial regions; Temporal dynamic modeling is performed on multiple consecutive frames of the facial expression images to obtain expression change features; Facial motion recognition is performed on the facial expression image to obtain facial motion features; The probability distribution of facial expression is obtained based on the local texture features, the expression change features, and the facial movement features.

4. The method according to claim 2, characterized in that, The step of obtaining fused emotional features based on the facial expression emotion probability distribution, the physiological emotion probability distribution, and the speech emotion probability distribution includes: Based on the current environmental state and the user's historical physical state, calculate the confidence scores corresponding to the emotional probability distributions of the facial expressions, physiological emotions, and voice emotions, as well as the emotional probability distributions of each modality. The weights of each modality are determined based on the confidence scores. Based on the weights of each modality, the probability distributions of facial expressions, physiological emotions, and speech emotions are weighted to obtain the fused emotional features.

5. The method according to claim 1, characterized in that, The dialogue speech also includes: questioning speech; the extraction of semantic features and prosodic features from the dialogue speech includes: Speech recognition is performed on the question and answer voices to obtain the question text and answer text; Semantic features are extracted from the question and answer texts to obtain the semantic features; Extract the prosodic features from the questioning speech.

6. The method according to claim 5, characterized in that, The semantic features include: question semantic features and answer semantic features; the step of scoring semantic relevance based on the facial expression features and the semantic features to obtain a semantic relevance score includes: Based on the facial expression features, attention enhancement is performed on the question semantic features to obtain enhanced question semantic features; The semantic relevance score is obtained by scoring the semantic relevance based on the enhanced question semantic features and the answer semantic features.

7. The method according to claim 1, characterized in that, The step of obtaining the current dialogue quality assessment result of the companion robot based on the semantic relevance score, the emotional consistency score, and the contextual coherence score includes: Based on the user profile, determine the user's personalized correction coefficient; Based on the personalized correction coefficient, the preset weights of multiple evaluation dimensions are adjusted to obtain the evaluation weights of each evaluation dimension. Based on the evaluation weights of each evaluation dimension, the semantic relevance score, the emotional consistency score, and the contextual coherence score are weighted to obtain the current dialogue quality evaluation result of the companion robot.

8. The method according to claim 7, characterized in that, The step of adjusting the preset weights of multiple evaluation dimensions according to the personalized correction coefficient to obtain the evaluation weights of each evaluation dimension includes: The adjustment range is calculated based on the personalized correction coefficient. Based on the adjustment range, obtain the correction coefficient range for each of the evaluation dimensions; Based on the correction coefficient range of each evaluation dimension, the preset weights of the multiple evaluation dimensions are adjusted to obtain the evaluation weights of each evaluation dimension.

9. The method according to claim 1, characterized in that, The process of extracting emotions from the user's physiological signals to obtain physiological emotional features includes: Local pattern extraction is performed on the user's physiological signals to obtain short-time local pattern features; Periodic pattern extraction is performed on the user's physiological signals to obtain rhythmic pattern features; The user's physiological signals are captured in time series to obtain trend characteristics. Emotion extraction is performed based on the short-term local pattern features, the rhythmic pattern features, and the trend features to obtain the physiological emotion features.

10. A computer device, characterized in that, include: The processor and memory, the memory storing machine-readable instructions executable by the processor, wherein when the computer device is running, the processor executes the machine-readable instructions to perform the steps of the multimodal companion robot dialogue quality assessment method as described in any one of claims 1 to 9.

Images