Robot interactive question-answering method and system based on large service model

Abstract

The invention discloses a robot interactive question-answering method and system based on a large service model, and belongs to the technical field of intelligent robots, and the method comprises the steps: carrying out the feature extraction and fusion of the original input of a user, obtaining a multi-modal vector, building a recognition model, outputting an emotion recognition tag, an intention recognition tag, a user state vector and a session history, fusing the user state vector with a domain knowledge graph, outputting a structure reasoning vector and a joint condition vector, and splicing a knowledge abstract vector with a condition fusion vector to obtain the user state vector as the input of a text generator; and outputting a natural language answer and a user state vector, mapping the natural language answer into a multi-modal behavior action, and finally outputting a voice answer signal and the multi-modal behavior action. According to the method, the answer strategy of the robot is dynamically adjusted according to the emotional state of the user by combining context awareness and knowledge reasoning technologies, so that intelligent interaction of emotional resonance and professional depth coexistence is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence robots, and particularly relates to a robot interaction question and answer method and system based on a service large model. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, service robots based on large-scale language models have gradually become the core component of various industry services, especially in customer service, intelligent assistants, and robot interaction fields. In recent years, natural language processing (NLP) technology based on deep learning has made significant breakthroughs, and large pre-training models (such as GPT series, BERT, etc.) have performed well in understanding and generating natural language. These models, trained on a large amount of text data, can generate accurate and fluent language output, greatly improving the dialogue ability and interaction experience of robots. However, although these technologies perform well in single task execution, existing service robots still face some problems that need to be solved, mainly in two aspects.

[0003] Firstly, in terms of emotional intelligence, most current service robots mainly rely on rule-based sentiment analysis or simple emotion detection models. Although these sentiment recognition technologies can identify the emotional state of users (such as happy, angry, confused, etc.), they usually lack sufficient depth and dynamic adjustment capabilities. Therefore, when the robot faces users with high emotional fluctuations, it often cannot adaptively adjust its tone, content, or expression according to the user's emotional changes. For example, when the user is in a low mood, the robot may still respond in a mechanical tone, lacking appropriate empathy for emotions, resulting in poor user experience. For scenarios that require high emotional interaction, such as psychological counseling, customer service, etc., the existing robot technology obviously cannot fully meet the needs.

[0004] Secondly, although large language models perform well in handling regular knowledge, there are still significant limitations in the depth of domain knowledge reasoning. Many existing robot systems can only answer user questions through simple keyword matching or template generation, which makes them often appear incompetent when dealing with problems involving complex domain knowledge. For example, in professional fields such as medicine, law, technology, etc., robots are difficult to conduct in-depth professional reasoning and cannot dynamically obtain relevant data from knowledge graphs and reason. This makes the robot often unable to provide accurate and in-depth answers when facing user questions that are highly professional and require reasoning and analysis. In addition, since the knowledge graph and the language model usually operate independently, how to make the robot both effectively interact emotionally and deeply understand and reason professional knowledge has become a major challenge in technology implementation.

[0005] Currently, although some attempts combine sentiment analysis and knowledge reasoning to solve this problem, they are mostly fragmented solutions, lack a complete and systematic technical framework, and do not effectively combine the synergies between the two.

[0006] To this end, we propose a robot interactive question and answer method and system based on a large service model to solve the above problems. SUMMARY

[0007] The purpose of the present application is to solve the problem of coordination between sentiment understanding and professional reasoning in the prior art, and to propose a robot interactive question and answer method and system based on a large service model.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] The robot interactive question and answer method based on a large service model comprises:

[0010] S1: input the user's original input, perform feature extraction and fusion on the user's original input to obtain a multi-modal vector;

[0011] S2: establish a recognition model to recognize the emotional state and semantic intent of the user, and finally output an emotional probability distribution and an intent probability distribution, obtain an emotional recognition label and an intent recognition label by maximizing the probability, combine the multi-modal vector with the emotional recognition label and the intent recognition label to construct a user state vector, and save the conversation history in text form;

[0012] S3: input the domain knowledge graph, and fuse the user state vector with the domain knowledge graph;

[0013] According to the intent recognition label, use a task classification index function to retrieve a subgraph from the domain knowledge graph, and the subgraph retains nodes and edges highly related to the current task intent;

[0014] Extract vector representations of all user input sentences from the conversation history using a pre-trained language model BERT;

[0015] Construct node embedding representations of the subgraph and introduce an innovative emotion-sensitive attention mechanism; encode the nodes using a two-layer graph neural network;

[0016] Get each node embedding, do cosine matching with the conditional fusion vector, select the top-K node embeddings to form a structured reasoning result; the conditional fusion vector contains the splicing results of multi-modal semantics, emotion labels, intent labels, and context representations;

[0017] S4: average pooling the structured entity vector in the structured inference result to obtain a knowledge summary vector, and concatenating the knowledge summary vector with a condition fusion vector to obtain a user state vector as an input of a text generator; outputting a natural language answer and the user state vector;

[0018] S5: mapping the natural language answer into a multi-modal behavior action, the multi-modal behavior action including a speech tone, a facial expression, a body action, and a GUI display;

[0019] Finally, outputting a speech answer signal and the multi-modal behavior action.

[0020] Preferably, the user original input includes text input by the user, speech data of the user, and a video frame image.

[0021] Preferably, the recognition model adopts a multi-task modeling structure, and emotion recognition and intent recognition are modeled as two branch tasks, each branch being composed of three fully connected networks, and a bidirectional attention structure being inserted into a middle layer to take the middle hidden state of the current branch as a Query and the other branch as a Key and a Value.

[0022] Preferably, the loss function in the recognition model includes a standard cross-entropy classification loss of emotion and intent and a collaborative modeling regular.

[0023] Preferably, a graph adjacency masking matrix is introduced in step S3, the graph adjacency masking matrix indicating whether a connection between nodes is allowed, and used to improve the model's inhibition ability to an error inference path under a high emotional state of the user.

[0024] Preferably, the multi-modal behavior action is a set of behavior control vectors of all multi-modal channels, the behavior control vector being generated by channel mapping of the user state vector based on a behavior decoupling mapping network to generate a control vector of each channel.

[0025] Preferably, a cross-channel emotion balance regular term is introduced in the process of generating the control vector of each channel, the cross-channel emotion balance regular term constraining the control ability of emotion on each channel to keep collaborative consistency, and when it tends to 0, it indicates that the response trends of different channels to changes in emotion input are consistent.

[0026] The robot interactive question and answer system based on a service large model comprises:

[0027] A user input recognition module, the user input recognition module inputting a user original input, performing feature extraction and fusion on the user original input to obtain a multi-modal vector;

[0028] An emotion intention recognition module, which establishes a recognition model to identify the emotional state and semantic intention of a user, finally outputs an emotion probability distribution and an intention probability distribution, and obtains emotion recognition labels and intention recognition labels through the maximum probability; the emotion recognition labels and the intention recognition labels are combined with a multi-modal vector to construct a user state vector; and a conversation history is saved in a text form;

[0029] A semantic reasoning module, which inputs a domain knowledge graph, fuses the user state vector with the domain knowledge graph, retrieves a subgraph from the domain knowledge graph according to the intention recognition labels using a task classification index function, the subgraph retains nodes and edges highly relevant to the current task intention, extracts vector representations of all user input sentences through a pre-trained language model BERT for the conversation history, constructs node embedding representations of the subgraph and introduces an innovative emotion-sensitive attention mechanism, encodes nodes using a two-layer graph neural network, obtains each node embedding, performs cosine matching with a conditional fusion vector, and selects Top-K node embeddings to constitute a structured reasoning result; the conditional fusion vector includes a splicing result of multi-modal semantics, emotion labels, intention labels and context representations;

[0030] An output generation module, which performs average pooling on structured entity vectors in the structured reasoning result to obtain a knowledge summary vector, splices the knowledge summary vector with a conditional fusion vector to obtain a user state vector as an input of a text generator, and outputs a natural language answer and the user state vector;

[0031] An output execution module, which maps the natural language answer into a multi-modal behavior action, the multi-modal behavior action includes a voice tone, a facial expression, a body movement and a GUI display, and finally outputs a voice answer signal and the multi-modal behavior action.

[0032] In summary, the technical effects and advantages of the present application are as follows: the present application can adjust the tone and content of the answer generated by the robot according to the real-time emotion analysis result, ensuring that the user can resonate with the robot in emotional communication; at the same time, by deeply combining the domain knowledge graph with the reasoning engine, the robot can not only provide surface answers when answering questions in a professional field, but also can reason and extract information from the knowledge graph for deep answers, and through multi-task learning, the performance of the robot in practical application is optimized, the robot can perform professional field reasoning tasks while performing emotion analysis, thereby improving its response ability in multiple dimensions in the same conversation. This multi-task cooperative work not only improves the answer quality of the robot, but also optimizes itself adaptively through a real-time feedback mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1The method flowchart in the present application;

[0034] Figure 2 The system structure diagram in the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application.

[0036] As Figure 1 shown, the robot interactive question and answer method based on a service large model comprises:

[0037] S1: inputting a user original input, performing feature extraction and fusion on the user original input to obtain a multi-modal vector;

[0038] S2: establishing an identification model to identify the emotional state and semantic intention of the user, finally outputting an emotional probability distribution and an intention probability distribution, and obtaining an emotional recognition label and an intention recognition label through a probability maximum value; combining the emotional recognition label and the intention recognition label with the multi-modal vector to construct a user state vector; saving a conversation history in a text form;

[0039] S3: inputting a domain knowledge graph, and fusing the user state vector with the domain knowledge graph;

[0040] retrieving a subgraph from the domain knowledge graph according to the intention recognition label using a task classification index function, the subgraph retaining nodes and edges highly relevant to a current task intention;

[0041] extracting vector representations of all user input sentences from the conversation history through a pre-trained language model BERT;

[0042] constructing node embedding representations of the subgraph, and introducing an innovative emotion-sensitive attention mechanism; encoding nodes using a two-layer graph neural network;

[0043] obtaining each node embedding, performing cosine matching with a conditional fusion vector, and selecting Top-K node embeddings to constitute a structured reasoning result; the conditional fusion vector comprises a splicing result of multi-modal semantics, an emotional label, an intention label, and a context representation;

[0044] S4: performing average pooling on a structured entity vector in the structured reasoning result to obtain a knowledge summary vector, splicing the knowledge summary vector with a conditional fusion vector to obtain a user state vector as an input of a text generator; and outputting a natural language answer and the user state vector;

[0045] S5: Map the natural language answer to multi-modal behavior actions, including speech tone, facial expression, body movement, GUI display;

[0046] Finally output the speech answer signal and multi-modal behavior actions.

[0047] The specific implementation steps are as follows:

[0048] Step 1: Multi-modal user input analysis and unified modeling

[0049] The goal of this step is to convert the original multi-modal user input collected by the service robot, including text (semantic main axis), speech (tone and emotion), image (facial expression), into a unified semantic-emotion fusion vector X. This vector will serve as the basis for "user state modeling" in the entire system. Considering the real interaction scenarios faced by service robots, there are quality fluctuations in multi-modal input (such as speech environment noise, light blocking face, etc.) and inconsistency in emotional expression (for example, positive semantics but sad face), we introduce a modal attention mechanism and a cross-modal emotion alignment regularization term as two key innovative designs to enhance the adaptability and expression robustness of the system to real complex interaction scenarios.

[0050] Input:

[0051] The input processed in this step comes from the collection devices on the service robot side, including:

[0052] x text : Text input by the user, or converted from x audio by the speech recognition module;

[0053] x audio : User speech data collected by a microphone array with a sampling rate of 16 kHz;

[0054] x vis : Video frame images collected by a front-facing camera and extracted by a face detection module (such as MTCNN) to obtain facial region images.

[0055] Details of the step:

[0056] We use standard pre-trained models for feature extraction for each of the three modalities:

[0057] f text : Text semantic vector extracted by BERT-base, taking its [CLS] bit output, dimension 768;

[0058] Speech emotion features extracted by Mel spectrogram + three-layer bidirectional GRU network, output dimension 128;

[0059] The 512-dimensional features of the facial image processed by the ResNet18 model are used as visual emotion representations.

[0060] where f text represents the semantic principal axis, and and represent the speech and visual emotion features, respectively. We use a modal weighting fusion mechanism to integrate the three modal information to form a fusion vector X. This mechanism combines the semantic and emotion features in a linear weighted form, and obtains the modal weights through training, thereby realizing the integrated modeling of semantics and emotions.

[0061] To address the inconsistency of information expression between modalities and the problem of missing modalities, we propose the following fusion method:

[0062]

[0063] where:

[0064] α text ,α audio ,α vis are the weighting coefficients of the three modalities, representing the importance of the current modal information;

[0065] Each α i is calculated by a layer of multi-layer perception (MLP) from its modal feature and normalized (only calculated in the training stage);

[0066] is a learnable fusion matrix, is the bias term, and the output dimension d = 512;

[0067] All features are extracted by pre-trained networks according to their original sources, and all models are publicly reproducible.

[0068] The fusion process is to express the semantic features and emotion features in a unified weighted manner in high-dimensional space, which can more comprehensively reflect the user's true intention and emotional state. This modal fusion mechanism has been verified in the field of multi-modal sentiment analysis, and experimental verification shows that it can significantly improve the understanding robustness of service interaction systems.

[0069] To enhance the consistency of different modal emotion expressions in the training process, especially to prevent the speech and image modalities from deviating in the emotion direction, leading to confusion in downstream recognition, we introduce a regularization term as follows:

[0070]

[0071] where:

[0072] is the sentiment consistency regularization term;

[0073] λ is an adjustable regularization weight, and is recommended to be set to 0.1 to 0.3;

[0074] This option is only effective during the model training phase and is used to promote the consistency of the two emotion channels, speech and vision, in the feature space.

[0075] This design ensures that in the service robot emotion adaptation task, even if some modalities are damaged or semantically conflicting, the fusion result X still has expression stability and integrity.

[0076] This step outputs a unified fusion multimodal vector This serves as input for the subsequent user state modeling module. X simultaneously encodes the semantic backbone, voice emotion, and facial emotional expression, weighted by modal importance, ensuring robustness and contextual adaptability. A modal attention mechanism is designed to achieve the fusion of semantic and emotional information. A residual consistency regularization term is introduced to address the "cross-modal conflict of emotion" in service robot applications, effectively improving the robustness and expressive integrity of the features.

[0077] In the present invention, the fusion vector X is a clear and quantifiable real number vector, which is used to represent the comprehensive state of the user in the current interaction round, including the semantic backbone and multimodal emotional information. The quantization process of this vector mainly includes the following two stages: First, the original input data from the three modalities of text, voice, and image are extracted into vector features in numerical form through a standard deep neural network model. These features themselves have clear dimensions and numerical structures; secondly, the system performs a weighted combination of these three types of features according to the weights obtained by dynamic learning, and uniformly maps them into a real number vector of fixed length through a fully connected network to form the final fusion vector X. The numerical value of each dimension in this vector represents the expression intensity and emotional tendency of the information under different modalities in the current context. For example, the front dimension can focus more on semantic information, the middle part may integrate voice and visual emotional activation, and the back dimension captures cross-modal consistency or contradiction features. This vector form not only meets the input requirements of neural network calculations, but can also be directly used for subsequent state modeling and semantic reasoning.

[0078] Step 2: Emotion-intent joint recognition and user state modeling

[0079] The core goal of this step is to jointly identify the user's current emotional state y based on the fusion feature vector X output in the previous step emo With semantic intention intent , and combined with historical multi-round dialogue H t Construct a structured user state vector S t, for the subsequent knowledge graph reasoning module. This step undertakes the key responsibility of "user cognitive modeling" in this patent scheme, directly connecting input information and professional knowledge calling path. Unlike common emotion recognition or intent recognition tasks, service robots face complex contexts where emotions and tasks are intertwined and expressed mixedly, such as users expressing requests with emotional agitation or embedding decision-making intentions when venting, which poses a challenge to traditional "emotion / intent independent recognition" schemes.

[0080] We adopt a multi-task modeling structure, modeling emotion recognition and intent recognition as two branch tasks, but introducing a cross-attention mechanism to capture the potential synergistic relationship between them. The model structure is as follows:

[0081] Each branch consists of three fully connected networks, with dimensions of [512, 256, 64] at each layer and ReLU activation.

[0082] A bidirectional attention structure is inserted in the middle layer, with the current branch's intermediate hidden state as Query and the other branch as Key and Value.

[0083] The final outputs are the emotion probability distribution p(y emo |X) and the intent probability distribution p(y intent |X).

[0084] To suppress the common problem of "accurate intent prediction but completely deviated emotion", we introduce the following joint optimization term:

[0085]

[0086] Where:

[0087] KL(·||·) represents the KL divergence;

[0088] is a set of fixed intent-emotion joint distribution mapping functions, derived from real interaction data statistics, such as "complaint type intent" corresponding to negative emotion with high probability, and "knowledge inquiry" biased towards neutral emotion;

[0089] λ1 is the regularization term coefficient, controlling the weight of this constraint in the loss function (stable when set to 0.3);

[0090] This term takes effect during training, ensuring that the output emotion result does not conflict with the intent prediction logic.

[0091] At the same time, to avoid the "emotion dominating intent" phenomenon (such as users getting angry, the model misjudges as complaint rather than consultation), we design a semantic temperature control term:

[0092]

[0093] in:

[0094] p k (y intent ) and p k (y emo ) are the intention and emotion probabilities of the kth category respectively;

[0095] This item encourages the two distributions to maintain the necessary distinction under the premise of semantic linkage;

[0096] λ2 controls its weight and is usually set to 0.1;

[0097] ∈ is a small constant to prevent division by zero, set to 10 -6 ;

[0098] This mechanism ensures that the model does not simply rely on emotions to make intention judgments, and has stronger semantically independent modeling capabilities.

[0099] Final loss function:

[0100]

[0101] The first two are the standard cross-entropy classification losses for sentiment and intent, and the last two are innovatively designed collaborative modeling regularizations.

[0102] During model inference, the label is obtained by the Softmax maximum value:

[0103] y emo =argmaxp(y emo |X);

[0104] y intent =argmaxp(y intent |X).

[0105] The user state vector is constructed as follows:

[0106] S t ={X,y emo ,y intent ,H t}, as a dictionary structure for use by subsequent knowledge graph reasoning modules.

[0107] Output:

[0108] This step outputs the structured user state vector S t ,include:

[0109] The feature vector X of the previous step;

[0110] Current emotion recognition label y emo (e.g., happy, sad, angry, neutral);

[0111] Current intent recognition label y intent (e.g., request for help, confirmation, complaint, suggestion, etc.);

[0112] Current session history H t (text form);

[0113] This step not only completes the conversion of the multi-modal vector to the structural semantic label, but also solves the problems of "emotional-intention recognition split, overfitting of emotion, and intention ignoring context" commonly seen in traditional models in service interaction through the triple innovation mechanism of multi-task coordination structure + joint distribution regularization + temperature difference constraint. Especially in the scenario of this patent, the service robot needs to identify the "request" type of intention while perceiving the user's emotions such as impatience and disappointment, and combine these two types of information in subsequent responses to achieve a response strategy of "semantic accuracy + emotional adaptation". Therefore, this step is in the core position of "human state modeling" in the entire system and is the premise for the subsequent graph reasoning and language generation.

[0114] Step 3: Knowledge graph semantic reasoning combined with user state

[0115] This step aims to fuse the user state vector S t = {X, y emo , y intent , H t} generated in the previous step with the domain knowledge graph K (where: K: domain knowledge graph; E: entity set, such as "dizziness", "medical advice", "contraindicated drugs"; R: relationship set, defining the semantic relationship between entities, such as "symptom-corresponding-disease", "disease-advice-treatment method) to construct a dynamic semantic reasoning path affected by emotion and intention, and output a structured reasoning vector r kg and a joint condition vector R t . This is a bridge module connecting "user perception" and "knowledge generation" in the patent system, and its task is not only semantic matching but also dynamic knowledge extraction sensitive to emotion and context-dependent in service dialogue. Unlike traditional graph reasoning schemes, we first introduce an emotion-sensitive attention mechanism and a dynamic masking mechanism for graph adjacency matrices in this step to improve the robot's knowledge filtering and path guiding ability in high emotional states of the user, thereby solving the problem of "failing to hit professional knowledge points when the user's question is ambiguous or overly emotional".

[0116] This step includes four sub-modules: intent-oriented graph screening, context-enhanced graph embedding, emotion-sensitive propagation control, and fusion to generate structured reasoning results.

[0117] First, according to the user's intention y intent , a task classification index function Φ(yintent ) Retrieving subgraph from graph atlas This subgraph retains nodes and edges that are highly relevant to the current task intent. For example, the "request help" intent will preferentially hit "FAQ", "solution" class nodes.

[0118] Second, the historical context H t Extract the vector representation of all user input sentences through the pre-trained language model BERT, and average pooling to get Reflect the user's current dialogue background semantics.

[0119] Next, we construct the node embedding representation of the graph atlas subgraph, and introduce an innovative emotion-sensitive attention mechanism. Use two-layer graph neural network (GAT) to encode nodes, where neighbor aggregation adopts the following weight control:

[0120]

[0121] Where:

[0122] h i ,h j is the embedding representation of node i and its neighbor node j;

[0123] is a linear transformation matrix, is an attention projection vector;

[0124] is an emotion embedding vector, y meo Look up through the Embedding table, used to control the influence of emotion on graph propagation attention;

[0125] is an emotion mapping matrix;

[0126] α ij represents the emotion-aware attention weight of node j to node i.

[0127] The core of the weight control mechanism is that when calculating the attention weight between each target node and its adjacent node, not only their respective graph embedding features are considered, but also an additional vector encoded by the user's current emotional state is introduced to dynamically adjust the bias of attention. That is, the system will input the semantic representation of the target node, the semantic representation of the neighbor node, and the user emotion embedding into a nonlinear transformation function, and output the importance score of the neighbor node according to their relationship. All scores form an attention distribution after standardization, which is used as the weight of information propagation in the graph. This mechanism makes the graph neural network no longer a structure-driven average information transmission, but has the ability to adapt to the user's current psychological state. For example, when the user expresses confusion or anxiety, the attention weight of the "explanation type" and "comfort type" knowledge node will automatically increase, so as to preferentially guide the knowledge reasoning path to focus on content that relieves emotions and provides guidance.

[0128] This design realizes a structure-aware propagation path control mechanism based on emotion regulation. For example, when the user is in an "angry" state, focus on "problem explanation type" nodes rather than "basic definition" nodes, thereby achieving dynamic response to the user's focus.

[0129] In addition, to further enhance the model's ability to suppress incorrect reasoning paths under high emotional states of the user, we propose a graph adjacency masking matrix M∈{0,1} |ε|×|ε| and introduce a masking regularization term under emotional regulation:

[0130]

[0131] where:

[0132] M ij represents whether to allow nodes i and j to establish a connection, based on the emotional control strategy setting (for example, when y emo = anxious, mask the connection with "chitchat" type nodes);

[0133] α ij is the attention weight;

[0134] This term avoids information leakage by penalizing the residual attention on the masked edges, enhancing the effect of emotion-constrained graph structure control.

[0135] After GAT encodes the subgraph, it obtains the embedding of each node and the input condition vector R t = [X; y emo ; y intent ; h ctx ] do cosine matching, and select the top-K node embeddings to form the structured reasoning result:

[0136]

[0137] Final structure vector R t Together with the inference result r kg as the conditional input of the downstream natural language generation module.

[0138] This step outputs two variables:

[0139] r kg : structured graph inference result composed of Top-K node representations, retaining semantic and structural information;

[0140] R t : conditional fusion vector containing the splicing results of multi-modal semantics X, emotion label y emo , intent label y intent , and context representation h ctx .

[0141] This step innovatively introduces an emotion regulation attention mechanism and a graph masking regularization mechanism in knowledge graph inference in the context of service robots in this patent, achieving dynamic control of graph structure propagation paths and solving the problem of path misdirection when traditional graph inference faces "ambiguous expression" and "extreme emotion". The proposed attention structure not only dynamically controls information aggregation, but also makes the inference result more consistent with the user's potential focus under the influence of emotion.

[0142] Step 4: Conditional natural language generation and emotion regulation output

[0143] The goal of this step is to generate a natural language answer that meets the current user's intent and emotional state based on the output of Step 3 This step emphasizes the "knowledge-driven + emotion regulation" dual mechanism to ensure that the robot-generated answer is not only semantically accurate and emotionally appropriate, but also truly reflects the professional content brought by graph inference. To this end, we introduce a structured knowledge vector r kg into the text generation model through embedding integration to inject generation control while retaining multi-modal state and context information R t . In addition, by introducing an emotion regulation factor, the output result presents different styles, achieving a personalized question and answer experience.

[0144] The input of this step is the two output variables of Step 3:

[0145] r kg : structured entity result from the graph inference module, representing the K most relevant graph nodes (such as definitions, suggestions, and abnormalities) to the current user state;

[0146] R t = [X; e emo ; eintent ;h ctx ], wherein:

[0147] Multi-modal fusion representation;

[0148] Derived from the emotion label y emo embedding;

[0149] Derived from the intent label y intent embedding;

[0150] Historical dialogue context vector, encoded by BERT.

[0151] First, average pool the K structured entity vectors in r kg to get the knowledge summary vector

[0152]

[0153] wherein:

[0154] is the embedding vector for each knowledge graph node (e.g. encoded using TransE or BERT);

[0155] is the final graph knowledge summary representation.

[0156] Then, we concatenate it with R t to build the conditioned augmentation vector as input to the text generator:

[0157]

[0158] wherein:

[0159] Combining knowledge reasoning and user state;

[0160] Using a Transformer decoder with augmented condition vector;

[0161] a(y emo ): adjustment strength looked up from emotion label (e.g. anxious = 1.2, calm = 0.8);

[0162] Applying adjustment weights to the word distribution of generated text, making it more user-emotion-congruent in expression, e.g. using more mitigating language under an angry emotion.

[0163] For example:

[0164] When the emotional label is "anxiety", the model will tend to generate language with "Please be reassured", "We will help you";

[0165] When the emotional label is "neutral", the answer is more concise, such as "Please try to restart the device".

[0166] This design makes the generation process have clear control over the emotional style, while the graph structure entity ensures the semantic content to be professional.

[0167] Output:

[0168] The generated natural language answer with emotional style and graph knowledge content can be converted to speech or displayed as text;

[0169] Used to record the combination of this interaction conditions, which can be used for subsequent user satisfaction evaluation, adaptive optimization, etc.

[0170] Step 5: Robot reply behavior execution and multi-modal output presentation

[0171] The goal of this step is to use the answer generated by the natural language generation module together with the user state vector to drive the service robot to complete the multi-modal behavior presentation at the physical level. Unlike the conventional "text to speech" output, this step needs to implement a full-channel, multi-modal, emotion-adjustable, cross-modal style consistent output system. Considering that the service robot should have style distinction, humanized adaptability and scene robustness when facing different semantic intentions and emotional states, we designed a dynamic behavior regulation generator for unified generation and collaborative control of multi-modal behavior. Its essence is to convert "semantic content + emotional driving + context history + graph reasoning information" into a structured action sequence, realizing "semantic to action" cross-modal behavior modeling.

[0172] The innovation focus of this step is: not simply broadcasting text, but mapping the robot answer to multi-modal behavior actions, covering voice tone, facial expression, body movement, GUI display, etc. In actual interaction, such multi-channel response behavior determines the user's experience satisfaction and the robot's humanization degree.

[0173] First, build the behavior control vector for each channel. We introduce a "behavior decoupling mapping network" to map the input state to generate the control vector z c of each channel:

[0174]

[0175] wherein:

[0176] c denotes the channel type, is speech, is expression, is action, is graphical interface;

[0177] is the parameter matrix corresponding to the channel;

[0178] is the joint feature input of step four output;

[0179] output is the behavior instruction template (such as TTS parameters, expression ID, motion trajectory, UI template, etc.);

[0180] The structure can be implemented by a shallow linear transformation or MLP (2 layers, 64 units).

[0181] Emotion regulation mechanism design:

[0182] After generating the behavior control vector, in order to avoid the uneven distribution of the influence of the emotional state on different channels, a cross-channel emotion balancing regularization term is introduced to ensure that the response intensity of all channels to the emotion embedding dimension remains consistent, avoiding problems such as speech emphasis and flat expression imbalance:

[0183]

[0184] wherein:

[0185] z i ,z j are the behavior vectors of the two channels (such as speech and expression);

[0186] denotes the gradient sensitivity of the channel behavior to the emotion embedding;

[0187] tends to 0, indicating that the response trend of different channels to the change in emotional input is consistent;

[0188] λ is the regularization coefficient (recommended range 0.1-0.5).

[0189] This regularization term is an innovative design, which does not constrain the content of the behavior itself, but rather constrains the control ability of the emotion on each channel to maintain consistent consistency.

[0190] Then, each channel executes output according to its z c Execution output:

[0191] speech channel TTS module synthesizes speech according to Setting tone, speed, style, and synthesizing

[0192] Expression channel Expression control module maps To facial action codes, such as eye opening amplitude, mouth corner movement trajectory;

[0193] Action channel Robot controller converts To servo joint trajectory, realizing feedback such as nodding, retreating, etc.

[0194] GUI channel According to Activate screen prompts, icons or light to express emotional feedback.

[0195] After behavior synthesis, send to corresponding execution module through hardware API interface, such as:

[0196] After speech synthesis, send to sound card interface through hardware API interface;

[0197] After expression and action synthesis, send to rudder control module through hardware API interface;

[0198] GUI output result is sent to touch screen UI service module.

[0199] All behavior modules are triggered cooperatively to form an output behavior set that is consistent in emotional semantics, coordinated in multi-modal, and stable in response.

[0200] The final output result is:

[0201] The adjusted speech answer signal;

[0202] z c : Behavior control vector set of all multi-modal channels Directly drive robot multi-modal execution module.

[0203] This step introduces multi-channel behavior control mechanism on the basis of traditional speech broadcast, and builds a behavior generation framework that integrates semantic-emotion-graph state. In particular, we design an emotional consistency regularization term Innovatively ensures that the response style of each output channel of the robot is coordinated and consistent, solving the technical bottleneck of traditional service robots that "only talk but behave stiffly". The whole process not only has complete structure and clear execution, but also has practical deployment feasibility, which is the key link of this patent to land to "semantic-behavior collaborative service robot with emotional regulation".

[0204] The technical solutions in the embodiments of the present application have at least the following technical effects or advantages: the present application can adjust the tone and content of the answer generated by the robot according to the real-time emotion analysis result, so as to ensure that the user can resonate with the robot in emotional communication; meanwhile, by deeply combining the domain knowledge graph and the reasoning engine, the robot can not only provide surface answers when answering questions related to a professional field, but also can perform reasoning and extract information from the knowledge graph for deep answering, and by multi-task learning, the performance of the robot in actual application is optimized, the robot can perform a professional field reasoning task while performing emotion analysis, so that the response capability of the robot is improved in multiple dimensions in the same conversation. The multi-task cooperative work can not only improve the answer quality of the robot, but also can be self-adaptively optimized through a real-time feedback mechanism.

[0205] The embodiments of the present application also provide a robot interaction question and answer method and system based on a service large model, as shown in Figure 2 The embodiments of the present application also provide a robot interaction question and answer method and system based on a service large model, as shown in

[0206] A user input recognition module is configured to input a user original input, perform feature extraction on the user original input, and obtain a multi-modal vector by fusion;

[0207] An emotion and intention recognition module is configured to establish a recognition model, recognize an emotion state and a semantic intention of a user, finally output an emotion probability distribution and an intention probability distribution, obtain an emotion recognition label and an intention recognition label by a maximum probability, construct a user state vector by combining the emotion recognition label and the intention recognition label with the multi-modal vector, and save a conversation history in a text form;

[0208] A semantic reasoning module is configured to input a domain knowledge graph, fuse the user state vector with the domain knowledge graph, retrieve a subgraph from the domain knowledge graph according to the intention recognition label by using a task classification index function, the subgraph retains nodes and edges highly related to a current task intention, extract vector representations of all user input sentences of the conversation history by using a pre-training language model BERT, construct node embedding representations of the subgraph, and introduce an innovative emotion-sensitive attention mechanism; encode nodes by using a two-layer graph neural network; obtain each node embedding, perform cosine matching with a conditional fusion vector, and select Top-K node embeddings to constitute a structured reasoning result; the conditional fusion vector includes a splicing result of multi-modal semantics, an emotion label, an intention label, and a context representation;

[0209] An output generation module is configured to perform average pooling on structured entity vectors in the structured reasoning result, obtain a knowledge summary vector, splice the knowledge summary vector with a conditional fusion vector to obtain a user state vector as an input of a text generator, and output a natural language answer and the user state vector.

[0210] an output execution module that maps the natural language answer to a multi-modal behavior action, the multi-modal behavior action including a speech tone, a facial expression, a body action, a GUI presentation; and finally outputs a speech answer signal and the multi-modal behavior action.

[0211] The above merely provides the preferred but not limiting embodiments of the present application, and the protection scope of the present application should not be limited thereto, and any person skilled in the art should understand that any equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application should be covered within the protection scope of the present application.

Claims

1. A robot interactive question-answering method based on a large service model, characterized in that: include: S1: Input the user's original input, extract features from the user's original input and fuse them to obtain a multimodal vector; S2: Build a recognition model to identify the user's emotional state and semantic intent, ultimately outputting an emotion probability distribution and an intent probability distribution, and obtaining an emotion recognition label and an intent recognition label by maximizing the probability; combine the emotion recognition label and the intent recognition label with a multimodal vector to construct a user state vector; and save the conversation history in text form. S3: Input the domain knowledge graph and fuse the user state vector with the domain knowledge graph; Retrieving a subgraph from the domain knowledge graph using a task classification index function according to the intent identification label, wherein the subgraph retains nodes and edges that are highly relevant to the current task intent; Extracting vector representations of all user input sentences from the conversation history using the pre-trained language model BERT; Construct node embedding representations of subgraphs and introduce an innovative emotion-sensitive attention mechanism; encode nodes using a two-layer graph neural network; Obtain each node embedding, perform cosine matching with the conditional fusion vector, and select the top-K node embeddings to form the structured reasoning result; the conditional fusion vector contains the splicing result of multimodal semantics, emotion labels, intent labels, and context representation; S4: performing average pooling on the structured entity vectors in the structured reasoning result to obtain a knowledge summary vector, and concatenating the knowledge summary vector with the conditional fusion vector to obtain a user state vector as input to the text generator; Output natural language answer and user state vector; S5: Mapping the natural language response into multimodal behavioral actions, wherein the multimodal behavioral actions include voice intonation, facial expressions, body movements, and GUI displays; Finally, the output is a voice answer signal and multimodal behavior action.

2. The robot interactive question-answering method based on the service big model according to claim 1 is characterized in that: The user original input includes text input by the user, user voice data, and video frame images.

3. The robot interactive question-answering method based on the service big model according to claim 1 is characterized in that: The recognition model adopts a multi-task modeling structure, modeling emotion recognition and intent recognition as two branch tasks respectively. Each branch consists of a three-layer fully connected network, with a bidirectional attention structure inserted in the middle layer. The intermediate hidden state of the current branch is used as the query, and the other branch is used as the key and value.

4. The robot interactive question-answering method based on the service big model according to claim 1 is characterized in that: The loss function in the recognition model includes the standard cross-entropy classification loss and collaborative modeling regularization for emotion and intent.

5. The robot interactive question-answering method based on the service big model according to claim 1 is characterized in that: In step S3, a graph adjacency masking matrix is ​​introduced. The graph adjacency masking matrix indicates whether connections are allowed between nodes, and is used to improve the model's ability to suppress erroneous reasoning paths when the user is in a high-emotion state.

6. The robot interactive question-answering method based on the service big model according to claim 1 is characterized in that: The multimodal behavior action is a set of behavior control vectors for all multimodal channels. The behavior control vectors are generated by performing channel mapping on the user state vector based on a behavior decoupling mapping network to generate a control vector for each channel.

7. The robot interactive question-answering method based on the service big model according to claim 6 is characterized in that: In the process of generating the control vector for each channel, a cross-channel emotion balance regularization term is introduced. The cross-channel emotion balance regularization term constrains the control ability of emotions on each channel to maintain coordinated consistency. When it approaches 0, it indicates that the response trends of different channels to changes in emotion input are consistent.

8. A robot interactive question-answering system based on a large service model, characterized by: include: A user input recognition module, which receives the user's original input, extracts features from the user's original input, and fuses them to obtain a multimodal vector; An emotion and intention recognition module, which establishes a recognition model to identify the user's emotional state and semantic intent, ultimately outputs an emotion probability distribution and an intent probability distribution, obtains an emotion recognition label and an intent recognition label by maximizing the probability; combines the emotion recognition label and the intent recognition label with a multimodal vector to construct a user state vector; and saves the conversation history in text form; A semantic reasoning module takes a domain knowledge graph as input and fuses the user state vector with the domain knowledge graph. Based on the intent recognition label, it uses a task classification index function to retrieve a subgraph from the domain knowledge graph, where the subgraph retains nodes and edges that are highly relevant to the current task intent. It uses the pre-trained BERT language model to extract vector representations of all user input sentences from the conversation history. It constructs a node embedding representation of the subgraph and introduces an innovative emotion-sensitive attention mechanism. It uses a two-layer graph neural network to encode nodes. Obtain each node embedding, perform cosine matching with the conditional fusion vector, and select the top-K node embeddings to form the structured reasoning result; the conditional fusion vector contains the splicing result of multimodal semantics, emotion labels, intent labels, and context representation; An output generation module, which performs average pooling on the structured entity vectors in the structured reasoning result to obtain a knowledge summary vector, and concatenates the knowledge summary vector with the conditional fusion vector to obtain a user state vector as input to a text generator; Output natural language answer and user state vector; An output execution module maps the natural language answer into multimodal behavioral actions, wherein the multimodal behavioral actions include voice intonation, facial expressions, body movements, and GUI displays; and finally outputs a voice answer signal and multimodal behavioral actions.

Images