Response method, device and equipment for customer service robot

Abstract

The invention provides a response method, device and equipment for a customer service robot, and relates to the technical field of customer service robots, and the method comprises the steps: obtaining multi-mode request data inputted into a current session with the customer service robot by a user, and a conversation state of the current session; according to the request data of different modals and the modal features of the request data of the corresponding modals, constructing a multi-modal heterogeneous graph; fusing the different modal features based on the request data of the different modalities, the corresponding modal features, the dialogue states and the multi-modal heterogeneous graph to obtain multi-modal fusion features; inputting the multi-modal fusion feature into a pre-constructed user intention response model based on a causal relationship to obtain a response strategy of the customer service robot to the multi-modal request data; wherein the user intention response model is used for representing the causal relationship between the multi-modal interaction data input by the user and the response strategy of the customer service robot. The response accuracy of the customer service robot can be improved, and the user interaction experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of customer service robots, and more specifically, to a response method, apparatus, and device for a customer service robot. Background Art

[0002] Currently, most robot customer service can only process data of a single modality during conversations, and generally use linear splicing or fixed-weight attention mechanisms for modal fusion for multimodal data. However, the above methods make it difficult to dynamically adjust the weights of each modality according to the conversation scenario, resulting in the system being unable to effectively respond to environmental changes; it is unable to accurately understand complex context-related requests and cannot provide comprehensive and accurate responses; it lacks a dialogue state tracking mechanism with adjustable granularity, which affects the accuracy of dialogue understanding and task execution effects; and it mostly outputs responses to requests based on statistical correlations, which cannot accurately understand the user's intentions, resulting in large deviations in response content and a poor interactive experience. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a response method, device and equipment for a customer service robot, so as to solve the above-mentioned problems existing in the prior art, improve the response accuracy of the customer service robot, and enhance the user interaction experience.

[0004] In a first aspect, a response method of a customer service robot is provided, which may include:

[0005] Obtain multimodal request data entered by the user in the current session with the customer service robot, as well as the conversation state of the current session;

[0006] For request data of any modality, performing modal feature extraction on the request data of the modality to obtain modal features;

[0007] Constructing a multimodal heterogeneous graph based on the request data of different modalities and the corresponding modal features; wherein the multimodal heterogeneous graph is used to characterize the relationship between the request data of different modalities;

[0008] Based on the request data of the different modalities and the corresponding modal features, the conversation state and the multimodal heterogeneous graph, fusing the different modal features to obtain a multimodal fusion feature;

[0009] Inputting the multimodal fusion features into a pre-built causal-based user intention response model to obtain the customer service robot's response strategy for the multimodal request data; wherein the user intention response model is used to characterize the causal relationship between the multimodal interaction data input by the user and the customer service robot's response strategy;

[0010] Control the customer service robot to respond to the user according to the response strategy.

[0011] In an optional implementation, performing modal feature extraction on modal request data includes:

[0012] According to the configured modal feature extraction method corresponding to the request data of the modality, modal feature extraction is performed on the request data of the modality to obtain the modal features of the request data of the modality.

[0013] In an optional implementation, a multimodal heterogeneous graph is constructed based on request data of different modalities and corresponding modal features, including:

[0014] For request data of any modality, extracting a modality segment corresponding to a corresponding modality feature from the request data of the modality;

[0015] The modal segments corresponding to each modal feature are taken as nodes, and the modal features corresponding to each node are taken as attributes of the corresponding node;

[0016] If the correlation between any two nodes is greater than the configured correlation threshold, an edge is generated between the two nodes;

[0017] Based on each node and each edge, an initial multimodal heterogeneous graph is obtained;

[0018] The graph attention mechanism is used to update each node in the initial multimodal heterogeneous graph to obtain a multimodal heterogeneous graph.

[0019] In an optional implementation, a graph attention mechanism is used to update each node in the initial multimodal heterogeneous graph to obtain a multimodal heterogeneous graph, including:

[0020] For any node in the initial multimodal heterogeneous graph, calculate the attention coefficient of the node and all neighboring nodes;

[0021] According to the attention coefficients of the node and each neighboring node, the attributes of the node are updated to obtain an updated node;

[0022] Based on the updated nodes, a multimodal heterogeneous graph is obtained.

[0023] In an optional implementation, based on the request data of different modalities and the corresponding modal features, the conversation state, and the multimodal heterogeneous graph, the different modal features are fused to obtain a multimodal fusion feature, including:

[0024] The modal features, conversation states, and multimodal heterogeneous graphs of request data of different modalities are input into a pre-trained modal confidence estimation network to obtain the weight vector of each modal feature.

[0025] Based on the weight vector of each modal feature, the features of each modal are weighted and fused to obtain multimodal fusion features.

[0026] In an optional implementation, the method for obtaining the conversation status includes:

[0027] Inputting the multimodal request data into a pre-built dialog state tracking model to obtain the dialog state of the current session;

[0028] Among them, the dialogue state tracking model is used to process the historical multimodal interaction data of the user and the customer service robot in the historical rounds of the current conversation through a time series modeling mechanism, and combine the attention mechanism to obtain the dialogue state of the current conversation based on the multimodal request data.

[0029] In an optional implementation, the causal relationship-based user intention response model includes:

[0030] Input layer, used to input multimodal fusion features;

[0031] The causal feature enhancement layer is used to calculate the association weights of the multimodal fusion features using the Transformer multi-head attention, and use the association weights to weight and embed the multimodal fusion features into a matrix to obtain a causal enhancement feature vector;

[0032] The intention causal reasoning layer is used to perform dimension reduction clustering on the causal enhancement feature vector to generate multiple causal elements. Based on the multiple causal elements and the configured causal rule library, it constructs the user intention causal graph and the probability distribution of each causal element.

[0033] The intent confidence assessment layer determines the user intent category based on the user intent causal graph and causal enhancement feature vector;

[0034] The multimodal response generation layer controls the customer service robot to execute the business processing rules corresponding to different user intent categories according to the configured business processing rules to obtain a response strategy;

[0035] The output layer is used to output the response strategy.

[0036] In a second aspect, a response device of a customer service robot is provided, which may include:

[0037] An acquisition unit, configured to acquire multimodal request data input by a user into a current session with a customer service robot, and a dialog state of the current session; wherein the interaction data is multimodal interaction data;

[0038] an extraction unit, configured to extract modal features from request data of any modality in the multimodal request data to obtain modal features;

[0039] A construction unit, configured to construct a multimodal heterogeneous graph based on request data of different modalities and corresponding modal features; wherein the multimodal heterogeneous graph is used to characterize the relationship between request data of different modalities;

[0040] a fusion unit, configured to fuse the different modal features based on the request data of the different modalities and the corresponding modal features, the conversation state, and the multimodal heterogeneous graph to obtain a multimodal fusion feature;

[0041] A determination unit is used to input the multimodal fusion feature into a pre-built user intention response model based on causality to obtain the response strategy of the customer service robot to the multimodal request data; wherein the user intention response model is used to characterize the causal relationship between the multimodal interaction data input by the user and the response strategy of the customer service robot.

[0042] In a third aspect, an electronic device is provided, the electronic device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0043] Memory for storing computer programs;

[0044] The processor is configured to implement any of the method steps described in the first aspect when executing a program stored in the memory.

[0045] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the method steps described in the first aspect is implemented.

[0046] This application implements the processing of multimodal request data, and accurately and comprehensively perceives the user's emotions and intentions by extracting modal features and constructing heterogeneous graphs on multimodal request data. It then combines the dialogue state tracking of the current session to accurately understand complex context-related requests, introduces causal graphs to generate accurate and comprehensive response strategies, improves the response efficiency of the customer service robot, and improves the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0048] Figure 1An architectural diagram of a response system for a customer service robot provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of a response method of a customer service robot provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a response device of a customer service robot provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following, in conjunction with the accompanying drawings, provides a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the described embodiments represent only a portion of the embodiments of this application and do not constitute a complete set of embodiments. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this application without inventive effort are intended to fall within the scope of protection of this application. Unless otherwise defined, technical or scientific terms used in this application should have the same ordinary meanings as those understood by persons of ordinary skill in the art. The terms "first," "second," and similar expressions used in this application do not denote any order, quantity, or importance; they are merely used to distinguish between different components. Terms such as "include" or "comprising" mean that the element or object preceding the term includes the elements or objects listed after the term, and their equivalents, without excluding other elements or objects. Terms such as "connect," "couple," or "connected" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used solely to indicate relative positional relationships. When the absolute position of the described objects changes, the relative positional relationships may also change accordingly.

[0053] The response method of the customer service robot provided in the embodiment of the present application can be applied to Figure 1 In the system architecture shown in Figure 1As shown, the system may include: a terminal access layer (edge ​​side), an edge intelligent processing layer, a middle platform semantic and decision layer (cloud) and a customer service robot dialogue controller; the terminal access layer is communicated with the edge intelligent processing layer; the terminal device collects raw data and sends it to the terminal access layer (edge ​​side), which is pre-processed (noise reduction, framing) on ​​the edge side and uploaded to the edge intelligent processing layer; the edge intelligent processing layer performs real-time calculations (ASR, image recognition, emotion recognition), filters redundant data and reports key results to the middle platform semantic and decision layer (cloud); the middle platform semantic and decision layer (cloud) integrates multimodal data, generates response measurements and feeds back to the user through the customer service robot dialogue controller to complete closed-loop interaction.

[0054] The terminal access layer is used to receive the original multimodal data (voice, video) collected by the user through the terminal device (mobile phone, computer, etc.), and perform basic preprocessing (such as noise reduction, framing, encoding compression) on the original multimodal data to obtain the preprocessed multimodal data, and transmit the preprocessed multimodal data to the edge intelligent processing layer; wherein, the terminal device uses the local noise reduction model (deployed in the terminal App or edge node) to perform noise suppression and silence detection on the original voice data in the original multimodal data; obtain the preprocessed original voice data; transmit the preprocessed original voice data ( PCM (16kHz / 16bit) is compressed using G.711 or Opus codecs (e.g., Opus compression to 1 / 10 of its original size) and transmitted as binary audio frames over WebSocket. The pre-processed raw voice data is truncated into small frames of 20-50 milliseconds per frame to facilitate real-time transmission and processing. Data is cached on terminal devices or edge nodes, supporting resumable transmission and recording encryption to ensure data privacy. Video frames are trimmed, retaining only key frames (e.g., frames with significant facial expressions) to reduce the amount of transmitted data. A packet loss retransmission mechanism is used to ensure data integrity.

[0055] The edge intelligent processing layer is used to deploy AI models (ASR, image recognition, emotion recognition) at edge computing nodes close to users (such as urban data centers), perform real-time computing tasks, perform in-depth processing on data uploaded by terminals, extract key information (such as speech-to-text, user emotions, visual clues), and filter redundant data to reduce cloud load; specifically, real-time speech recognition (ASR): converts audio signals into text, supports streaming recognition (transcribe while listening), and outputs partial results in real time; video frame extraction and image understanding model (OCR / object recognition): extracts key visual information (such as facial expressions, product logos, environmental features) from video frames; emotion recognition model (expression + intonation): combines the timbre / intonation features of speech with the characteristics of the video The edge nodes use a dimensionality reduction filtering mechanism to report only summary features (such as emotion tags and key events) and key frames (such as video frames with significant changes in expression), reducing the amount of data uploaded to the cloud and reducing bandwidth consumption. The edge node distribution strategies include: proximity deployment: deploying edge nodes in user-dense areas, routing user WebSocket connections to the nearest node through DNS resolution or access gateways to reduce transmission delays (voice round-trip delays are reduced to tens of milliseconds); computing and storage sinking: edge nodes cache high-frequency models (such as speech recognition and dialogue models) and business data (such as answers to frequently asked questions and product knowledge), and can directly reply locally when the cache is hit, reducing cloud interactions. For video analysis, image recognition models or reference images for high-frequency products are cached to accelerate local recognition. Layered network transmission: Original data (voice / video) is transmitted from the terminal to the edge, and only results (such as text transcription and intent summary) are transmitted from the edge to the cloud, reducing backbone network bandwidth usage and protecting user privacy. Model pruning and lightweight deployment: Different model versions are deployed based on the hardware capabilities of edge nodes. Nodes with weak computing power (such as embedded devices) deploy lightweight models (such as pruned / quantized small-scale speech recognition models), sacrificing a small amount of accuracy in exchange for real-time performance. Nodes with abundant computing resources (such as edge data centers) deploy lightweight models (such as pruned / quantized small-scale speech recognition models), sacrificing a small amount of accuracy in exchange for real-time performance. Deploy complete models (such as large, high-precision speech recognition models) to provide the highest-precision services. Model pruning techniques, including reducing the number of neural network layers, bit-width quantization, and distillation, can reduce the number of parameters in large models by over 50% with manageable accuracy loss, supporting real-time inference on the CPU. Scenario-specific model selection: For example, deploying noise-resistant training models in noisy factory environments ensures recognition rates. Elastic scaling and coverage: Adding edge nodes to densely populated areas during peak business hours. When there is no local edge in remote areas, requests are routed to the cloud or nearby edge nodes, dynamically evaluating load and business volume to adjust the number and distribution of nodes.

[0056] The semantic and decision-making layer of the middle platform (cloud) is used for global semantic understanding, conversation management, business decision-making and action execution, integrating multimodal data (speech transcription text, image analysis results, emotion labels), generating the final reply strategy and triggering business actions (such as creating a work order, sending a text message); server-side preprocessing tasks: Voice endpoint detection (VAD): combining silence duration threshold, voice activity energy detection and syntactic completion judgment (dependent on the language model) to detect the end of the sentence. For example, when more than 800 milliseconds of silence are detected and the sentence has an ending tone, the endpoint is determined, and the ASR end event is triggered to notify the dialogue management module to generate a reply; Speech Separation: When overlapping speech is detected (such as two people speaking at the same time), the source separation network is started to divide the mixed speech into independent tracks, discard the background / irrelevant speech tracks, and retain the main track input ASR. A lightweight time-frequency mask model (window length of tens of milliseconds) is used to ensure low latency;

[0057] The customer service robot dialogue controller is used to implement two-way voice stream synchronization based on WebSocket OpenAPI, customer service robot participation in audio and video dialogues, and session event scheduling (such as retries, interruptions, and multi-round questions and answers). It is the core control module for direct interaction between the system and users, responsible for coordinating resources at all levels to ensure the smoothness and real-time nature of the dialogue process; specifically, two-way voice stream synchronization: real-time synchronization of voice streams between the terminal and the cloud is achieved through WebSocket long connections, ensuring seamless connection between user voice input and system responses; customer service robot participation in audio and video dialogues: customer service robots (such as virtual customer service images) conduct multimodal interactions with users through audio and video channels (such as voice responses and expression feedback) to enhance interactive immersion; session event scheduling: manage key events in the conversation life cycle (such as retries: re-initiating connection after network interruption; interruptions: safe exit under abnormal circumstances; multi-round questions and answers: dynamically adjusting response strategies based on user questions) to ensure the continuity and stability of the dialogue process; unified communication protocol: based on WebSocket OpenAPI designs a unified data interaction protocol that supports the classified transmission of multimodal data such as audio, video, and control instructions (for example, using the event_type field to distinguish audio.stream, video.frame, and emotion.data), simplifying the complexity of client development.

[0058] In some embodiments of the present application, the system uses an edge inference scheduling algorithm on the device side to dynamically decide whether to execute computing tasks at the edge or in the cloud, balancing latency and bandwidth consumption. The decision strategy formula is as follows:

[0059]

[0060] Among them, Execute_Locally represents a Boolean value, indicating whether the task is executed on the edge; Le Indicates the time required for local inference on the edge node (unit: milliseconds); L c represents the total time taken to upload and complete the task in the cloud (including transmission + processing); ε represents the allowed edge time redundancy (redundancy tolerance threshold); D represents the task complexity score (such as the number of model calls and data dimension); B represents the current network bandwidth status (such as Mbps); θ represents the upper threshold of task complexity; β represents the minimum acceptable bandwidth threshold (lower than this value, local processing is recommended).

[0061] In other embodiments of the present application, the system further constructs a context retention and adaptive optimization mechanism, including:

[0062] Conversation context memory: Maintains conversation context at the system level. For example, names of people, places, or proper nouns mentioned by the user in the previous round are stored in the conversation state. During the current round of recognition, the corresponding context information is used to customize the language model or hotword list and inject it into the ASR engine.

[0063] Multi-round contextual disambiguation: When a user uses pronouns (he, it) or omits the subject, the system can infer the referential relationship based on previous rounds of conversation and make corrections after the recognition result;

[0064] Active confirmation mechanism: The system adopts an active confirmation strategy to ensure accurate identification of important information;

[0065] Optimization in noisy environments: In noisy environments, the system dynamically adjusts recognition strategies to improve accuracy. For example, when the background noise level rises, the system temporarily raises the VAD (endpoint detection) threshold to prevent excessive noise fragments from entering the recognition. At the same time, a stronger noise reduction model can be enabled, and the recognition engine will load an acoustic model optimized for noisy environments.

[0066] Continuous learning and model correction: The system uses conversation logs for continuous learning and optimization. After each conversation, if there are any recognition errors that are subsequently corrected manually or discovered through offline analysis, they will be fed back to the model training module. Regularly (for example, weekly), these actual conversation data are used to fine-tune the acoustic model and language model, so that the model gradually adapts to the language characteristics of the current customer base (accents, new industry terms, etc.). For edge-deployed models, after centralized training and updating in the cloud, a new version of the model is released or the parameters are incrementally updated. Before the model is updated, A / B testing is performed using some real-time traffic to ensure that the new model has indeed improved in recognition accuracy before it is fully released. This closed loop allows recognition accuracy to be continuously optimized over time.

[0067] Fusion of multiple recognition engines: In mission-critical scenarios, the outputs of two different ASR engines can be used in parallel to improve accuracy. For example, one engine excels in acoustic modeling, while the other offers a wider vocabulary coverage. When a user inputs voice, the two engines each generate transcriptions, and then a more reliable combined result is obtained through a result fusion algorithm (for example, selection based on confidence, or through edit distance comparison). Although this multi-engine architecture consumes more resources, it can serve as a safeguard in scenarios where extremely high accuracy is required. If edge computing resources permit, a second engine can be deployed for important conversations or as a backup when the confidence level of the first engine is low, to further optimize the recognition effect. Through the above multiple rounds of context and optimization strategies, the system becomes more and more accurate with continued use during conversations, maintaining semantic consistency and high transcription accuracy, and improving the credibility of the user experience.

[0068] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0069] Figure 2 This is a flow chart of a response method of a customer service robot provided in an embodiment of the present application. Figure 2 As shown, the method may include:

[0070] Step S210: Obtain multimodal request data input by the user in the current session with the customer service robot, as well as the conversation status of the current session.

[0071] In a specific implementation, the user initiates a customer service session request in different conversational forms in the response system of the customer service robot; after verifying the user's authority, the response system of the customer service robot establishes a corresponding dialogue channel based on the conversational form contained in the user's customer service session request; the user and the customer service robot conduct multiple rounds of interaction in the established dialogue channel; in each round of interaction, the user sends multimodal request data to the customer service robot through the input method corresponding to the conversational form, and the multimodal request data is used to describe the user's problem or demand; wherein, the conversational form includes voice conversation and video conversation; when the conversational form is a video conversation, the customer service robot is displayed in the conversation with a configured virtual image.

[0072] Among them, the multimodal request data is the multimodal request data sent by the user to the customer service robot in the current round of interaction; the multimodal interaction data includes: text modal request data, image modal request data and audio modal request data; the audio modal request data is obtained by preprocessing the data collected by the audio acquisition device, and the audio modal request data is an audio time series; the text modal request data is obtained by performing text recognition on the audio modal request data; the image modal request data is collected by the image device, and the image modal request data is a video frame sequence.

[0073] Specifically, the method for obtaining the requested data in the audio mode includes:

[0074] Obtaining an original audio signal (including ambient noise, echo, etc.) captured by an audio capture device; preprocessing the original audio signal using a real-time noise cancellation algorithm and digital signal processing methods (spectral subtraction, adaptive filtering) to obtain a first audio signal, thereby filtering out stable background noise (such as air conditioning sound and street noise) from the original audio signal; processing the first audio signal using a deep learning noise reduction model (such as RNNoise) to convert noisy speech into clear speech through spectrum mapping optimization, focusing on processing non-stationary noise (such as sudden human voices), thereby obtaining a third audio signal; spatially filtering (beamforming) the third audio signal using a microphone array or stereo recording to obtain a fourth audio signal, thereby enhancing the voice from the main speaker, i.e., the user, and attenuating irrelevant sounds from other directions (such as side conversations and background conversations);

[0075] A lightweight VAD algorithm is used to detect speech start boundaries in the fourth audio signal in real time to obtain a fifth audio signal, thereby avoiding recording of silence segments. The lightweight VAD algorithm is used to detect speech start boundaries in the fourth audio signal in real time, and a silence duration threshold (e.g., silence exceeding 800 milliseconds), voice activity energy detection, and language model syntactic completion judgment (e.g., sentence ending tone) are combined to determine the speech end boundary.

[0076] The response voice played by the customer service robot to the user is used as a reference signal. The echo component is subtracted from the fifth audio signal using a classic acoustic echo cancellation algorithm to obtain a sixth audio signal.

[0077] Based on the pre-acquired user's voiceprint and the customer service robot's voiceprint, the user's audio signal is extracted from the sixth audio signal through the timbre recognition model; when overlapping speech is detected (such as a sudden increase in instantaneous energy or abnormal spectral characteristics), a lightweight speech separation model (such as a network based on time-frequency masking) is started to divide the mixed speech into independent tracks and discard irrelevant tracks; the extracted user's audio signal is used as the request data of the audio modality.

[0078] Specifically, the method for obtaining the dialog status of the current session includes:

[0079] The multimodal request data is input into a pre-built conversation state tracking model to obtain the conversation state of the current session. The conversation state tracking model is used to determine the conversation state of the current session based on the historical multimodal interaction data between the user and the customer service robot in the current session and the multimodal request data input by the user in the current round. The conversation state tracking model continuously tracks the interaction between the user and the customer service robot (including sending request data and responses). Therefore, each time the multimodal request data of the current round is output, the conversation state tracking model can determine the conversation state.

[0080] The conversation state tracking model is used to process the historical multimodal interaction data between the user and the customer service robot in the historical rounds of the current conversation through a time series modeling mechanism. It combines the attention mechanism with the multimodal request data to obtain the conversation state of the current conversation. Specifically, it includes:

[0081] Input layer, used for multimodal request data input by users in the current round;

[0082] The multimodal data processing layer is used to preprocess, extract and fuse the historical multimodal interaction data of each historical round to obtain the historical multimodal fusion features corresponding to each historical round; and preprocess, extract and fuse the multimodal request data to obtain the current multimodal fusion features;

[0083] A multi-granularity state encoder is used to generate state feature vectors of different granularities based on the historical multimodal fusion features corresponding to each historical round using fully connected layers of different dimensions.

[0084] The granularity weight calculation layer is used to calculate the similarity of state feature vectors of different granularities; calculate the input complexity based on the current multimodal fusion features; and determine the weight vectors of state feature vectors of different granularities based on the similarity and input complexity of state feature vectors of different granularities;

[0085] The multi-granularity state fusion layer is used to perform weighted fusion of state feature vectors of different granularities based on their weight vectors to obtain a multi-granularity fused feature vector.

[0086] The feature vector extraction layer is used to calculate the similarity between the current multimodal fusion feature and the historical multimodal fusion features corresponding to each historical round; based on the calculated similarity, the weight of the historical multimodal fusion feature corresponding to each historical round is determined; based on the weight of the historical multimodal fusion feature corresponding to each historical round, the historical multimodal fusion features corresponding to each historical round are weighted and summed to obtain the historical fusion feature vector;

[0087] The cross-granularity attention calculation layer is used to reduce the dimensionality of the historical fusion feature vector. The self-attention mechanism is used to calculate the similarity between the multi-granularity fusion feature vector and the historical fusion feature vector to obtain the attention weight. Based on the attention weight, the multi-granularity fusion feature vector is weighted and adjusted to obtain the target feature vector.

[0088] The residual gated update layer is used to input the dialogue state feature vector and target feature vector of the previous round of the current round into the gating function to obtain the gating value. Based on the gating value, the dialogue state feature vector and target feature vector of the previous round of the current round are fused and updated to obtain the dialogue state feature vector of the current round.

[0089] The conversation state decoding layer is used to input the conversation state feature vector of the current round into a fully connected layer. The fully connected layer performs feature transformation through matrix multiplication and activation function, and then inputs it into another fully connected layer. The Softmax function is then used to calculate the probability of each intent category and output the intent category with the highest probability as the intent of the current conversation. The multi-layer perceptron is used to process the conversation state feature vector of the current round and predict the values ​​of each slot involved in the conversation. The conversation state of the current conversation in the current round is obtained based on the recognized intent and the filled-in slot information.

[0090] In some embodiments of the present application, the dialogue state tracking model further includes:

[0091] The historical state cache unit is used to store the fused features of the last N rounds of conversations in a queue. New rounds of data are added to the end of the queue, and the earliest data is deleted when the length exceeds the limit.

[0092] Multi-granularity state encoder, coarse-grained encoding: use convolutional neural network to extract the overall theme of historical multimodal interaction data; medium-grained encoding: use LSTM to capture the task type of historical multimodal interaction data; fine-grained encoding: use Transformer model to parse the specific details in historical multimodal interaction data.

[0093] In some embodiments of the present application, the update formula of the dialogue state tracking model is as follows:

[0094] s t =f DST (s t-1 ,u t , a t-1 );

[0095] Among them, s t Indicates the conversation status of the current session in the current round; s t-1 Indicates the historical conversation state of the current conversation in the previous round; u t Represents the multimodal request data entered by the user in the current round; at-1 Indicates the historical response strategy of the customer service robot in the previous round of the current round; f DST (·) denotes the state transition function (which can be implemented by hand-crafted rules, finite state machines, or a fine-tuned large language model).

[0096] Step S220: For the request data of any modality in the multimodal request data, perform modal feature extraction on the request data of the modality to obtain modal features; and construct a multimodal heterogeneous graph based on the request data of different modalities and the corresponding modal features.

[0097] In an embodiment of the present application, modal feature extraction is performed on modal request data, including:

[0098] According to the configured modal feature extraction method corresponding to the request data of the modality, modal feature extraction is performed on the request data of the modality to obtain the modal features of the request data of the modality.

[0099] In a specific implementation, when the interactive data is request data in a text mode, modal features are extracted from the modal request data to obtain modal features, including:

[0100] Clean the request data in the text modality to obtain a cleaned first text; perform word segmentation and normalization on the first text to obtain a second text; input the second text into a pre-trained user intent recognition model to obtain first modality features; the first modality features are used to represent user intent;

[0101] When the interaction data is request data in image mode, modal features are extracted from the modal request data to obtain modal features, including:

[0102] For any video frame in the video frame sequence, the video frame is preprocessed to obtain a preprocessed video frame; face detection is performed on the preprocessed video frame to obtain a face area and an environment area of ​​the preprocessed video frame; a sequence composed of the face areas of each preprocessed video frame is used as a first sequence; a sequence composed of the environment areas of each preprocessed video frame is used as a second sequence; the first sequence is input into a pre-trained expression recognition model to obtain an expression recognition result; the second sequence is input into a pre-trained environment recognition model to obtain an environment recognition result; based on the expression recognition result and the environment recognition result, a second modal feature is obtained; the second modal feature is used to characterize the user expression recognition result and the environment recognition result of the user's current environment;

[0103] When the interaction data is request data in the audio modality, modal features are extracted from the modal request data to obtain modal features, including:

[0104] The time domain features of the audio time series are extracted to obtain the time domain features. The time domain features include short-time energy, short-time zero-crossing rate, fundamental frequency and pronunciation duration of different syllables, words or phrases in the audio time series. The time domain features include short-time energy and short-time zero-crossing rate. The short-time energy is calculated by summing the squares of each frame of audio signal and then taking the average, which is used to reflect the strength of the speech signal in the frame. The short-time zero-crossing rate is the number of positive and negative changes of the signal in each frame. The zero-crossing rate of the voiced signal is low, and the zero-crossing rate of the unvoiced signal is high. By analyzing the short-time zero-crossing rate, the voiced and unvoiced parts of the speech can be preliminarily distinguished, thereby assisting in judging the speech rhythm. The frequency domain features of the audio time series are extracted to obtain the frequency domain features. The time domain features and the frequency domain features are fused to obtain the audio fusion features. The audio fusion features are input into a pre-trained emotion recognition model to obtain the emotion recognition results. Based on the emotion recognition results and the audio fusion features, the third modal features are obtained. The third modal features are used to characterize the user's voice pitch and emotional state.

[0105] In the embodiment of the present application, a multimodal heterogeneous graph is constructed based on request data of different modalities and corresponding modal features, including:

[0106] For the request data of any modality, the modal segments corresponding to the corresponding modal features are extracted from the request data of the modality; the modal segments corresponding to each modal feature are used as nodes, and the modal features corresponding to each node are used as attributes of the node; the correlation between any two nodes is calculated to be greater than the configured correlation threshold. If the correlation between any two nodes is greater than the configured correlation threshold, an edge is generated between the two nodes; if the correlation between any two nodes is not greater than the configured correlation threshold, no edge is generated between the two nodes; based on each node and each edge, an initial multimodal heterogeneous graph is obtained; the graph attention mechanism is used to update each node in the initial multimodal heterogeneous graph to obtain a multimodal heterogeneous graph; wherein, the multimodal heterogeneous graph is used to characterize the relationship between request data of different modalities.

[0107] Specifically, the modal segments corresponding to each modal feature are taken as nodes, including:

[0108] The first modal segment corresponding to each first modal feature is taken as the first node; the attribute of each first node is the corresponding first modal feature; the second modal segment corresponding to the expression recognition result in each second modal feature is taken as the second node; the attribute of each second node is the corresponding second modal feature; the second modal segment corresponding to the environment recognition result in each second modal feature is taken as the third node; the attribute of each third node is the corresponding second modal feature; the third modal segment corresponding to the emotion recognition result in each third modal feature is taken as the fourth node; the attribute of each fourth node is the corresponding third modal feature; the third modal segment corresponding to the audio fusion feature in each third modal feature is taken as the fifth node; the attribute of each fifth node is the corresponding third modal feature.

[0109] Specifically, the calculation of the correlation between any two nodes is greater than the configured correlation threshold, including:

[0110] If any two nodes are nodes of the same modality, the feature similarity between the modal features corresponding to the two nodes is calculated; the calculated feature similarity is used as the correlation between the two nodes; if any two nodes are not nodes of the same modality, the modal features corresponding to the two nodes are mapped to the same semantic space through a cross-modal encoder to obtain the mapped modal features; the vector similarity of the mapped modal features corresponding to the two nodes is calculated; the calculated vector similarity is used as the correlation between the two nodes.

[0111] Specifically, when an edge is generated between two nodes, the initial weight of the edge corresponding to the corresponding degree of association is determined according to a comparison table of different degrees of association and different initial weights.

[0112] Specifically, the graph attention mechanism is used to update each node in the initial multimodal heterogeneous graph to obtain a multimodal heterogeneous graph, including:

[0113] For any node in the initial multimodal heterogeneous graph, the attention coefficient of the node and all neighboring nodes is calculated; based on the attention coefficient of the node and each neighboring node, the attributes of the node (i.e., modal features) are updated to obtain an updated node; based on each updated node, a multimodal heterogeneous graph is obtained.

[0114] Step S230: Based on the request data of different modalities and the corresponding modal features, the conversation state and the multimodal heterogeneous graph, the different modal features are fused to obtain a multimodal fusion feature.

[0115] In the specific implementation, based on the request data of different modalities and the corresponding modal features, the conversation state and the multimodal heterogeneous graph, the different modal features are fused to obtain the multimodal fusion features, including:

[0116] The conversation state of the current session, the modal features of request data of different modalities, and the multimodal heterogeneous graph are input into a pre-trained modal confidence estimation network to obtain the weight vector of each modal feature; based on the weight vector of each modal feature, the modal features are weightedly fused to obtain the multimodal fusion feature.

[0117] Among them, the modal confidence estimation network includes:

[0118] The input layer is used to input the conversation state of the current session, the modal features of the request data of different modalities, and the multimodal heterogeneous graph;

[0119] The feature enhancement layer is used to perform feature enhancement processing on the modal features of the request data of each modality to obtain the enhanced first feature vector of each modality;

[0120] The graph convolution layer is used to update the node representation based on the modal features and the first eigenvectors of each modality in the multimodal heterogeneous graph through a graph convolutional network (GCN). This layer obtains the second eigenvectors of each modality that integrate the graph structure information to capture the dependencies between modal segments (e.g., the association between the text "temperature" and the thermometer in the image).

[0121] The conversation state fusion layer is used to concatenate or weightedly fuse the conversation state of the current session and the second eigenvectors of each modality, and map them to a unified dimension using a fully connected layer to obtain the third eigenvectors of each modality of the fused conversation state.

[0122] The attention mechanism layer is used to calculate the attention weight of the third eigenvector of each modality in the fused conversation state, and perform attention weighting on the third eigenvector of each modality to obtain the fourth eigenvector of each modality;

[0123] The confidence estimation layer uses a fully connected layer and a sigmoid activation function to output a confidence score (0-1) for each modality based on the fourth eigenvector of each modality. The confidence score indicates the reliability of the modality in the current conversation (for example, speech confidence decreases in noisy environments).

[0124] The normalization layer is used to normalize the confidence scores of each modality using the Softmax function to ensure that the sum of the weights is 1, and to obtain the weight vector of each modality feature;

[0125] The output layer is used to output the weight vector of each modal feature.

[0126] Step S240: Input the multimodal fusion features into a pre-built causal-based user intention response model to obtain the customer service robot's response strategy for the multimodal request data; and control the customer service robot to respond to the user according to the response strategy.

[0127] In an embodiment of the present application, the causal relationship-based user intention response model includes:

[0128] Input layer, used to input multimodal fusion features;

[0129] The causal feature enhancement layer is used to input the multimodal fusion features into the Transformer's multi-head attention module. The association weights of each internal dimension of the multimodal fusion features are calculated through three sets of learnable weight matrices (query matrix, key matrix, value matrix), and the query vector, key vector, value vector and association weight matrix are obtained. The attention enhancement feature vector is obtained based on the value vector and the association weight matrix. The causal enhancement feature vector is obtained by multiplying the attention enhancement feature vector with the predefined embedding matrix.

[0130] The intention causal reasoning layer is used to input the causal enhancement feature vector into a two-layer fully connected network for dimensionality reduction processing to obtain the reduced-dimensional features; the reduced-dimensional features are grouped through a clustering algorithm to generate multiple causal elements with semantic meaning; the extracted causal elements are matched with a predefined causal rule library to generate rule-based causal edges; the causal probability between each causal element is calculated using a Bayesian network model to generate data-based causal edges; the causal elements are used as nodes, and the rule-based and data-based causal edges are merged to construct a directed acyclic graph to obtain the user intention causal graph and the probability distribution of each causal element;

[0131] The intent confidence assessment layer is used to calculate the coverage of the extracted causal elements and the key element set based on the configured key element set. The causal enhancement feature vector is input into the fully connected network and the probability distribution of the intent category is output through the classification algorithm. Based on the coverage and the intent category probability distribution, the intent category is determined and whether clarification of information to the user is required (if the coverage or the highest intent probability is below the threshold, a clarification request is generated).

[0132] The multimodal response generation layer is used to determine the target business processing rules corresponding to the intent categories based on a configured comparison table of different intent categories and different business processing rules. Based on the target business processing rules, the response strategy is determined. For query intent, the knowledge base content is located based on the causal graph, and a language model is used to generate natural language responses. For handling intent, the step text is generated according to the causal graph process. If the "apology strategy" has a high weight, an apology statement is added. If the causal graph contains an "image evidence" node, an image annotation tool is used to generate annotated images. For complex processes, step screenshots are generated and spliced ​​according to the causal graph node sequence. For voice response generation, TTS parameters (speech rate, intonation, and modal particles) are adjusted based on the emotion tags.

[0133] The output layer is used to output the response strategy.

[0134] Specifically, based on the target business processing rules, determine the response strategy, including:

[0135] Control the customer service robot to generate an initial response strategy based on the response type (such as text, image, and voice) in the configured target business processing rules and the key elements in the causal graph;

[0136] Based on the emotion strategy weight determined by the emotion recognition result in the multimodal fusion feature, the initial response strategy is adjusted to obtain the response strategy; wherein, if the response content in the response strategy is only text, the causal graph element value is filled into the text template; if the response content in the response strategy includes an image, the image annotation tool is called according to the spatial relationship elements in the causal graph to generate a schematic diagram with arrows and text descriptions; if the response content in the response strategy includes a voice response: then according to the emotion strategy parameters, the text response content is input into the TTS engine and modal particles are added; wherein, the target business rules also include the response method.

[0137] Among them, the user intention response model is used to characterize the causal relationship between the multimodal interaction data input by the user and the response strategy of the customer service robot; the response strategy includes the response method and response content; the response method includes two types: customer service robot response and manual customer service response; when the response method is manual customer service response, the customer service robot will synchronize the current conversation and multimodal fusion features to the manual customer service.

[0138] In specific implementations, the response content can be retrieved by the customer service robot from the FAQ knowledge base or by calling the reasoning engine or business process (such as querying the order status).

[0139] In specific implementations, after obtaining the customer service robot's response strategy for multimodal request data, the robot will output a response to the user based on the configured output parameters, or transfer the call to a human operator. Output parameters include the robot's corresponding avatar's tone, wording, and actions. The robot determines its output method based on the user's input method: if the user inputs via voice, the robot responds via voice; if the user inputs via video, the robot responds via video. If the user inputs via voice or video, the robot also converts the response into text and displays it on the conversation interface. This means that during the conversation, the user not only hears the voice response but also sees the corresponding text or image on the terminal device.

[0140] In actual applications, after determining the response strategy for the current round, the customer service robot waits again for the multimodal request data input by the user to determine the response strategy for the next round until the user's problem is solved or transferred to manual customer service.

[0141] In some embodiments of this application, after each conversation between a user and a customer service robot, the system evaluates service effectiveness through user feedback or conversation results. If misunderstandings or dissatisfaction are detected, the system uses the conversation record to train dialogue management and language models, continuously optimizing strategies. For individual users, the system appropriately memorizes their preferences (e.g., address, past questions), making personalized adjustments in subsequent interactions, ensuring that the customer service robot's behavior increasingly aligns with user needs.

[0142] The customer service robot of this application can demonstrate flexibility and affinity similar to that of human customer service while ensuring business accuracy, achieving a service experience that is both professional, efficient and satisfactory.

[0143] In an embodiment of the present application, the customer service robot is controlled to output the response content to the user according to the response method in the response strategy to complete the response.

[0144] For example, in an insurance customer service center, a user initiates an audio or video customer service request through an app. The customer service robot identifies the uploaded image of the accident scene (image recognition), detects the user's agitated tone (emotion recognition), and outputs a soothing response containing the automatically extracted accident address (semantic understanding). If the response strategy the customer service robot receives indicates a transfer to a human operator, the robot transfers the call to a human operator.

[0145] In the scenario of online banking consultation, users enter the bank app and consult financial products through video connection; the user shows the ID card photo, and the customer service robot recognizes and verifies the information at the edge; at the same time, it analyzes the user's investment preferences based on the voice content and images and recommends products.

[0146] In some embodiments of the present application, the speech recognition model is an end-to-end speech recognition model that supports streaming decoding, such as a model based on RNN-Transducer or streaming Transformer. This type of model can output partial recognition results in real time under incomplete speech input, and is suitable for interactive scenarios. Compared with the offline model that post-processes the entire speech segment, streaming ASR can control the delay within a few hundred milliseconds. At the same time, the model structure introduces a look-ahead mechanism in reasoning to ensure accuracy. This function can be easily implemented in combination with the streaming recognition interface provided by Button Intelligent Voice. The speech recognition model can also be a Unified model trained with mixed Chinese and English corpus.

[0147] Corresponding to the above method, the embodiment of the present application also provides a response device of a customer service robot, such as Figure 3 As shown, the device includes:

[0148] An acquisition unit 310 is configured to acquire multimodal request data input by a user into a current conversation with a customer service robot, as well as a conversation state of the current conversation; wherein the interaction data is multimodal interaction data;

[0149] The extraction unit 320 is configured to extract modal features from the request data of any modality in the multimodal request data to obtain modal features;

[0150] A construction unit 330 is configured to construct a multimodal heterogeneous graph based on the request data of different modalities and corresponding modal features; wherein the multimodal heterogeneous graph is used to represent the relationship between the request data of different modalities;

[0151] A fusion unit 340 is configured to fuse the different modal features based on the request data of different modalities and the corresponding modal features, the conversation state, and the multimodal heterogeneous graph to obtain a multimodal fusion feature;

[0152] Determining unit 350 is configured to input the multimodal fusion features into a pre-built causal-based user intent response model to obtain the customer service robot's response strategy for the multimodal request data; wherein the user intent response model is configured to characterize the causal relationship between the multimodal interaction data input by the user and the customer service robot's response strategy;

[0153] The response unit 360 is used to control the customer service robot to respond to the user according to the response strategy.

[0154] The functions of each functional unit of the response device of the customer service robot provided in the above embodiments of the present application can be achieved through the above-mentioned method steps. Therefore, the specific working process and beneficial effects of each unit in the response device of the customer service robot provided in the embodiments of the present application will not be repeated here.

[0155] The present application also provides an electronic device, such as Figure 4 As shown, it includes a processor 410 , a communication interface 420 , a memory 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 .

[0156] Memory 430, for storing computer programs;

[0157] The processor 410 is configured to execute the program stored in the memory 430 by performing the following steps:

[0158] Obtain the multimodal request data entered by the user in the current conversation with the customer service robot, as well as the conversation status of the current conversation;

[0159] For request data of any modality, modal features are extracted from the request data of the modality to obtain modal features;

[0160] Construct a multimodal heterogeneous graph based on the request data of different modalities and the corresponding modal features; wherein the multimodal heterogeneous graph is used to represent the relationship between the request data of different modalities;

[0161] Based on the request data of different modalities and the corresponding modal features, conversation state and multimodal heterogeneous graph, the different modal features are fused to obtain multimodal fusion features;

[0162] The multimodal fusion features are input into a pre-built causal-based user intent response model to obtain the customer service robot's response strategy for multimodal request data. The user intent response model is used to characterize the causal relationship between the multimodal interaction data input by the user and the customer service robot's response strategy.

[0163] Control the customer service robot to respond to users according to the response strategy.

[0164] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0165] The communication interface is used for communication between the above electronic device and other devices.

[0166] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0167] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0168] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments to solve the problems can be found in Figure 2 The various steps in the embodiment shown are implemented, therefore, the specific working process and beneficial effects of the electronic device provided by the embodiment of the present application are not repeated here.

[0169] In another embodiment provided in the present application, a computer-readable storage medium is further provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the response method of the customer service robot described in any of the above embodiments.

[0170] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the response method of the customer service robot described in any of the above embodiments.

[0171] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application can be implemented in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of the present application can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0176] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims and their equivalents, the embodiments of the present application are also intended to include these modifications and variations.

Claims

1. A response method of a customer service robot, characterized in that: The method comprises: Obtain multimodal request data entered by the user in the current session with the customer service robot, as well as the conversation state of the current session; For request data of any modality, performing modal feature extraction on the request data of the modality to obtain modal features; Constructing a multimodal heterogeneous graph based on the request data of different modalities and the corresponding modal features; wherein the multimodal heterogeneous graph is used to characterize the relationship between the request data of different modalities; Based on the request data of the different modalities and the corresponding modal features, the conversation state and the multimodal heterogeneous graph, fusing the different modal features to obtain a multimodal fusion feature; Inputting the multimodal fusion features into a pre-built causal-based user intention response model to obtain the customer service robot's response strategy for the multimodal request data; wherein the user intention response model is used to characterize the causal relationship between the multimodal interaction data input by the user and the customer service robot's response strategy; Control the customer service robot to respond to the user according to the response strategy.

2. The method according to claim 1, wherein Perform modal feature extraction on modal request data, including: According to the configured modal feature extraction method corresponding to the request data of the modality, modal feature extraction is performed on the request data of the modality to obtain the modal features of the request data of the modality.

3. The method according to claim 1, wherein Based on the request data of different modalities and the corresponding modal features, a multimodal heterogeneous graph is constructed, including: For request data of any modality, extracting a modality segment corresponding to a corresponding modality feature from the request data of the modality; The modal segments corresponding to each modal feature are taken as nodes, and the modal features corresponding to each node are taken as attributes of the corresponding node; If the correlation between any two nodes is greater than the configured correlation threshold, an edge is generated between the two nodes; Based on each node and each edge, an initial multimodal heterogeneous graph is obtained; The graph attention mechanism is used to update each node in the initial multimodal heterogeneous graph to obtain a multimodal heterogeneous graph.

4. The method according to claim 3, wherein The graph attention mechanism is used to update each node in the initial multimodal heterogeneous graph to obtain a multimodal heterogeneous graph, including: For any node in the initial multimodal heterogeneous graph, calculate the attention coefficient of the node and all neighboring nodes; According to the attention coefficients of the node and each neighboring node, the attributes of the node are updated to obtain an updated node; Based on the updated nodes, a multimodal heterogeneous graph is obtained.

5. The method according to claim 1, wherein Based on the request data of different modalities and the corresponding modal features, conversation state, and multimodal heterogeneous graph, the different modal features are fused to obtain multimodal fusion features, including: The modal features, conversation states, and multimodal heterogeneous graphs of request data of different modalities are input into a pre-trained modal confidence estimation network to obtain the weight vector of each modal feature. Based on the weight vector of each modal feature, the features of each modal are weighted and fused to obtain multimodal fusion features.

6. The method according to claim 1, wherein The method for obtaining the conversation state includes: Inputting the multimodal request data into a pre-built dialog state tracking model to obtain the dialog state of the current session; Among them, the dialogue state tracking model is used to process the historical multimodal interaction data of the user and the customer service robot in the historical rounds of the current conversation through a time series modeling mechanism, and combine the attention mechanism to obtain the dialogue state of the current conversation based on the multimodal request data.

7. The method according to claim 1, wherein The causal relationship-based user intention response model includes: Input layer, used to input multimodal fusion features; The causal feature enhancement layer is used to calculate the association weights of the multimodal fusion features using the Transformer multi-head attention, and use the association weights to weight and embed the multimodal fusion features into a matrix to obtain a causal enhancement feature vector; The intention causal reasoning layer is used to perform dimension reduction clustering on the causal enhancement feature vector to generate multiple causal elements. Based on the multiple causal elements and the configured causal rule library, it constructs the user intention causal graph and the probability distribution of each causal element. The intent confidence assessment layer determines the user intent category based on the user intent causal graph and causal enhancement feature vector; The multimodal response generation layer controls the customer service robot to execute the business processing rules corresponding to different user intent categories according to the configured business processing rules to obtain a response strategy; The output layer is used to output the response strategy.

8. A response device for a customer service robot, characterized in that: The device comprises: An acquisition unit, configured to acquire multimodal request data input by a user into a current session with a customer service robot, and a dialog state of the current session; wherein the interaction data is multimodal interaction data; an extraction unit, configured to extract modal features from request data of any modality in the multimodal request data to obtain modal features; A construction unit, configured to construct a multimodal heterogeneous graph based on request data of different modalities and corresponding modal features; wherein the multimodal heterogeneous graph is used to characterize the relationship between request data of different modalities; a fusion unit, configured to fuse the different modal features based on the request data of the different modalities and the corresponding modal features, the conversation state, and the multimodal heterogeneous graph to obtain a multimodal fusion feature; A determination unit is used to input the multimodal fusion feature into a pre-built user intention response model based on causality to obtain the response strategy of the customer service robot to the multimodal request data; wherein the user intention response model is used to characterize the causal relationship between the multimodal interaction data input by the user and the response strategy of the customer service robot.

9. An electronic device, characterized in that: The electronic device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images