Multi-modal feature collaborative emotion semantic reasoning method
By employing a multimodal feature-based sentiment semantic reasoning method, we can predict emotional turning points in medical customer service conversations in real time and provide causal explanations. This solves the problem of emotional turning points that cannot be predicted and explained in real time in existing technologies, thereby improving the response capabilities of customer service personnel.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI EMOTION RECOGNITION TECHNOLOGY CO LTD
- Filing Date
- 2026-01-28
- Publication Date
- 2026-05-08
AI Technical Summary
Existing technologies cannot predict emotional turning points in medical customer service conversations in real time and lack causal explanations, making it impossible for customer service personnel to adjust their response strategies in advance.
A sentiment semantic reasoning method based on multimodal feature collaboration is adopted. A temporal multimodal feature sequence is generated by a multimodal encoder, a causal triple sequence is generated by an entity relation extraction algorithm, and a causal potential energy distribution map is generated by incremental update of causal graph and potential energy allocation algorithm. Combined with sliding window sentiment analysis and potential energy gradient calculation, the prediction of turning points and causal explanation are realized.
It enables real-time prediction of emotional shifts, provides in-depth causal explainability, and helps customer service personnel understand the chain of reasons for patients' emotional deterioration and adjust communication strategies accordingly.
Smart Images
Figure CN121996706A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of emotion shift prediction technology, and more specifically, to an emotion semantic reasoning method based on multimodal feature collaboration. Background Technology
[0002] In healthcare customer service conversations, patients engage in multiple rounds of dialogue with customer service personnel, covering topics such as medical condition consultation, cost explanations, and appointment arrangements. The patient's emotional state is influenced by a combination of factors, potentially evolving from initial anxiety to dissatisfaction or trust. Existing methods for predicting emotional turning points primarily suffer from the following technical problems: First, existing methods for predicting emotional shifts based on time-series statistics rely solely on the numerical trend of emotional intensity to make predictions, outputting the probability of a shift but not explaining the reasons for the shift, making it impossible for customer service personnel to adjust their response strategies accordingly.
[0003] Second, existing causal attribution methods can only conduct post-hoc analysis after an emotional shift has occurred, and cannot provide predictions before the shift occurs.
[0004] Third, when a patient's emotions are about to deteriorate, the system is unable to provide early warnings or predictively identify the underlying causal chain that may lead to the deterioration (such as "high examination costs → economic pressure → distrust of the hospital"), causing customer service personnel to miss the best time to intervene and lack effective intervention basis.
[0005] The root cause of the above problems is that time-series prediction methods only focus on superficial changes in emotional intensity and ignore the causal driving mechanism of emotional evolution, while causal analysis methods can only trace past emotional changes and lack the ability to predict causal transmission trends. The two types of methods operate independently, resulting in the inability to unify prediction and attribution. Summary of the Invention
[0006] This invention provides a multimodal feature-coordinated sentiment semantic reasoning method, which solves the technical problem in related technologies where simple temporal prediction methods rely solely on changes in sentiment intensity values while ignoring causal driving mechanisms.
[0007] This invention provides a multimodal feature-coordinated sentiment semantic reasoning method, comprising the following steps: Acquire real-time multimodal data streams of the dialogue and generate temporal multimodal feature sequences using a multimodal encoder; The system analyzes dialogue text using an entity relation extraction algorithm, identifies event entities, state entities and their causal relationships, generates a real-time causal triple sequence, and labels sentiment outcome nodes. The causal triplet sequence is integrated into a dynamic causal link graph using the causal graph incremental update algorithm; The causal potential energy allocation algorithm is based on the topology of the dynamic causal link graph. The maximum potential energy value is assigned to the emotional result node, and the decayed potential energy value is assigned to other nodes according to the shortest path length to the emotional result node, thus generating a causal potential energy distribution graph. The sliding window sentiment analysis algorithm is used to analyze the temporal multimodal feature sequence, generate the historical sentiment intensity value sequence, and generate the sentiment acceleration sequence by calculating the second difference. The current dialogue position node is determined by using the potential energy gradient calculation algorithm, the causal potential energy gradient components along each outgoing edge direction are calculated, and the causal potential energy gradient vector is concatenated with the emotion acceleration sequence to generate a fused prediction feature vector. The inflection point joint prediction model is used to calculate the inflection probability value and the predicted causal potential energy gradient direction based on the fused prediction feature vector. When the inflection probability value exceeds the preset threshold and the direction is negative, an inflection warning signal is generated. The multi-hop backtracking algorithm is used to traverse backwards from the current node along the causal edge to the root node, generating a set of multi-hop causal paths, and calculating the attribution significance score of each path. The multi-hop causal path with the highest attribution significance score is converted into causal explanation text, and the turning point warning signal and causal chain explanation are output.
[0008] Furthermore, the causal potential energy allocation algorithm includes: Assign the maximum potential energy value to the emotional outcome node and determine the potential energy sign based on the emotional polarity; For non-emotional result nodes, breadth-first search is used to calculate the shortest path length from them to all emotional result nodes, and the minimum value is taken as the distance value. The potential energy value is calculated based on the shortest path length. The potential energy value is equal to the larger of the lower limit of potential energy and the maximum potential energy value minus the difference between the potential energy decay coefficient and the distance value. The potential energy sign is inherited from the emotional polarity of the nearest emotional result node.
[0009] Furthermore, the maximum potential energy value ranges from 1 to 10, the potential energy decay coefficient ranges from greater than zero to less than the maximum potential energy value, and the lower limit of the potential energy value ranges from 0.1 to 1.
[0010] Furthermore, the potential energy gradient calculation algorithm includes: The newly identified entity is matched with the nodes in the causal potential energy distribution map by word vector cosine similarity to determine the node at the current position. Iterate through all outgoing edges of the current node and get the next node that each outgoing edge points to; For each outgoing edge direction, the causal potential gradient component is calculated as the potential energy value of the next node minus the potential energy value of the current node. The gradient components in all directions are combined to form a causal potential gradient vector.
[0011] Furthermore, before concatenating the vectors, the causal potential gradient vector and the emotional acceleration sequence are respectively subjected to Z-score standardization. The emotional acceleration sequence takes values from a number of recent moments, with the number of moments ranging from 3 to 8.
[0012] Furthermore, the sliding window sentiment analysis algorithm is implemented based on a recurrent neural network. The recurrent neural network receives a sequence of multimodal feature vectors within the sliding window. The output layer maps the hidden state to a scalar sentiment intensity value through a linear mapping layer, and constrains the output to the range of negative one to positive one through a hyperbolic tangent activation function. The emotional acceleration is calculated by subtracting twice the emotional intensity value of the previous moment from the current emotional intensity value and then adding the emotional intensity values of the two moments before.
[0013] Furthermore, the joint prediction model for the inflection point is a multilayer perceptron model, including a probability output head and a direction output head; The probability output header passes through a fully connected layer and is followed by a Sigmoid activation function to output the transition probability value. The directional output head passes through a fully connected layer and is followed by a Softmax activation function to output the probability distribution of three directional categories: positive, negative, and neutral.
[0014] Furthermore, the multi-hop backtracking algorithm includes: Initialize an empty set of multi-hop causal paths and the current path stack, and push the current position node onto the path stack; Get the set of predecessor nodes corresponding to all incoming edges of the top node of the stack; If the predecessor node set is empty, the node sequence in the path stack is reversed and added to the set as the complete path. Then, the top node of the stack is popped and backtracking is performed. If it is not empty, push each predecessor node onto the path stack in turn and continue backtracking; Repeat until the path stack is empty.
[0015] Furthermore, the calculation of the attribution significance score includes: The number of nodes in a multi-hop causal path is used as the path length; Traverse each directed edge on the path and calculate the confidence weight product; The consistency coefficient between the potential energy polarity of the path endpoint node and the predicted causal potential energy gradient direction is calculated; it is 1 when they are consistent and 0 when they are inconsistent. The attribution significance score is equal to the weighted product multiplied by the consistency coefficient and then divided by the path length.
[0016] This invention provides a multimodal feature-coordinated sentiment semantic reasoning system for performing the aforementioned method, comprising: The multimodal feature extraction module is used to acquire real-time multimodal data streams of dialogues and generate temporal multimodal feature sequences; The causal relationship extraction module is used to analyze dialogue text to identify event entities, state entities and causal relationships, generate causal triple sequences and label sentiment result nodes; The dynamic causal graph construction module is used to integrate causal triple sequences into a dynamic causal link graph; The causal potential energy allocation module is used to generate a causal potential energy distribution map by assigning potential energy values to each node based on the graph topology. The sentiment analysis module is used to analyze time-series multimodal feature sequences to generate sentiment intensity value sequences and sentiment acceleration sequences; The potential energy gradient calculation module is used to calculate the causal potential energy gradient vector and fuse it with the emotion acceleration sequence to generate a fused prediction feature vector; The turning point prediction module is used to calculate the turning point probability value and direction based on the fused prediction feature vector and generate a turning point early warning signal. The multi-hop attribution module is used to generate a set of multi-hop causal paths by traversing backwards from the current position node and calculating the significance score; The explanation generation module is used to convert the path with the highest significance score into causal explanation text and output warning results.
[0017] The beneficial effects of this invention are as follows: By assigning potential energy values to each node in the dynamic causal link graph using a causal potential energy allocation algorithm, the static causal graph structure is transformed into a dynamic potential energy field with a causal potential energy gradient. This allows the direction of dialogue progress in the causal space to be quantified and tracked in real time. Since the direction of the causal potential energy gradient indicates the most likely direction of causal transmission, when the causal potential energy gradient points to a node with a negative emotional outcome and the emotional acceleration is also negative, the signals from the two independent dimensions corroborate each other, indicating that the emotion will evolve in a negative direction along this multi-hop causal path. This allows for the prediction of the occurrence of a turning point before it actually happens.
[0018] By fusing the causal potential gradient vector with the emotion acceleration sequence, the problem of relying solely on numerical changes in emotion intensity and ignoring the causal driving mechanism in simple time-series prediction methods is overcome. It also overcomes the problem that simple causal analysis methods can only perform post-hoc attribution and cannot make predictions, thus realizing the synergistic use of causal and temporal information.
[0019] By using a multi-hop backtracking algorithm to traverse backward along the causal potential gradient to the root node of the causal link graph, the complete transmission path from the root cause to the emotional turning point is recorded, making the turning point prediction results have deep causal interpretability. Customer service personnel can use this to understand the deep causal chain that may lead to the patient's emotional deterioration, and thus adjust their communication strategies accordingly. Attached Figure Description
[0020] Figure 1 This is a flowchart of the multimodal feature-coordinated affective semantic reasoning method of the present invention; Figure 2 This is a diagram showing the relationship between causal potential energy distribution and node distance in the present invention, illustrating the distribution of causal potential energy values of each node in the dynamic causal link diagram; Figure 3 This is a comparison and analysis diagram of the significance of multi-hop causal paths in this invention, which shows the key evaluation indicators of two multi-hop causal paths. Figure 4 This is a heatmap of the performance verification of the inflection point prediction model of the present invention, which shows the F1 score performance verification results of the joint prediction model of inflection points under different parameter configurations. Figure 5 The figure shows the results of the ablation experiment comparing the method of this invention with the baseline method on the task of predicting emotional shifts in medical customer service dialogue. Detailed Implementation
[0021] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.
[0022] This implementation provides a multimodal feature-coordinated sentiment semantic reasoning method, such as... Figure 1 As shown, it includes the following steps: Step 100: Obtain the real-time multimodal data stream of the medical customer service dialogue, and use a multimodal encoder to analyze the voice data and text input data at each time point to generate a temporal multimodal feature sequence.
[0023] In this step, the multimodal data stream includes the patient's speech data and text input data. The multimodal encoder extracts acoustic features from the speech data and semantic features from the text input data. The two types of features are then aligned by timestamps and concatenated to generate multimodal feature vectors for each time step. The temporal multimodal feature sequence is a set of multimodal feature vectors arranged in chronological order, denoted as . ,in Indicates the first Multimodal feature vectors at time step.
[0024] The aforementioned multimodal encoder comprises two sub-networks: a speech encoder and a text encoder. The speech encoder is a multi-layer recurrent neural network. Its input layer receives the Mel-spectral feature sequence corresponding to the speech data, the hidden layer encodes temporal acoustic information through recurrent units, and the output layer takes the hidden state at the last moment as the speech feature vector. The text encoder is a multi-layer recurrent neural network. Its input layer receives the word embedding vector sequence corresponding to the text data, the hidden layer encodes semantic information through recurrent units, and the output layer takes the hidden state at the last time step as the text feature vector. Since speech feature vectors and text feature vectors come from different modalities and have different value ranges, L2 norm normalization needs to be performed on both types of feature vectors before vector concatenation to eliminate the impact of differences in feature scale between different modalities on subsequent models. Multimodal feature vectors It is obtained by concatenating the normalized speech feature vector and the text feature vector: The aforementioned multimodal encoder is trained using supervised learning, with the training data being a medical dialogue dataset labeled with sentiment categories. During training, a linear classification layer is appended to the multimodal feature vectors to map the features to sentiment category predictions, and the loss function used is cross-entropy loss. in, For the number of emotion categories, One-hot encoding for the real label. This represents the predicted class probability. The Adam optimization algorithm is used for parameter updates during training.
[0025] In a practical application at the customer service center of a top-tier hospital, patient Li inquired about a CT scan via the customer service hotline. The system collected the patient's voice data and real-time transcribed text data. In the... At that moment, when the patient says, "When can I schedule the test again?", the speech encoder extracts the Mel-spectral features of that turn of speech and outputs a speech feature vector. The text encoder performs word embedding encoding on the transcribed text and outputs text feature vectors. The two types of feature vectors are normalized by the L2 norm and then concatenated to obtain the multimodal feature vector at that time. As of the [date] At time 1, the system has generated a time-series multimodal feature sequence. The dialogue content and extracted feature dimensions at each time point are shown in the table below.
[0026] Table 1. Results of Dialogue Multimodal Feature Extraction Step 200: Analyze the dialogue text content using an entity relation extraction algorithm, identify the event entities, state entities and their causal relationships in the dialogue, and generate a real-time causal triple sequence.
[0027] In this step, the entity relation extraction algorithm takes the dialogue text as input and outputs a sequence of causal triples. Event entities refer to specific events mentioned in the dialogue (e.g., "informing about examination fees," "making an appointment"), while state entities refer to states or attributes reflected in the dialogue (e.g., "financial pressure," "long waiting time"). The causal triple takes the form of... ,in As the causal entity, For the result entity, This is a causal relationship type.
[0028] Furthermore, the entity relation extraction algorithm employs a joint model of sequence labeling and relation classification. The input layer of the joint model receives the word embedding vector sequence corresponding to the dialogue text, the encoding layer uses a bidirectional recurrent neural network to encode the text, the entity recognition layer uses a conditional random field to label the encoded sequence to identify entity boundaries and types, and the relation classification layer inputs the contextual representation of the identified entity pairs into a fully connected layer for causal relation type classification.
[0029] This step also includes step 201, which uses an emotion node labeling algorithm to identify entities expressing emotions in the dialogue text and label them as emotion result nodes. The input of the emotion node labeling algorithm is the set of identified entities, and the output is a subset of entities with emotion labels. Emotion result nodes are the endpoint nodes of causal transmission, representing the final emotional state (such as "dissatisfaction" or "trust").
[0030] Furthermore, the sentiment node labeling algorithm achieves this by matching the text representation of each entity with a predefined sentiment dictionary. Specifically, for each identified entity, the cosine similarity between the identified entity text and the word vectors of each sentiment entry in the sentiment dictionary is calculated. If the maximum similarity exceeds a preset matching threshold and the corresponding entry belongs to the sentiment category, the identified entity is marked as a sentiment result node, and the sentiment result node is labeled as positive or negative sentiment based on the sentiment polarity of the matched entry.
[0031] Furthermore, the matching threshold for sentiment node annotation ranges from [value missing]. The matching threshold is used to determine whether the semantic similarity between entity text and sentiment terms is sufficient to establish a matching relationship.
[0032] The system extracts entity relations from the aforementioned dialogue text. The customer service response identified the event entity "Notification of Inspection Fee" and the status entity "Fee 800 Yuan" from the following information: The status entity "economic stress perception" was identified from the patient's responses at any time. The system identifies the event entity "explaining the reimbursement ratio" and the status entity "low reimbursement ratio" in customer service replies. Based on contextual semantic analysis, the system generates a causal triple sequence and uses an emotion node labeling algorithm to mark "dissatisfaction" as a negative emotion result node (this entity has a cosine similarity of 0.87 with the word "dissatisfaction" in the emotion dictionary, exceeding the matching threshold of 0.75). As of [date / time]... The causal triples extracted at each time point are shown in the table below.
[0033] Table 2 Results of Causal Triple Extraction Step 300: Use the causal graph incremental update algorithm to accumulate and integrate the real-time causal triplet sequence into a dynamic causal link graph, generating a graph structure with event entities and state entities as nodes and causal relationships as directed edges.
[0034] In this step, the causal graph incremental update algorithm takes the newly generated causal triples and the existing causal link graph as input, and outputs the updated dynamic causal link graph. The dynamic causal link graph is denoted as... ,in It is a collection of nodes, including event entities and state entities. It is a set of directed edges, each edge representing a causal relationship. The dynamic causal link graph supports the representation of multi-hop causal paths, that is, starting from the root node, it can reach the sentiment outcome node through multiple intermediate nodes.
[0035] The aforementioned causal graph incremental update algorithm includes the following steps: Step 301: Receive the newly generated causal triples Extract the cause entities respectively and result entity .
[0036] Step 302: Traverse the current set of nodes in the causal link graph. Determine the cause entity Does it already exist in the node set? If not, then... Add to node collection ; For the result entity Perform the same operation.
[0037] Step 303: Traverse the edge set of the current causal link graph , judge from point to If the directed edge already exists, add the directed edge and initialize its confidence weight to 1. If it already exists, increase the confidence weight of the directed edge by 1.
[0038] Step 304: Output the updated dynamic causal link graph .
[0039] Step 400: Using the causal potential energy allocation algorithm based on the topology of the dynamic causal link graph, assign potential energy values to each node to generate a causal potential energy distribution graph.
[0040] In this step, the input to the causal potential energy allocation algorithm is the dynamic causal link graph. The set of emotion outcome nodes is used as the output, which is the potential energy value of each node. The causal potential energy distribution graph is a data structure that adds a potential energy attribute to each node based on the dynamic causal link graph.
[0041] The aforementioned causal potential energy allocation algorithm includes the following steps: Step 401: Obtain all sentiment result nodes from the sentiment result node set, and assign the maximum potential energy value to each sentiment result node. The potential energy symbol is determined based on the emotional polarity of the emotional outcome node, with positive emotions corresponding to positive potential energy and negative emotions corresponding to negative potential energy.
[0042] Step 402: For the node set Each non-emotional outcome node in The shortest path length from non-sentiment result nodes to all sentiment result nodes is calculated using breadth-first search, and the minimum value among them is taken as the shortest path length. At the same time, record the corresponding most recent sentiment outcome node. .
[0043] Step 403: Calculate the potential value of each non-emotional outcome node based on the shortest path length: in, The potential energy decay coefficient, This is the lower limit of the potential energy value, and the sign of the potential energy value is inherited from the emotional polarity of the most recent emotional outcome node.
[0044] Furthermore, the lower limit of potential energy The range of values is This is used to ensure that the potential value does not decay to zero or a negative value when the node is far from the emotional result node, thereby maintaining the effectiveness of the causal potential gradient.
[0045] Step 404: Append the calculated potential energy values to each node of the dynamic causal link graph and output the causal potential energy distribution graph.
[0046] Furthermore, the maximum potential energy value The range of values is Potential energy decay coefficient The range of values is This is to ensure that the potential energy value does not decay to a negative value or become too small after a finite number of jumps, thus losing its distinguishability.
[0047] Based on the dynamic causal link graph constructed from the aforementioned dialogue, the system executes a causal potential energy allocation algorithm. A maximum potential energy value is set. Potential energy decay coefficient Potential energy lower limit The emotional outcome node "dissatisfaction" is marked as a negative emotion and assigned a potential energy value. For other nodes, the system calculates the shortest path length to the "dissatisfaction" node using a breadth-first search, and then applies the formula: The potential energy value is calculated, and the sign of the potential energy is inherited from the negative polarity of the most recent sentiment outcome node. The potential energy distribution results for each node are shown in the table below.
[0048] Table 3. Results of Causal Potential Energy Allocation Step 500: Analyze the historical window data of the time-series multimodal feature sequence using the sliding window sentiment analysis algorithm, generate a historical sentiment intensity value sequence, and use second-order difference to calculate and generate a sentiment acceleration sequence.
[0049] In this step, the input to the sliding window sentiment analysis algorithm is a temporal multimodal feature sequence. China recently The feature vector at each time point ( (where the window size is 1), the output is the sentiment intensity value at the corresponding time point. The historical sentiment intensity value sequence is denoted as... ,in Negative values represent negative emotions, positive values represent positive emotions, and absolute values represent the intensity of emotions.
[0050] The aforementioned sliding window sentiment analysis algorithm is implemented based on a recurrent neural network. The input layer of the recurrent neural network receives the data within the sliding window. The multimodal feature vector sequence at time 1 The hidden layer processes features at each time step sequentially and updates the hidden state through recurrent units. The output layer is a linear mapping layer that maps the hidden state at the last time step to a scalar sentiment intensity value, and constrains the output to a certain value using a hyperbolic tangent activation function. Within the range: in, For recurrent neural networks in the first... The hidden state at all times and These are the weights and bias parameters of the linear mapping layer.
[0051] The aforementioned sliding window sentiment analysis network is trained using supervised learning, with the training data being a medical dialogue dataset labeled with sentiment intensity values. The loss function used is mean squared error loss. in, For the first The predicted sentiment intensity value for each training sample. For the first The true sentiment intensity values labeled on each training sample. This represents the number of training samples. The Adam optimization algorithm is used for parameter updates during training.
[0052] Furthermore, the sliding window size The range of values is At a certain point in time, ensure that the window contains enough historical information for sentiment intensity prediction, while avoiding excessively large windows that could cause computational delays and affect real-time performance.
[0053] The input for the second-order difference calculation is a sequence of historical sentiment intensity values. The output is an emotional acceleration sequence. Emotional acceleration value The calculation formula is: in, , , The first , , The emotional intensity value at any given moment. The emotional acceleration value reflects the trend of the rate of change of emotion; a positive value indicates that the emotion is accelerating in a positive direction, and a negative value indicates that the emotion is accelerating in a negative direction.
[0054] The system performs sliding window sentiment analysis on the temporal multimodal feature sequence of the aforementioned dialogue, with the window size being... The recurrent neural network processes the feature vector sequence within each window sequentially and outputs the sentiment intensity value at each time step. The patient's emotional state remained nearly neutral when inquiring about the examination process. The intensity of the emotion begins to decrease when the patient mentions "it's a bit expensive". The emotional intensity further decreased when the patient's tone conveyed urgency. An emotional acceleration sequence was obtained by calculating the second-order difference based on the emotional intensity value sequence. The emotional acceleration of a moment: Negative values indicate that the emotion is accelerating in a negative direction. The emotion intensity and emotion acceleration at each time point are shown in the table below.
[0055] Table 4 Calculation Results of Emotion Intensity and Acceleration Step 600: Analyze the potential energy change trend of the corresponding node in the causal potential energy distribution map using the potential energy gradient calculation algorithm, generate the causal potential energy gradient vector, and perform feature splicing with the emotion acceleration sequence to generate the fused prediction feature vector.
[0056] In this step, the potential gradient calculation algorithm takes as input a causal potential energy distribution map and a sequence of node positions corresponding to the current dialogue progress, and outputs a causal potential energy gradient vector.
[0057] The aforementioned potential gradient calculation algorithm includes the following steps: Step 601: Perform text matching between the newly identified event entity or state entity and the nodes in the causal potential energy distribution graph to determine the current position node corresponding to the current dialogue progress. And obtain the potential energy value of the node at the current position. .
[0058] Furthermore, text matching is achieved using word vector cosine similarity calculation. Specifically, the newly identified entity text and the entity text of each node in the causal potential energy distribution graph are converted into word vector representations, and the cosine similarity between them is calculated. The node with the highest similarity exceeding the matching threshold is selected as the current position node. If the similarity of all nodes does not exceed the matching threshold, the newly identified entity is added as a new node to the causal potential distribution map.
[0059] Furthermore, the range of the text matching threshold is as follows: The matching threshold is used to determine whether the newly identified entity and the existing node refer to the same concept.
[0060] Step 602: Traverse the nodes starting from the current position For all outgoing edges, obtain the set of next nodes pointed to by each outgoing edge. ,in Indicates the first The next node that the outgoing edge points to. This represents the number of outgoing edges.
[0061] Step 603: For each outgoing edge direction Calculate the causal potential gradient components along this direction: The causal potential energy gradient components in all directions are combined to form a causal potential energy gradient vector: .
[0062] Step 604: Since the causal potential gradient vector and the emotional acceleration sequence are different physical quantities with different value ranges, Z-score standardization needs to be performed on both types of features before vector concatenation to eliminate the impact of differences in dimensions and numerical ranges on the prediction model. The standardized causal potential gradient vector is then matched with the closest... The values are concatenated into a vector to generate a fused prediction feature vector: in, This indicates a vector concatenation operation.
[0063] Furthermore, the length of the emotion acceleration sequence The range of values is This ensures that the fused predictive feature vector contains sufficient information about the trend of emotional changes, while avoiding the introduction of noise with low relevance to the current state due to excessively long historical sequences.
[0064] exist At this moment, the system identifies the implicit state entity "waiting for an appointment" from the patient's utterance, "When can I schedule an appointment for the test recently?" It then uses word vector cosine similarity calculation to match this with the "economic pressure perception" node in the causal potential energy distribution graph (similarity is 0.62, not exceeding the matching threshold of 0.75). Therefore, "waiting for an appointment" is added as a new node to the graph, and a causal edge is established from "economic pressure perception" to "waiting for an appointment". The current node is determined to be "economic pressure perception" (potential energy value). This node has two outgoing edges, one pointing to "dissatisfaction" (potential energy value). ) and "Waiting for Reservation" (potential energy value to be allocated) The causal potential energy gradient components along the two directions are respectively and This forms the causal potential gradient vector: The standardized causal potential gradient vector is compared with the nearest... Emotional acceleration value at a given moment The features are concatenated to generate a fused predictive feature vector. .
[0065] Step 700: Calculate the future based on the fused prediction feature vector using the inflection point joint prediction model. The probability value of an emotional turning point within a time step and the predicted causal potential gradient direction are used to generate a turning point warning signal when the turning point probability value exceeds a preset threshold and the direction is negative.
[0066] In this step, the input to the joint prediction model for inflection points is the fused prediction feature vector. The output is the turning probability value. and the predicted causal potential gradient direction The inflection point joint prediction model is a pre-trained multilayer perceptron model, consisting of two output heads: a probability output head and a probability output head. Within the range of turning probability values, the directional output head outputs the causal potential gradient direction (positive, negative, neutral) category prediction.
[0067] The input layer of the aforementioned joint prediction model for turning points receives fused prediction feature vectors. The hidden layer consists of multiple fully connected layers, with ReLU activation functions used for non-linear transformations between them. The output layer has two parallel output heads: a probability output head, consisting of a fully connected layer followed by a Sigmoid activation function, maps the hidden layer output to transition probability values; and a direction output head, consisting of a fully connected layer followed by a Softmax activation function, maps the hidden layer output to probability distributions for three direction categories. The probability and direction output heads are calculated as follows: in, The output of the hidden layer, For the Sigmoid function, , These are the parameters for the probability output header. , These are the parameters for the direction output head. Let be the probability vectors for the three directional categories. The category with the highest probability is selected as the prediction result.
[0068] The aforementioned joint prediction model for turning points is trained using supervised learning. The training data consists of a medical dialogue dataset labeled with both the occurrence and direction of the turning point. The loss function is a weighted sum of the probability prediction loss and the direction prediction loss. in, The binary cross-entropy loss is the probability output head. For the multi-class cross-entropy loss of the directional output head, To balance the weights of the two loss terms, the Adam optimization algorithm is used for parameter updates during training.
[0069] Furthermore, predict the number of time steps. The range of values is Each time step, turning point probability threshold The range of values is Loss weighting coefficient The range of values is .in, Higher values can reduce false positives but may result in missed positives, while lower values can increase recall but may increase false positives. In practical applications, the tolerance for false positives and missed positives should be adjusted according to the medical customer service scenario.
[0070] The conditions for generating a turning point warning signal are: in, The preset turning point probability threshold, This indicates the generation of a turning point warning signal. This indicates that no warning will be generated.
[0071] The inflection point joint prediction model receives the aforementioned fused prediction feature vector. After nonlinear transformation through a multi-layer fully connected network, the probability output head outputs the inflection probability value. The probability distribution of the three directional categories output by the directional output head is as follows: These correspond to positive, negative, and neutral, respectively. The category with the highest probability is selected to obtain the predicted direction. Set a turning point probability threshold. ,because and The system generates a turning point warning signal. The customer service staff were reminded that the patient's emotions might change in the future. The situation reversed in a negative direction within a certain time step.
[0072] Based on step 700, the following steps are also included: Step 800: Using a multi-hop backtracking algorithm, traverse the dynamic causal link graph in reverse based on the predicted causal potential gradient direction, backtracking from the current dialogue position node along the causal edge to the root node, generating a set of multi-hop causal paths.
[0073] In this step, the input to the multi-hop backtracking algorithm is a dynamic causal link graph. The current node position and the predicted causal potential gradient direction are used to output a set of multi-hop causal paths.
[0074] The aforementioned multi-hop backtracking algorithm includes the following steps: Step 801: Initialize an empty set of multi-hop causal paths and the current path stack, and push the current position node onto the path stack as the backtracking starting point.
[0075] Step 802: Obtain all incoming edges (i.e. directed edges pointing to the top node of the path stack) of the top node, and filter out the set of predecessor nodes corresponding to the incoming edges.
[0076] Step 803: Determine if the predecessor node set is empty. If it is empty, it means that the current path stack top node is the root node. Reverse the node sequence in the path stack and add it to the multi-hop causal path set as a complete multi-hop causal path. Then pop the top node of the stack and backtrack. If it is not empty, push each predecessor node onto the path stack in turn and continue to execute step 802.
[0077] Step 804: Repeat steps 802 and 803 until the path stack is empty, and output the set of multi-hop causal paths.
[0078] This step also includes step 805, which calculates the attribution significance score of each multi-hop causal path using a path significance evaluation algorithm. The path significance evaluation algorithm takes the set of multi-hop causal paths as input and outputs the significance score of each multi-hop causal path.
[0079] The aforementioned path saliency evaluation algorithm includes the following steps: Step 806: For each multi-hop causal path in the set of multi-hop causal paths, count the number of nodes contained in the multi-hop causal path as the length of the multi-hop causal path. .
[0080] Step 807: Traverse each directed edge on the multi-hop causal path, obtain the confidence weight of each edge, and calculate the weight product. ,in For the first hop in a multi-hop causal path Confidence weights of the edges.
[0081] Step 808: Calculate the consistency between the potential polarity of the endpoint node of the multi-hop causal path and the direction of the predicted causal potential gradient. If consistent, calculate the consistency coefficient. If they are inconsistent, then .
[0082] Step 809: Calculate the attribution significance score for each multi-hop causal path: Furthermore, in the attribution significance score formula, the length of the multi-hop causal path... For positive integers, the product of confidence scores and weights. (Since the initial confidence weights for each side are 1 and only increase), consistency coefficient Therefore, the attribution significance score , and when A time score of zero indicates that the multi-hop causal path is inconsistent with the predicted direction.
[0083] Step 810: Output each multi-hop causal path and its corresponding significance score.
[0084] The system executes a multi-hop backtracking algorithm starting from the current node "Perception of Economic Pressure," traversing backwards along the causal edges to each root node. During the backtracking process, two complete multi-hop causal paths are found: Path 1 is "Informed about examination fees → Fee 800 yuan → Perception of economic pressure → Dissatisfaction," and Path 2 is "Explained reimbursement ratio → Low reimbursement ratio → Perception of economic pressure → Dissatisfaction." The system calculates the attribution significance score for each path, where the length of Path 1 is... The product of confidence weights on each side The potential polarity of the endpoint "dissatisfaction" is negative, consistent with the predicted direction, hence the consistency coefficient is: Score Length of path two The product of confidence weights on each side (Due to the "low reimbursement rate → perceived economic pressure" side) After the time is identified, the confidence weight increases to 2), and the consistency coefficient... ,Score The multi-hop causal paths and their evaluation results are shown in the table below.
[0085] Table 5. Significance Assessment Results of Multi-hop Causal Paths Step 900: Use a natural language generation algorithm to convert the multi-hop causal path with the highest attribution significance score into causal explanation text, and output the turning point warning signal, the predicted turning point time window, and the complete causal chain explanation from the root cause to the emotional turning point.
[0086] In this step, the input to the natural language generation algorithm is the multi-hop causal path with the highest attribution significance score and its node information, and the output is causal explanatory text. The causal explanatory text adopts a chain-like sentence structure of "A leads to B, B further leads to C, and therefore may lead to D", converting the node entities and relationships in the multi-hop causal path into natural language descriptions.
[0087] Furthermore, the natural language generation algorithm is implemented using a template-based filling method. Specifically, a predefined causal chain sentence template is used, which contains entity slots and relation slots. Following the node order of the multi-hop causal path, the entity text of each node is sequentially filled into the corresponding entity slot, and the causal relationship type of each side is mapped to conjunctions (such as "cause", "lead to", "further") and filled into the relation slot. Finally, the complete causal explanation text is generated by splicing the text together.
[0088] The predicted turning point window is estimated based on the path length from the current location node to the sentiment outcome node and the average time interval of historical conversations.
[0089] Furthermore, the formula for calculating the turning point time window is as follows: ,in This represents the shortest path length from the current node to the nearest sentiment result node. The average time interval between adjacent turns in a historical dialogue. This indicates the length of the time window in which the predicted emotional shift may occur.
[0090] Furthermore, the average time interval The calculation method is as follows: count the timestamp sequence of each turn that has occurred in the current dialogue session. ,in Indicates the first Timestamps of round-based conversations; calculate the arithmetic mean of the time differences between adjacent rounds. , This represents the number of turns that have occurred in the current conversation.
[0091] The system selects path two, which has the highest attribution significance score: "Explanation of reimbursement ratio → Low reimbursement ratio → Perceived economic pressure → Dissatisfaction," and uses a natural language generation algorithm to convert it into causal explanatory text. Based on a predefined causal chain sentence template, it sequentially fills in the entity text and relational connectives for each node, generating the causal explanatory text as: "Customer service explained that the medical insurance reimbursement ratio leads to the patient's perception of a low reimbursement ratio, which in turn exacerbates the perceived economic pressure and may trigger dissatisfaction." The current dialogue has completed 5 rounds, with the timestamp sequence for each round as follows: (Unit: seconds), average time interval Seconds, the shortest path length from the current location node to the sentiment result node. Predicting the turning point window Seconds. The complete alert information output by the system to customer service personnel is shown in the table below.
[0092] Table 6. Results of Emotional Turning Point Early Warning It is understood that data preprocessing methods known to those skilled in the art include data cleaning, data transformation, and data reduction. Data transformation includes type conversion and normalization and standardization. Although the dimensions and types of data were omitted in the description of the preceding embodiments, data preprocessing is a technical knowledge known to those skilled in the art and a prerequisite step in data processing. Therefore, the previously described well-known data preprocessing steps were not described independently.
[0093] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.
Claims
1. A multimodal feature-coordinated affective semantic reasoning method, characterized in that, Includes the following steps: Acquire real-time multimodal data streams of the dialogue and generate temporal multimodal feature sequences using a multimodal encoder; The system analyzes dialogue text using an entity relation extraction algorithm, identifies event entities, state entities and their causal relationships, generates a real-time causal triple sequence, and labels sentiment outcome nodes. The causal triplet sequence is integrated into a dynamic causal link graph using the causal graph incremental update algorithm; The causal potential energy allocation algorithm is based on the topology of the dynamic causal link graph. The maximum potential energy value is assigned to the emotional result node, and the decayed potential energy value is assigned to other nodes according to the shortest path length to the emotional result node, thus generating a causal potential energy distribution graph. The sliding window sentiment analysis algorithm is used to analyze the temporal multimodal feature sequence, generate the historical sentiment intensity value sequence, and generate the sentiment acceleration sequence by calculating the second difference. The current dialogue position node is determined by using the potential energy gradient calculation algorithm, the causal potential energy gradient components along each outgoing edge direction are calculated, and the causal potential energy gradient vector is concatenated with the emotion acceleration sequence to generate a fused prediction feature vector. The inflection point joint prediction model is used to calculate the inflection probability value and the predicted causal potential energy gradient direction based on the fused prediction feature vector. When the inflection probability value exceeds the preset threshold and the direction is negative, an inflection warning signal is generated. The multi-hop backtracking algorithm is used to traverse backwards from the current node along the causal edge to the root node, generating a set of multi-hop causal paths, and calculating the attribution significance score of each path. The multi-hop causal path with the highest attribution significance score is converted into causal explanation text, and the turning point warning signal and causal chain explanation are output.
2. The method according to claim 1, characterized in that, The causal potential energy allocation algorithm includes: Assign the maximum potential energy value to the emotional outcome node and determine the potential energy sign based on the emotional polarity; For non-emotional result nodes, breadth-first search is used to calculate the shortest path length from them to all emotional result nodes, and the minimum value is taken as the distance value. The potential energy value is calculated based on the shortest path length. The potential energy value is equal to the larger of the lower limit of potential energy and the maximum potential energy value minus the difference between the potential energy decay coefficient and the distance value. The potential energy sign is inherited from the emotional polarity of the nearest emotional result node.
3. The method according to claim 2, characterized in that, The maximum potential energy value ranges from 1 to 10, the potential energy decay coefficient ranges from greater than zero to less than the maximum potential energy value, and the lower limit of the potential energy value ranges from 0.1 to 1.
4. The method according to claim 1, characterized in that, The potential energy gradient calculation algorithm includes: The newly identified entity is matched with the nodes in the causal potential energy distribution map by word vector cosine similarity to determine the node at the current position. Iterate through all outgoing edges of the current node and get the next node that each outgoing edge points to; For each outgoing edge direction, the causal potential gradient component is calculated as the potential energy value of the next node minus the potential energy value of the current node. The gradient components in all directions are combined to form a causal potential gradient vector.
5. The method according to claim 4, characterized in that, Before vector concatenation, Z-score standardization was performed on the causal potential gradient vector and the emotional acceleration sequence, respectively. The emotional acceleration sequence takes values from a number of recent moments, with the number of moments ranging from 3 to 8.
6. The method according to claim 1, characterized in that, The sliding window sentiment analysis algorithm is based on a recurrent neural network. The recurrent neural network receives a sequence of multimodal feature vectors within the sliding window. The output layer maps the hidden state to a scalar sentiment intensity value through a linear mapping layer, and constrains the output to the range of negative one to positive one through a hyperbolic tangent activation function. The emotional acceleration is calculated by subtracting twice the emotional intensity value of the previous moment from the current emotional intensity value and then adding the emotional intensity values of the two moments before.
7. The method according to claim 1, characterized in that, The joint prediction model for the inflection point is a multilayer perceptron model, which includes a probability output head and a direction output head; The probability output header passes through a fully connected layer and is followed by a Sigmoid activation function to output the transition probability value. The directional output head passes through a fully connected layer and is followed by a Softmax activation function to output the probability distribution of three directional categories: positive, negative, and neutral.
8. The method according to claim 1, characterized in that, The multi-hop backtracking algorithm includes: Initialize an empty set of multi-hop causal paths and the current path stack, and push the current position node onto the path stack; Get the set of predecessor nodes corresponding to all incoming edges of the top node of the stack; If the predecessor node set is empty, the node sequence in the path stack is reversed and added to the set as the complete path. Then, the top node of the stack is popped and backtracking is performed. If it is not empty, push each predecessor node onto the path stack in turn and continue backtracking; Repeat until the path stack is empty.
9. The method according to claim 1, characterized in that, The calculation of the attribution significance score includes: The number of nodes in a multi-hop causal path is used as the path length; Traverse each directed edge on the path and calculate the confidence weight product; The consistency coefficient between the potential energy polarity of the path endpoint node and the predicted causal potential energy gradient direction is calculated; it is 1 when they are consistent and 0 when they are inconsistent. The attribution significance score is equal to the weighted product multiplied by the consistency coefficient and then divided by the path length.
10. A multimodal feature-coordinated affective semantic reasoning system, used to execute the method according to any one of claims 1 to 9, characterized in that, include: The multimodal feature extraction module is used to acquire real-time multimodal data streams of dialogues and generate temporal multimodal feature sequences; The causal relationship extraction module is used to analyze dialogue text to identify event entities, state entities and causal relationships, generate causal triple sequences and label sentiment result nodes; The dynamic causal graph construction module is used to integrate causal triple sequences into a dynamic causal link graph; The causal potential energy allocation module is used to generate a causal potential energy distribution map by assigning potential energy values to each node based on the graph topology. The sentiment analysis module is used to analyze time-series multimodal feature sequences to generate sentiment intensity value sequences and sentiment acceleration sequences; The potential energy gradient calculation module is used to calculate the causal potential energy gradient vector and fuse it with the emotion acceleration sequence to generate a fused prediction feature vector; The turning point prediction module is used to calculate the turning point probability value and direction based on the fused prediction feature vector and generate a turning point early warning signal. The multi-hop attribution module is used to generate a set of multi-hop causal paths by traversing backwards from the current position node and calculating the significance score; The explanation generation module is used to convert the path with the highest significance score into causal explanation text and output warning results.