Multi-turn dialogue emotion quadruple extraction method using large language model reasoning

Through dynamic word pruning, knowledge distillation and multi-level teacher model, combined with structural entropy minimization algorithm, the problem of large-scale language models consumes a lot of computing resources in multi-round dialogue sentiment analysis is solved, and efficient and accurate extraction of dialogue emotions is achieved.

CN118982030AActive Publication Date: 2024-11-19SU ZHOU DING YI ZHI NENG JI SHU YOU XIAN GONG SI

Patent Information

Application Number
CN202411465374.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-21
Publication Date
2024-11-19
Estimated Expiration
2044-10-21

AI Technical Summary

Technical Problem

The existing large language models consume a lot of computing resources and slow inference speed in multi-round dialogue sentiment analysis, which limits their application in dialogue systems.

Method used

Using dynamic word pruning model and knowledge distillation model, combining multi-level teacher model and two-dimensional structure entropy minimization algorithm, preprocessing and partitioning of dialogue data, constructing dialogue diagrams, extracting emotional quadruples, and matching through maximum pooling operation Emotional relationship.

Benefits of technology

It improves the accuracy and computing efficiency of dialogue emotion extraction, reduces the consumption of computing resources, and enhances the fine-grained understanding of dialogue emotion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982030A_ABST
    Figure CN118982030A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-turn dialogue emotion quadruple extraction method using large language model reasoning, and relates to the technical field of emotion quadruple extraction, and the method comprises the steps: collecting multi-turn dialogue data and context information, and carrying out the preprocessing of the data; performing lexical element trimming on the context information through a dynamic lexical element trimming model; establishing a knowledge distillation model, and constructing a multi-level teacher model according to a reasoning result in combination with multi-turn dialogue data; guiding the student model to learn detail information in the dialogue from multi-level dialogue knowledge through the multi-level teacher model; based on the dialogues processed by the LazyLLM and the knowledge distillation model, constructing a dialogue graph; dividing the dialogue into minimum sub-dialogues by using a two-dimensional structure entropy minimization algorithm, combining the sub-dialogues, realizing optimal division, and forming a sub-dialogue set; constructing a two-dimensional table annotation entity relationship, extracting entities in the utterance, and matching the entities based on a sub-dialogue set to form an emotion tetrad; and finally matching the emotion tetrad by predicting the relationship between the entities.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of emotion quadruple extraction, and in particular to a method for multi-round dialogue emotion quadruple extraction using large language model reasoning. Background Art

[0002] With the rapid development of natural language processing technology, multi-turn dialogue systems have been widely used in the fields of intelligent customer service, virtual assistants, and social robots. In these applications, sentiment analysis, as an important means to understand the user's emotional state, is crucial to improving user experience and system response quality. Sentiment quadruple extraction is an advanced task in sentiment analysis, which not only requires identifying the sentiment polarity (such as positive, negative, and neutral) in the conversation, but also extracting the target, aspect, and opinion of the sentiment.

[0003] In recent years, large-scale language models based on deep learning (such as BERT and GPT series) have shown superior performance in sentiment analysis tasks. However, these models still face problems such as high computing resource consumption and slow reasoning speed when processing long texts or multi-turn dialogues, which limits their application in dialogue systems. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a multi-round dialogue emotion quadruple extraction method using large language model reasoning to solve the problem of improving the accuracy of emotion extraction and reducing computational costs.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: The embodiment of the present invention provides a method for extracting emotions from a multi-round conversation using a large language model inference, which includes collecting multi-round conversation data and context information between a customer service representative and a customer, and preprocessing the collected data; The collected context information is pruned using a dynamic lemma pruning model; Establish a knowledge distillation model and build a multi-level teacher model based on the reasoning results and multi-round dialogue data; The multi-level teacher model guides the student model to learn the detailed information in the conversation from the multi-level conversation knowledge; Construct a conversation graph based on the conversations processed by the LazyLLM and knowledge distillation models; Use a two-dimensional structural entropy minimization algorithm to divide the conversation into the smallest sub-conversations, and perform a merge operation to achieve the optimal division to form a sub-conversation set; Construct a two-dimensional table, annotate entity relationships, extract entities in discourse, match entities based on sub-dialogue sets, and form sentiment quadruples; Through the maximum pooling operation, the relationship between entities is predicted and finally the sentiment quadruple is matched.

[0007] As a preferred solution of the multi-round dialogue emotion quadruple extraction method using large language model inference described in the present invention, wherein: the multi-round dialogue information includes customer ID, customer service ID, dialogue content, timestamp and dialogue turn; The context information includes historical interaction records, conversation topics, relevant business status and customer information; The preprocessing includes data cleaning and text standardization.

[0008] As a preferred solution of the method for extracting emotions from multi-round dialogues using large language model inference according to the present invention, the method of performing word pruning on the collected context information by using a dynamic word pruning model comprises the following steps: In each generation step, use the layer The attention map determines the input word The importance relative to the next token to be predicted is expressed as,

[0009] in, To pay attention to the number of avatars, is the sequence length, Word In the Words in the head The probability of attention, Word In the Confidence score of the layer; In each transformer layer, the Top-k percentile is selected as the threshold for word pruning based on the importance score of the word. In the Confidence score of the layer Less than the first percentile, the token will be pruned. Once a token is pruned, it will be excluded from the calculations of all subsequent layers. Indicated in The percentage of the layer used to determine the threshold; Prune different numbers of tokens in each layer; Apply a progressive pruning strategy that keeps more tokens in earlier transformer layers, progressively removes more tokens in later layers, and gradually reduces the number of tokens toward the end of the transformer. In the pre-filling phase, the hidden state is calculated for each token, and progressive token pruning is achieved by removing the hidden state of the pruned token. For tokens pruned in subsequent layers, the hidden states are stored in an auxiliary cache; During the decoding phase, each transformer layer retrieves the KV pairs of past tokens from the KV cache. For tokens that do not exist in the KV cache, the LazyLLM retrieves the hidden states from the auxiliary cache.

[0010] As a preferred solution of the method for extracting emotions from multi-round dialogues using large language model reasoning described in the present invention, the method of establishing a knowledge distillation model and constructing a multi-level teacher model based on the reasoning results combined with multi-round dialogue data includes the following steps: Teachers with different levels of understanding of the components of the dialogue document were constructed; The input of all teachers consists of each round of speech in the dialogue dataset, expressed as,

[0011] in, represents the i-th speech in the entire data set, is the length of the speech, Indicates a word in a speech; Word Level Slot Filling Teacher Predict the slot type for each word, word-level slot filling teacher The output of is represented as the predicted slot type for each word; Sentence-level intent detection tutor Predicting the intention of speech, For prediction express,

[0012] in, represents the number of intents in the dataset, For the intention detection task; Dialogue Document-Level Domain Classification Teacher Predicting the domain of the conversation, from document-level domain classification teachers The predicted value is expressed as ,

[0013] in, is the number of domains in the dataset, Categorize tasks for domains; Apply cross entropy loss and fine-tune the pre-trained model for a fixed number of rounds, using the checkpoint of the last round as the teacher model. The specific expression is,

[0014] in, and is the number of training samples, is the true value, Indicates that the length of the speech in the training set is A collection of speeches, Indicates that the length of the speech in the training set is The first speaker in the collection, Indicates that the length of the speech in the training set is The first in the collection A speech, Indicates that the length of the speech in the training set is The first in the collection A speech, represents the cross entropy loss, represents the teacher model, where ; The teachers at different levels of understanding of dialogue document components include word-level slots, sentence-level intent, and dialogue-level domain knowledge.

[0015] As a preferred solution of the multi-round dialogue emotion quadruple extraction method using large language model reasoning described in the present invention, wherein: the multi-level teacher model guides the student model to learn detailed information in the dialogue from multi-level dialogue knowledge, including the following steps: The KD loss is calculated by comparing the average probability produced by the teacher model ensemble with the probability derived by the student model; Use KD loss to align the predicted probability distribution between the student model and the teacher model; By using the cross-entropy loss, the student model receives a direct supervisory signal from the ground truth; During the training process, triplets are randomly generated for each batch of data, and the internal relations of the triplets are determined by aggregating the votes of the teacher model combinations; Use TripletMarginLoss to minimize the distance between positive samples and anchor samples, while maximizing the distance between negative samples and anchor samples; The similarity loss is calculated by maximizing the similarity of the prediction vectors (logits) between the student model and the teacher model; Incorporate the teacher model’s predictions as pseudo labels and use the probabilities assigned by the teacher model for each class.

[0016] As a preferred solution of the method for extracting emotions from multi-round dialogues using large language model inference described in the present invention, the dialogue graph is constructed based on the dialogue processed by the dynamic word gram pruning model (LazyLLM) and the knowledge distillation model, including the following steps: In the dialogue data, each round of speech is regarded as a sentence, and these sentences are combined to form the input of the model. Assuming that the current dialogue data contains multiple rounds, each sentence and utterance is recorded as , connect these words in series as input, construct the input format, expressed as,

[0017] in, Represents a serial operation, is a special category identifier, is a separator used to distinguish between words, represents the nth utterance, i.e. the last utterance in the sequence; After encoding by the pre-trained language model, the output form is expressed as,

[0018] in, It is a coded symbol of discourse ,make Corresponding to each [CLS] as the embedding of each utterance; The embedding of each utterance is assigned as a node in the conversation graph, and edges are constructed based on the reply relations and semantic similarity in the conversation; For any two utterances and ,if Yes The reply is in the corresponding node and Create a bidirectional edge between them; For any two nodes that are not directly connected and , if there is an intermediate node Make and pass To form a path, and Add an edge between them; The weight of each edge is calculated by the cosine similarity of the node embeddings; Add a self-loop edge to each node, that is, at each node Create an edge with weight 1.

[0019] As a preferred solution of the method for extracting emotions from a multi-round dialogue using a large language model inference according to the present invention, the method uses a two-dimensional structural entropy minimization algorithm to divide the dialogue into the smallest sub-dialogues, and performs a merging operation to achieve the optimal division to form a sub-dialogue set, including the following steps: Initialize the conversation graph as an encoding tree ,Each leaf node represents a vertex in the conversation graph. If two leaf nodes share the same parent node, it means that the two leaf nodes belong to the same child conversation cluster.,Initially, each leaf node is regarded as a separate cluster; For the encoding tree The two sub-coding trees in and subcoding tree Share the same parent node and perform a merge operation Generate a new encoding tree , before and after the merge operation, the encoding tree The SE change calculation formula is:

[0020] in, and Sub-coding trees and subcoding tree After the merger , , and They are , and Before the merge operation ,like , represents the merged coding tree of Reduce, represents the structural entropy after the merging operation, represents the structural entropy before the merging operation, Represents the coding tree The change in structural entropy is express and The sub-coding tree of express The nodes in Remove from the tree The leaf node after the node; Traverse all edges in the coding tree and calculate the difference between each pair of sub-coding trees before and after merging. Changes, record each merge operation Each edge of corresponds to the connection between two leaf nodes, and merging these leaf nodes merges the two clusters into one cluster; choose The largest change value Merge the edges. For each selected edge, merge the corresponding two sub-coding trees into a new sub-coding tree and update the coding tree ; In each iteration, the structural entropy of the coding tree will gradually decrease, and the number of sub-dialog clusters will be gradually reduced by continuously merging sub-coding trees; When the encoding tree of When it no longer decreases, the iteration stops and the encoding tree structure finally stabilizes; When dividing a conversation, the root utterance is included in each sub-conversation cluster to form a sub-conversation set.

[0021] As a preferred solution of the method for extracting emotion quadruple from multi-round dialogues using large language model reasoning described in the present invention, the steps of constructing a two-dimensional table, annotating entity relationships, extracting entities from discourse, matching entities based on sub-dialogue sets, and forming emotion quadruple include the following steps: Represent the relationship between any two boundary locations in each utterance as cells in a two-dimensional table; Use two dense multilayer perceptrons to learn the start and end boundaries of utterances; The boundary relationship table is constructed using the start boundary and the end boundary of the utterance, and the maximum pooling operation is performed on the boundary relationship table; Applying dual affine attention to further model boundary relations in discourse; Input the boundary relationship table into a dense layer and a CNN layer to obtain the final entity relationship table; Use a linear layer to predict the label for each position; Introducing a cross-entropy based loss function to perform supervised learning for each position in each utterance; Extract all target, aspect, and opinion entity spans from each utterance, and group entities belonging to the same sub-dialogue together based on the partitioned sub-dialogue set; Pair the target, aspect and opinion entities to form three types of entity pairs: target-aspect pair TA, target-opinion pair TO and aspect-opinion pair AO; For target-aspect pair TA entity and target-opinion pair TO entity, use labels to determine whether they belong to the same sentiment quadruple; For target-opinion pairs TO entity pairs, labels are used to determine their inclusion in the same sentiment quartet and the sentiment polarity of the quartet.

[0022] As a preferred solution of the multi-round dialogue emotion quadruple extraction method using large language model reasoning described in the present invention, wherein: the relationship between entities is predicted through maximum pooling operation, and finally the emotion quadruple is matched, including the following steps: For two span spaces that do not belong to the same entity category in the same sub-dialogue set, perform maximum pooling operations on their span ranges; Perform a linear transformation on each pair of span representations to obtain the final span representation; Use a labeled linear layer to predict the relationship label between each pair of entities based on the final spanned representation; For each sentiment triple, traverse the combination of target, aspect, and opinion in turn; If and only if all conditions are met simultaneously, the triple forms a successful sentiment quadruple.

[0023] The beneficial effects of the present invention are as follows: through the preprocessing of dialogue data, dynamic word pruning, knowledge distillation, the introduction of a multi-level teacher model, and the dialogue division and entity matching method based on the entropy minimization algorithm, the fine-grained extraction of dialogue sentiment is achieved as a whole, and each step cooperates with each other, which not only improves the computational efficiency of the model, but also enhances the accuracy of dialogue sentiment extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0025] Figure 1 This is a flowchart of the method for extracting quadruple emotions from multi-turn dialogues using large language model reasoning in Example 1.

[0026] Figure 2 This is a diagram of the dynamic word pruning model based on a large language model in Example 1.

[0027] Figure 3 This is a diagram of the knowledge distillation model used for multi-round dialogue understanding in Example 1.

[0028] Figure 4 This is a four-fold extraction graph of conversational emotions based on structural entropy minimization in Example 1. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0031] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0032] Example 1, reference Figure 1 , Figure 2 , Figure 3 and Figure 4 , which is the first embodiment of the present invention, and provides a method for extracting emotions from a multi-turn dialogue using a large language model inference, including the following steps: S1. Collect multi-round conversation data and context information between customer service and customers, and pre-process the collected data, including the following steps: Choose a fully functional online customer service platform. These platforms generally record all conversations between customers and customer service. Before starting to collect data, you need to ensure that you comply with relevant data protection laws and regulations to ensure that customer privacy is protected; Extract conversation records, including customer ID, customer service ID, conversation content, timestamp, and conversation rounds, through the API provided by the platform or direct database access. For example, the data structure may include the following fields: customer ID: a unique number that identifies the customer, customer service ID: a unique number that identifies the customer service, conversation content: the text content of each round of conversation, timestamp: the time when each message was sent, conversation rounds: the round information of the conversation is recorded to analyze emotional changes. In the process of data collection, special attention should be paid to the collection of contextual information. For example, the customer's historical interaction records before the conversation began can be recorded. This historical information can provide important background and help understand the customer's emotional changes. When extracting data, it is recommended to cover conversations in different time periods as much as possible to capture different situations and emotional expressions. Contextual information includes conversation topics, related business status, and customer information; To enrich the dataset, you can collect customer feedback data, such as satisfaction ratings and follow-up reviews. These feedbacks can help annotate the emotional state of the conversation; Clean the data to remove irrelevant information and noise, including: deleting automatically generated system prompts, welcome messages, etc., to ensure that only the actual conversation between the customer and the customer service is retained; checking the missing values ​​in the data and deciding whether to fill in (such as using the mean or median) or delete missing records; eliminating obvious abnormal conversations, such as obvious meaningless text or overly brief conversations; The conversation text is standardized to improve the accuracy of the analysis, including converting all text to lowercase or uppercase to eliminate duplication caused by different cases; cleaning up special characters, punctuation marks and extra spaces in the text to keep the text neat; expanding common abbreviations and correcting spelling errors to ensure the readability of the text.

[0033] S2, performing word pruning on the collected context information through a dynamic word pruning model, including the following steps: The LazyLLM starts with a full context and gradually reduces tokens to gradually reduce the number of computations at the end of the model. The LazyLLM allows the model to select different subsets of tokens from the context at different generation steps, even though some of them may have been removed in previous steps. Compared to static pruning, which prunes all tokens at once, dynamic pruning optimizes the next prediction of each generation token, which is critical to maintaining performance.

[0034] The dynamic token pruning model (LazyLLM) "lazily" computes tokens that are important for predicting the next token only in the first round of inference (pre-population phase). A key challenge in pruning tokens in the first round is to determine their importance.

[0035] Progressive token pruning methods need to accumulate the full attention maps of the first few tokens predicted in order to analyze the importance of the hint tokens before starting pruning. Therefore, they are not suitable for reducing the time to the first token because they still need to calculate all KV caches in the pre-filling stage.

[0036] In each generation step, use the layer The attention map determines the input word The importance relative to the next token to be predicted is expressed as,

[0037] in, To pay attention to the number of avatars, is the sequence length, Word In the Words in the head The probability of attention, Word In the Confidence score of the layer; After calculating the confidence score of the token, the threshold for pruning tokens is determined by using a top-k percentile selection strategy, which varies with the distribution of attention scores between different layers and different tasks; In each transformer layer, the Top-k percentile is selected as the threshold for word pruning based on the importance score of the word. In the Confidence score of the layer Less than the first percentile, the word will be pruned. Once the word is pruned, it will be excluded from the calculation of all subsequent layers, and the words used in the subsequent layers will be a subset of the previous layers. Indicated in The percentage of the layer used to determine the threshold, for example, The value is 30, which means 30% of the words in the layer have confidence scores below this value; The position of different pruned layers and the number of pruned tokens have an impact on performance. When pruning is performed on the same transformer layer, the performance of the model gradually decreases as the number of tokens retained decreases. Pruning in later transformer layers always performs better than pruning in earlier layers, which indicates that later layers are less sensitive to token pruning. In order to achieve a better balance between acceleration and accuracy, a progressive pruning strategy is applied, retaining more tokens in earlier transformer layers, gradually removing more tokens in later layers, and gradually reducing the number of tokens until the end of the transformer; In the pre-filling phase, hidden states are calculated for each token, and progressive token pruning is achieved by removing the hidden states of pruned tokens. In the decoding stage, an auxiliary cache (Aux Cache) is introduced. In addition to the original KV cache, the auxiliary cache stores the hidden states of those pruned tokens. If their KV does not exist in the KV cache of the subsequent layer, these hidden states can be retrieved in subsequent iterations; In each decoding step, each transformer layer (e.g. layer ) First, the KV cache of past tokens is retrieved. For those tokens that do not exist in the KV cache, their hidden states can be directly retrieved from the auxiliary cache of their previous layer without passing through the previous layer again. The introduction of the auxiliary cache ensures that each token is calculated at most once in each transformer layer and ensures that the worst running time of the dynamic token pruning model (LazyLLM) is no slower than the baseline.

[0038] S3, the establishment of the knowledge distillation model, building a multi-level teacher model based on the inference results combined with multi-round dialogue data, includes the following steps: To enhance natural language understanding in multi-turn dialogues, a new multi-level dialogue teacher knowledge distillation framework is introduced, which enhances natural language understanding capabilities by training student models through multi-level teachers. We built teachers with different levels of understanding of conversational document components, including word-level slots, sentence-level intent, and conversation-level domain knowledge; The input of all teachers consists of each round of speech in the dialogue dataset, expressed as,

[0039] in, represents the i-th speech in the entire data set, is the length of the speech, Indicates a word in a speech; Word Level Slot Filling Teacher Predict the slot type for each word, provide the student model with knowledge about key slots in the conversation, and the word-level slot filling teacher The output represents the predicted slot type for each word , the expression is,

[0040] in, , is the number of slot types; Sentence-level intent detection tutor Predict the intent of the utterance, help the student model understand the overall intent of each turn, and detect the sentence-level intent of the teacher For prediction express,

[0041] in, represents the number of intents in the dataset, For the intention detection task; Dialogue Document-Level Domain Classification Teacher Predicting the domain of the conversation provides information for its classification and understanding of background knowledge, from document-level domain classification teachers The predicted value is expressed as ,

[0042] in, is the number of domains in the dataset, Categorize tasks for domains; Through these three levels of teachers, the student model is guided to understand the conversation from multiple perspectives, which enhances the student model's grasp of different levels of conversation. Each pre-trained model focuses on learning a specific level of knowledge from the conversation dataset, so that the teacher has different levels of understanding of the components of the conversation document; Each teacher focuses on one level of conversational knowledge. This approach is motivated by two factors; first, learning knowledge from a single task is simpler than integrating knowledge from all tasks, simplifying the fine-tuning of pre-trained models. Second, rather than burdening a single model with the challenge of grasping knowledge from all aspects of the conversation, each teacher focuses on a specific level of understanding. Apply cross entropy loss and fine-tune the pre-trained model for a fixed number of rounds, using the checkpoint of the last round as the teacher model. The specific expression is,

[0043] in, and is the number of training samples, is the true value, Indicates that the length of the speech in the training set is A collection of speeches, Indicates that the length of the speech in the training set is The first speaker in the collection, Indicates that the length of the speech in the training set is The first in the collection A speech, Indicates that the length of the speech in the training set is The first in the collection A speech, represents the cross entropy loss, represents the teacher model, where .

[0044] S4. Guide the student model to learn detailed information in the conversation from multi-level conversation knowledge through a multi-level teacher model, including the following steps: After obtaining multi-level teachers, the student model is guided by a mixture of these teachers through the distillation of multi-level teacher knowledge. The teacher combination includes different levels, such as BERT-Base ID, BERT-Base SF, and BERT-Base DC. The student model is trained separately for each task, enabling it to master the complexity of a single task with the help of different levels of conversation knowledge. The KD loss is calculated by comparing the average probability produced by the teacher model combination with the probability obtained by the student model. The KD loss is used to align the predicted probability distribution between the student model and the teacher model, so that the student model can learn from multiple teachers. The expression is,

[0045] in, is the number of teachers, KL divergence loss (KLDivLoss) represents KD loss, represents the teacher model, represents the log odds of the teacher model, represents the log odds of the student model, is the normalized exponential function (softmax) of the teacher model, i.e., the activation function, is the normalized exponential function (softmax) of the student model; By using the cross entropy loss , the student model receives direct supervision signals from the real situation to help its learning process. This loss function is calculated by comparing the predictions of the student model with the basic facts of each task. The specific formula is,

[0046] in, represents the log odds of the student model, represents the true value; During the training process, triplets are randomly generated for each batch of data, and the internal relations of the triplets are determined by aggregating the votes of the teacher model combinations; Use TripletMarginLoss to minimize the distance between positive samples and anchor samples, while maximizing the distance between negative samples and anchor samples. Using TripletMarginLoss, the student model learns the internal relationship between batches of data, so that its understanding is consistent with that of the teacher model and ensures a consistent view of the data set. The expression is,

[0047] in, Indicates relationship loss, is the batch size, is a triple; The similarity loss is calculated by maximizing the similarity of the prediction vectors (logits) between the student model and the teacher model. In this way, the student model can learn the knowledge of the feature space from the teacher model, not just the prediction probability. The expression is,

[0048] in, represents the similarity loss, represents a similarity function, represents the log odds of the teacher model; The predictions of the teacher model are incorporated as pseudo labels to facilitate the training of the student model, and the probability assigned by the teacher model is used for each class to ensure that the student fully acquires the knowledge embedded in the teacher model, expressed as, .

[0049] S5. Based on the conversations processed by the LazyLLM and knowledge distillation models, a conversation graph is constructed, which includes the following steps: In the dialogue data, each round of speech is regarded as a sentence, and these sentences are combined to form the input of the model. Assuming that the current dialogue data contains multiple rounds, each sentence and utterance is recorded as ,in Indicates A word, belong ; These utterances are concatenated as input and the input format is constructed, which is expressed as:

[0050] in, Represents a serial operation, is a special category identifier, is a separator used to distinguish between words, represents the nth utterance, i.e. the last utterance in the sequence; After encoding by the pre-trained language model, the output form is expressed as,

[0051] in, It is a coded symbol of discourse ,make Corresponding to each [CLS] as the embedding of each utterance; The embedding of each utterance is assigned as a node in the conversation graph, and edges are constructed based on the reply relations and semantic similarity in the conversation; For any two utterances and ,if Yes The reply is in the corresponding node and Create a bidirectional edge between them; For any two nodes that are not directly connected and , if there is an intermediate node Make and pass To form a path, and Add an edge between them; The weight of each edge is calculated by the cosine similarity of the node embeddings; Add a self-loop edge to each node, that is, It establishes an edge with a weight of 1, and after adding a self-loop, an undirected conversation graph G is constructed.

[0052] S6. Using a two-dimensional structural entropy minimization algorithm to divide the conversation into the smallest sub-conversations, and performing a merge operation to achieve optimal division to form a sub-conversation set, including the following steps: Initialize the conversation graph as an encoding tree ,Each leaf node represents a vertex in the conversation graph G. If two leaf nodes share the same parent node, it means that the two leaf nodes belong to the same child conversation cluster. Initially, each leaf node is regarded as a separate cluster; For the encoding tree The two sub-coding trees in and subcoding tree Share the same parent node and perform a merge operation Generate a new encoding tree , which can be understood as clustering Merge into , before and after the merge operation, the encoding tree The SE change calculation formula is:

[0053] in, and Sub-coding trees and subcoding tree After the merger , , and They are , and Before the merge operation ,like , represents the merged coding tree of Reduced, thereby reducing uncertainty, represents the structural entropy after the merging operation, represents the structural entropy before the merging operation, Represents the coding tree The change in structural entropy is express and The sub-coding tree of express The nodes in Remove from the tree The leaf node after the node; Traverse all edges in the coding tree and calculate the difference between each pair of sub-coding trees before and after merging. Changes, record each merge operation Each edge of corresponds to the connection between two leaf nodes, and merging these leaf nodes can be regarded as merging two clusters into one cluster; choose The largest change value Merge the edges. For each selected edge, merge the corresponding two sub-coding trees into a new sub-coding tree and update the coding tree ,After each merging operation, the overall structural entropy of the coding tree will be reduced; In each iteration, the structural entropy of the coding tree will gradually decrease, and the number of sub-dialog clusters will be gradually reduced by continuously merging sub-coding trees; When the encoding tree of When it stops decreasing, the iteration stops and the coding tree structure finally stabilizes, thus achieving the optimal division of sub-dialogues. Since the root discourse contains common topic information, when dividing the conversation, the root discourse is included in each sub-conversation cluster to form a sub-conversation set; Encoding Tree Define the following properties: Encoding tree Each node in There is an associated subset ;for The root node in , ; For the encoding tree Any leaf node in , ,in yes A single node in of The child nodes are represented as , mutually exclusive yes Node The height is recorded as ; =0, in yes The parent node of The height is recorded as ; Intuitively, the encoding tree reflects the hierarchical division of the graph, The dimensional encoding tree means ; By encoding tree Determine the graph G Defined as:

[0054] in, is from The sum of the edge weights to all external vertices, , and Represents vertices , and The degree.

[0055] Depending on the order, graph G has multiple different dimensional coding tree, optimal The dimensional encoding tree is defined as having the minimum SE, so the definition of the graph G is The dimension SE is: .

[0056] S7, construct a two-dimensional table, mark entity relationships, extract entities in the discourse, match entities based on sub-dialogue sets, and form sentiment quadruplets, including the following steps: Represent the relationship between any two boundary locations in each utterance as cells in a two-dimensional table; Label the entities in a two-dimensional table, where the vertical axis represents the start of the span and the horizontal axis represents the end of the span. Within the range (where ) can be uniquely labeled at their coordinates in the table, with a total of four different labels: ; The start and end boundaries of an utterance are learned using two dense multilayer perceptrons, represented as,

[0057] in, It is the boundary representation; The boundary relationship table is constructed using the start boundary and end boundary of the speech, and the maximum pooling operation is performed on the boundary relationship table. The expression is:

[0058] in, Entity Relationship Table of For the first time, It is the maximum pooling operation, the purpose is to obtain the span range in the speech All information, is the BiaffineAttention operation, is the parameter matrix; Applying dual affine attention to further model boundary relations in discourse; To further enhance the boundary relations, the boundary relation table is input into a dense layer and then into a separate CNN layer to obtain the final entity relation table, which is expressed as,

[0059] in is a dense layer, is a single CNN layer with a kernel size of 3×3 and a stride of 1; Using a linear layer predict The label for each position in , ,if belong , you can extract the range The span within is taken as the corresponding entity; A cross-entropy-based loss function is introduced to perform supervised learning on each position in each utterance, expressed as:

[0060] in The length is The loss of a word, For location The label, is the indicator function, is the number of utterances in the conversation, is the total number of all utterances; Take the target, aspect and opinion entity spans from each utterance, and group the entities belonging to the same sub-dialogue together based on the divided sub-dialogue set; Pair the target, aspect and opinion entities to form three types of entity pairs: target-aspect pair (TA), target-opinion pair (TO) and aspect-opinion pair (AO); For each pair of TA entity and TO entity, use the tag To determine whether they belong to the same sentiment quadruple; For TO pairs, use the label Determine their inclusion in the same emotion quartet and the emotion polarity of that quartet.

[0061] S8, predict the relationship between entities through the maximum pooling operation, and finally match the sentiment quadruple, including the following steps: For two span spaces in the same sub-dialogue that do not belong to the same entity category and , and perform the maximum pooling operation on their span range, we can get their span representation as, To express It is expressed as:

[0062] in, for The representation of all words in is a learnable parameter matrix that linearly transforms each pair of span representations to obtain the final span representation; Using labeled linear layer To predict Tags ; Use a labeled linear layer to predict the relationship label between each pair of entities based on the final spanned representation; For each sentiment triple, traverse the combination of target, aspect, and opinion in turn; If and only if , and When ; The loss function is introduced to measure whether the model's prediction of entity-to-entity relationships is accurate, and the expression is:

[0063] in, Indicates a subdialog of loss, is the number of traversal pairs; The loss of the quadruple matching module is combined with the losses of other modules to form the final overall loss function, which is expressed as: .

[0064] In summary, the present invention collects multi-round dialogue data and context information, and removes noise data and redundant messages in combination with preprocessing operations, thereby improving the accuracy and standardization of data and reducing model understanding errors; through dynamic word pruning, the operating efficiency of the model is improved, making the reasoning speed faster, while reducing the resource consumption of the model, especially for long text dialogues, it can effectively reduce the interference of redundant information; through multi-level knowledge distillation, the student model can acquire the knowledge of the teacher model with fewer parameters, and the multi-level teacher model can provide a more comprehensive dialogue understanding, significantly improving the model's ability to capture the content of the dialogue; by dividing the dialogue into the smallest sub-dialogue set, the model can be clearer The hierarchical structure of the conversation is understood accurately, ensuring the refined processing of the emotion quadruple extraction. The introduction of the entropy minimization algorithm further optimizes the conversation division, avoids the problem of too fine or too coarse conversation division, and improves the accuracy of conversation emotion extraction. By using two-dimensional tables to label and predict entity relationships, the model can extract entities in the conversation more finely and accurately match the emotion quadruple, effectively reducing the possible errors in the entity relationship extraction process and ensuring the accuracy and completeness of the emotion quadruple. Through the maximum pooling operation, the model can quickly capture the key information between entities, improve the accuracy of relationship prediction, achieve efficient matching of emotion quadruple, and ensure the comprehensive extraction and expression of conversation emotion information.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for extracting emotion quadruple from multi-turn dialogues using large language model reasoning, characterized by: include, Collect multi-round conversation data and context information between customer service and customers, and pre-process the collected data; The collected context information is pruned using a dynamic lemma pruning model; Establish a knowledge distillation model and build a multi-level teacher model based on the reasoning results and multi-round dialogue data; The multi-level teacher model guides the student model to learn the detailed information in the conversation from the multi-level conversation knowledge; Construct a conversation graph based on the conversations processed by LazyLLM and the knowledge distillation model; Use a two-dimensional structural entropy minimization algorithm to divide the conversation into the smallest sub-conversations, and perform a merge operation to achieve the optimal division to form a sub-conversation set; Construct a two-dimensional table, annotate entity relationships, extract entities in discourse, match entities based on sub-dialogue sets, and form sentiment quadruples; Through the maximum pooling operation, the relationship between entities is predicted and finally the sentiment quadruple is matched.

2. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 1, characterized in that: The multi-round conversation information includes customer ID, customer service ID, conversation content, timestamp, and conversation round number; The context information includes historical interaction records, conversation topics, relevant business status and customer information; The preprocessing includes data cleaning and text standardization.

3. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 2, characterized in that: The word pruning of the collected context information by the dynamic word pruning model includes the following steps: In each generation step, use the layer The attention map determines the input word The importance relative to the next token to be predicted is expressed as, in, To pay attention to the number of avatars, is the sequence length, Word In the Words in the head The probability of attention, Word In the Confidence score of the layer; In each transformer layer, the Top-k percentile is selected as the threshold for word pruning based on the importance score of the word. In the Confidence score of the layer Less than the first percentile, the token will be pruned. Once a token is pruned, it will be excluded from the calculations of all subsequent layers. Indicated in The percentage of the layer used to determine the threshold, for example, The value is 30, which means 30% of the words in the layer have confidence scores below this value; Prune different numbers of tokens in each layer; Apply a progressive pruning strategy, retaining more tokens in earlier transformer layers, gradually removing more tokens in later layers, and gradually reducing the number of tokens until the end of the transformer; In the pre-filling phase, the hidden state is calculated for each token, and progressive token pruning is achieved by removing the hidden state of the pruned token. For tokens pruned in subsequent layers, the hidden states are stored in an auxiliary cache; During the decoding phase, each transformer layer retrieves the KV pairs of past tokens from the KV cache. For tokens that do not exist in the KV cache, LazyLLM retrieves the hidden states from the auxiliary cache.

4. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 3, characterized in that: The knowledge distillation model is established to build a multi-level teacher model based on the reasoning results combined with multi-round dialogue data, including the following steps: Teachers with different levels of understanding of the components of the dialogue document were constructed; The input of all teachers consists of each round of speech in the dialogue dataset, expressed as, in, represents the i-th speech in the entire data set, is the length of the speech, Indicates a word in a speech; Word Level Slot Filling Teacher Predict the slot type for each word, word-level slot filling teacher The output of is represented as the predicted slot type for each word; Sentence-level intent detection tutor Predicting the intent of an utterance, and detecting intent at the sentence level For prediction express, in, represents the number of intents in the dataset, For the intention detection task; Dialogue Document-Level Domain Classification Teacher Predicting the domain of the conversation, from document-level domain classification teachers The predicted value is expressed as , in, is the number of domains in the dataset, Categorize tasks for domains; Apply cross entropy loss and fine-tune the pre-trained model for a fixed number of rounds, using the checkpoint of the last round as the teacher model. The specific expression is, in, and is the number of training samples, is the true value, Indicates that the length of the speech in the training set is A collection of speeches, Indicates that the length of the speech in the training set is The first speaker in the collection, Indicates that the length of the speech in the training set is The first in the collection A speech, Indicates that the length of the speech in the training set is The first in the collection A speech, represents the cross entropy loss, represents the teacher model, where ; The teachers at different levels of understanding of dialogue document components include word-level slots, sentence-level intent, and dialogue-level domain knowledge.

5. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 4, characterized in that: The method of guiding the student model to learn detailed information in the conversation from multi-level conversation knowledge through a multi-level teacher model includes the following steps: The KD loss is calculated by comparing the average probability produced by the teacher model ensemble with the probability derived by the student model; Use KD loss to align the predicted probability distribution between the student model and the teacher model; By using the cross-entropy loss, the student model receives direct supervision signals from the ground truth; During the training process, triplets are randomly generated for each batch of data, and the internal relations of the triplets are determined by aggregating the votes of the teacher model combinations; Use TripletMarginLoss to minimize the distance between positive samples and anchor samples, while maximizing the distance between negative samples and anchor samples; The similarity loss is calculated by maximizing the similarity of logits between the student model and the teacher model; Incorporate the teacher model’s predictions as pseudo labels and use the probabilities assigned by the teacher model for each class.

6. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 5, characterized in that: The dialogue processed based on LazyLLM and knowledge distillation model is used to construct a dialogue graph, including the following steps: In the dialogue data, each round of speech is regarded as a sentence, and these sentences are combined to form the input of the model. Assuming that the current dialogue data contains multiple rounds, each sentence and utterance is recorded as , connect these words in series as input, construct the input format, expressed as, in, Represents a serial operation, is a special category identifier, is a separator used to distinguish between words, represents the nth utterance, i.e. the last utterance in the sequence; After encoding by the pre-trained language model, the output form is expressed as, in, It is discourse The encoding symbol is Corresponding to each [CLS] as the embedding of each utterance; The embedding of each utterance is assigned as a node in the conversation graph, and edges are constructed based on the reply relations and semantic similarity in the conversation; For any two utterances and ,if Yes The reply is in the corresponding node and Create a bidirectional edge between them; For any two nodes that are not directly connected and , if there is an intermediate node Make and pass To form a path, and Add an edge between them; The weight of each edge is calculated by the cosine similarity of the node embeddings; Add a self-loop edge to each node, that is, at each node Create an edge with weight 1.

7. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 6, characterized in that: The method of using a two-dimensional structural entropy minimization algorithm to divide a conversation into minimum sub-conversations and performing a merging operation to achieve optimal division to form a sub-conversation set includes the following steps: Initialize the conversation graph as an encoding tree ,Each leaf node represents a vertex in the conversation graph. If two leaf nodes share the same parent node, it means that the two leaf nodes belong to the same child conversation cluster.,Initially, each leaf node is regarded as a separate cluster; For the encoding tree The two sub-coding trees in and subcoding tree Share the same parent node and perform a merge operation Generate a new encoding tree , before and after the merge operation, the encoding tree The SE change calculation formula is: in, and Sub-coding trees and subcoding tree After the merger , , and They are , and Before the merge operation ,like , represents the merged coding tree of Reduce, represents the structural entropy after the merging operation, represents the structural entropy before the merging operation, Represents the coding tree The change in structural entropy is express and The sub-coding tree of express The nodes in Remove from the tree The leaf node after the node; Traverse all edges in the coding tree and calculate the difference between each pair of sub-coding trees before and after merging. Changes, record each merge operation Each edge of corresponds to the connection between two leaf nodes, and merging these leaf nodes merges the two clusters into one cluster; choose The largest change value Merge the edges. For each selected edge, merge the corresponding two sub-coding trees into a new sub-coding tree and update the coding tree ; In each iteration, the structural entropy of the coding tree will gradually decrease, and the number of sub-dialog clusters will be gradually reduced by continuously merging sub-coding trees; When the encoding tree of When it no longer decreases, the iteration stops and the encoding tree structure finally stabilizes; When dividing a conversation, the root utterance is included in each sub-conversation cluster to form a sub-conversation set.

8. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 7, characterized in that: The method of constructing a two-dimensional table, marking entity relationships, extracting entities in discourse, matching entities based on sub-dialogue sets, and forming sentiment quadruplets includes the following steps: Represent the relationship between any two boundary locations in each utterance as cells in a two-dimensional table; Use two dense multilayer perceptrons to learn the start and end boundaries of utterances; The boundary relationship table is constructed using the start boundary and the end boundary of the utterance, and the maximum pooling operation is performed on the boundary relationship table; Applying dual affine attention to further model boundary relations in discourse; Input the boundary relationship table into a dense layer and a CNN layer to obtain the final entity relationship table; Use a linear layer to predict the label for each position; Introducing a cross-entropy based loss function to perform supervised learning for each position in each utterance; Extract all target, aspect, and opinion entity spans from each utterance, and group entities belonging to the same sub-dialogue together based on the partitioned sub-dialogue set; Pair the target, aspect and opinion entities to form three types of entity pairs: target-aspect pair TA, target-opinion pair TO and aspect-opinion pair AO; For target-aspect pair TA entity and target-opinion pair TO entity, use labels to determine whether they belong to the same sentiment quadruple; For target-opinion pairs TO entity pairs, labels are used to determine their inclusion in the same sentiment quartet and the sentiment polarity of the quartet.

9. The method for extracting emotions from multi-turn dialogues using large language model reasoning as claimed in claim 8, characterized in that: The method of predicting the relationship between entities through the maximum pooling operation and finally matching the sentiment quadruple includes the following steps: For two span spaces that do not belong to the same entity category in the same sub-dialogue set, perform maximum pooling operations on their span ranges; Perform a linear transformation on each pair of span representations to obtain the final span representation; Use a labeled linear layer to predict the relationship label between each pair of entities based on the final spanned representation; For each sentiment triple, traverse the combination of target, aspect, and opinion in turn; If and only if all conditions are met simultaneously, the triple forms a successful sentiment quadruple.

Citation Information

Patent Citations

  • Dynamic reasoning dialogue generation method based on graph neural network

    CN113961691A

  • Aspect-level sentiment analysis method based on knowledge distillation framework

    CN116384373A

  • Question constraint and focus entity transfer knowledge graph dialogue type question answering method based on reinforcement learning

    CN116414956A

  • Method for constructing training corpus of dialogue system, training method and computing equipment

    CN116504229A

  • Language model lightweight method combining knowledge distillation and dynamic word pruning

    CN117709395A

Cited By

  • Data generation method and device, storage medium and computing equipment

    CN119250041A

  • Conversation system interaction method and device based on KV cache multiplexing

    CN119556862A

  • Collaborative response method, device and equipment for large and small models, medium and program

    CN120430386A

  • A method, apparatus, device, medium and program for cooperative response of a large model

    CN120430386B

  • Dialogue scene-oriented aspect emotion tetrad prediction method and system

    CN120653756A