A few-shot implicit discourse relation identification method of retrieval-based context learning

Abstract

This invention discloses a few-shot implicit discourse relation recognition method based on retrieval-based context learning, comprising: S1. A pre-trained model containing a cue learning template T is trained using a positive and negative cue contrast learning method to obtain a discourse relation indexer; S2. Based on the discourse relation indexer, the representation vectors of positive and negative MASK lexical units at each level in the training samples are extracted to construct a positive and negative index database; S3. The top K most similar positive and negative training samples are retrieved; S4. The union of the sub-label set of the predicted label in the previous hierarchical structure of the input argument pair and the sub-label set of the truth label of the positive and negative training samples is taken as the candidate label set for the current level; S5. For each level of the label hierarchical structure tree, a cue learning template t is constructed based on the task instruction, positive and negative training samples, candidate label set, and input argument pair, and input to a large language model for inference to obtain the final implicit discourse relation recognition result.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a few-shot implicit discourse relation recognition method based on retrieval-based context learning. Background Technology

[0002] Implicit discourse relation identification aims to identify discourse relationships between a pair of text segments in the absence of explicit conjunctions. Due to the lack of explicit conjunctions, models can only identify specific discourse relationships through entity references and semantic cues between arguments, making implicit discourse relation identification relatively difficult and a challenging task. Furthermore, this task is crucial for text coherence and is considered a necessary step in many downstream tasks involving context, such as question answering, text summarization, and event relation extraction. Meanwhile, the labeling structure of discourse relations follows a hierarchical structure, and implicit conjunctions are inserted during annotation to assist in identification; these conjunctions are considered the finest-grained discourse relation labels in the implicit discourse relation task.

[0003] Early research on implicit discourse relation recognition largely relied on constructing probabilistic statistical models based on human-defined linguistic features. Improved models based on pre-trained language models achieved significant performance gains in discourse relation recognition tasks. With the emergence of larger-scale language models, which unified the input-output formats of various NLP tasks through generative architectures and demonstrated powerful effects, the increasing size and number of parameters of these models have led to very high computational and data requirements for training. Therefore, more and more researchers are attempting to solve downstream tasks through context learning without altering the parameters of the large model. This is also true for implicit discourse relation recognition; some researchers have begun to explore the ability of large models to learn context in the discourse relation domain. However, previous work has primarily focused on the design of prompt learning templates, neglecting the importance of selecting appropriate sample examples in context learning. Appropriate samples can guide the reasoning of large models, which is a crucial step in context learning. Furthermore, due to the hierarchical nature of discourse relation labels, implicit discourse relation recognition has lacked a unified definition of few samples, resulting in insufficient exploration of low-resource scenarios.

[0004] Based on the above analysis, it is crucial to fully explore the ability of large models to retrieve suitable samples in low-resource scenarios by utilizing the hierarchical relationship of tags to guide large models and to fully leverage the capabilities of large models in terms of textual relationships. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a retrieval-based context learning method for identifying implicit discourse relations with few samples. This method utilizes a small model to retrieve suitable samples when reasoning for a large model, fully explores the performance of implicit discourse relation identification tasks in context learning, and injects hierarchical constraint relations into the process.

[0006] The objective of this invention is achieved through the following technical solution: A retrieval-based context learning method for few-shot implicit text relation identification includes the following steps: S1. Text Relationship Indexer Training: Construct a pre-trained model containing a cue learning template T, and train the pre-trained model using a positive and negative cue contrast learning method to obtain a text relationship indexer in order to learn the hierarchical matching relationship between samples; S2. Positive and Negative Index Database: Based on the trained text relation indexer, positive and negative [MASK] lexical representations of each level in the training samples are extracted to construct a positive and negative index database; S3. Sample retrieval: Using the discourse relation indexer, generate representation vectors of positive and negative [MASK] words for input argument pairs at each level, calculate their similarity with vectors in the positive and negative index databases, and retrieve the top K most similar positive and negative training samples. S4. Decoupling hierarchical iteration strategy: Based on the hierarchical structure tree of the label, prediction is performed by layer-by-layer reasoning. The union of the sub-label set of the predicted label in the previous hierarchical structure of the input argument pair and the sub-label set of the true label of the positive and negative training samples retrieved in step S3 is taken as the candidate label set of the current level. S5. Contextual Learning: For each level of the hierarchical structure tree of the tags, based on the task instructions, the positive and negative training samples retrieved in step S3, the candidate tag set generated in step S4, and the input argument pairs, a prompt learning template t is constructed and input into the large language model for inference to obtain the final implicit discourse relation recognition result.

[0007] Furthermore, step S1 specifically includes: The cue learning template T is constructed as "[CLS]x[SEP]The positive representation:[MASK],[MASK],[MASK];The negative representation:[MASK],[MASK],[MASK].[SEP]"; where the number of positive or negative [MASK] tokens is equal to the depth of the label hierarchy structure tree; positive [MASK] tokens are used to retrieve positive samples, and negative [MASK] tokens are used to retrieve negative samples; the cue learning template T is input into the pre-trained model RoBERTa to obtain the hidden layer vectors: (1); in and Let represent the representation vectors of positive and negative [MASK] tokens at the d-th level, where d = 1, 2, 3…D; Let represent the vector of the nth lexical unit; the learning template T is input into the pre-trained model to capture the representation vectors of discourse argument pairs, and the representation vectors of all positive [MASK] lexical units are extracted and denoted as . And input into a multi-level teleprompter. This multi-level teleprompter maps the representation vectors of positive [MASK] tokens to the corresponding level of labeled words, thus obtaining the cross-entropy loss for each input sample. : (2); in This represents the predicted label of layer d, i.e. This represents the i-th real label in layer d. This represents the number of tags at level d, where D represents the level number. Indicates the loss at layer d; The representation vector of the positive [MASK] term is used to retrieve the most relevant positive samples, and the representation vector of the negative [MASK] term is used to retrieve the least relevant negative samples. The formula for the positive contrastive loss function is as follows: (3); in It is the representation vector of the i-th layer positive label in the learning template T. Let represent the representation vector of the i-th positive [MASK] word, and τ be the temperature coefficient of contrastive learning. Includes the representation vector of the negative [MASK] word in the i-th layer, the representation vectors of the positive and negative labels, where exp represents the exponential function, log represents the logarithmic function, and D represents the number of layers; The formula for the negative contrastive learning loss function is shown below: (4); in It is the representation vector of the negative label of the i-th layer in the learning template T. Let represent the representation vector of the i-th negative [MASK] term, and τ be the temperature coefficient of the contrastive learning. , including the representation vector of the positive [MASK] word in the i-th layer, the representation vector of the positive label, and the representation vector of the negative label, where exp represents the exponential function, log represents the logarithmic function, and D represents the number of layers; Finally, the overall training objective function for the document relation indexer is as follows: (5); Where λ1 and λ2 are hyperparameters that control the corresponding loss weights. Represents cross-entropy loss, This indicates positive contrast learning loss. This represents negative contrastive learning loss; after training, a discourse relation indexer is obtained. E .

[0008] Furthermore, step S2 specifically includes: Based on the discourse relation indexer E, the representation vectors of [MASK] words at each level of the training samples are stored separately. and This is used to build a positive index database. and negative index database This method can obtain different vector representations for each layer, making it convenient to perform retrieval and adaptation for each layer separately.

[0009] Furthermore, step S3 specifically includes: For an input argument pair Using the chapter relation indexer E Let be the representation vectors of the positive [MASK] lexical units and the negative [MASK] lexical units generated from the input arguments, denoted as and For each training sample Positive index vectors are stored in the positive and negative index databases. and negative index vector Therefore, for the i-th layer, and The similarity between them is calculated using the following formula: (6); (7); in, and Each refers to a training sample The representation vectors of the i-th positive [MASK] word and the i-th negative [MASK] word are given. These refer to the input argument pairs respectively. The representation vectors of the i-th positive [MASK] word and the i-th negative [MASK] word are obtained from the positive index database through similarity calculation. and negative index database The top K most similar positive and negative training samples are retrieved as sample examples.

[0010] Furthermore, step S4 specifically includes: In the process of marking multi-level implicit discourse relations, all tags form a hierarchical structure graph G with D levels. Several elements are selected from this graph to form a hierarchical structure tree of depth D. For an argument pair y, the tag i at the d-th level structure has a set of related sub-tags at the (d+1)-th level structure. Represented as: (8); Where d+1<=D, and Parent(∙) represents the parent node of a given node. This represents the i-th label of the argument pair in the d-th hierarchical structure. Let represent the j-th label of an argument pair in the (d+1)-th hierarchical structure; divide the label hierarchy into different sub-clusters, each with a common parent node; generate sub-labels layer by layer using an iterative method until leaf labels are obtained; the Large Language Model (LLM) performs iterative inference at each hierarchical structure; the input prompt learning template t of the LLM includes the task instruction, the top K most similar positive and negative training samples, the input argument pair, and the candidate label set for the current hierarchical structure; the LLM predicts the label of the argument pair in the current hierarchical structure based on the candidate label set; during the inference process for the (d+1)-th hierarchical structure of the label, the candidate label set... It is the subset of labels i predicted by LLM in the previous hierarchical structure d. The ground truth labels of the K most similar positive and negative training samples in the previous hierarchical structure d child tag set The union of sets is represented as : (9).

[0011] Preferably, the present invention also provides a retrieval-based context learning device for few-shot implicit discourse relation recognition, comprising: The document relation indexer training unit is used to construct a pre-trained model containing a cue learning template T, and to train the pre-trained model using a positive and negative cue contrast learning method to obtain a document relation indexer in order to learn the hierarchical matching relationship between samples. The positive and negative index database construction unit is used to extract positive and negative [MASK] lexical representations of each level in the training samples based on the trained text relation indexer, and construct positive and negative index databases; The sample retrieval unit is used to generate representation vectors of positive and negative [MASK] words for input argument pairs at each level using the discourse relation indexer, calculate their similarity with vectors in the positive and negative index databases, and retrieve the top K most similar positive training samples and negative training samples. The decoupled hierarchical iterative unit is used for label-based hierarchical structure tree prediction. It adopts a layer-by-layer reasoning approach to make predictions. The union of the sub-label set of the predicted label in the previous hierarchical structure of the input argument pair and the sub-label set of the ground value label of the positive and negative training samples retrieved in step S3 is used as the candidate label set of the current level. The context learning unit is used to construct a cue learning template t for each level of the hierarchical structure tree of the tags, based on task instructions, retrieved positive and negative training samples, generated candidate tag sets, and input argument pairs. This template is then input into the large language model for inference to obtain the final implicit discourse relation recognition result.

[0012] Preferably, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the retrieval-based context learning few-sample implicit discourse relation recognition method.

[0013] Preferably, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the retrieval-based context learning method for identifying few-sample implicit textual relationships.

[0014] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: 1. A Positive and Negative Cue Contrast Learning (PNPCL) approach is introduced to train the discourse relation indexer. A bidirectional cue learning template is used to extract positive and negative [MASK] lexical representations, which are then jointly trained using positive contrastive loss (bringing closer to truth values ​​and away from negative values) and negative contrastive loss (moving away from truth values ​​and closer to false values). This approach yields independent vector representations for each level, achieving finer-grained hierarchical matching rather than using a single, generalized vector to represent all levels of relations. The trained positive representations cluster near the truth labels, while the negative representations are far from them. This allows the discourse relation indexer to accurately retrieve the most helpful "positive samples" (those that best match the input semantics and relations) and "negative samples" (samples that are most easily confused but provide boundary information through contrast) for inference. This addresses the problem that traditional methods (such as using [CLS] or average word vectors) cannot capture subtle semantic differences in multi-level discourse relations, and that the retrieval index cannot effectively distinguish between "most relevant positive examples" and "least relevant negative examples."

[0015] 2. A bidirectional positive and negative example retrieval mechanism is constructed. Based on a pre-trained discourse relation indexer, both a positive vector index database and a negative vector index database are built simultaneously. During inference, the top K positive and negative samples are retrieved and input into the large model, respectively. This contrasting contextual information significantly enhances the large model's ability to discriminate implicit relations, especially in difficult samples lacking explicit connectors. During retrieval, examples with identical labels are filtered out, increasing the diversity of context and preventing the model from overfitting to a single pattern. This addresses the problem of large models lacking boundary constraints in context learning (ICL), which can easily lead to overgeneralization or misjudgment under ambiguous semantics by relying solely on positive examples.

[0016] 3. A decoupled hierarchical iterative strategy (DLIS) is proposed. Label prediction is decomposed into a layer-by-layer process, and the candidate label set of each layer consists of the union of the sub-label set of the predicted labels in the previous layer and the sub-label sets of the ground truth labels of the retrieved positive and negative training samples. By using the pruned label subset as a simplified candidate label set, the hierarchical constraints between labels are maximized, reducing the classification difficulty of large models. Even if the parent node prediction is biased, or the labels of highly similar samples retrieved can be considered as candidates, avoiding the risk of perpetuating errors and maximizing the use of sample information. This solves the problem of confusion that large models are prone to when facing a huge full label set, as well as the problem that traditional hierarchical methods may lead to irreversible errors in subsequent layers or ignore the implicit label biases in the retrieved examples.

[0017] 4. By combining the aforementioned features to improve inference performance with few samples, this approach addresses the issues of expensive data annotation and poor model performance due to hierarchical imbalance in low-resource scenarios during implicit discourse relation recognition tasks. Experimental data shows that, with very few samples, this method outperforms baseline models (such as DiscoPrompt-ChatGPT) at both the first and second levels of the PDTB2.0 and 3.0 datasets; demonstrating that when data volume is extremely limited, assisting large model inference with high-quality retrieval is more advantageous than simply fine-tuning small models. Attached Figure Description

[0018] Figure 1 This diagram illustrates a few-shot implicit text relation recognition model based on retrieval-based context learning provided by the present invention.

[0019] Figure 2 This demonstrates the effect of the global hierarchical bottom-up probability propagation method in this invention on enhancing the identification of a specific instance. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0021] Example 1 This embodiment focuses on the task of recognizing multi-level implicit discourse relationships. Given training samples and a set of chapter relationships with D hierarchical structures. ,in Indicates the first d A set of hierarchical chapter relationship tags; the ultimate goal of this task is to determine the relationship between the layers of text and the text. Predict the sequence of chapter relationships ,in It is the first d Hierarchical prediction labels. This embodiment selects two versions of the standard English text relations dataset PDTB, 2.0 and 3.0, for validation. The pre-trained language model RoBERTa is used for text encoding. The overall model architecture diagram is shown below. Figure 1 As shown in the diagram, the prompt module is also the prompt learning module, and the decoder refers to the text relationship indexer.

[0022] A retrieval-based context learning method for identifying few-shot implicit text relations includes the following steps: S1, Training of the text relation indexer: S101. Positive and Negative Cue Contrast Learning: To better retrieve the most relevant positive samples and the least relevant negative samples for context learning, this embodiment uses dual cue contrast learning to fine-tune the pre-trained model, training a good retriever for context learning. The effect of cue fine-tuning has been specifically introduced in the previous section. To adapt to the requirements of the current task, the cue learning template T is constructed as "[CLS]x[SEP]The positive representation:[MASK],[MASK],[MASK];The negative representation:[MASK],[MASK],[MASK].[SEP]". The number of positive or negative [MASK] tokens is equal to the depth of the label hierarchy structure tree. Positive [MASK] tokens are used to retrieve positive samples, and negative [MASK] tokens are used to retrieve negative samples. Then, T is input into the pre-trained model RoBERTa to obtain the hidden layer vector: (1) in and Let d represent the vector representations of positive and negative [MASK] tokens at the d-th level, where d = 1, 2, 3…D; Let represent the vector of the nth word. Hint: The learning template T is input into the pre-trained model RoBERTa, which can then capture the representation vectors of discourse argument pairs and extract the representation vectors of all positive [MASK] words, denoted as . The representation vector of the positive sample is input into the multi-level teleprompter. This teleprompter maps the representation vectors of positive [MASK] tokens to the corresponding level of tag words, and finally obtains the cross-entropy loss for each input instance. : (2) in This represents the predicted label of layer d, i.e. This represents the i-th real label in layer d. This represents the number of tags at level d, where D represents the level number. This represents the loss at layer d.

[0023] S102. Contrastive Learning: To train index vectors for retrieving positive and negative samples based on cue tuning, the representation vectors of positive and negative [MASK] tokens must satisfy the following: the representation vector of the [MASK] token in the label should be close to the representation vector of its positive label, far from the representation vector of the negative label in the feature space, and also far from the representation vector of the negative [MASK] token. The same requirement applies to the representation vector of the negative [MASK] token. A positive and negative index contrastive learning method is used to capture hierarchical matching relationships of arguments with the same label at corresponding levels. The objective function of contrastive learning consists of positive contrastive learning loss and negative contrastive learning loss. Positive label contrastive learning is performed on the representation vectors of positive [MASK] tokens. The representation of the ground truth label at the corresponding level is used as an anchor point. The representation vectors of different labels and the representation vectors of negative [MASK] tokens at the corresponding level are used as negative samples. This approach brings the positive [MASK] token representation vectors closer to the ground truth label representation vectors and widens the distance between the positive [MASK] token representation vectors and the representation vectors of both negative labels and negative [MASK] tokens. This clusters the positive [MASK] token representation vectors of all instances near the ground truth label representations, while keeping the negative [MASK] token representation vectors further away. This facilitates subsequent retrieval of the most relevant positive samples using the positive [MASK] token representation vectors and the least relevant negative samples using the negative [MASK] token representation vectors. The positive contrastive loss function formula is shown below: (3) in It is the representation vector of the i-th layer positive label in the learning template T. Let represent the representation vector of the i-th positive [MASK] word, and τ be the temperature coefficient of contrastive learning. Includes the representation vector of the negative [MASK] term in the i-th layer, the representation vectors of the positive and negative labels, where exp represents the exponential function, log represents the logarithmic function, and D represents the number of layers.

[0024] Negative contrastive learning is applied to the representation vectors of negative [MASK] vocabularies, and... Conversely, the target positive sample is the representation vector of the negative label of this input argument pair, while the negative sample is the representation vector of the ground truth label and the positive [MASK] word. This is because the representation vector of the negative [MASK] word should be far away from the representation vector of the ground truth label and close to the representation vector of the false label. The formula for the negative contrastive learning loss function is as follows: (4) in It is the representation vector of the negative label of the i-th layer in the learning template T. Let represent the representation vector of the i-th negative [MASK] term, and τ be the temperature coefficient of the contrastive learning. , including the representation vector of the positive [MASK] word in the i-th layer, the representation vector of the positive label, and the representation vector of the negative label, where exp represents the exponential function, log represents the logarithmic function, and D represents the number of layers.

[0025] Finally, the overall training objective function of the passage relation indexer is as follows: (5) Where λ1 and λ2 are hyperparameters that control the corresponding loss weights. Represents cross-entropy loss, This indicates positive contrast learning loss. This represents the negative contrastive learning loss. After training, a discourse relation indexer can be obtained. E .

[0026] S2. Construct a database with positive and negative indexes: After the above steps, a discourse relation indexer E is obtained, which stores the representation vector of each level of [MASK] word in the training samples. and This is used to build a positive index database. and negative index database This is used for subsequent sample retrieval. Compared to previous methods that used [CLS] for sentence representation or averaged all word vector representations, this method is more suitable for multi-level tasks because it can obtain different vector representations for each level, facilitating separate retrieval and adaptation for each level, resulting in finer-grained matching relationships, rather than using a single vector to represent matching relationships across multiple levels. Subsequently, suitable samples need to be retrieved for the input argument pairs.

[0027] S3, Sample Retrieval: With the increasing capabilities of large language models, context learning has emerged as a new paradigm in natural language processing. Large models are capable of making predictions based on context and a few examples. For an input argument pair... Using the trained text relation indexer E Let be the representation vectors of the positive [MASK] lexical units and the negative [MASK] lexical units generated from the input arguments, denoted as and For each training sample Positive index vectors are stored in the index database. and negative index vector Therefore, for the i-th layer, and The similarity between them can be calculated using the following formula: (6) (7) in, and These refer to the input argument pairs respectively. The representation vectors of the i-th positive [MASK] word and the i-th negative [MASK] word are given. Each refers to a training sample The representation vectors of the i-th positive [MASK] word and the i-th negative [MASK] word can be obtained by similarity calculation. and The top K most similar positive and negative samples are retrieved as sample examples. It's worth noting that sample examples with the same label are filtered out to ensure that the labels of the examples are different, thus providing a relatively diverse range of sample examples for the context learning process.

[0028] S4. Decoupling hierarchical iteration strategy: To avoid overwhelming a large model with all hierarchical labels and to retrieve finer-grained sample examples hierarchically, while also integrating label constraints into context learning, a decoupled hierarchical iterative strategy is introduced. This strategy decouples and iteratively utilizes the label structure. Specifically, all labels form a hierarchical graph G with D levels. Several labels are selected from this graph to form a hierarchical structure tree of depth D. For a specific label i at the d-th level of argument pair y, its related sub-labels at the (d+1)-th level are... Represented as: (8) Where d+1<=D, and Parent(∙) represents the parent node of a given node. This represents the i-th label of the argument pair in the d-th hierarchical structure. Let represent the j-th label of the argument pair in the (d+1)-th hierarchical structure; divide the label hierarchy into different sub-clusters, each sub-cluster sharing a common parent node; generate sub-labels layer by layer using an iterative method until leaf labels are obtained; the Large Language Model (LLM) performs iterative inference at each hierarchical structure; the input prompt learning template t of the LLM includes the task instruction, the top K most similar positive and negative training samples, the input argument pair, and the candidate label set for the current hierarchical structure; the LLM predicts the label of the argument pair in the current hierarchical structure based on the candidate label set; during the inference process of the (d+1)-th hierarchical structure, the candidate label set... It is the subset of labels i predicted by LLM in the previous hierarchical structure d. The ground truth labels of the K most similar positive and negative training samples in the previous hierarchical structure d child tag set The union of sets is represented as : (9) Typically, when a Large Language Model (LLM) is used for inference in a classification task, it always takes the complete label set of the dataset as input, and the inference result is derived from this complete label set. In contrast, the method of this invention provides a pruned subset of labels as a streamlined candidate label set, which maximizes the use of sample information and avoids the influence of redundant labels.

[0029] S5, Contextual Learning: Using the discourse relation indexer in S1 and the positive and negative retrieval databases in S2, and after retrieving the desired positive and negative samples using the retrieval method in S3, a cue learning template t for large-scale model inference is constructed. The cue learning template t mainly consists of task instructions, retrieved positive and negative samples, and input argument pairs. The task instructions describe the task, the positive and negative samples are important guidance for the large model in context learning, and the input argument pairs are the argument pairs that need to be predicted in the end. By inputting the constructed cue learning template t into the large model, the inference result of the current level is obtained. Then, the decoupled hierarchical iteration strategy in S4 is used to complete the sample retrieval and cue learning template t construction for each level, and finally, the inference for each level is completed.

[0030] Specifically, to verify the effectiveness of the method of this invention in a real-world system, this embodiment conducts experimental verification on the English datasets PDTB2.0 and 3.0. Specific details are as follows: (1) Dataset: This embodiment verifies the effectiveness of the method based on the English discourse relation corpora PDTB2.0 and PDTB3.0. For dataset segmentation, this embodiment adopts the PDTB-Ji segmentation method, that is, selecting K samples from partitions 2-20 as the training set as the support set, partitions 0-1 as the validation set, and partitions 21-22 as the test set. In this embodiment, path-based few-sample sampling is performed. This invention follows the previous approach of selecting representative connectors at the connector level to form a hierarchical structure tree of labels. The results of the selection for the two datasets are shown in Tables 1 and 2: Table 1. Label set after linker selection on the PDTB2.0 dataset Table 2. Label set after linker selection on the PDTB3.0 dataset (2) Few-sample sampling algorithm based on path sampling: In tasks such as text classification, image segmentation, and named entity recognition, the few-shot problem has been explored in depth. However, few works have focused on the task of identifying multi-level implicit discourse relations with few shots. This is because sampling at a single level of discourse relations is easy; a strategy of directly selecting K samples for each class and adding them to the training set is sufficient. However, this sampling method is difficult to apply directly to all levels because the input samples contain a different number of labels for each level. For example, selecting one sample for each label in the second level may result in some labels having a common parent label, leading to some labels in the first level not being sampled with a single number of samples. Therefore, the number of samples for multi-level discourse relations cannot be defined as simply as for a single level. To address this, this invention selects representative labels from the connector word level to form a hierarchical structure tree (Tree). The results of the selection for the two datasets are shown in Tables 1 and 2. Based on the hierarchical structure tree, the few-shot setting is defined through path sampling. That is, K non-repeating label paths are randomly sampled to form the training set or support set Sp. This eliminates the focus on the number of samples for a certain label at a single level, but rather uses the label path as the distinguishing factor, ensuring that each label path has an accurate K-shot sample example. (3) Model details: This embodiment was developed using the PyTorch and Huggingface libraries. To train the discourse relation indexer, a pre-trained RoBERTa was used as the Transformer encoder, and K training samples of paths from the root node to the leaf node were randomly sampled from the training set. The learning rate was 5e-5, the batch size was 1, the optimizer was AdamW, and updates were performed over 10 iterations on a single RTX 3090 GPU. For context learning, experiments were conducted using the OpenAI GPT-4o-mini state-of-the-art GPT model. The experiments were implemented using APIs provided by OpenAI.

[0031] (3) Evaluation indicators: Evaluation metrics include accuracy (Accuracy, Acc) and macro-F1 score (Macro-F1). If a set of argument pairs contains multiple text relations, the model is considered correct as long as it hits one of the correct classes.

[0032] (4) Comparison system: To verify the effectiveness of the method of this invention, recent representative context learning methods were selected for comparative analysis. Due to existing work on few-shot implicit discourse relation recognition, the selected context learning methods were randomly assigned few-shot examples for reproduction, and their performance on this task was tested.

[0033] 1) RoBERTa-FT: [1] Improved BERT by removing the NSP task, and completed the classification experiment by training and fine-tuning each level with a small amount of sample data.

[0034] 2)DiscoPrompt-chatGPT:[2] First, the zero-sample performance of the large language model on IDRR was evaluated. In this embodiment, its method was used to conduct a comparative experiment by randomly sampling a small number of samples.

[0035] 3) Two-step DCinsertio:[3] Based on DiscoPrompt-chatGPT, complex prompting engineering is used to further evaluate the ability of LLM on IDRR. This embodiment uses its prompting template and randomly gives a small number of samples for comparative experiments.

[0036] The experimental results of the baseline model and the model proposed in this invention at various levels are shown in Tables 3 and 4 (- indicates that the comparison model has a gap in this index): Table 3 shows the average experimental results of three random number seeds on PDTB2.0 and PDTB3.0. The bolded values ​​are the highest values, and the underlined values ​​are the second highest values. Table 3 The following observations can be made from Table 3 above: 1) The model of this invention achieved the highest results in most cases at the first and second levels of PDTB2.0 and PDTB3.0 with a small number of samples; 2) As the number of training samples increases, it can be seen that the performance metrics of the Roberta-ft model at both levels are improving on both datasets. This shows that for small models, the amount of training data has a significant impact on performance improvement. Although using all the data to train and fine-tune the small model performs better, when the amount of data is extremely small, context learning of the large model shows its advantage. Therefore, the method of this invention is more suitable for low-resource scenarios with limited data.

[0037] 3) Additionally, as can be seen from the two tables, the performance of DiscoPrompt-chatGPT and Two-stepDCinsertion did not improve with the increase of the number of training samples. This is because even if these two methods have a certain number of training samples, they lack a way to search for the best context example. Their context examples are randomly obtained, so their performance is volatile. 4) Finally, on both datasets, as the number of training samples increases, the performance metrics of the method of this invention continuously improve at both levels and outperform other methods. This is mainly because as the training data increases, the training effect of the index model is also better, and it can find better samples for the large model in context learning, thus improving the performance metrics.

[0038] To explore the contribution of each part of the model to the overall performance improvement, this invention decomposes the various innovative points of the model, designs multiple ablation models, and verifies the performance of each part on PDTB2.0, as shown in Table 4. The main part of this invention involves decoupling the training objectives such as the Decoupled Layered Iterative Strategy (DLIS), the training of the discourse relation encoder (i.e., Positive and Negative Prompt Contrastive Learning (PNPCL)), and In-Context Learning demonstration retrieval (ICLDR). Each model is gradually removed to test its effectiveness; deleting any module impairs performance at all levels, thus verifying the effectiveness of each module. Table 4 shows that: Table 4. Ablation experimental results on PDTB2.0. w / o"" means "without"; DLIS refers to the decoupling hierarchical iterative strategy; PNPCL refers to positive and negative cue contrast learning; ICLDR refers to sample retrieval based on context learning.

[0039] Table 4 1) After removing DLIS, the model directly infers and predicts label sequences at each level, while using the average similarity across all levels for sample retrieval. In most few-sample settings, the model's performance degrades, suggesting that finer-grained sample retrieval for each layer of LLM inference can provide more suitable sample examples.

[0040] 2) After removing PNPCL, this invention directly fine-tunes the Roberta model to replace it, and performs sample retrieval with the same similarity. It can be seen that the effect is not as effective as using PNPCL for positive and negative sample retrieval. This shows that PNPCL can learn more fine-grained matching relationships for each layer.

[0041] 3) After removing ICLDR, LLM made predictions without samples at each level, which is equivalent to zero-sample evaluation, resulting in a performance degradation. This also shows the importance of appropriate samples in the context learning (ICL) process.

[0042] 4) When both DLIS+PNPCL and DLIS+ICLDR are removed, the model performs worse than if either module is removed alone, which confirms that the two methods can work together and are effective when used simultaneously.

[0043] Preferably, Figure 2 An example is provided to analyze the inference process of LLM at the second level. The complete prompt template is as follows: Figure 2As shown, the model mainly consists of instructions, retrieved positive and negative samples, and input argument pairs. The instructions explicitly define the task's objective. The retrieved samples are the top K most similar positive and negative training samples that match the input argument pairs in the positive and negative index database; here, the number of samples is 1. The previous layer label refers to the prediction result of the previous layer in the hierarchical iterative inference process. Based on the hierarchical dependency of labels, the candidate label set of the current hierarchical structure is formed by the union of the sub-label set of the predicted labels in the previous hierarchical structure and the sub-label set of the ground truth labels of the retrieved positive and negative training samples. Originally, there were 11 types of second-level labels in PDTB2. After pruning based on label dependencies, there are only 6 types in the candidate label set. At the same time, because the first-level label set of hierarchical iterative inference does not need to be displayed together, a more concise label set is provided to reduce label interference. The model selects the final output from these candidate labels. It is worth mentioning that before each layer of inference, the corresponding positive and negative [MASK] lexical representations are used for retrieval. This allows for a more granular retrieval of samples suitable for the current level, which is more beneficial to the model's inference.

[0044] Example 2 Based on the same inventive concept, this application also provides a retrieval-based context learning few-shot implicit discourse relation recognition device, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of the retrieval-based context learning few-shot implicit discourse relation recognition device is similar to that of the few-shot implicit discourse relation recognition method, the implementation of this device can refer to the aforementioned method implementation, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0045] Embodiments of the present invention provide a specific implementation of a recognition device capable of realizing a few-sample implicit text relation recognition method based on retrieval-based context learning, specifically including the following: The document relation indexer training unit is used to construct a pre-trained model containing a cue learning template T, and to train the pre-trained model using a positive and negative cue contrast learning method to obtain a document relation indexer in order to learn the hierarchical matching relationship between samples. The positive and negative index database construction unit is used to extract positive and negative [MASK] lexical representations of each level in the training samples based on the trained text relation indexer, and construct positive and negative index databases; The sample retrieval unit is used to generate representation vectors of positive and negative [MASK] words for input argument pairs at each level using the discourse relation indexer, calculate their similarity with vectors in the positive and negative index databases, and retrieve the top K most similar positive training samples and negative training samples. The decoupled hierarchical iterative unit is used for label-based hierarchical structure tree prediction. It adopts a layer-by-layer reasoning approach to make predictions. The union of the sub-label set of the predicted label in the previous hierarchical structure of the input argument pair and the sub-label set of the ground value label of the positive and negative training samples retrieved in step S3 is used as the candidate label set of the current level. The context learning unit is used to construct a cue learning template t for each level of the hierarchical structure tree of the tags, based on task instructions, retrieved positive and negative training samples, generated candidate tag sets, and input argument pairs. This template is then input into the large language model for inference to obtain the final implicit discourse relation recognition result.

[0046] Preferably, embodiments of this application also provide a specific implementation of an electronic device capable of implementing all steps of the few-shot implicit discourse relation recognition method using retrieval-based context learning in the above embodiments. The electronic device specifically includes the following: Processor, memory, communications interface, and bus; The processor, memory, and communication interface communicate with each other via a bus; the communication interface is used to realize information transmission between server-side devices, metering devices, and user-side devices.

[0047] The processor is used to call the computer program in memory. When the processor executes the computer program, it implements all the steps in the retrieval-based context learning few-sample implicit discourse relation recognition method in the above embodiments.

[0048] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the retrieval-based context learning few-shot implicit discourse relation recognition method in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the retrieval-based context learning few-shot implicit discourse relation recognition method in the above embodiments.

[0049] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0050] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0051] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed in the order shown in the embodiments or drawings or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0052] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0054] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0055] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

[0056] References: [1]LiuY,OttM,GoyalN,etal.RoBERTa:ARobustlyOptimizedBERTPretrainingApproach.ArXiv2019:1907.11692. [2]ChunkitChan, [3]YungF,AhmadM,ScholmanM,etal.PromptingImplicitDiscourseRelationAnnotation[C].InACL2024:150–165.

Claims

1. A method for identifying few-shot implicit text relations based on retrieval-based context learning, characterized in that, Includes the following steps: S1. Text Relationship Indexer Training: Construct a pre-trained model containing a cue learning template T, and train the pre-trained model using a positive and negative cue contrast learning method to obtain a text relationship indexer in order to learn the hierarchical matching relationship between samples; S2. Positive and Negative Index Database: Based on the trained text relation indexer, positive and negative [MASK] lexical representations of each level in the training samples are extracted to construct a positive and negative index database; S3. Sample retrieval: Using the discourse relation indexer, generate representation vectors of positive and negative [MASK] words for input argument pairs at each level, calculate their similarity with vectors in the positive and negative index databases, and retrieve the top K most similar positive and negative training samples. S4. Decoupling hierarchical iteration strategy: Based on the hierarchical structure tree of the label, prediction is performed by layer-by-layer reasoning. The union of the sub-label set of the predicted label in the previous hierarchical structure of the input argument pair and the sub-label set of the true label of the positive and negative training samples retrieved in step S3 is taken as the candidate label set of the current level. S5. Contextual Learning: For each level of the hierarchical structure tree of the tags, based on the task instructions, the positive and negative training samples retrieved in step S3, the candidate tag set generated in step S4, and the input argument pairs, a prompt learning template t is constructed and input into the large language model for inference to obtain the final implicit discourse relation recognition result.

2. The few-sample implicit discourse relation identification method according to claim 1, characterized in that, Step S1 specifically includes: The cue learning template T is constructed as "[CLS]x[SEP]The positive representation:[MASK],[MASK],[MASK];The negative representation:[MASK],[MASK],[MASK].[SEP]"; where the number of positive or negative [MASK] tokens is equal to the depth of the label hierarchy structure tree; positive [MASK] tokens are used to retrieve positive samples, and negative [MASK] tokens are used to retrieve negative samples; the cue learning template T is input into the pre-trained model RoBERTa to obtain the hidden layer vectors: （1）； in and Let represent the representation vectors of positive and negative [MASK] tokens at the d-th layer, where d = 1, 2, 3…D; Let represent the vector of the nth lexical unit; the learning template T is input into the pre-trained model to capture the representation vectors of discourse argument pairs, and the representation vectors of all positive [MASK] lexical units are extracted and denoted as . And input into a multi-level teleprompter. This multi-level teleprompter maps the representation vectors of positive [MASK] tokens to the corresponding level of labeled words, thus obtaining the cross-entropy loss for each input sample. : （2）； in This represents the predicted label of layer d, i.e. This represents the i-th real label in layer d. This represents the number of tags at level d, where D represents the level number. Indicates the loss at layer d; The representation vector of the positive [MASK] term is used to retrieve the most relevant positive samples, and the representation vector of the negative [MASK] term is used to retrieve the least relevant negative samples. The formula for the positive contrastive loss function is as follows: （3）； in It is the representation vector of the i-th layer positive label in the learning template T. Let represent the representation vector of the i-th positive [MASK] word, and τ be the temperature coefficient of contrastive learning. Includes the representation vector of the negative [MASK] word in the i-th layer, the representation vectors of the positive and negative labels, where exp represents the exponential function, log represents the logarithmic function, and D represents the number of layers; The formula for the negative contrastive learning loss function is shown below: （4）； in It is the representation vector of the negative label of the i-th layer in the learning template T. Let represent the representation vector of the i-th negative [MASK] term, and τ be the temperature coefficient of contrastive learning. , including the representation vector of the positive [MASK] word in the i-th layer, the representation vector of the positive label, and the representation vector of the negative label, where exp represents the exponential function, log represents the logarithmic function, and D represents the number of layers; Finally, the overall training objective function for the document relation indexer is as follows: （5）； Where λ1 and λ2 are hyperparameters that control the corresponding loss weights. Represents cross-entropy loss, This indicates positive contrast learning loss. This represents negative contrastive learning loss; after training, a discourse relation indexer is obtained. E .

3. The method for identifying few-shot implicit discourse relations based on retrieval-based context learning according to claim 1, characterized in that, Step S2 specifically includes: Based on the discourse relation indexer E, the representation vectors of [MASK] words at each level of the training samples are stored separately. and This is used to build a positive index database. and negative index database This method can obtain different vector representations for each layer, making it convenient to perform retrieval and adaptation for each layer separately.

4. The method for identifying few-shot implicit discourse relations based on retrieval-based context learning according to claim 3, characterized in that, Step S3 specifically includes: For an input argument pair Using the chapter relation indexer E Let be the representation vectors of the positive [MASK] lexical units and the negative [MASK] lexical units generated from the input arguments, denoted as and For each training sample Positive index vectors are stored in the positive and negative index databases. and negative index vector Therefore, for the i-th layer, and The similarity between them is calculated using the following formula: （6）； （7）； in, and Each refers to a training sample The representation vectors of the i-th positive [MASK] word and the i-th negative [MASK] word are given. These refer to the input argument pairs respectively. The representation vectors of the i-th positive [MASK] word and the i-th negative [MASK] word are obtained from the positive index database through similarity calculation. and negative index database The top K most similar positive and negative training samples are retrieved as sample examples.

5. The method for identifying few-shot implicit discourse relations based on retrieval-based context learning according to claim 1, characterized in that, Step S4 specifically includes: In the process of marking multi-level implicit discourse relations, all tags form a hierarchical structure graph G with D levels. Several elements are selected from this graph to form a hierarchical structure tree of depth D. For an argument pair y, the tag i at the d-th level structure has a set of related sub-tags at the (d+1)-th level structure. Represented as: （8）； Where d+1<=D, and Parent(∙) represents the parent node of a given node. This represents the i-th label of the argument pair in the d-th hierarchical structure. Let represent the j-th label of an argument pair in the (d+1)-th hierarchical structure; divide the label hierarchy into different sub-clusters, each with a common parent node; generate sub-labels layer by layer using an iterative method until leaf labels are obtained; the Large Language Model (LLM) performs iterative inference at each hierarchical structure; the input prompt learning template t of the LLM includes the task instruction, the top K most similar positive and negative training samples, the input argument pair, and the candidate label set for the current hierarchical structure; the LLM predicts the label of the argument pair in the current hierarchical structure based on the candidate label set; during the inference process for the (d+1)-th hierarchical structure of the label, the candidate label set... It is the subset of labels i predicted by LLM in the previous hierarchical structure d. The ground truth labels of the K most similar positive and negative training samples in the previous hierarchical structure d child tag set The union of sets is represented as : （9）。 6. A retrieval-based context learning device for identifying few-shot implicit text relationships, characterized in that, include: The document relation indexer training unit is used to construct a pre-trained model containing a cue learning template T, and to train the pre-trained model using a positive and negative cue contrast learning method to obtain a document relation indexer in order to learn the hierarchical matching relationship between samples. The positive and negative index database construction unit is used to extract positive and negative [MASK] lexical representations of each level in the training samples based on the trained text relation indexer, and construct positive and negative index databases; The sample retrieval unit is used to generate representation vectors of positive and negative [MASK] words for input argument pairs at each level using the discourse relation indexer, calculate their similarity with vectors in the positive and negative index databases, and retrieve the top K most similar positive training samples and negative training samples. The decoupled hierarchical iterative unit is used for label-based hierarchical structure tree prediction. It adopts a layer-by-layer reasoning approach to make predictions. The union of the sub-label set of the predicted label in the previous hierarchical structure of the input argument pair and the sub-label set of the ground value label of the positive and negative training samples retrieved in step S3 is used as the candidate label set of the current level. The context learning unit is used to construct a cue learning template t for each level of the hierarchical structure tree of the tags, based on task instructions, retrieved positive and negative training samples, generated candidate tag sets, and input argument pairs. This template is then input into the large language model for inference to obtain the final implicit discourse relation recognition result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the few-sample implicit discourse relation recognition method of any one of claims 1 to 5 using retrieval-based context learning.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the few-sample implicit discourse relation identification method of retrieval context learning as described in any one of claims 1 to 5.

Images