Implicit discourse relation identification method for global and local level prompt fine tuning

Abstract

The invention discloses an implicit chapter relation identification method for global and local level prompt fine tuning, which comprises the following steps: coding a prompt template constructed by an argument pair, capturing the instance representation of the argument pair, and designing a multi-level classifier for classification; the label probability of the current level is obtained through the child node probability in the next layer and the weight of the edge according to the global label level graph, and then the pre-training language model and parameters in the learnable classifier are updated through loss, so that the label probability distribution of the current level can be obtained through the label probability distribution in the lower-layer classifier; realizing the injection of global hierarchical knowledge; and the comparison loss is controlled through the number of shared nodes, so that the semantic representation distance is changed by the argument, and local hierarchical labels are utilized. And finally, setting different weight coefficients for loss functions of different methods to carry out joint learning, and updating model parameter information to improve system identification performance.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an implicit chapter relationship recognition method with global and local level prompt fine-tuning. Background Art

[0002] Implicit discourse relation identification aims to identify discourse relations between a pair of text segments in the absence of explicit connectives. Due to the lack of explicit connectives, the model can only identify specific discourse relations through entity references, semantic clues, etc. between arguments, making implicit discourse relation identification relatively difficult and a challenging task. In addition, this task is crucial for text coherence and is considered a necessary step for many downstream tasks involving context, such as question answering, text summarization, and event relation extraction. At the same time, the label structure of discourse relations follows a hierarchical structure. In order to facilitate identification during annotation, implicit connectives are inserted to assist the criteria. The connectives are also considered to be the most fine-grained discourse relation labels in implicit discourse relation tasks.

[0003] Most of the initial studies on implicit discourse relation recognition were based on the construction of probabilistic statistical models based on relevant linguistic features formulated by humans; with the widespread application of pre-trained language models, implicit discourse relation recognition has made considerable progress. Under the pre-training and fine-tuning paradigm, researchers encode the representation of parameter pairs by designing complex neural networks for relation classification. Although these task-specific neural networks can effectively learn the contextual semantics of a parameter, they introduce some additional parameters and rely on a large amount of data for training. In addition, some studies have shown that the task objectives are often inconsistent with the objectives of pre-trained language models, which limits the utilization of the knowledge of pre-trained models. Inspired by prompt learning, some studies have reformulated the implicit discourse relation recognition task as a fill-in-the-blank problem to bridge the gap between masked language models and downstream tasks. Despite the excellent performance, the inherent discourse label hierarchy is ignored.

[0004] To solve this problem, the latest studies have tried to utilize hierarchical information by converting hierarchical structures into paths and adding them as prompt template prefixes; however, they select a few special connection nodes to form path prompt templates, which lacks the completeness of hierarchical label modeling. Other studies have proposed a hierarchical label refinement method for prompt learning, which deeply integrates hierarchical guidance into prompt tuning.

[0005] However, existing methods still have two limitations. 1) Although they exploit the static dependencies between labels (global hierarchy), the hierarchical relationship modeling is independent of the classifier, resulting in the inability to effectively utilize the probability distribution information of the classifier output. 2) Under the pre-training and fine-tuning paradigm, the dynamic label structure information (local hierarchy) corresponding to each sample shows its importance in text classification tasks. However, the hierarchy-aware cue fine-tuning method only considers the entire global label hierarchy and ignores the utilization of local labels.

[0006] Based on the above analysis, it is very important to explore how to utilize both global and local hierarchical relationships and fully utilize hierarchical knowledge. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide an implicit chapter relationship recognition method that can utilize both global and local hierarchical structures, thereby making full use of hierarchical knowledge.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] A method for implicit chapter relationship recognition with global and local level prompt fine-tuning includes the following steps:

[0010] (1) Based on the distribution characteristics that implicit discourse relation connectives are mostly distributed at the connection position of two argument sentences, a prompt learning template is constructed for all argument pairs in a certain data set, and the prompt learning template is input into a pre-trained language model to obtain the semantic interaction information of argument pairs. After capturing the semantic interaction information of argument pairs, the vector corresponding to the position of the mask word between the argument pairs is obtained as the semantic vector representation of the argument pair; learnable classifiers of different levels are designed, and the obtained semantic vector representation of the argument pair is mapped into the probability distribution of discourse relation categories of different levels through learnable classifiers of different levels;

[0011] (2) Define the dependency relationship of all labels in the data set as the global level, and construct a global bottom-up probability propagation module. The label probability of the current level is obtained through the node probability of the next level of the learnable classifier and the global level, and then the parameters in the pre-trained language model and the learnable classifier are updated through the cross entropy calculation loss, so as to obtain the label probability distribution of the current level through the label probability distribution in the lower level learnable classifier, thereby generating global level constraint relationships in learnable classifiers at different levels and realizing the injection of global level knowledge;

[0012] (3) Define the argument pair labels as local levels and construct a local level driven contrastive learning module. The number of shared nodes in the local level between different argument pairs is different. The more shared nodes there are, the closer the semantic vector representations of the argument pairs are. Conversely, the farther they are, the more the contrastive loss between different argument pairs is controlled by the number of shared nodes. This changes the distance between the semantic vector representations of different argument pairs and achieves the utilization of the local level.

[0013] (4) The cross entropy loss function and contrast loss function are used to calculate the loss values ​​of different modules in steps (2) and (3), and then different weight coefficients are set for joint learning to update the parameter information of the pre-trained language model to improve the recognition performance.

[0014] Furthermore, step (1) specifically includes:

[0015] (101) Obtaining semantic vector representation of argument pairs based on a pre-trained language model;

[0016] Let the two arguments be Arg1 and Arg2, and construct a prompt learning template T = [CLS] + Arg1 + [MASK] + Arg2 + [SEP] as the input of the pre-trained language model. After the pre-trained language model PLM, the vectors h of all words in the argument pair are obtained. 1:n , as shown in formula (1), and extract the vector h corresponding to the position of [MASK] [MASK] Used for argument pair semantic vector representation h m , as shown in formula (2):

[0017] h 1:n =PLM(T) (1)

[0018] h m =h [MASK] (2)

[0019] Among them, [CLS] and [SEP] indicate the first character and separator of the argument pair respectively, [MASK] is the mask word, PLM refers to the pre-trained language model, T is the prompt learning template, and h 1:n Represents the vector of all words in the argument pair, n is the word length of the input prompt learning template;

[0020] (102) Design different levels of learnable classifiers and create a virtual learnable vector W for each learnable classifier d , where d represents the dth layer, and after initializing with the label vector of the corresponding layer, we get a list V of learnable classifiers at different levels, as shown in formula (3):

[0021] V={V1,...V d ,...,V D} (3)

[0022] Where V d represents the learnable classifier at the dth level; D is the number of levels of learnable classifiers; obtains the semantic vector representation of the argument pair h m After that, they are sent to different levels of learnable classifiers and the probability distribution value of each layer label is calculated by the softmax function.

[0023] Furthermore, step (2) specifically includes:

[0024] Each layer in the learnable classifier is classified independently without hierarchical constraints. The probability distribution of the label of the next layer in the global hierarchy is used to obtain the probability distribution of the label of the current layer in the global hierarchy. According to the global hierarchy, the i-th specific label of a d-th layer is Its child node set at layer d+1 From formula (4), we can get:

[0025]

[0026] Among them, Parent(*) represents the parent node of the node corresponding to *; in the bottom-up probability propagation process, several learnable weight units are used to balance the influence of several factors; therefore, for the argument pair, the i-th label of the d-th layer The bottom-up propagation probability of Through formula (5), we get:

[0027]

[0028] is the weight coefficient of the jth edge corresponding to the i-th label in the d-th layer, P(y d+1 )express Child node y at layer d+1 d+1 The corresponding probability value.

[0029] Furthermore, step (3) specifically includes:

[0030] For a batch of training sets B = {(X n ,Y n )}, where X n Indicates input, Y n represents the corresponding label set, n represents the batch size; several training sets B are encoded with different dropouts through the pre-trained language model to obtain a replica set B+, so as to obtain a total of 2N sets of hidden vectors Z = {(X n ,Y n )}, and then calculate the score l of the local label shared between the i-th and j-th argument pair by formula (6) ij :

[0031]

[0032] in Indicates that when the i-th label of the d-th layer With the jth label The value is 1 when they are equal, otherwise it is 0; d represents the coefficient of layer d, and D represents the number of layers of the learnable classifier; thus, the number of shared nodes between pairs of arguments with different inputs is obtained, and then l is converted into ij Normalize to get the similarity score α ij :

[0033]

[0034] Finally, for each input argument pair (i, j), the local level contrast loss is defined as:

[0035]

[0036] where h i and h j refers to the semantic vector of the argument pair, sim represents cosine similarity, τ represents the temperature coefficient of contrastive learning; exp represents exponential function, log represents logarithmic function. Under contrastive loss, for the argument pair (i, j), the more sub-labels they share, the higher their similarity score α ij The larger it is, the more loss terms the argument pairs have. The value of h i and h j The distance between them is closer; on the contrary, the distance is farther; therefore, the contrast loss uses the similarity score α ij Guide the distance between the semantic vector representations of different argument pairs.

[0037] Furthermore, step (4) specifically includes:

[0038] The loss function of each module is calculated separately and joint learning is performed. The loss of the learnable classifier is calculated using the cross entropy loss, which is obtained by formula (9):

[0039]

[0040] in represents the cross entropy loss value of the learnable classifier at layer d, and D represents the number of learnable separator layers;

[0041] The loss of the bottom-up probability propagation module at the global level also uses the cross entropy loss and is calculated using formula (10):

[0042]

[0043] in represents the cross entropy loss obtained by the global bottom-up probability propagation module at the dth layer, and D represents the number of learnable separator layers;

[0044] The loss of the local layer-driven contrastive learning module is calculated using formula (11):

[0045]

[0046] in represents the local hierarchical contrast loss of each set of input argument pairs (i, j), B represents a batch of training sets, B+ represents a replica set obtained by performing different dropout encodings on a batch of training sets B through a pre-trained language model, and N represents the size of each batch of training sets B;

[0047] Finally, the above three losses are jointly learned, and the overall training target is calculated by formula (12):

[0048]

[0049] in represents the learnable classifier loss, represents the global level bottom-up probability propagation module loss, represents the contrastive learning module loss driven by the local hierarchy, and λ1 and λ2 represent hyperparameters.

[0050] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the implicit chapter relationship recognition method with global and local level prompt fine-tuning are implemented.

[0051] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the implicit chapter relationship recognition method with global and local level prompt fine-tuning.

[0052] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0053] 1. The global and local hierarchical structure knowledge is integrated into the prompt fine-tuning at the same time, and the probability information output by the classifier is more fully utilized for hierarchical modeling. The hierarchical knowledge is more comprehensively utilized, which improves the recognition performance on this task.

[0054] 2. The proposed implicit chapter relationship recognition method with global and local hierarchical prompt fine-tuning uses the output of a multi-level classifier to model hierarchical relationships and injects hierarchical constraint relationships, narrowing the task gap between the pre-training task in the pre-trained language model and the downstream multi-level implicit chapter relationship recognition task. It makes fuller use of the prior knowledge and capabilities of the pre-trained language model, resulting in improved accuracy and F1 value on this task.

[0055] 3. The number of shared nodes at the local level is used to guide the distance between the semantic vector representations of different argument pairs, which improves the representation distribution of masked words. The local level knowledge is used in prompt fine-tuning to further improve the recognition performance of the multi-level classifier.

[0056] 4. The results were verified on two versions of English text relationship standard datasets, and the experimental results verified the effectiveness of the present invention. Comparisons were made with other classification models of the same category, proving that the comprehensive results of the present invention on different levels of evaluation indicators are superior to similar classification models. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the implicit chapter relationship recognition model with global and local level prompt fine-tuning provided by the present invention is shown.

[0058] Figure 2 It shows the effect of the global level bottom-up probability propagation method in the present invention on the recognition enhancement of a certain instance.

[0059] Figure 3a and Figure 3b The distribution diagram of [MASK] embedding in each example under the local hierarchical contrastive learning method of the present invention is shown. Figure 3a [mask] representation without local level contrast learning; Figure 3b To add [mask] representation for local level contrastive learning. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0061] The present invention focuses on the multi-level implicit text relation recognition task. Given an input argument pair instance x = (Arg1, Arg2) and a text relation set with D levels, the ultimate goal of this task is to predict the text relation sequence based on x. This embodiment selects two versions 2.0 and 3.0 of the standard English text relation dataset PDTB for verification, and uses the pre-trained language model RoBERTa for text encoding. The overall model architecture diagram is shown in the figure below. Figure 1 shown.

[0062] An implicit chapter relationship recognition method integrating parameter validation and relationship label hierarchical semantic priors specifically comprises the following steps:

[0063] (1) Obtaining semantic vector representation of argument pairs based on pre-trained language model

[0064] Let the two arguments be Arg1 and Arg2, and construct the prompt learning template T = [CLS] + Arg1 + [MASK] + Arg2 + [SEP] as the input of the pre-trained language model. After the pre-trained language model (PLM), the vector h1 of all words in the argument pair is obtained. :n , as shown in formula (1), and extract the vector h corresponding to the position of [MASK] [MASK] Used for argument pair semantic vector representation h m , as shown in formula (2):

[0065] h 1:n =PLM(T) (1)

[0066] h m =h [MASK] (2)

[0067] Among them, [CLS] and [SEP] indicate the first character and separator of the argument pair respectively, [MASK] is the mask word, PLM refers to the pre-trained language model, T is the prompt learning template, and h 1:n Represents the vector of all words in the argument pair, and n is the word length of the input prompt learning template.

[0068] Further design different levels of learnable classifiers and create a virtual learnable vector W for each learnable classifier d , where d represents the dth layer, and after initializing with the label vector of the corresponding layer, we get a list V of learnable classifiers at different levels, as shown in formula (3):

[0069] V={V1,...V d ,...,V D} (3)

[0070] Where V d represents the learnable classifier at the dth level; D is the number of levels of learnable classifiers; obtains the semantic vector representation of the argument pair h m After that, they are sent to different levels of learnable classifiers and then calculated by the softmax function to obtain the probability distribution value of each layer label. Therefore, the learnable classifier loss uses the cross entropy loss and is calculated by formula (4):

[0071]

[0072] in represents the cross entropy loss value of the learnable classifier at the dth layer, and D represents the number of learnable separator layers.

[0073] (2) Global bottom-up probability propagation method

[0074] In the above method, each layer in the learnable classifier is classified independently without integrating hierarchical constraints. According to the global hierarchy, the probability distribution of the label of the current layer can be obtained by using the probability distribution of the label of the next layer; according to the global hierarchy, for a specific label of the ith layer of d Its child node set at layer d+1 From formula (5), we can get:

[0075]

[0076] Among them, Parent(*) represents the parent node of the node corresponding to *; in the bottom-up probability propagation process, several learnable weight units are used to balance the influence of several factors; therefore, for the argument pair, the i-th label of the d-th layer The bottom-up propagation probability of Through formula (6), we get:

[0077]

[0078] Where x is an argument pair, is the weight coefficient of the jth edge corresponding to the i-th label in the d-th layer, P(y d+1 )express Child node y at layer d+1 d+1 The corresponding probability value. Therefore, the loss of the bottom-up probability propagation module at the global level is calculated by formula (7):

[0079]

[0080] in represents the cross entropy loss obtained by the global bottom-up probability propagation of the dth layer, and D represents the number of learnable separator layers. By optimizing the global bottom-up probability propagation loss, a hierarchical constraint relationship is achieved in the hierarchical classifier.

[0081] (3) Local Hierarchy Driven Contrastive Learning Method

[0082] For a batch of training sets B = {(X n ,Y n )}, where X n Indicates input, Y nrepresents the corresponding label set, n represents the batch size; a batch of training sets B is encoded with different dropouts through the pre-trained language model to obtain a replica set B+, so as to obtain a total of 2N sets of hidden vectors Z = {(X n ,Y n )}, and then calculate the fraction l of the local labels shared between the i-th and j-th argument pairs using formula (8): ij :

[0083]

[0084] in Indicates that when the i-th label of the d-th layer With the jth label The value is 1 when they are equal, otherwise it is 0; d represents the coefficient of layer d, and D represents the number of layers of the learnable classifier; thus, the number of shared nodes between pairs of arguments with different inputs is obtained, and then l is converted into ij Normalize to get the similarity score α ij :

[0085]

[0086] Finally, for each input argument pair (i, j), the local level contrast loss is defined as:

[0087]

[0088] where h i and h j They refer to the semantic vectors of the argument pair, sim represents cosine similarity, τ represents the temperature coefficient of contrastive learning; exp represents exponential function, and log represents logarithmic function; under contrastive loss, for the argument pair (i, j), the more sub-labels they share, the greater their weight α ij The larger it is, the more loss terms the argument pairs have. The value of h i and h j The distance between them will be closer; on the contrary, the distance will be farther; therefore, the contrast loss uses the similarity score α ij To guide the distance between the semantic vector representations of different argument pairs.

[0089] Finally, the above losses are jointly learned, and the overall training target is calculated by formula (11):

[0090]

[0091] in represents the learnable classifier loss, represents the bottom-up probability propagation loss at the global level, represents the local hierarchical driven contrastive learning loss, and λ1 and λ2 represent hyperparameters.

[0092] Specifically, in order to verify the effectiveness of the method of the present invention in a real system, this embodiment is experimentally verified on the English datasets PDTB2.0 and 3.0. The specific details are as follows:

[0093] (1) Dataset:

[0094] For data set segmentation, this method adopts the segmentation method of PDTB-Ji, that is, 2-20 partitions are selected as training sets, 0-1 partitions are selected as validation sets, and 21-22 partitions are selected as test sets. PDTB originally had three levels of chapter relations. Since multiple categories in the third level lack data for implicit relations, the third level is discarded and implicit conjunctions are used as the third level. Each level of PDTB2.0 contains 4 (first level), 11 (second level) and 102 (third level) categories respectively. PDTB 3.0 contains 4 (first level), 14 (second level) and 150 (third level) categories. The data statistics of PDTB 2.0 and 3.0 are shown in Tables 1 and 2:

[0095] Table 1 Statistics of the second-level category data volume of PDTB 2.0 through Ji segmentation

[0096]

[0097] Table 2 Statistics of the second-level category data volume in PDTB 3.0 through Ji segmentation

[0098]

[0099]

[0100] (2) Model details:

[0101] The encoder of the model is the BASE version of the pre-trained language model RoBERTa, which has 12 layers, 12 attention heads, and 768 hidden embedding dimensions. During training, the AdamW optimizer with a learning rate of 1e-6 and a batch size of 8 updates the model parameters for 10 epochs. The number of evaluation steps is set to 400, and the model is selected based on the best result in the validation set. The invention is trained, validated, and tested on an RTX 3090.

[0102] (3) Evaluation indicators:

[0103] Evaluation indicators include accuracy score (Acc) and multi-classification macro-average F1 score (Macro-F1). If a set of argument pairs contains multiple textual relations, the model is considered correct as long as it hits one of the correct categories.

[0104] (4) Comparison system:

[0105] In order to verify the effectiveness of the present invention, the method of the present invention (Ours) is compared with a variety of currently advanced and representative methods. According to the verification results of different versions of the PDTB dataset (PDTB 2.0, PDTB 3.0), the relevant models are introduced here respectively.

[0106] First, we introduce the model verified on PDTB 2.0:

[0107] FT-RoBERTa: Fine-tuned RoBERTa model [1] The RoBERTa model improves the performance of BERT by removing the NSP task and pre-training on a wider corpus.

[0108] HierMTN-CRF: A multi-level chapter relationship prediction network based on conditional random fields [2] :The model first predicts multi-level implicit chapter relations as multiple subtasks and performs label link prediction through a conditional random field network. The BERT and RoBERTa versions of this baseline model are compared here.

[0109] BMGF: Proposed bilateral multi-view matching encoder [3] , to enhance the interaction of arguments at the text span and sentence level.

[0110] LDSGM: A hierarchical label dependency based sequence generation model [4] :The model regards implicit chapter relationship recognition as a label sequence prediction task and models the hierarchical dependencies between labels through a graph convolutional network, and performs label sequence prediction through a GRU decoder.

[0111] PCP: The model first builds a manual template [5] , mining the strong correlation between connectives and discourse relations.

[0112] ChatGPT: A ChatGPT-based approach [6] , equipped with contextual learning prompt templates.

[0113] GOLF: A comparative framework for global and local hierarchical perception [7], these two levels of information are modeled and captured with the help of contrastive learning.

[0114] DiscoPrompt: A prompt-based path prediction model [6] , utilizing the mutual information and intrinsic perception among layers in IDRR.

[0115] ·PEMI [8] : A model that uses efficient parameter hint tuning to drive parameter matching in the pre-training space to achieve low-parameter approximation.

[0116] At the same time, the models verified on the PDTB 3.0 dataset include:

[0117] ·RoBERTa-FT [1] : This embodiment conducts experiments on each level on PDTB3.0.

[0118] ·MANF [9] : A dual-attention model that encodes word pair offsets to enhance semantic interactions.

[0119] ConnPrompt

[10] : A model that converts the relation prediction task into a connection cloze prediction task.

[0120] ·TEprompt

[11] : A dual-attention model that encodes word pair offsets to enhance semantic interactions.

[0121] GOLF [7] : A contrastive framework for global and local level perception that models and captures information at both levels with the help of contrastive learning.

[0122] ·PEMI [8] : A model that uses efficient parameter hint tuning to drive parameter matching in the pre-training space to achieve low-parameter approximation.

[0123] The experimental results of the above baseline model and the model proposed in the present invention at various levels are shown in Tables 3 and 4 (- indicates that the comparison model is missing in this indicator):

[0124] Table 3: Results on the PDTB 2.0 dataset (Acc score, F1 score, and trainable parameter metrics)

[0125]

[0126]

[0127] Table 4: Results on the PDTB 3.0 dataset (Acc score, F1 score, and trainable parameter metrics)

[0128]

[0129] (5) Category-related indicator verification

[0130] In order to further verify the classification ability of the model for different categories, the label-wise F1 scores of each category at the second level are tested here, and compared with the baseline models PEMI and GOLF introduced above. The results are shown in Table 5:

[0131] Table 5: Label-related F1 scores on the PDTB 2.0 dataset

[0132]

[0133]

[0134] The following observations can be made from Tables 3, 4, and 5 above: 1) Whether on PDTB2.0 or 3.0, the present invention achieves comparable performance using only 0.1% of the trainable parameters of recent advanced methods. On PDTB 2.0, the improvement mainly occurs at the third level (connection level), which shows that the present invention is more accurate in grasping the more fine-grained hierarchical semantics; however, on PDTB 3.0, the performance improvement mainly occurs at the first and second levels, which shows that there are significant differences in the label distribution of the two versions of the data set, and PDTB 3.0 has too many connectives, which significantly increases the parameters of the label embedding part, which may not be conducive to the ability of parameter effective prompt learning. 2) In Tables 3 and 4, compared with the baseline model, the present invention exceeds all existing fine-tuning models, which proves that the way to improve the model side is conducive to solving the problem of data scarcity. 3) In Table 5, the improvement of the performance of the present invention is mainly reflected in the minority classes (long-tail classes) with less data. For PDTB 2.0, the improvement depends on three small categories: Comp.Concession, Expa.List, and Expa.Instantiation, which shows that by using parameter efficiency methods and using fewer trainable parameters to approximate the results, the model will pay more attention to minority classes and significantly improve the performance of minority classes.

[0135] To further illustrate, Figure 2 As shown, an example is used to explain the impact of the global hierarchical bottom-up probability propagation method. Figure 2 Part (a) shows the predicted probability distribution of the multi-level classifier at the first level label. Figure 2Part (b) shows the probability distribution calculated by the probability propagation method. It can be clearly observed that their probability distributions have a high degree of similarity, which indicates that the prediction of the upper layer can be obtained by the classifier of the lower layer based on the global hierarchical structure. There is global hierarchical information between multi-level classifiers. In addition, Figure 2 The probability prediction in part (c) has higher discriminative power, indicating the effectiveness of the global hierarchical probability propagation method. Figure 3a and Figure 3b As shown in the figure, the [MASK] representation of the local hierarchical contrastive learning method is visualized. It can be seen that Label0 and label1 share 2 node labels, Label0 and label2 share 1 node label, and Label0 and label3 do not share a node label. Figure 3a and Figure 3b , we can see that label0 is closest to label1 and farthest from label3. Therefore, the contrastive learning method guides the distance between [MASK] representations of input instances by utilizing the difference in the number of shared nodes between local labels.

[0136] Preferably, the embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps in the implicit chapter relationship identification method with global and local level prompt fine-tuning in the above embodiment, and the electronic device specifically includes the following contents:

[0137] Processor, memory, communications interface and bus;

[0138] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0139] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the implicit chapter relationship recognition method with global and local level prompt fine-tuning in the above embodiment are implemented.

[0140] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the implicit chapter relationship identification method with global and local level prompt fine-tuning in the above-mentioned embodiments. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all the steps of the implicit chapter relationship identification method with global and local level prompt fine-tuning in the above-mentioned embodiments.

[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0144] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0147] The above contents are intended to illustrate the technical solution of the present invention schematically, and the present invention is not limited to the embodiments described above. Without departing from the scope of the present invention and the scope protected by the claims, a person skilled in the art may make many specific changes under the guidance of the present invention, which are all within the protection scope of the present invention.

[0148] Finally, it should be pointed out that the above examples are only used to illustrate the calculation process of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above examples, a person skilled in the art should understand that the calculation process described in the above examples can still be modified, or some parameters can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding calculation method deviate from the spirit and scope of the calculation method of the present invention.

[0149] References:

[0150] [1]Liu Y,Ott M,Goyal N,et al.RoBERTa:A Robustly Optimized BERTPretraining Approach.ArXiv 2019:1907.11692

[0151] [2]Wu C, Hu C, Li R, et al. Hierarchical multi-task learning with CRF for implicit discourse relation recognition. Knowledge-Based System 2020,195,105637

[0152] [3] Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. On the importance of word and sentence representation learning in implicit discourse relation classification. IJCAI-20.

[0153] [4]Wu C,Cao L,Ge Y,et al.A Label Dependence-aware Sequence GenerationModel for Multi-level Implicit Discourse Relation Recognition.AAAI 2022:11486-11494.

[0154] [5]Hao Zhou,Man Lan,Yuanbin Wu,Yuefeng Chen,and Meirong Ma.Prompt-based connective prediction method for fine-grained implicit discourserelationrecognition.In EMNLP Findings.

[0155] [6]Chunkit Chan,Xin Liu,Jiayang Cheng,Zihan Li,Yangqiu Song,GinnyWong,and Simon See.DiscoPrompt:Path prediction prompt tuning forimplicitdiscourse relation recognition.In ACL Findings.

[0156] [7]Yuxin Jiang,Linhan Zhang,and Wei Wang.Global and local hierarchy-awarecontrastive framework for implicit discourse relation recognization.InACL 2023Findings.

[0157] [8]Haodong Zhao,Ruifang He,Mengnan Xiao,and JingXu.2023.Infusinghierarchical guidance into prompt tuning:A parameter-efficient framework formulti-level implicit discourse relation recognition.InACL 2023.

[0158] [9]Wei Xiang,Bang Wang,Lu Dai,and Yijun Mo.Encoding and fusingsemanticconnection and linguistic evidence for implicit discourse relationrecognition.InACL 2022 Findings.

[0159]

[10] Wei Xiang,Zhenglin Wang,Lu Dai,and Bang Wang.ConnPrompt:Connective-cloze prompt learning for implicit discourse relationrecognization.In COLING 2022.

[0160]

[11] Wei Xiang,Chao Liang,and Bang Wang.TEPrompt:Taskenlightenmentprompt learning for implicit discourse relation recognition.InACL 2023 Findings。

Claims

1. A method for identifying implicit chapter relations with global and local level hint fine-tuning, characterized in that: The following steps are involved: (1) Construct a prompt learning template for all argument pairs in a data set, and input the prompt learning template into a pre-trained language model to obtain the semantic interaction information of the argument pairs, and then obtain the vector corresponding to the position of the mask word between the argument pairs as the semantic vector representation of the argument pairs; design different levels of learnable classifiers, and map the semantic vector representation of the argument pairs into the probability distribution of different levels of text relationship categories through different levels of learnable classifiers; (2) Define the dependency relationship of all labels in the data set as the global level, and construct a global bottom-up probability propagation module. The label probability of the current level is obtained through the node probability of the next level of the learnable classifier and the global level, and then the parameters in the pre-trained language model and the learnable classifier are updated through the cross entropy calculation loss, so as to obtain the label probability distribution of the current level through the label probability distribution in the lower level learnable classifier, thereby generating global level constraint relationships in learnable classifiers at different levels and realizing the injection of global level knowledge; (3) Define the argument pair labels as local levels and construct a local level driven contrastive learning module. The number of shared nodes in the local level between different argument pairs is different. The more shared nodes there are, the closer the semantic vector representations of the argument pairs are. Conversely, the farther they are, the more the contrastive loss between different argument pairs is controlled by the number of shared nodes. This changes the distance between the semantic vector representations of different argument pairs and achieves the utilization of the local level. (4) The cross entropy loss function and contrast loss function are used to calculate the loss values ​​of different modules in steps (2) and (3), and then different weight coefficients are set for joint learning to update the parameter information of the pre-trained language model to improve the recognition performance.

2. The implicit chapter relationship recognition method with global and local level prompt fine-tuning according to claim 1 is characterized in that: Step (1) specifically includes: (101) Obtaining semantic vector representation of argument pairs based on a pre-trained language model; Let the two arguments be Arg1 and Arg2, and construct a prompt learning template T = [CLS] + Arg1 + [MASK] + Arg2 + [SEP] as the input of the pre-trained language model. After the pre-trained language model PLM, the vectors h of all words in the argument pair are obtained. 1:n , as shown in formula (1), and extract the vector h corresponding to the position of [MASK] [MASK] Used for argument pair semantic vector representation h m , as shown in formula (2): h 1:n =PLM(T) (1) h m =h [MASK] (2) Among them, [CLS] and [SEP] indicate the first character and separator of the argument pair respectively, [MASK] is the mask word, PLM refers to the pre-trained language model, T is the prompt learning template, and h 1:n Represents the vector of all words in the argument pair, n is the word length of the input prompt learning template; (102) Design different levels of learnable classifiers and create a virtual learnable vector W for each learnable classifier d , where d represents the dth layer, and after initializing with the label vector of the corresponding layer, we get a list V of learnable classifiers at different levels, as shown in formula (3): V={V1,...V d ,...,V D } (3) Where V d represents the learnable classifier at the dth level; D is the number of levels of learnable classifiers; obtains the semantic vector representation of the argument pair h m After that, they are sent to different levels of learnable classifiers and the probability distribution value of each layer label is calculated by the softmax function.

3. The implicit chapter relationship recognition method with global and local level prompt fine-tuning according to claim 1 is characterized in that: Step (2) specifically includes: Each layer in the learnable classifier is classified independently without hierarchical constraints. The probability distribution of the label of the next layer in the global hierarchy is used to obtain the probability distribution of the label of the current layer in the global hierarchy. According to the global hierarchy, the i-th specific label of a d-th layer is Its child node set at layer d+1 From formula (4), we can get: Among them, Parent(*) represents the parent node of the node corresponding to *; in the bottom-up probability propagation process, several learnable weight units are used to balance the influence of several factors; therefore, for the argument pair, the i-th label of the d-th layer The bottom-up propagation probability of Through formula (5), we get: is the weight coefficient of the jth edge corresponding to the i-th label in the d-th layer, P(y d+1 )express Child node y at layer d+1 d +1 The corresponding probability value.

4. The implicit chapter relationship recognition method with global and local level prompt fine-tuning according to claim 1 is characterized in that: Step (3) specifically includes: For a batch of training sets B = {(X n ,Y n )}, where X n Indicates input, Y n represents the corresponding label set, n represents the batch size; several training sets B are encoded with different dropouts through the pre-trained language model to obtain a replica set B+, so as to obtain a total of 2N sets of hidden vectors Z = {(X n ,Y n )}, and then calculate the score l of the local label shared between the i-th and j-th argument pair by formula (6) ij : in Indicates that when the i-th label of the d-th layer With the jth label The value is 1 when they are equal, otherwise it is 0; d represents the coefficient of layer d, and D represents the number of layers of the learnable classifier; thus, the number of shared nodes between pairs of arguments with different inputs is obtained, and then l is converted into ij Normalize to get the similarity score α ij : Among them l ij represents the score of the local label, B+ represents the replica set obtained by performing different dropout encodings on several training sets B through the pre-trained language model, k represents the kth, l ik represents the fraction of local labels shared between the i-th and k-th argument pairs; Finally, for each input argument pair (i, j), the local level contrast loss is defined as: where h i and h j refers to the semantic vector of the argument pair, sim represents cosine similarity, τ represents the temperature coefficient of contrastive learning; exp represents exponential function, log represents logarithmic function. Under contrastive loss, for the argument pair (i, j), the more sub-labels they share, the higher their similarity score α ij The larger it is, the more loss terms the argument pairs have. The value of h i and h j The distance between them is closer; on the contrary, the distance is farther; therefore, the contrast loss uses the similarity score α ij Guide the distance between the semantic vector representations of different argument pairs.

5. The implicit chapter relationship recognition method with global and local level prompt fine-tuning according to claim 1 is characterized in that: It is characterized in that Step (4) specifically includes: The loss function of each module is calculated separately and joint learning is performed. The loss of the learnable classifier is calculated using the cross entropy loss, which is obtained by formula (9): in represents the cross entropy loss value of the learnable classifier at layer d, and D represents the number of learnable separator layers; The loss of the bottom-up probability propagation module at the global level also uses the cross entropy loss and is calculated using formula (10): in represents the cross entropy loss obtained by the global bottom-up probability propagation module at the dth layer, and D represents the number of learnable separator layers; The loss of the local layer-driven contrastive learning module is calculated using formula (11): in represents the local hierarchical contrast loss of each set of input argument pairs (i, j), B represents a batch of training sets, B+ represents a replica set obtained by performing different dropout encodings on a batch of training sets B through a pre-trained language model, and N represents the size of each batch of training sets B; Finally, the above three losses are jointly learned, and the overall training target is calculated by formula (12): in represents the learnable classifier loss, represents the global level bottom-up probability propagation module loss, represents the contrastive learning module loss driven by the local hierarchy, and λ1 and λ2 represent hyperparameters.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the implicit chapter relationship recognition method with global and local level prompt fine-tuning as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the implicit chapter relationship recognition method with global and local level prompt fine-tuning as described in any one of claims 1 to 5 are implemented.

Images