An implicit discourse relation identification method with global and local hierarchical cue refinement

Abstract

The application discloses an implicit discourse relation recognition method with global and local hierarchical prompt fine-tuning, comprising the following steps: constructing a prompt template of a discourse pair, encoding the prompt template, capturing a discourse pair instance representation, and designing a multi-level classifier for classification; acquiring a label probability of a current level by a child node probability and an edge weight in a next level according to a global label hierarchical graph, and updating parameters in a pre-trained language model and a learnable classifier through a loss, so that the label probability distribution in the lower classifier can be used to acquire the label probability distribution of the current level, and global hierarchical knowledge injection is realized; the size of a comparative loss is controlled by the number of shared nodes, so that the semantic representation distance of the discourse pair is changed, and the local hierarchical label is utilized; finally, different weight coefficients are set for loss functions of different methods for joint learning, and model parameter information is updated to improve system recognition performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly to an implicit discourse relation recognition method based on global and local hierarchical prompt fine-tuning. BACKGROUND

[0002] Implicit discourse relation recognition aims to identify the discourse relation between a pair of text segments in the absence of explicit connectives. Due to the lack of explicit connectives, the model can only identify specific discourse relations through entity references, semantic clues, and other means between arguments, making implicit discourse relation recognition relatively difficult and a challenging task. In addition, this task is crucial for text coherence and is considered a necessary step for many downstream tasks involving context, such as question answering, text summarization, and event relation extraction. At the same time, the label structure of discourse relations follows a hierarchical structure, and an inserted implicit connective is used to assist the standard for ease of identification when labeling. This connective is also considered the most fine-grained discourse relation label in the implicit discourse relation task.

[0003] The initial research on implicit discourse relation recognition is mostly based on human-established linguistic features to construct a probabilistic statistical model; with the widespread application of pre-trained language models, implicit discourse relation recognition has made considerable progress. Under the pre-training and fine-tuning paradigm, researchers encode the representation of parameter pairs through the design of complex neural networks to perform relation classification. Although these task-specific neural networks can effectively learn the contextual semantics of a parameter, they introduce some additional parameters and rely on a large amount of data for training. In addition, some studies have shown that the task goal often does not align with the goal of the pre-trained language model, which limits the utilization of the knowledge of the pre-trained model. Inspired by prompt learning, some studies reformulate the implicit discourse relation recognition task as a fill-in-the-blank problem to bridge the gap between masked language models and downstream tasks. Although the performance is outstanding, the inherent discourse label hierarchy is ignored.

[0004] To address this issue, recent research attempts to convert the hierarchical structure into a path and add it as a prompt template prefix to utilize hierarchical information; however, their approach of forming a path prompt template by selecting a few special connection nodes lacks the completeness of hierarchical label modeling. Some other research proposes a hierarchical label refinement method for prompt learning, which integrates hierarchical guidance depth into prompt tuning.

[0005] However, the existing methods still have two limitations. 1) Although they utilize the static dependency relationship between labels (global hierarchy), the hierarchy relationship modeling is independent of the classifier, resulting in the inability to effectively utilize the probability distribution information output by the classifier. 2) In the pre-training and fine-tuning paradigm, the dynamic label structure information corresponding to each sample (local hierarchy) shows its importance in the text classification task. However, the hierarchy-aware prompt fine-tuning method only considers the entire global label hierarchy structure, and ignores the use of local labels.

[0006] Based on the above analysis, how to simultaneously utilize the global and local hierarchical structure relationship and fully explore the full use of hierarchical structure knowledge is very important. SUMMARY

[0007] The purpose of the present application is to overcome the deficiencies in the prior art and provide an implicit discourse relationship recognition method that can simultaneously utilize global and local hierarchical structure, and fully utilize hierarchical knowledge.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] A global and local hierarchical prompt fine-tuning implicit discourse relationship recognition method, comprising the following steps:

[0010] (1) Based on the distribution characteristics of implicit discourse relationship connecting words that are mostly distributed in the linking position of two argument sentences, a prompt learning template is constructed for all input argument pairs in a certain data set, and the prompt learning template is input into a pre-trained language model to obtain argument pair semantic interaction information. After capturing the argument pair semantic interaction information, the vector of the position corresponding to the mask word between the argument pair is obtained as the argument pair semantic vector representation; a learnable classifier of different levels is designed, and the obtained argument pair semantic vector representation is mapped to the probability distribution of different levels of discourse relationship categories through the learnable classifier of different levels;

[0011] (2) The dependency relationship of all labels in the data set is defined as a global hierarchy, and a global hierarchy bottom-up probability propagation module is constructed. The label probability of the current level is obtained through the next layer node probability of the learnable classifier and the global hierarchy, and then the loss is updated by calculating the cross entropy to update the parameters in the pre-trained language model and the learnable classifier, so as to obtain the label probability distribution of the current level through the label probability distribution in the lower learnable classifier, thereby generating a global hierarchy constraint relationship in the learnable classifier of different levels, and realizing the injection of global hierarchy knowledge;

[0012] (3) defining the label of the argument pair as a local hierarchy, constructing a local hierarchy driven contrastive learning module, the number of shared nodes between different argument pairs of local hierarchies is different, the more the number of shared nodes between the argument pairs, the closer the semantic vector representation between the argument pairs, and vice versa, the contrastive loss between different argument pairs is controlled by the number of shared nodes, the distance between the semantic vector representations of different argument pairs is changed, and the local hierarchy is utilized;

[0013] (4) calculating the loss value of different modules in steps (2) and (3) by using cross-entropy loss function and contrastive loss function, then setting different weight coefficients for joint learning, updating the parameter information of the pre-trained language model to improve the recognition performance.

[0014] Further, step (1) specifically comprises:

[0015] (101) obtaining the semantic vector representation of the argument pair based on the pre-trained language model;

[0016] Let the two arguments be Arg1 and Arg2, and the prompt learning template T = [CLS] + Arg1 + [MASK] + Arg2 + [SEP] is constructed for the input of the pre-trained language model, and the vector h of all words in the argument pair is obtained after the pre-trained language model PLM. 1:n As shown in formula (1), and the vector h corresponding to [MASK] is extracted. [MASK] The semantic vector representation of the argument pair is used for m As shown in formula (2):

[0017] h 1:n = PLM (T) (1)

[0018] h m = h [MASK] (2)

[0019] Wherein, [CLS], [SEP] respectively indicate the first symbol of the argument pair, the separator, [MASK] is the mask word, PLM indicates the pre-trained language model, T is the prompt learning template, h 1:n represents the vector of all words in the argument pair, and n is the length of the input prompt learning template.

[0020] (102) designing different levels of learnable classifiers, creating a virtual learnable vector W d for each learnable classifier, initializing the corresponding level label vector after the initialization, obtaining the list V of different levels of learnable classifiers, as shown in formula (3):

[0021] V = {V1,...V d ,...,V D} (3)

[0022] where V d represents the d-th level of learnable classifier; D represents the number of levels of learnable classifiers; h m After that, the semantic vector representation of each argument pair is sent to different levels of learnable classifiers to obtain the probability distribution value of each layer label through the softmax function.

[0023] Further, step (2) specifically includes:

[0024] Each layer of the learnable classifier is independently classified without hierarchical constraint relationship. The probability distribution of the label of the current layer of the global level is obtained by using the probability distribution of the label of the next layer of the global level. According to the global level, the probability distribution of the i-th specific label of the d-th layer is obtained by using the probability distribution of the i-th specific label of the d+1-th layer. The child node set of the d+1-th layer is obtained by formula (4):

[0025]

[0026] where Parent(*) represents the parent node of the node corresponding to *; in the bottom-up probability propagation process, several learnable weight units are used to balance the influence of several factors; therefore, the bottom-up propagation probability of the argument pair, the i-th label of the d-th layer is obtained by formula (5):

[0027]

[0028] is the weight coefficient of the j-th edge corresponding to the i-th label of the d-th layer, and P(y d+1 ) represents the probability value corresponding to the child node y d+1 of the d+1-th layer.

[0029] Further, step (3) specifically includes:

[0030] For a batch of training sets B={(X n ,Y n )}, where X n represents the input, Y n represents the corresponding label set, and n represents the batch size; several training sets B are encoded by the pre-training language model to obtain the replicated sets B+, so as to obtain all the semantic vector representations of argument pairs, a total of 2N groups of hidden vectors Z={(X n ,Y n )}, and then the score l ij of the local label shared between the i-th and j-th argument pairs is calculated by formula (6).​

[0031]

[0032] wherein denotes 1 when the i-th label of the d-th layer is equal to the j-th label , otherwise 0; λ d denotes the coefficient of the d-th layer, D denotes the number of layers of the learnable classifier; the number of shared nodes between different input argument pairs is obtained, and then l ij is normalized to obtain the similarity score α ij :

[0033]

[0034] Finally, for each input argument pair (i, j), the local hierarchical contrastive loss is defined as:

[0035]

[0036] wherein h i and h j denote the semantic vectors of the argument pairs, sim denotes the cosine similarity, τ denotes the temperature coefficient of the contrastive learning; exp denotes the exponential function, log denotes the logarithmic function; under the contrastive loss, the more shared sub-labels the argument pair (i, j) has, the larger the similarity score α ij is, which increases the value of the loss term L of the argument pair; therefore, the distance between h i and h j is closer; on the contrary, the distance is farther; therefore, the contrastive loss guides the distance between the semantic vector representations of different argument pairs. ij

[0037] Further, step (4) specifically comprises:

[0038] calculating the loss function of each module respectively and performing joint learning; the loss of the learnable classifier utilizes the cross-entropy loss, which is calculated by formula (9):

[0039]

[0040] wherein denotes the cross-entropy loss value of the d-th layer of the learnable classifier, D denotes the number of layers of the learnable classifier;

[0041] The loss of the global hierarchical bottom-up probability propagation module also utilizes the cross-entropy loss, which is calculated by formula (10):

[0042] ​

[0043] wherein represents the cross-entropy loss obtained by the global hierarchical bottom-up probability propagation module of the dth layer, and D represents the number of learnable separators;

[0044] For the loss of the local hierarchical driven contrastive learning module, it is calculated by formula (11):

[0045]

[0046] wherein represents the local hierarchical contrastive loss of the argument pair (i, j) of each group of input, B represents a batch of training sets, B+ represents a replicated set obtained by performing different dropout encodings on the batch of training sets B through the pre-trained language model, and N represents the size of each batch of training sets B;

[0047] Finally, the above three losses are jointly learned, and the overall training target is calculated by formula (12):

[0048]

[0049] wherein represents the learnable classifier loss, represents the global hierarchical bottom-up probability propagation module loss, represents the local hierarchical driven contrastive learning module loss, and λ1 and λ2 represent hyperparameters.

[0050] The application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the global and local hierarchical cue fine-tuning implicit discourse relationship identification method when executing the program.

[0051] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the global and local hierarchical cue fine-tuning implicit discourse relationship identification method.

[0052] Compared with the prior art, the technical scheme of the application has the beneficial effects that:

[0053] 1. The global and local hierarchical structure knowledge is integrated into the cue fine-tuning, the probability information output by the classifier is more fully utilized for hierarchical modeling, the hierarchical knowledge is more fully utilized, and the recognition performance on the task is improved.

[0054] 2. The proposed global and local hierarchical prompt fine-tuning implicit discourse relation identification method uses the output of the multi-level classifier to model the hierarchical relationship, injects hierarchical constraint relationship, reduces the task gap between the pre-training task in the pre-training language model and the downstream multi-level implicit discourse relation identification task, and more fully utilizes the prior knowledge and ability of the pre-training language model, thereby improving the accuracy and F1 value on the task.

[0055] 3. The number of shared nodes in the local hierarchy is used to guide the distance between the semantic vector representations of different argument pairs, improving the representation distribution of the mask word. The local hierarchical knowledge is utilized in the prompt fine-tuning, further improving the recognition performance of the multi-level classifier.

[0056] 4. The results are verified on two versions of English discourse relation standard data sets, and the experimental results verify the effectiveness of the present application; compared with other similar classification models, it is proved that the comprehensive results of the present application on different hierarchical evaluation indicators are better than similar classification models. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The global and local hierarchical prompt fine-tuning implicit discourse relation identification model provided by the present application is shown.

[0058] Figure 2 The recognition enhancement effect of the global hierarchical bottom-up probability propagation method on a certain instance in the present application is shown.

[0059] Figure 3a And Figure 3b The distribution diagram of the [MASK] embedding in each instance under the action of the local hierarchical contrast learning method in the present application is shown. Among them Figure 3a is the [mask] representation without local hierarchical contrast learning; Figure 3b is the [mask] representation with local hierarchical contrast learning. DETAILED DESCRIPTION

[0060] The present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0061] The present application focuses on the multi-level implicit discourse relation identification task. Given an input argument pair instance x=(Arg1,Arg2) and a discourse relation set with D levels, the final goal of the task is to predict the discourse relation sequence according to x. This embodiment selects two versions 2.0 and 3.0 of the English discourse relation standard data set PDTB for verification, uses the pre-training language model RoBERTa for text encoding, and the overall model architecture diagram is as shown in Figure 1 .

[0062] An implicit discourse relation recognition method fusing parameter validation and relation label hierarchical semantic prior, specifically comprising the following steps:

[0063] (1) Obtaining semantic vector representation of argument pair based on pre-trained language model

[0064] Two arguments are denoted as Arg1 and Arg2, and a prompt learning template T = [CLS] + Arg1 + [MASK] + Arg2 + [SEP] is constructed for input of the pre-trained language model. After the pre-trained language model (PLM), the vector h1 of all words in the argument pair is obtained :n , as shown in formula (1), and the vector h [MASK] corresponding to the [MASK] position is extracted m , as shown in formula (2):

[0065] h 1:n = PLM (T) (1)

[0066] h m = h [MASK] (2)

[0067] Wherein, [CLS], [SEP] respectively indicate the first symbol of the argument pair and the separator, [MASK] is a mask word, PLM indicates the pre-trained language model, T is the prompt learning template, h 1:n represents the vector of all words in the argument pair, and n is the word length of the input prompt learning template.

[0068] Further design different levels of learnable classifiers, create a virtual learnable vector W d for each learnable classifier, where d represents the dth layer, and after initialization with the corresponding level label vector, a list of different levels of learnable classifiers V is obtained, as shown in formula (3):

[0069] V = {V1,...V d ,...,V D} (3)

[0070] Wherein V d represents the dth level of learnable classifier; D is the number of levels of learnable classifiers; After obtaining the semantic vector representation h m of the argument pair, it is respectively sent to different levels of learnable classifiers, and the probability distribution value of each layer label is calculated after the softmax function. Therefore, the loss of the learnable classifier is calculated by cross-entropy loss, as shown in formula (4):

[0071]

[0072] Wherein represents the cross-entropy loss value of the learnable classifier of the d-th layer, and D represents the number of learnable separator layers.

[0073] (2) Global hierarchical bottom-up probability propagation method

[0074] In the above method, each layer in the learnable classifier is independently classified, without incorporating hierarchical constraint relationship, and the probability distribution of the label of the next layer can be obtained according to the probability distribution of the label of the current layer of the global hierarchy; according to the global hierarchy, the i-th specific label of the d-th layer its child node set in the d+1-th layer obtained by formula (5):

[0075]

[0076] where Parent (*) represents the parent node of the node corresponding to *; in the bottom-up probability propagation process, a plurality of learnable weight units are used to balance the influence of a plurality of factors; therefore, the bottom-up propagation probability of the i-th label of the d-th layer , obtained by formula (6):

[0077]

[0078] where x is the argument pair, is the weight coefficient of the j-th edge corresponding to the i-th label of the d-th layer, and P(y d+1 ) represents the probability value corresponding to the child node y d+1 in the d+1-th layer. Therefore, the loss of the global hierarchy bottom-up probability propagation module is calculated by formula (7):

[0079]

[0080] where represents the cross-entropy loss obtained by the global hierarchy bottom-up probability propagation of the d-th layer, and D represents the number of learnable separator layers. By optimizing the loss of the global hierarchy bottom-up probability propagation, the hierarchical constraint relationship in the hierarchical classifier is obtained.

[0081] (3) Local hierarchical driven contrastive learning method

[0082] For a batch of training sets B = {(X n , Y n )}, where X n represents the input, and Y ndenotes the corresponding set of labels, n denotes the batch size; a batch of training set B is encoded by the pre-trained language model with different dropout to obtain the replicated set B+, so as to obtain 2N groups of hidden vectors Z = {(X n ,Y n )} representing the semantic vector representation of all argument pairs, and then the score l ij of the local label shared between the i-th and j-th argument pairs is calculated by formula (8):

[0083]

[0084] wherein denotes 1 when the i-th label of the d-th layer is equal to the j-th label , otherwise 0; λ d denotes the coefficient of the d-th layer, and D denotes the number of layers of the learnable classifier; thus the number of shared nodes between different input argument pairs is obtained, and then l ij is normalized by formula (9) to obtain the similarity score a ij :

[0085]

[0086] Finally, for each group of input argument pairs (i, j), the local hierarchical contrastive loss is defined as:

[0087]

[0088] wherein h i and h j denote the semantic vectors of the argument pairs, sim denotes the cosine similarity, and τ denotes the temperature coefficient of the contrastive learning; exp denotes the exponential function, and log denotes the logarithmic function; under the contrastive loss, the more sub-labels shared by the argument pairs (i, j), the greater the weight a ij , which increases the value of the loss term of the argument pairs; therefore, the distance between h i and h j will be closer; on the contrary, the distance will be farther; thus, the contrastive loss uses the similarity score a ij to guide the distance between the semantic vector representations of different argument pairs.

[0089] Finally, the above losses are jointly learned, and the overall training target is calculated by formula (11):

[0090]

[0091] wherein L denotes the loss of the learnable classifier, represents a global hierarchical bottom-up probabilistic propagation loss, represents a local hierarchical driven contrastive learning loss, λ1, λ2 represent hyperparameters.

[0092] Specifically, to verify the effectiveness of the method in the real system, the embodiment carries out experimental verification on the English data sets PDTB2.0 and 3.0. The specific details are as follows:

[0093] (1) Data set:

[0094] For data set segmentation, the method adopts the segmentation method of PDTB-Ji, that is, 2-20 partitions are selected as the training set, 0-1 partitions are selected as the validation set, and 21-22 partitions are selected as the test set. PDTB originally has three levels of discourse relations. Since the third level has a lack of data for multiple classes of implicit relations, the third level is discarded, and the implicit connecting words are taken as the third level. PDTB2.0 contains 4 (first level), 11 (second level) and 102 (third level) classes respectively. PDTB 3.0 contains 4 (first level), 14 (second level) and 150 (third level) classes. The data statistics of PDTB 2.0 and 3.0 are shown in Table 1 and Table 2:

[0095] Table 1: PDTB 2.0 second level class data volume statistics by Ji segmentation method

[0096]

[0097] Table 2: PDTB 3.0 second level class data volume statistics by Ji segmentation method

[0098]

[0099]

[0100] (2) Model details:

[0101] The encoder of the model is the BASE version of the pre-trained language model RoBERTa, which has 12 layers, 12 attention heads and 768 hidden embedding dimensions. In the training process, the learning rate is 1e-6, the batch size is 8, the AdamW optimizer updates the model parameters for 10 epochs. The evaluation step is set to 400, and the model is selected according to the best result in the validation set. The present application is trained, verified and tested on an RTX 3090.

[0102] (3) Evaluation index:

[0103] The evaluation metrics include the accuracy score (Accuracy, Acc) and the macro-averaged F1 score (Macro-F1 score, Macro-F1). If a set of argument pairs contains multiple discourse relations, the model is considered correct as long as it hits one of the correct categories.

[0104] (4) Comparative system:

[0105] In order to verify the effectiveness of the present application, the method of the present application (Ours) is compared with a plurality of methods which are currently advanced and representative. According to the verification conditions of different versions (PDTB 2.0, PDTB 3.0) of the PDTB data set, the related models are introduced respectively here.

[0106] First, the model verified on PDTB 2.0 is introduced:

[0107] · FT-RoBERTa: Fine-tuned RoBERTa model [1] The RoBERTa model improves the performance of BERT by removing the NSP task and pre-training on a more extensive corpus.

[0108] · HierMTN-CRF: Multilevel discourse relation prediction network based on conditional random field [2] The model first predicts the multilevel implicit discourse relation as a plurality of subtasks, and performs label link prediction through a conditional random field network. Here, the BERT and RoBERTa versions of the baseline model are compared.

[0109] · BMGF: A bilateral multi-view matching encoder is proposed [3] To enhance the interaction of arguments at the text span and sentence level.

[0110] · LDSGM: Sequence generation model based on hierarchical label dependency [4] The model identifies implicit discourse relations as a label sequence prediction task and models the hierarchical dependency between labels through a graph convolution network, and predicts the label sequence through a GRU decoder.

[0111] · PCP: The model first constructs a manual template [5] To mine the strong correlation between conjunctions and discourse relations.

[0112] · ChatGPT: A method based on ChatGPT [6] Equipped with a context learning prompt template.

[0113] · GOLF: A global and local level perception comparison framework [7], which utilizes the inter-level interaction information and intrinsic perception in IDRR.

[0114] DiscoPrompt: Prompt-based path prediction model [6] , which utilizes the inter-level interaction information and intrinsic perception in IDRR.

[0115] PEMI [8] : A model that utilizes parameter-efficient prompt tuning to drive parameter matching pre-training space to achieve few-parameter approximation.

[0116] At the same time, the model verified on the PDTB 3.0 dataset also includes:

[0117] RoBERTa-FT [1] : This embodiment respectively conducts experiments on each level on the PDTB 3.0.

[0118] MANF [9] : A dual attention model that encodes the word pair offset to enhance semantic interaction.

[0119] ConnPrompt

[10] : A model that converts the relationship prediction task into a connection complete form prediction task.

[0120] TEprompt

[11] : A dual attention model that encodes the word pair offset to enhance semantic interaction.

[0121] GOLF [7] : A global and local level perception contrast framework that models and captures the information of the two levels by means of contrast learning.

[0122] PEMI [8] : A model that utilizes parameter-efficient prompt tuning to drive parameter matching pre-training space to achieve few-parameter approximation.

[0123] The experimental results of the above baseline model and the model proposed by the present application on each level are shown in Tables 3 and 4 (- indicates that the contrast model has a gap on this indicator):

[0124] Table 3: Results on the PDTB 2.0 dataset (Acc score, F1 score and trainable parameter index)

[0125]

[0126]

[0127] Table 4: Results on the PDTB 3.0 dataset (Acc score, F1 score and trainable parameter index)

[0128]

[0129] (5) Category-related indicator verification

[0130] To further verify the model's classification ability for different categories, the label-wise F1 score of each category at the second level, i.e., Label-wise F1, is tested here and compared with the baseline models PEMI and GOLF introduced above, and the results are shown in Table 5:

[0131] Table 5: Label-wise F1 scores on the PDTB 2.0 dataset

[0132]

[0133]

[0134] From the above Tables 3, 4, and 5, the following observations can be made: 1) The present application achieves comparable performance to the state-of-the-art methods on PDTB 2.0 and 3.0 using only 0.1% of their trainable parameters. On PDTB 2.0, the improvement mainly occurs at the third level (the connective level), which indicates that the present application is more accurate in capturing the semantics of finer-grained levels; however, on PDTB 3.0, the performance improvement mainly occurs at the first and second levels, which shows that there are significant differences in the label distribution between the two versions of the dataset, and the excessive connectives in PDTB 3.0 significantly increase the parameters of the label embedding part, which may not be conducive to the ability of parameter effectiveization hint learning. 2) In Tables 3 and 4, the present application outperforms all existing fine-tuning models compared to the baseline models, which proves that the way of improving the model side is beneficial to solving the data scarcity problem. 3) In Table 5, the improvement of the present application mainly reflects on the minority classes (long-tail classes) with less data. For PDTB 2.0, the improvement depends on three small categories: Comp. Concession, Expa. List, and Expa. Instantiation, which indicates that by using the parameter effectiveization method to approach the results with fewer trainable parameters, the model pays more attention to the minority classes and significantly improves the performance of the minority classes.

[0135] To further illustrate, as shown in Figure 2 , an example is used to explain the impact of the global level bottom-up probability propagation method. Figure 2 Part (a) of the figure shows the predicted probability distribution of the multi-level classifier at the first level label, Figure 2The probability distribution calculated by the probability propagation method in part (b) shows that the probability distributions of them have high similarity, which indicates that the prediction of the upper layer can be obtained by the classifier of the lower layer based on the global hierarchical structure. There is global hierarchical information between the multi-level classifiers. In addition, Figure 2 The probability prediction in part (c) has higher discriminative ability, indicating the effectiveness of the global hierarchical probability propagation method. As shown in Figure 3a and Figure 3b The [MASK] representation of the local hierarchical contrast learning method is visualized. It can be seen that Label0 and label1 share 2 node labels, Label0 and label2 share 1 node label, and Label0 and label3 do not share node labels. By comparing Figure 3a and Figure 3b It can be seen that label0 is closest to label1 and farthest from label3. Therefore, the contrast learning method guides the distance between the [MASK] representations of the input instances by utilizing the difference in the number of shared nodes between local labels.

[0136] Preferably, the embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps of the global and local hierarchical prompt fine-tuning implicit discourse relationship identification method in the above embodiments, and the electronic device specifically includes the following contents:

[0137] A processor, a memory, a communications interface, and a bus;

[0138] Wherein, the processor, the memory, the communications interface complete the communication between each other through the bus; the communications interface is used to realize the information transmission between the server-side device, the metering device and the user-side device and other related devices.

[0139] The processor is used to call the computer program in the memory, and the processor executes the computer program to realize all steps of the global and local hierarchical prompt fine-tuning implicit discourse relationship identification method in the above embodiments.

[0140] The embodiments of the present application also provide a computer readable storage medium capable of implementing all steps of the global and local hierarchical prompt fine-tuning implicit discourse relationship identification method in the above embodiments, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize all steps of the global and local hierarchical prompt fine-tuning implicit discourse relationship identification method in the above embodiments.

[0141] The various embodiments described in this specification are described in the context of progressive development of embodiments. Like reference numerals in different embodiments describe the same or similar elements. Each embodiment is directed to the different aspects of the application. In particular, the hardware + software embodiment is described in less detail as it is substantially similar to the method embodiment.

[0142] The above description discloses, among other things, various embodiments of the application. Additional embodiments are within the scope of the following claims. In some cases, acts or steps recited in the claims can be performed in a different order than that described in the embodiments, and still achieve desirable results. Additionally, the process depicted in the figures does not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0143] Although the present application provides method operations steps as in the embodiments or flowcharts, more or fewer operations steps can be included based on routine or non-creative labor. The order in which the steps are recited in the embodiments is only one of many possible execution sequences of the steps. The embodiments can be executed in the order recited or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the methods recited in the embodiments or the figures.

[0144] Those skilled in the art will appreciate that embodiments of the application can be devised for a system or a method or a computer program product. Accordingly, embodiments of the application can be embodied in a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon for use by or in connection with an instruction execution system.

[0145] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart Figure 1 The flowchart and block diagram in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable

[0146] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart Figure 1one or more processes and / or blocks Figure 1 the steps of the functions specified in one or more blocks.

[0147] The above is intended to illustrate the technical solutions of the present application, and the present application is not limited to the embodiments described above. Those skilled in the art can make many specific modifications under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these all belong to the protection scope of the present application.

[0148] Finally, it should be pointed out that: the above examples are only used to illustrate the calculation process of the present application, and not to limit it. Although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the calculation process recorded in the foregoing examples, or make equivalent replacement to part of the parameters, and these modifications or replacements do not make the essence of the corresponding calculation method deviate from the spirit and scope of the calculation method of the present application.

[0149] References:

[0150] [1] Liu Y, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv 2019: 1907.11692

[0151] [2] Wu C, Hu C, Li R, et al. Hierarchical multi-task learning with CRF for implicit discourse relation recognition. Knowledge-Based System 2020, 195, 105637

[0152] [3] Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. On the importance of word and sentence representation learning in implicit discourse relation classification. IJCAI-20.

[0153] [4] Wu C, Cao L, Ge Y, et al. A Label Dependence-aware Sequence Generation Model for Multi-level Implicit Discourse Relation Recognition. AAAI 2022: 11486-11494.

[0154] [5] Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. Prompt-based connective prediction method for fine-grained implicit discourse relation recognition. In EMNLP Findings.

[0155] [6] Chunkit Chan, Xin Liu, Jiayang Cheng, Zihan Li, Yangqiu Song, Ginny Wong, and Simon See. DiscoPrompt: Path prediction prompt tuning for implicit discourse relation recognition. In ACL Findings.

[0156] [7] Yuxin Jiang, Linhan Zhang, and Wei Wang. Global and local hierarchy-aware contrastive framework for implicit discourse relation recognization. In ACL 2023 Findings.

[0157] [8] Haodong Zhao, Ruifang He, Mengnan Xiao, and Jing Xu. 2023. Infusing hierarchical guidance into prompt tuning: A parameter-efficient framework for multi-level implicit discourse relation recognition. In ACL 2023.

[0158] [9] Wei Xiang, Bang Wang, Lu Dai, and Yijun Mo. Encoding and fusing semantic connection and linguistic evidence for implicit discourse relation recognition. In ACL 2022 Findings.

[0159]

[10] Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. ConnPrompt: Connective-cloze prompt learning for implicit discourse relation recognization. In COLING 2022.

[0160]

[11] Wei Xiang, Chao Liang, and Bang Wang. TEPrompt: Task enlightenment prompt learning for implicit discourse relation recognition. In ACL 2023 Findings.

Claims

1. A method for implicit discourse relation identification with global and local hierarchical cueing fine-tuning, characterized in that, The method comprises the following steps: (1) constructing a prompt learning template for all argument pairs in the input of a certain data set, inputting the prompt learning template into a pre-trained language model to obtain semantic interaction information of the argument pairs, and then obtaining vectors of positions corresponding to mask words between the argument pairs as semantic vector representations of the argument pairs; designing different levels of learnable classifiers, and mapping the semantic vector representations of the argument pairs into probability distributions of different levels of discourse relation categories through the different levels of learnable classifiers; (2) defining the dependency relationship of all labels in the data set as a global level, constructing a global level bottom-up probability propagation module, obtaining the label probability of the current level through the probability of the next level node of the learnable classifier and the global level, and updating the parameters in the pre-trained language model and the learnable classifier through cross-entropy loss calculation to obtain the label probability distribution of the current level through the label probability distribution in the lower learnable classifier, thereby generating a global level constraint relationship in the different levels of learnable classifiers and achieving the injection of global level knowledge; (3) defining the argument pair label as a local level, constructing a local level driven contrastive learning module, and the number of shared nodes between different argument pairs is different, the more the number of shared nodes between the argument pairs, the closer the semantic vector representations of the argument pairs, and vice versa, the contrastive loss between different argument pairs is controlled by the number of shared nodes, the distance between the semantic vector representations of different argument pairs is changed, and the local level is utilized; (4) calculating the loss values of different modules in steps (2) and (3) using cross-entropy loss functions and contrastive loss functions, then setting different weight coefficients for joint learning, and updating the parameter information of the pre-trained language model to improve the recognition performance.

2. The method of claim 1, wherein the global and local hierarchical cueing of the implicit discourse relation recognition is performed by: Step (1) specifically comprises: ​ (101) obtaining semantic vector representations of argument pairs based on a pre-trained language model; Two arguments are recorded as Arg1 and Arg2, and a prompt learning template T = [CLS] + Arg1 + [MASK] + Arg2 + [SEP] is constructed for the input of the pre-trained language model. After the pre-trained language model PLM, the vector h of all words in the argument pair is obtained 1:n As shown in formula (1), and the vector h of the [MASK] corresponding position is extracted [MASK] For the semantic vector representation h of the argument pair m As shown in formula (2): h 1:n = PLM(T) (1) h m = h [MASK] (2) wherein [CLS], [SEP] respectively indicate the first token symbol and the separator symbol of the token pair, [MASK] is a mask word, PLM is a pre-trained language model, T is a prompt learning template, h 1:n represents the vector of all words in the token pair, and n is the word length of the input prompt learning template; (102) Designing different levels of learnable classifiers, creating a virtual learnable vector W for each learnable classifier d where d denotes the dth level, and after initializing with the corresponding level's label vector, we obtain a list of different levels of learnable classifiers V as shown in equation (3): V = {V1,...V d ,...,V D} (3) where V d represents the d-th level of learnable classifier; D is the number of levels of learnable classifiers; the semantic vector representation h m After that, the probability distribution value of each level label is calculated by the softmax function after being sent to the different levels of learnable classifiers respectively.

3. The method of claim 1, wherein the global and local hierarchical cueing of the implicit discourse relation recognition is performed by: Step (2) specifically comprises: ​ Each layer in the learnable classifier is independently classified, without hierarchical constraint relationship, and the probability distribution of the label of the current layer in the global hierarchy is obtained by using the probability distribution of the label of the next layer in the global hierarchy; according to the global hierarchy, the i-th specific label of the d-th layer The child node set of the d+1-th layer is obtained by formula (4): where Parent(*) denotes the parent node of *; several learnable weight units are used to balance the effects of several factors in the bottom-up probability propagation process; thus the pair of arguments, the dth layer i label bottom-up propagated probabilities is obtained by formula (5): Let P(y) be the weight coefficient of the j-th edge corresponding to the i-th label in the d-th layer. d+1 )express The child node y at level d+1 d +1 The corresponding probability value.

4. The method of claim 1, wherein the global and local hierarchical cueing of the implicit discourse relation recognition is performed by: Step (3) specifically comprises: ​ A batch of training sets B = {(X n ,Y n )} is obtained, where X n represents the input, Y n represents the corresponding label set, and n represents the batch size; a plurality of training sets B are encoded by a pre-trained language model with different dropouts to obtain a replication set B+, so as to obtain a total of 2N groups of hidden vector representations of argument pairs Z = {(X n ,Y n )}; then the score l ij of the local label shared between the i and j argument pairs is calculated by formula (6). wherein denotes the value of 1 when the dth layer ith label is equal to the jth label and 0 otherwise; λ d denotes the coefficients of the dth layer, D denotes the number of layers of the learnable classifier; the number of shared nodes between the arguments of different inputs is thus obtained, and then l ij is normalized to obtain the similarity score a ij : wherein l ij represents the fraction of local labels, B+represents replicated sets obtained by different dropout encodings of several training sets B by a pre-trained language model, k represents the kth, l ik represents the fraction of local labels shared between the ith and kth argument pair; Finally, for each group of input argument pairs (i, j), the local level contrastive loss is defined as: Where h i and h j Here, α represents the semantic vector of the argument pair, sim represents the cosine similarity, τ represents the temperature coefficient of contrastive learning, exp represents the exponential function, and log represents the logarithmic function. Under contrastive loss, for argument pair (i,j), the more sub-labels they share, the higher their similarity score α. ij The larger the value, the greater the loss term for argument pairs. The value of h; therefore, h i and h j The closer the two sides are, the greater the distance is; conversely, the farther the two sides are, the more similar the two sides are. Therefore, the contrastive loss utilizes the similarity score α. ij Guides the distance between the semantic vector representations of different argument pairs.

5. The method of claim 1, wherein the global and local hierarchical cueing of the implicit discourse relation refinement is performed by: identifying a discourse relation between the first and second discourse units; identifying a discourse relation between the first and third discourse units; and identifying a discourse relation between the second and third discourse units. characterized in that Step (4) specifically comprises: Calculate the loss function of each module and perform joint learning; the loss of the learnable classifier uses cross-entropy loss, which is calculated by formula (9): wherein represents the cross-entropy loss value of the learnable classifier of the dth layer, and D represents the number of learnable separator layers. The loss of the global level bottom-up probability propagation module also uses cross-entropy loss, which is calculated by formula (10): wherein represents the cross-entropy loss obtained by the global hierarchical bottom-up probabilistic propagation module of the dth layer, and D represents the number of learnable separators. The loss of the local level driven contrastive learning module is calculated by formula (11): wherein represents the local hierarchical contrastive loss of each group of input argument pairs (i, j), B represents a batch of training sets, B+ represents a replicated set obtained by encoding the batch of training sets B through the pre-trained language model with different dropouts, and N represents the size of each batch of training sets B; Finally, the above three losses are jointly learned, and the overall training target is calculated by formula (12): wherein denotes the learnable classifier loss, denotes the global hierarchical bottom-up probability propagation module loss, denotes the local hierarchical driven contrastive learning module loss, λ1, λ2denote hyperparameters.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the global and local level prompt fine-tuning implicit discourse relation identification method of any one of claims 1 to 5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the global and local level prompt fine-tuning implicit discourse relation identification method of any one of claims 1 to 5.

Images