A method and system for document-level relation extraction under resource-constrained conditions based on self-distillation

Abstract

The application discloses a method and system for document-level relation extraction under resource-limited conditions based on self-distillation, which comprises the following steps: using a symbol "*" to mark all entity mentions in a document; using a pre-trained BERT model as an encoder to encode the document; constructing an entity representation enhancement module by combining a first backbone model and a first teacher model, and obtaining entity embeddings according to the marked and encoded document; constructing an entity pair representation enhancement module by combining a second backbone model and a second teacher model, and obtaining entity pair embeddings according to the obtained entity embeddings; training and optimizing the model according to an adaptive threshold loss function to obtain a trained document-level relation extraction model; and extracting document-level relations based on the document-level relation extraction model. The application successfully learns the knowledge of the teacher model without increasing the size of the backbone model by introducing the self-distillation idea, and can be deployed in a resource-limited condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a method and system for document-level relation extraction under resource-limited conditions based on self-distillation. BACKGROUND

[0002] Relation extraction (RE) aims to identify the relationship between two entities in a given text, and plays a crucial role in many information extraction tasks. For example, in the current various researches around large language models (LLM), researchers use a large amount of data to train the model. However, such training will pass the discriminatory ideas, false information and even some dangerous assumptions in free text into the large language model, resulting in some common sense errors and even hallucination problems in the large language model. Since June 2023, Microsoft has released three versions of the "Phi" series model, among which the "Phi-2" model has only 2.7 billion parameters, but can have the most advanced performance among models with less than 13 billion parameters. The reason is that Microsoft realized the absolute influence of data quality on model performance, so it constructed a "textbook-like" data set containing multi-domain common sense for training, which is also the reason for the outstanding performance of the "Phi" series model. The same idea was also proposed by Andrew Ng and others, who believe that the current deep learning should be shifted from "model-centered" to "data-centered", and more attention should be paid to the quality of training data in the relatively stagnant stage of model development. In the process of constructing high-quality training data sets, the related method of relation extraction is used to capture the relationship between two entities in the text, and plays a decisive role in the quality of the data set.

[0003] Currently, the main goal of relation extraction is to predict the relations between entities in a single sentence through sentence-level analysis. However, previous studies mostly focused on finding the connections between single sentences, ignoring the detection of connections between multiple sentences. In real life, many cooperative relations are often stated in multiple sentences. According to previous studies, more than 40.7% of the connections are identified separately in multiple sentences. Therefore, the model must extract the relations at the document level. The pre-training model based on the Transformer can capture the relationship information between different entities in multiple sentences through the underlying attention mechanism, so it has recently attracted great attention in the field of document-level relation extraction. However, compared with sentence-level relation extraction, the implementation of document-level relation extraction is more challenging. In particular, the subject and object of the connection cannot be inferred from a single sentence because they can exist in different sentences. Secondly, the model needs to be able to represent entities across sentences because the same entity can appear in multiple places in the text. Finally, to facilitate reasoning, the connection between certain entity pairs is often discovered by using other entities. To determine these multi-hop relationships, logical reasoning must be performed on the interactions between various entities.

[0004] In summary, in the current research on relation extraction, most methods are directed at sentence-level relation extraction, that is, only the entity pairs in a single sentence are analyzed, and the relationship is extracted based on a small amount of context information. In common Chinese and English texts, the subject entity and the object entity corresponding to a relationship may be located in different sentences, which requires the relation extraction model to have the ability to extract relationships from multiple sentences or even multiple paragraphs. The current several document-level relation extraction methods improve the performance of the model by simply adding a representation enhancement module to the existing model. Although the acceleration performance has been improved to some extent, these models are not suitable for deployment in resource-constrained situations because these new modules increase resource consumption. SUMMARY

[0005] The present application is directed to the above problems, and proposes a method and system for document-level relation extraction in a resource-constrained situation based on self-distillation, which creates a self-distillation framework that can improve the performance of the document-level relation extraction model while only requiring one training and without the need to increase additional parameters, thereby enabling deployment in resource-constrained situations.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] The present application proposes a method for document-level relation extraction in a resource-constrained situation based on self-distillation. The document-level relation extraction task of the present application can be formally defined as follows: a document contains a set of entities The task of relation extraction is to identify the set of entity pairs (e in the sets , e o ) between them, where represents a predefined set of relations, e s and e o denote the subject entity and the object entity, respectively. In a document, any entity e i may appear multiple times in the form of mentions , and denotes the number of mentions related to entity e i . A mention refers to the diverse representations of the same entity in different contexts. If there is a segment of text that contains a mention of two entities, it is considered that there is a relation between the two entities (e s , e o ). The label N A is used to mark that there is no special relation between the two entities. In the testing process, the framework predicts the relation label of all entity pairs (e s , e o ) s,o＝1...n；s≠o in the document.

[0008] The method comprises:

[0009] Step 1: using the symbol "*" to mark all entity mentions in the document;

[0010] Step 2: using a pre-trained BERT model as an encoder to encode the document;

[0011] Step 3: constructing an entity representation enhancement module by combining a first backbone model and a first teacher model, and obtaining entity embeddings according to the annotations in step 1 and the encoded document in step 2 through the entity representation enhancement module;

[0012] Step 4: constructing an entity pair representation enhancement module by combining a second backbone model and a second teacher model, and obtaining entity pair embeddings according to the entity embeddings obtained in step 3 through the entity pair representation enhancement module;

[0013] Step 5: training and optimizing the model according to the adaptive threshold loss function to obtain a trained document-level relation extraction model;

[0014] Step 6: extracting document-level relations based on the document-level relation extraction model.

[0015] Further, in step 1, the symbol "*" is placed at the beginning and end of each entity mention to locate each entity mention.

[0016] Further, in step 2, a pre-trained BERT model is used as an encoder to encode the document once; for documents with more than 512 word groups, a dynamic window method is used to encode the entire document, and then the average value of the overlapping embedding in different windows is calculated to obtain the final word embedding.

[0017] Further, the first backbone model uses "*" embedding at the beginning and end of the entity mention to represent the entity related to the mention embedding, and uses a Log-Sum-Exp pooling method to obtain the entity embedding of all mentions related to the entity.

[0018] Further, in the first teacher model, RSMAN is used to enhance the entity representation.

[0019] Further, in the second backbone model, linear and nonlinear activation layers are used to map entity embeddings to hidden states z, divide the embedding dimension into k equal groups, apply a bilinear function and sigmoid activation function in each group, and calculate the probability of the relationship r by the following formula:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] wherein represents a learnable weight matrix, tanh() represents a bilinear function, e s and e o respectively represent the subject entity and the object entity, z s and z o respectively represent the hidden state of e s and e o corresponding to, represents a learnable parameter, b r represents a bias unit in a neural network, g (s,o) represents the probability that the relationship r is adapted to the entity pair (e s , e o ).

[0027] Further, in the second teacher model, after arranging the entity representations corresponding to the specific relation pairs in the order of the relation types, the sorted entities are mapped to hidden states using a linear layer and an activation function, thereby obtaining representations of the subject entities and the object entities respectively.

[0028] Further, in the second teacher model, an axial attention mechanism is adopted to enhance the neighborhood information of each entity pair (e s ,e o ) in the axial direction, and the two-hop information represented by the adjacency matrix is captured by using the axial attention which is calculated by the external attention along the horizontal axis and the vertical axis.

[0029] Further, the adaptive threshold loss function is:

[0030]

[0031]

[0032] where TH represents the adaptive threshold class, r represents the actual relation, P T , and N T represent the positive example and the negative example respectively, the positive example represents the relation exceeding the adaptive threshold, the negative example represents the relation below the adaptive threshold, r' in L1 represents the positive example and the threshold relation defined by the user, exp() is an exponential function, logit is a logit model used as a classification layer in the network structure, and r' in L2 represents the negative example and the threshold relation defined by the user; the loss L teacher of the first teacher model / second teacher model and the loss L backbone of the first backbone model / second backbone model are obtained through the adaptive threshold loss function.

[0033] The overall loss function of the model is:

[0034] L (s,o) = L teacher + alpha L backbone + (1-alpha) L logits + beta L feature ,

[0035] where alpha and beta represent hyperparameters, L logits is the mean square loss between the final logs of the first backbone model / second backbone model and the first teacher model / second teacher model, and L feature represents the cosine similarity calculated according to the entity features and the entity pair features of the first backbone model / second backbone model and the first teacher model / second teacher model.

[0036] The application further provides a system for document-level relation extraction under resource-limited conditions based on self-distillation, comprising:

[0037] a data preprocessing unit configured to mark all entity mentions in the document using a symbol "*";

[0038] a text representation unit configured to encode the document using a pre-trained BERT model as an encoder;

[0039] an entity embedding obtaining unit configured to jointly construct an entity representation enhancement module with the first backbone model and the first teacher model, and obtain entity embeddings according to the marking in the data preprocessing unit and the encoded document in the text representation unit through the entity representation enhancement module;

[0040] an entity pair embedding obtaining unit configured to jointly construct an entity pair representation enhancement module with the second backbone model and the second teacher model, and obtain entity pair embeddings according to the entity embeddings obtained by the entity embedding obtaining unit through the entity pair representation enhancement module;

[0041] an adaptive training unit configured to train and optimize the model according to an adaptive threshold loss function, and obtain a trained document-level relation extraction model;

[0042] a document-level relation extraction unit configured to extract document-level relations based on the document-level relation extraction model.

[0043] Compared with the prior art, the present application has the beneficial effects that:

[0044] The present application marks all entity mentions in the document using a symbol "*", converts data into better features representing potential problems from feature engineering, and provides more distinctive features for the BERT model, so that the BERT model can obtain better performance without changing its own model structure.

[0045] The present application applies the self-distillation concept to the ongoing document-level relation extraction work, effectively improves the performance of the backbone model in the prediction stage without introducing any additional parameters. Unlike other self-distillation technologies, the present application constructs an entity representation enhancement module and an entity pair representation enhancement module according to the characteristics of the document-level relation extraction task, and then the above modules simultaneously and independently perform self-distillation.

[0046] The comprehensive loss function proposed in the present application includes the contrastive loss between the entity and entity pair representations of the teacher model and the backbone model, and the contrastive loss between the entity pair embeddings of the teacher model and the classification layer. Training the teacher model and the backbone model at the same time can make the former almost reach the performance level of the latter in one training.

[0047] The backbone model and the teacher model can be trained simultaneously, and the knowledge of the teacher model is successfully learned without increasing the size of the backbone model by introducing the self-distillation thought, and the method can be deployed in a resource-limited condition. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A method for document-level relation extraction in a resource-limited condition based on self-distillation is shown in the whole framework schematic diagram of an embodiment of the present application.

[0049] Figure 2 The workflow schematic diagram of an entity representation enhancement module is shown in an embodiment of the present application.

[0050] Figure 3 The axial attention calculation schematic diagram of a teacher model in an entity pair representation enhancement module is shown in an embodiment of the present application.

[0051] Figure 4 A system architecture schematic diagram of a method for document-level relation extraction in a resource-limited condition based on self-distillation is shown in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The present application will be further explained in combination with the drawings and specific embodiments:

[0053] As shown in the drawings, Figure 1 a method for document-level relation extraction in a resource-limited condition based on self-distillation, comprising:

[0054] Step 1: data preprocessing. Given a document The present application places a special symbol "*" at the beginning and end of each entity mention to locate each entity mention.

[0055] Step 2: text representation. Use a pre-trained language model to encode the document:

[0056] H=[h1,h2,...,h l ]=BERT([χ1,χ2,...,χ l ])

[0057] On the basis of previous research, the present application uses a BERT encoder to encode the document only once, and uses the same context embedding for classification. Here, h1 represents the input corresponding to the embedded word χ1 in the document. Considering the length limit of the BERT encoder, for documents with more than 512 word groups, we use a dynamic window approach to encode the entire document. Then, the average value of the overlapping embeddings in different windows is calculated to obtain the final word embedding.

[0058] Step three: Constructing entity representation enhancement module. In the entity representation enhancement module, the backbone model and the teacher model are introduced to achieve high-quality entity representation.

[0059] (1) Backbone model

[0060] The backbone model uses the "*" embedding at the beginning and end of the entity mention to represent the entity related to the mention embedding. In order to obtain the entity embedding of all mentions related to entity e i The Log-Sum-Exp pooling method is used:

[0061]

[0062] The pooling operation collects all the information of the references in the document, thereby generating the embedding representation of the entity. Experimental results show that this method performs better than the average pooling.

[0063] (2) Teacher model

[0064] Any method that can enhance entity representation can be included in the teacher model to enhance the overall model performance, and the performance of the backbone model can be improved by comparing the entity embedding with the teacher model. The invention uses an attention mechanism to generate different entity representations for multiple candidate relationships, which improves the performance of the backbone model in the teacher model, as shown in Figure 2 , where RSMAN is used to enhance entity representation. The semantics of the entity are represented based on different relationships. Then the entity representations are spliced according to their relationship types. Finally, a linear layer is used to reduce the dimension to match the specified dimension of the model.

[0065] For each candidate parameter, the randomly initialized parameter can learn p r as the representation of the relationship. Then the dot product method is used to calculate the semantic correlation between the relationship p r and each mention , as follows:

[0066]

[0067] where g represents a function for calculating the semantic correlation between two embeddings. The invention uses dot product to calculate the correlation, which has the same effect as using multi-layer perceptron (MLP) in some tasks.

[0068] Next, the attention weights of all mentions corresponding to different relationships are calculated using the softmax function , as follows:

[0069]

[0070] ​Once the attention of all entity mentions is obtained, the entity representation for a specific relation can be derived by weighted sum of the attention weights for different relations. Given an entity contains relation r, we have:

[0071]

[0072]

[0073] Step four: Construct the entity pair representation enhancement module. In the entity pair representation enhancement module, entities are embedded in a pair-wise combination. Given an entity (encoding) pair contains entities e s and e o . Similar to the entity representation enhancement module described above, the embedding of entity pair is jointly obtained by backbone model and teacher model.

[0074] (1) Backbone model

[0075] In the backbone model, the entity embedding is mapped to hidden state z by linear and nonlinear activation layers, and the bilinear function and sigmoid activation function are calculated for the probability of relation r.

[0076]

[0077]

[0078] P(r|e s ,e o )=σ(z s T W r z o +b r ),

[0079] where the model contains learnable weight matrix b r represents the bias unit in the neural network. The representation of the same entity remains consistent in different entity pairs. In order to reduce the parameter amount in the bilinear classifier, the embedding dimension is divided into k equal groups, and the bilinear function and sigmoid activation function are applied within each group, and the probability of relation r is calculated by the following formula:

[0080]

[0081]

[0082]

[0083]

[0084] where the learnable parameters g (s,o) denotes the probability that the relation r fits the entity pair (e s ,e o ). These groups are embedded in k ways to reduce the number of parameters from d 2 to d 2 / k.

[0085] (2) Teacher Model

[0086] In the teacher model, the present invention employs various methods to enhance the representation of entity pairs to improve the performance of the model. After arranging the entity representations corresponding to a specific relation in the order of relation types, a linear layer and an activation function are used to map these organized entities to hidden states z, thereby obtaining the representations of subject entities and object entities, respectively. The representations of subject entities and object entities are as follows:

[0087]

[0088]

[0089]

[0090] where the learnable parameters g′ (s,o) denotes the probability that the relation r fits the entity pair (e s ,e o ). Similar to the backbone model, these groups are also embedded in k ways here to reduce the number of parameters.

[0091] Secondly, an axial attention mechanism is employed to enhance the neighborhood information of each entity pair (e s ,e o ) in the axial direction. Although previous studies utilize axial attention to enhance neighborhood information for relation classification, the utilization of axial units has been proven to be more effective and intuitive. However, the present invention believes that the self-attention mechanism ignores the connection with other samples other than the axial direction. In the self-attention mechanism, the semantic correlation between the q vector and the k vector is first calculated, and then the weight from this attention mechanism is applied to the v vector to generate a new feature. In contrast, the external attention mechanism calculates the correlation between the q vector and a pre-defined common k vector, and then generates a feature map by multiplying this attention with another externally learnable v vector, thereby achieving different functions, such as Figure 3 As shown, the axial attention captures the multi-hop information between entity pairs, thereby enhancing their representation capabilities. After the pair of representation entities, the present invention utilizes axial attention to capture the information within two hops represented by the adjacency matrix.Figure 3 All entity pairs with two-hop in relation to the entity pair (e4, e6) are illustrated. Red indicates no correlation, and green indicates having correlation.

[0092] Axial attention is computed with external attention along the horizontal and vertical axes. Each computation along the axis adds a skip connection. Given an n x n entity list, the information of the axial entity pair elements (e s ,e o ) and (e s ,e i ) of the entity pair (e i ,e o ) is aggregated, where the entity pair (e s ,e o ) traverses all the relations within two hops around (e s ,e i ) and (e i ,e o ). In addition, external attention is used to incorporate all the information beyond two hops. Subsequently, these information is used to classify the relation between the entity pair (e s ,e o ), which not only contains the information of the two entities, but also incorporates the entity information beyond multiple hops. For the entity pair (e s ,e o ), the relation is classified as

[0093]

[0094]

[0095] Let q (i,j) ,k (i,j) ,v (i,j) be represented as W q ′g′ (i,j) , W k ′g′ (i,j) and W v ′g′ (i,j) , respectively, where denotes the learnable weight matrix in this model. Compared with the self-attention mechanism, denotes the semantic correlation between the element at position (s, o) and other elements in the p row. Parameters k (s,p) and v (s,p) are input-independent parameters, which are used as the weight of the entire training data set on the horizontal axis.

[0096] Step five: train and optimize the model according to the adaptive threshold loss function.

[0097] Relation extraction is essentially a multi-classification task. Traditionally, the relation classification problem is solved by using a cross-entropy loss function. However, this method relies on a universal threshold. To overcome this limitation, the present invention uses an adaptive threshold loss (ATL) as the loss function. A special class, TH, is introduced as the adaptive threshold at the same time. Relations above this threshold are labeled as positive examples, while relations below this threshold are considered negative examples.

[0098] The formula for calculating the adaptive threshold loss is as follows:

[0099]

[0100]

[0101] L = L1 + L2.

[0102] where r represents the actual relationship, P T , N T represent positive examples, negative examples, respectively, r' in L1 represents positive examples and the threshold relationship defined by the user, exp() is the exponential function, logit is the logit model used as the classification layer in the network structure, r' in L2 represents negative examples and the threshold relationship defined by the user. The L1 loss considers positive examples and the TH class. Since there may be multiple positive examples, the overall loss is calculated by adding the cross-entropy loss of all positive examples whose logarithmic values exceed the TH class. The L2 loss includes negative examples and the threshold class. This requires classification of the cross-entropy loss, where the TH class is explicitly labeled when the logarithm of the value is lower than the logarithm of the TH class.

[0103] In addition, in order to facilitate the backbone model to extract valuable information from the teacher model, the present invention adds three supplementary losses in the loss function. These losses include embedding contrast loss of the classification layer, embedding contrast loss of entity features, and embedding contrast loss of entity pair features between the backbone model and the teacher model.

[0104] To calculate the embedding contrast loss of the classification layer, the present invention uses the mean square error (MSE) loss function, which is defined as follows:

[0105]

[0106] To calculate the embedding contrast loss of entity features and entity pair features, the cosine embedding loss function is used, which is defined as follows:

[0107]

[0108] Therefore, the loss function of the application mainly includes the loss related to the final relation classification in the backbone model and the teacher model. In addition, it contains the embedding contrast loss of the classification layer and the entity and entity pair features, which is defined as follows:

[0109] L (s,o) =L teacher +αL backbone +(1-α)L logits +βL feature ,

[0110] wherein α and β represent hyperparameters. L teacher and L backbone respectively refer to the loss of the teacher model and the backbone model obtained by the adaptive threshold loss function. L logits is the mean square loss between the final logits of the backbone model and the teacher model. L feature represents the cosine similarity calculated according to the entity features and the entity pair features of the backbone model and the teacher model.

[0111] Step six: extracting the document-level relation based on the document-level relation extraction model.

[0112] On the basis of the above embodiment, as Figure 4 shown, the application also proposes a system for document-level relation extraction under resource-limited conditions based on self-distillation, comprising:

[0113] A data preprocessing unit is configured to mark all entity mentions in a document using the symbol "*".

[0114] A text representation unit is configured to use a pre-trained BERT model as an encoder to encode the document.

[0115] An entity embedding obtaining unit is configured to jointly use a first backbone model and a first teacher model to construct an entity representation enhancement module, and obtain entity embeddings from the entity representation enhancement module based on the markings in the data preprocessing unit and the encoded document in the text representation unit.

[0116] An entity pair embedding obtaining unit is configured to jointly use a second backbone model and a second teacher model to construct an entity pair representation enhancement module, and obtain entity pair embeddings from the entity pair representation enhancement module based on the entity embeddings obtained by the entity embedding obtaining unit.

[0117] An adaptive training unit is configured to train and optimize the model according to an adaptive threshold loss function to obtain a trained document-level relation extraction model.

[0118] A document-level relation extraction unit is configured to extract the document-level relation based on the document-level relation extraction model.

[0119] In summary, the present application makes changes to the selection of features that are invisible and blind in the deep neural network, and places the main feature mining in the data preprocessing process. Starting from feature engineering, the data is converted into better features that represent potential problems, providing the deep neural network with more distinctive features, so that it can achieve better performance without changing the model structure of the deep neural network. The present application introduces the concept of self-distillation and incorporates it into a new representation enhancement module, that is, the concept of self-refining is applied to the current document-level relation extraction work. Without introducing any additional parameters, the performance of the backbone model in the prediction stage is effectively improved. Unlike other self-distillation techniques, the present application constructs an entity representation enhancement module and an entity pair representation enhancement module according to the characteristics of the document-level relation extraction task. Then, these modules simultaneously and independently self-distill. The present application can train the backbone model and the teacher model simultaneously, and through the introduction of the self-distillation idea, the knowledge of the teacher model is successfully learned without increasing the size of the backbone model, and it can be deployed in resource-poor situations.

[0120] The above only shows the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered within the scope of protection of the present application.

Claims

1. A method for document-level relation extraction in resource-restricted scenarios based on self-distillation, characterized in that, The method comprises the steps of: Step 1: Annotate all entity mentions in the document using the symbol " Step 2: using a pre-trained BERT model as an encoder to encode the document; Step 3: combine the first backbone model and the first teacher model to build an entity representation enhancement module, and obtain entity embedding through the entity representation enhancement module according to the annotation in step 1 and the encoded document in step 2; the first backbone model uses the embedding of the start and end of the entity mention to represent the entity related to the mention embedding, and adopts a Log-Sum-Exp pooling method to obtain the entity embedding of all mentions related to the entity; ​ Step 4: combine the second backbone model and the second teacher model to build an entity pair representation enhancement module, and obtain an entity pair embedding through the entity pair representation enhancement module according to the entity embedding obtained in step 3; in the second backbone model, the entity embedding is mapped to a hidden state by using a linear and nonlinear activation layer, and the embedding dimension is divided into groups of equal size, and a bilinear function and a sigmoid activation function are applied in each group to obtain a probability that the relationship is adapted to the entity pair; Step 5: training and optimizing the model according to the adaptive threshold loss function to obtain a trained document-level relation extraction model; Step 6: extracting the document-level relation based on the document-level relation extraction model.

2. The method for document-level relation extraction in resource-restricted situations based on self-distillation according to claim 1, characterized in that, In step 1, the symbol is placed at the beginning and end of each entity mention to locate each entity mention.

3. The method for document-level relation extraction in resource-restricted scenarios based on self-distillation according to claim 1, characterized in that, In step 2, a pre-trained BERT model is used as an encoder to encode the document once; for documents with more than 512 word groups, a dynamic window method is used to encode the entire document, and then the average value of the overlapping embedding in different windows is calculated to obtain the final word embedding.

4. The method for document-level relation extraction in resource-restricted situations based on self-distillation according to claim 1, characterized in that, In the first teacher model, RSMAN is used to enhance the entity representation.

5. The method for document-level relation extraction in resource-restricted scenarios based on self-distillation according to claim 1, characterized in that, In the second backbone model, the probability of the relationship is calculated by the following equation: wherein denotes a learnable weight matrix, denotes a bilinear function, and denote subject and object entities, respectively, and denote and corresponding hidden states, denotes a learnable parameter, denotes a bias unit in a neural network, denotes a relation adapted to an entity pair .

6. The method for document-level relation extraction in resource-restricted scenarios based on self-distillation according to claim 1, characterized in that, In the second teacher model, after arranging the entity representations corresponding to specific relations in the order of relation types, a linear layer and an activation function are used to map the arranged entities to hidden states, thereby obtaining the representations of the subject entities and the object entities, respectively.

7. The method for document-level relation extraction in resource-restricted situations based on self-distillation according to claim 1, characterized in that, In the second teacher model, an axial attention mechanism is adopted to enhance each entity pair In the axial direction, the neighborhood information is captured within two hops represented by an adjacency matrix using axial attention, which is computed by external attention along the horizontal and vertical axes.

8. The method for document-level relation extraction in resource-restricted situations based on self-distillation according to claim 1, characterized in that, The adaptive threshold loss function is: wherein, represents an adaptive threshold class, represents an actual relationship, , respectively represent positive examples and negative examples, the positive examples represent relationships exceeding the adaptive threshold, and the negative examples represent relationships lower than the adaptive threshold, in represents a positive example and a self-defined threshold relationship, is an exponential function, is a model used as a classification layer in a network structure, in represents a negative example and a self-defined threshold relationship; the loss of the first teacher model / second teacher model is obtained through an adaptive threshold loss function , and the loss of the first backbone model / second backbone model ; The overall loss function of the model is: wherein, and denotes a hyperparameter, is a mean square loss between the final logit of the first / second backbone model and the first / second teacher model, denotes a cosine similarity calculated according to the entity feature and the entity pair feature of the first / second backbone model and the first / second teacher model.

9. A system for document-level relation extraction in resource-restricted scenarios based on self-distillation, characterized in that, The method comprises the steps of: a data pre-processing unit for annotating all entity mentions in a document using a symbol " a text representation unit configured to use a pre-trained BERT model as an encoder to encode the document; The entity embedding obtaining unit is configured to jointly construct an entity representation enhancement module with the first backbone model and the first teacher model, and obtain entity embedding according to the labels in the data preprocessing unit and the encoded documents in the text representation unit through the entity representation enhancement module. The first backbone model uses the entity mention start and end "embedding to represent the entity related to the mention embedding, and adopts a Log-Sum-Exp pooling method to obtain the entity embedding of all mentions related to the entity. The entity pair embedding acquisition unit is configured to jointly construct an entity pair representation enhancement module with a second backbone model and a second teacher model, and obtain an entity pair embedding according to the entity embedding obtained by the entity embedding acquisition unit through the entity pair representation enhancement module. In the second backbone model, the entity embedding is mapped to a hidden state by using linear and nonlinear activation layers, and the embedding dimension is divided into groups of equal size, and a bilinear function and a sigmoid activation function are applied in each group to obtain a probability that the relationship is adapted to the entity pair. an adaptive training unit configured to train and optimize the model according to the adaptive threshold loss function to obtain a trained document-level relation extraction model; a document-level relation extraction unit configured to extract the document-level relation based on the document-level relation extraction model.

Images