A two-stage document-level relation extraction method and related device

Abstract

This application provides a two-stage document-level relation extraction method and related equipment, belonging to the field of natural language processing technology. The method includes: First, inputting all entity pairs of the document to be processed into a pre-trained binary pre-classifier to filter out candidate entity pairs with potential semantic relationships; the pre-classifier is trained using the binary adaptive threshold focus loss function (BAFL), which combines a focus loss module and an adaptive threshold module to simultaneously address sample imbalance and adaptive classification threshold issues. Second, encapsulating the document to be processed, a predefined relation set, and candidate entity pairs according to a predetermined prompt template, and inputting them into a fine-tuned Large Language Model (LLM). The LLM performs multi-label relation classification on each candidate entity pair and outputs relation triples. This application effectively solves the problems of attention dispersion and performance degradation when LLM is directly applied to document-level relation extraction through a two-stage architecture.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a two-stage document-level relation extraction method and related equipment. Background Technology

[0002] Document-level Relation Extraction (DocRE) aims to identify the semantic relationships between all entity pairs in a document. It is one of the core technologies for building large-scale knowledge graphs and achieving deep text understanding, and has broad application prospects in fields such as intelligent information retrieval and biomedical text mining.

[0003] Existing DocRE methods can be mainly divided into the following three categories: The first category is based on traditional deep learning and graph structures. These methods, centered around convolutional neural networks (CNNs), long short-term memory networks (LSTMs), or graph convolutional networks (GCNs), capture long-distance dependencies between entities by constructing document graphs. However, their core shortcomings are: first, they can only explicitly model shallow syntactic dependencies and cannot capture complex implicit semantic associations and cross-sentence logical reasoning relationships within a document; second, when faced with multi-entity, long document scenarios, the complexity of graph structure construction increases exponentially, leading to a significant decrease in inference efficiency and accuracy; and third, they cannot effectively address the problem of imbalanced sample classes and have extremely poor ability to extract long-tail relationships.

[0004] The second category is methods based on Transformer pre-trained models such as BERT. These methods are the state-of-the-art (SOTA) solutions for traditional DocRE tasks, using models like BERT and RoBERTa as core encoders and enhancing entity representations through attention mechanisms. However, their inherent performance limitations are: first, the limited number of parameters in the pre-trained models results in significantly lower long-distance semantic understanding and multi-hop logical reasoning capabilities compared to large language models, leading to insufficient accuracy in classifying multi-label relationships in complex documents; second, regarding the extreme imbalance between positive and negative samples in DocRE tasks, traditional binary cross-entropy (BCE) loss and adaptive threshold loss (ATL) cannot simultaneously achieve adaptive threshold classification and reinforcement learning for difficult-to-classify positive samples, resulting in an imbalance between entity pair selection accuracy and recall during the pre-classification stage; and third, regarding the noise problem in remote supervised data, there is a lack of effective noise suppression mechanisms, resulting in insufficient model robustness and an inability to fully utilize large-scale weakly supervised data to improve performance.

[0005] The third category is methods directly based on Large Language Models (LLM). These methods rely on the powerful semantic understanding capabilities of LLM to achieve end-to-end relation extraction through prompting engineering, fine-tuning, and other methods. However, they have fatal technical defects: (1) Attention dilution and performance degradation: In the DocRE task, the number of entity pairs in a single document increases quadratically, of which more than 97% of the entity pairs are unrelated negative samples. The massive number of negative samples seriously dilutes the attention of LLM, resulting in insufficient learning of relation features of a small number of effective positive samples. The extraction performance is even lower than that of traditional BERT-like state-of-the-art models; (2) Defects in handling the "unrelated" category: LLM has a general bias in judging the "unrelated (NA)" category in context learning, which is very easy to miss or misjudge. (3) Poor computational efficiency and practicality: Directly extracting all entity pairs into the LLM end-to-end results in an exponential increase in computational load and inference time, extremely high hardware costs, and is not suitable for industrial-grade deployment scenarios with large-scale long documents; (4) Shortcomings of existing two-stage solutions: A few existing two-stage solutions simply split the “relationship judgment” and “relationship classification” processes. Their first-stage pre-classifier does not design a dedicated loss function for the extreme class imbalance problem of DocRE, resulting in poor screening effect of candidate entity pairs and failing to fundamentally solve the performance degradation problem of LLM. Summary of the Invention

[0006] The main objective of this application is to propose a two-stage document-level relation extraction method, electronic device, storage medium, and program product that integrates adaptive loss and large models. This aims to solve the technical problems of attention distraction, performance degradation, and huge computational overhead caused by massive unrelated entity pairs when large language models are directly applied to document-level relation extraction tasks. At the same time, it addresses the model training bias problem caused by the extreme imbalance of positive and negative samples in the DocRE task, as well as the problem of sensitivity to noise in remote supervision data.

[0007] To achieve the above objectives, one aspect of this application proposes a two-stage document-level relation extraction method, the method comprising: The first stage is the entity pair recognition stage. All entity pairs of the document to be processed are input into a pre-trained binary pre-classifier. The pre-classifier filters out candidate entity pairs with potential semantic relationships. The pre-classifier is trained using training data containing positive and negative samples and the binary adaptive threshold focus loss function BAFL. The BAFL loss function combines a focus loss module and an adaptive threshold module to simultaneously address the problems of sample imbalance and adaptive classification threshold. The second stage, the relation classification stage, involves encapsulating the document to be processed, the predefined relation set, and the candidate entity pairs selected in the first stage according to a predetermined prompt template, and then inputting them into the fine-tuned Large Language Model (LLM). The LLM performs multi-label relation classification on each candidate entity pair and outputs standardized relation triples.

[0008] In some embodiments, the BAFL loss function is specifically:

[0009]

[0010]

[0011]

[0012] in, For the focus loss module, For adaptive thresholding modules; and All of these are hyperparameters. For sample labels, This represents the class probability of whether a relationship exists between the samples; This is the adaptive threshold probability output by the model.

[0013] In some embodiments, the pre-classifier is further trained using a knowledge distillation strategy to suppress noise in the remote supervised data, specifically including: Generating soft labels for large-scale remote supervised datasets using a teacher model trained on a high-quality labeled dataset; The student model is jointly trained using the original hard labels of the remote supervision dataset and the soft labels generated by the teacher model.

[0014] In some embodiments, the total loss function for the joint training of the student model is:

[0015] in, It is based on the BAFL loss of the original hard labels. It is based on the mean squared error (MSE) loss between the soft labels of the teacher model and the predicted probabilities of the student learning model.

[0016] In some embodiments, in the first stage, the pre-classifier identifies entity pairs, specifically including: For each entity reference in the document, a special tag is inserted and input into a pre-trained language model to obtain the document's context embedding matrix; For each entity, aggregate all its mentioned features to obtain the entity embedding; For an entity pair consisting of a head entity and a tail entity to be identified, a localized context embedding is generated for the entity pair based on the context embedding matrix and the attention weights of the pre-trained language model. By fusing the entity embedding with the localized context embedding, a context-enhanced entity representation is generated; Based on the context-enhanced entity representation, the probability that the entity pair has a relationship is calculated using bilinear operations and the sigmoid activation function.

[0017] In some embodiments, the localized context embedding of the generated entity pairs specifically includes: For the head entity and the tail entity, aggregate all the attention weights mentioned to obtain their respective entity-level attention vectors; The entity-level attention vectors of the head entity and the tail entity are fused to obtain the fused attention weights for the entity pair; The localized context embedding is obtained by weighting the fused attention weights with the context embedding matrix of the document.

[0018] In some embodiments, the prompt template for the second stage explicitly removes the "No Relationship" or "NA" category option and instructs the LLM to select zero or one or more matching relationships for each candidate entity pair from a given set of predefined relationships.

[0019] In some embodiments, in the second stage, the low-rank adaptation (LoRA) method is used to efficiently fine-tune the parameters of the large language model (LLM), updating only the low-rank matrix parameters in the model.

[0020] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0021] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0022] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0023] Compared with the prior art, this application has the following beneficial effects: 1) Significant Improvement in Core Performance: The LRCAL method proposed in this application, through a two-stage "identification-reclassification" architecture, combines the efficient screening capabilities of traditional classifiers with the accurate inference capabilities of LLM, achieving a significant improvement in extraction accuracy on the two authoritative DocRE benchmark datasets, DocRED and Re-DocRED. On the DocRED dataset test set, the Ign F1 score reaches 59.92, and the F1 score reaches 61.47; on the Re-DocRED dataset, the Ign F1 score reaches 73.56, and the F1 score reaches 74.79, significantly outperforming all traditional BERT-like baseline models and methods directly based on LLM.

[0024] 2) Effectively solves the problem of imbalanced samples: The BAFL loss function designed in this application creatively combines focus loss with an adaptive threshold module, simultaneously solving the model training bias problem caused by extreme imbalance between positive and negative samples in the DocRE task, as well as the poor generalization ability of fixed threshold. Ablation experiments confirm that after replacing the BAFL loss with the traditional BCE loss, the model's Ign F1 decreases by 1.32, verifying the core role of this loss function.

[0025] 3) Significantly Improved Model Robustness: The knowledge distillation strategy introduced in this application effectively suppresses noise interference in remote supervised data through a teacher-student architecture, achieving efficient utilization of large-scale weakly supervised data. Ablation experiments confirm that after removing knowledge distillation, the model's Ign F1 score decreases by 1.02, verifying the effectiveness of this strategy in improving model robustness.

[0026] 4) Balancing accuracy and efficiency, with industrial applicability: Through the first-stage pre-classifier, this application can filter over 97% of irrelevant negative samples, reducing the inference input scale of LLM from quadratic to linear, significantly shortening inference time and reducing hardware overhead. Ablation experiments confirm that removing the first-stage pre-classifier and directly using LLM end-to-end extraction results in a sharp drop in Ign F1 to 22.83, demonstrating the core value of the two-stage architecture in unlocking LLM performance. Attached Figure Description

[0027] Figure 1 This is a flowchart of the two-stage document-level relationship extraction method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the overall framework of the two-stage document-level relation extraction method that integrates adaptive loss and large model provided in the embodiments of this application; Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0030] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0031] 1) Large Language Models (LLMs) are deep learning models trained on large amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics by being trained on massive datasets. The core idea is to learn patterns and structures of natural language through large-scale unsupervised training, thus mimicking the human language cognition and generation process to some extent.

[0032] To address the problems of existing technologies, this application proposes a two-stage document-level relation extraction method integrating adaptive loss and a large-scale model, named LRCAL (LLM-based Relation Classifier with Adaptive Loss). This method is based on a two-stage pipeline architecture of "identification-reclassification." First, an efficient pre-classifier accurately filters candidate entity pairs, and then a fine-tuned large-scale language model accurately classifies multi-label relations, fundamentally solving many shortcomings of existing technologies. Specifically, this method solves the following problems: 1) It addresses the core technical problem that when large-scale language models (LLM) are directly applied to document-level relation extraction (DocRE) tasks, the severe class imbalance caused by the massive number of unrelated entity pairs within documents leads to LLM attention distraction, insufficient learning of effective relation features, and a significant decrease in extraction performance. 2) It solves the model training bias problem caused by the extreme imbalance of positive and negative samples (negative samples accounting for over 97%) inherent in DocRE tasks, overcoming the shortcomings of traditional loss functions in underfitting a small number of positive samples (related entity pairs) and low classification accuracy. 3) It solves the technical problem that the remote supervised dataset in the DocRE task contains noise, which leads to the model learning incorrect relational patterns and poor robustness, and realizes the efficient use of large-scale weakly supervised data.

[0033] The two-stage document-level relation extraction method provided in this application relates to the field of natural language processing technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the two-stage document-level relation extraction method, but is not limited to the above forms.

[0034] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0035] like Figure 1 As shown, this embodiment provides a two-stage document-level relationship extraction method. This method is based on a two-stage pipeline architecture of "identification-reclassification" and executes the following two core stages sequentially: S1. First stage: Entity pair recognition stage. The full set of entity pairs of the document to be processed is input into a pre-trained binary pre-classifier. The pre-classifier filters out candidate entity pairs with potential semantic relationships. The pre-classifier is trained using training data containing positive and negative samples and the binary adaptive threshold focus loss function BAFL. The BAFL loss function combines a focus loss module and an adaptive threshold module to simultaneously address the problems of sample imbalance and classification threshold adaptation. The second stage: the relation classification stage, where the document to be processed, the predefined relation set, and the candidate entity pairs selected in the first stage are encapsulated according to a predetermined prompt template and then input into the fine-tuned Large Language Model (LLM). The LLM performs multi-label relation classification on each candidate entity pair and outputs standardized relation triples.

[0036] Below, in conjunction with Figure 2 The solutions of the embodiments of this application will be described in detail and explained with specific application examples.

[0037] like Figure 2 As shown, this embodiment provides a two-stage document-level relation extraction method that integrates adaptive loss and large models. This method consists of two core stages, which are executed sequentially as follows: The first stage is the relational entity pair identification stage (pre-classification stage). This stage uses the Transformer pre-trained model as the base encoder, and constructs it through customized entity embedding and local context embedding. Combined with the binary adaptive threshold focus loss function (BAFL) and knowledge distillation strategy designed in this invention, a highly efficient binary pre-classifier is trained. This pre-classifier selects candidate entity pairs with potential semantic relationships from all entity pairs in the document, filtering out more than 97% of irrelevant negative samples, thus achieving a significant dimensionality reduction of the input.

[0038] The second stage is the relation classification stage. This stage takes the candidate entity pairs selected in the first stage as input, constructs an input format adapted to the large language model through a customized Prompt project, fine-tunes the large language model using the LoRA low-rank adaptation method, performs accurate multi-label relation classification only on the candidate entity pairs, and finally outputs standardized relation triples, completing the entire document-level relation extraction process.

[0039] (1) First stage: Detailed technical solution for the identification of relational entity pairs The core objective of this stage is to complete the binary classification task of "relationship" or "non-relationship," accurately screening high-potential candidate pairs from a massive number of entity pairs, thus reducing the burden on the precise classification of the second stage, LLM. The core technical steps are as follows: 1.1) Document preprocessing and entity embedding construction a) Entity reference special markers: for the input document ( (a sequence of sentences within a document) and the set of entities annotated within it. By inserting the special word "*" at the beginning and end of each entity mention, the entity location is explicitly marked, enhancing the model's perception of entity boundaries.

[0040] b) Document context encoding: Input the tagged document word sequence into a pre-trained language model based on the Transformer architecture (this embodiment uses the RoBERTa series model), and output the document's context embedding matrix H and the corresponding attention matrix A, where H has a dimension of n×d, d is the hidden layer dimension of the pre-trained model, and A is the multi-head attention weight matrix with H attention heads.

[0041] c) Entity Embedding Aggregation: For each entity *e* in the entity set, the logsumexp pooling method is used to aggregate the feature information of all mentions of that entity to obtain the corresponding entity embedding. Specifically, the embedding vector of the special word "" at the beginning position of each mention of the entity is extracted, and logsumexp pooling is used to fuse the features of multiple mentions to obtain the final entity embedding representation, thus solving the problem of feature aggregation for multiple mentions of the same entity. 1.2) Entity Pair Localized Context Embedding Generation To accurately capture the semantic relationships between entity pairs and address the long-distance dependency problem of entity pairs in long documents, this embodiment employs a local context pooling method to generate localized context embeddings specific to each entity pair. The steps are as follows: a) Entity-level attention weight aggregation: For each entity, obtain the attention weights corresponding to all its mentions from the last layer of the pre-trained model, and aggregate the attention weights of all mentions of the entity into a single entity-level attention vector through average pooling; for the head entity in the entity pair... With tail entity Perform this operation separately to obtain the aggregated attention representations of the two entities.

[0042] b) Entity Pair Attention Fusion: The aggregated attention representations of the head entity and the tail entity are fused to obtain the average pooled attention weight of the entity pair, thereby achieving precise focus on the relevant context regions of the two entities.

[0043] c) Localized context embedding calculation: The average pooling attention weights of entity pairs are weighted with the document context embedding matrix H to obtain the localized context embedding corresponding to the entity pair, thereby achieving accurate extraction of relevant semantic information of entity pairs and filtering out irrelevant context noise.

[0044] 1.3) Binary classification prediction of entity pair relationship existence Based on the generated localized context embedding, the binary classification prediction of entity pairs for "whether there is a relationship" is completed, and the steps are as follows: a) Context-enhanced entity embedding generation: By using a trainable parameter matrix, entity embeddings are fused with localized context embeddings to generate context-enhanced entity embeddings for the head entity and tail entity respectively, thereby strengthening the semantic features related to the paired entity in the entity representation.

[0045] b) Calculation of the probability of the existence of a relationship: Through bilinear operation, the probability value of the existence of a semantic relationship between the head entity and the tail entity is calculated. The output is mapped to the 0-1 interval through the sigmoid activation function, and finally the predicted probability p of the existence of a relationship is obtained.

[0046] 1.4) Design of Binary Classification Adaptive Threshold Focus Loss Function (BAFL) To address the critical challenge of extreme imbalance between positive and negative samples in the DocRE task, this invention designs a dedicated BAFL loss function. This function consists of two parts: a focus loss module and an adaptive threshold module. It enables reinforcement learning for difficult-to-classify positive samples and dynamic adjustment of the sample-level adaptive classification threshold, as detailed below: a) Focus Loss Module This is used to mitigate model bias caused by an imbalance in the number of positive and negative samples. The calculation formula is as follows:

[0047]

[0048] in and All of these are hyperparameters. For sample labels, This represents the class probability of whether a relationship exists between the samples. This can reduce the loss contribution of easily separable samples (majority samples), thereby increasing the loss proportion of difficult-to-separate samples (minority samples).

[0049] b) Adaptive threshold module This module is used to implement dynamic classification thresholds at the sample level, overcoming the shortcomings of traditional fixed thresholds and improving model generalization performance. It introduces an adaptive threshold with corresponding probabilities for each class. The calculation formula is: in, The adaptive threshold probability output by the model is calculated using the sigmoid function. When the sample is a positive sample (y=1), this loss function drives the model to increase. and The difference makes When the sample is a negative sample (y=0), it drives the model to increase. and The difference makes This enables adaptive classification threshold determination for each sample, solving the problem of poor generalization of fixed thresholds in imbalanced data.

[0050] c) Total Loss Function: The final formula for calculating the BAFL loss function is:

[0051] By combining focus loss with adaptive threshold, the problem of class imbalance and adaptive classification threshold in the DocRE task is solved simultaneously, which greatly improves the accuracy and recall of the pre-classifier in identifying positive samples.

[0052] 1.5) Knowledge distillation strategy based on teacher-student architecture To address the noise issue in remote supervised datasets, this embodiment introduces a knowledge distillation strategy to improve the robustness of the pre-classifier and fully utilize large-scale weakly supervised data. The steps are as follows: a) Teacher model training: Based on a high-quality supervised dataset with manual annotation, a high-precision teacher model is trained using the RoBERTa-large pre-trained model and the BAFL loss function mentioned above.

[0053] b) Soft label generation: Using the trained teacher model, reasoning is performed on a large-scale remote supervision dataset to generate a probabilistic soft label for each entity pair. This soft label contains the semantic distribution information learned by the teacher model, which can effectively suppress noise in the original hard labels of the remote supervision data.

[0054] c) Joint training of student models: A RoBERTa-base model with the same structure as the teacher model but fewer parameters is used as the student model for joint training on a remote supervised dataset. The training process is guided by two loss signals. ① Hard label loss The original hard labels from remote supervision data were calculated using the BAFL loss function. ② Knowledge distillation loss Based on the soft labels generated by the teacher model, the difference between the predicted probabilities of the student model and the teacher model is calculated using the mean squared error (MSE). The formula is as follows: ,in The predicted probability for the student model. This represents the predicted probability for the teacher model.

[0055] The total loss during student model training is: .

[0056] d) Student model fine-tuning: After pre-training on a remote supervised dataset, the student model is fine-tuned using a high-quality, manually labeled dataset to obtain the final binary pre-classifier, which has both high screening accuracy and strong noise robustness.

[0057] 1.6) Candidate entity pair screening The pre-classifier, trained with all entity pairs from the input document, performs binary classification using an adaptive threshold corresponding to the BAFL loss. Candidate entity pairs predicted to be "related" are selected, while the vast majority of unrelated negative samples are filtered out, reducing the input size of the second stage from quadratic to linear.

[0058] (2) Second stage: Detailed technical solution for the relationship classification stage The core objective of this stage is to leverage the powerful semantic understanding and reasoning capabilities of large language models to perform accurate multi-label relationship classification on the candidate entity pairs selected in the first stage. The core technical steps are as follows: 2.1) Customization of Input Data and Prompt Template a) Input data preprocessing: Integrate the original input document text, the predefined relation set R of the dataset, and the candidate entity pairs obtained from the first stage of screening to construct the basic input data for LLM.

[0059] b) Customized Prompt Design: For multi-label classification tasks of candidate entity pairs, a dedicated Prompt template is designed. The core optimizations include: removing the "No Relationship (NA)" option to avoid the LLM's processing defects for this category; explicitly limiting the selection to matching relations only from the predefined relation set R; explicitly indicating that a single entity pair may have multiple valid relations to adapt to multi-label classification requirements; and standardizing the input and output formats to reduce the parsing error rate of LLM.

[0060] The core content of the Prompt template used in this embodiment is as follows: ① Instructions: Given text {text}, a set of relations {predefined relation set}, and candidate entity pairs {entity pairs}, select matching semantic relations from the given set of relations for each candidate entity pair based on the information in the text. Multiple matching relations can be selected for a single entity pair.

[0061] ② Input format: Entity pairs are presented in the format '(head entity | -| tail entity)'.

[0062] ③ Output format: The result is presented in the format '(head entity | -| relationship tail entity)'.

[0063] 2.2) LoRA Fine-tuning of Large Language Models To retain the native semantic reasoning capabilities of LLM while achieving low-cost adaptation of the model to the DocRE multi-label classification task, this embodiment employs the LoRA (Low-Rank Adaptation) parameter efficient fine-tuning method to fine-tune the large language model. The specific scheme is as follows: The LLaMA3.1-8B-Instruct large language model is selected as the fine-tuning base, possessing excellent semantic understanding, instruction following, and logical reasoning capabilities. A trainable low-rank matrix is ​​injected into the attention layer of the Transformer model, with hyperparameters set to: lora alpha=16, lora r=8. The native weights of the model are frozen, and only the parameters of the low-rank matrix are trained, significantly reducing the number of parameters and hardware costs for fine-tuning. Manually annotated DocRE datasets are used as the fine-tuning training data. The optimizer is AdamW, batch size=4, training epoch=8, and learning rate=1e-4. Training is completed on a single NVIDIA A100 GPU, resulting in a fine-tuned LLM model adapted for the document-level relation multi-label classification task.

[0064] 2.3) Label Relationship Classification Reasoning and Result Output The preprocessed document text, relation sets, and candidate entity pairs, combined with a customized Prompt template, are input into the fine-tuned LLM model for end-to-end inference. Following a preset output format, the LLM inference results are parsed, relation labels for each entity pair are extracted, and standardized relation triples are generated. set, in Here, r is the header entity, and r is the relation label in a predefined set of relations. This serves as the tail entity, ultimately completing the entire document-level relationship extraction process.

[0065] 2.4) Verification of the beneficial effects of the technical solution This embodiment has completed comprehensive experimental verification on the two authoritative DocRE benchmark datasets, DocRED and Re-DocRED, and the results are as follows: a) Core performance improvement: On the DocRED dataset test set, the LRCAL method in this embodiment achieved Ign F1=59.92 and F1=61.47, which is significantly better than all traditional BERT-like baseline models and methods directly based on LLM; on the Re-DocRED dataset, it achieved Ign F1=73.56 and F1=74.79, far exceeding existing LLM-like schemes, and achieved a leapfrog improvement in the extraction accuracy of the DocRE task.

[0066] b) Ablation experiment verification: Ablation experiments confirmed that the core innovations of this invention all bring significant performance gains: ①BAFL loss function: After replacing it with the traditional BCE loss, the model's Ign F1 decreased by 1.32, confirming the core role of this loss function in solving the class imbalance problem; ② Knowledge distillation strategy: After removing knowledge distillation, the model's Ign F1 decreased by 1.02, confirming the effect of this strategy on improving the model's robustness; ③ Two-stage architecture: By removing the first-stage pre-classifier and directly using LLM end-to-end extraction, the model's Ign F1 score dropped sharply to 22.83, confirming the core value of the two-stage architecture in unlocking the performance of LLM.

[0067] c) Efficiency improvement: Through the first-stage pre-classifier, more than 97% of irrelevant negative samples are filtered out, reducing the inference input size of LLM by two orders of magnitude, significantly shortening the inference time, reducing hardware overhead, and possessing excellent industrial-grade deployment capabilities.

[0068] In summary, compared with the prior art, this application has the following advantages and beneficial effects: 1) This application overcomes the problems of attention distraction and performance degradation caused by class imbalance when LLM is directly applied to the DocRE task. Through a two-stage architecture of "identification-reclassification", it combines the efficiency of traditional classifiers with the accurate reasoning ability of LLM, fully releases the technical potential of LLM in the DocRE task, and achieves a leapfrog improvement in extraction accuracy.

[0069] 2) Targeted solution to the imbalanced sample problem: Design a dedicated binary classification adaptive threshold focus loss function to solve the model training bias caused by the extreme imbalance of positive and negative samples in the DocRE task, improve the ability to identify a small number of positive samples, and at the same time realize adaptive classification threshold to enhance the generalization performance of the model.

[0070] 3) Improve the robustness of the model to noisy data: Introduce a knowledge distillation strategy based on a teacher-student architecture to suppress noise interference in remote supervised data, achieve efficient utilization of large-scale weakly supervised data, and further improve the screening accuracy and robustness of the pre-classifier.

[0071] 4) Balancing accuracy and efficiency: By filtering out the vast majority of irrelevant negative samples through the first-stage pre-classifier, the input scale of LLM is reduced from quadratic to linear, significantly reducing the inference computation overhead of LLM. At the same time, through customized Prompt design, the shortcomings of LLM in handling "irrelevant" categories are avoided, improving classification accuracy and enabling the industrial-grade implementation of the solution.

[0072] 5) Breaking through the performance ceiling of traditional technologies: By integrating the efficient screening capabilities of pre-trained models with the complex reasoning capabilities of LLM, it solves the dual pain points of insufficient reasoning capabilities of traditional pre-trained models and low accuracy of end-to-end LLM solutions, and promotes the application of DocRE technology in fields such as knowledge graph construction, intelligent information retrieval, and biomedical information extraction.

[0073] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0074] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0075] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 302 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called and executed by the processor 301 using the methods described above in the embodiments of this application. Input / output interface 303 is used to implement information input and output; The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304); The processor 301, memory 302, input / output interface 303, and communication interface 304 are connected to each other within the device via bus 305.

[0076] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0077] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0078] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0080] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0081] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0082] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0085] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0086] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0088] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A two-stage document-level relation extraction method, characterized in that, The method includes the following steps: The first stage is the entity pair recognition stage. All entity pairs of the document to be processed are input into a pre-trained binary pre-classifier. The pre-classifier filters out candidate entity pairs with potential semantic relationships. The pre-classifier is trained using training data containing positive and negative samples and the binary adaptive threshold focus loss function BAFL. The BAFL loss function combines a focus loss module and an adaptive threshold module to simultaneously address the problems of sample imbalance and adaptive classification threshold. The second stage, the relation classification stage, involves encapsulating the document to be processed, the predefined relation set, and the candidate entity pairs selected in the first stage according to a predetermined prompt template, and then inputting them into the fine-tuned Large Language Model (LLM). The LLM performs multi-label relation classification on each candidate entity pair and outputs standardized relation triples.

2. The method according to claim 1, characterized in that, The BAFL loss function is specifically as follows: in, For the focus loss module, For adaptive thresholding modules; and All of these are hyperparameters. For sample labels, This represents the class probability of whether a relationship exists between the samples; This is the adaptive threshold probability output by the model.

3. The method according to claim 1, characterized in that, The pre-classifier is also trained using a knowledge distillation strategy to suppress noise in the remote supervised data, specifically including: Generating soft labels for large-scale remote supervised datasets using a teacher model trained on a high-quality labeled dataset; The student model is jointly trained using the original hard labels of the remote supervision dataset and the soft labels generated by the teacher model.

4. The method according to claim 3, characterized in that, The total loss function for the joint training of the student models is: in, It is based on the BAFL loss of the original hard labels. It is based on the mean squared error (MSE) loss between the soft labels of the teacher model and the predicted probabilities of the student learning model.

5. The method according to claim 1, characterized in that, In the first stage, the pre-classifier identifies entity pairs, specifically including: For each entity reference in the document, a special tag is inserted and input into a pre-trained language model to obtain the document's context embedding matrix; For each entity, aggregate all its mentioned features to obtain the entity embedding; For an entity pair consisting of a head entity and a tail entity to be identified, a localized context embedding is generated for the entity pair based on the context embedding matrix and the attention weights of the pre-trained language model. By fusing the entity embedding with the localized context embedding, a context-enhanced entity representation is generated; Based on the context-enhanced entity representation, the probability that the entity pair has a relationship is calculated using bilinear operations and the sigmoid activation function.

6. The method according to claim 5, characterized in that, The localized context embedding of the generated entity pairs specifically includes: For the head entity and the tail entity, aggregate all the attention weights mentioned to obtain their respective entity-level attention vectors; The entity-level attention vectors of the head entity and the tail entity are fused to obtain the fused attention weights for the entity pair; The localized context embedding is obtained by weighting the fused attention weights with the context embedding matrix of the document.

7. The method according to claim 1, characterized in that, The second-stage prompt template explicitly removes the "No Relationship" or "NA" category options and instructs the LLM to select zero or one or more matching relationships for each candidate entity pair from a given predefined set of relationships.

8. The method according to claim 1 or 7, characterized in that, In the second stage, the low-rank adaptation (LoRA) method is used to efficiently fine-tune the parameters of the large language model (LLM), updating only the low-rank matrix parameters in the model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images