Joint entity relation extraction method based on semi-supervised learning and large language model

Abstract

The invention discloses a combined entity relationship extraction method based on semi-supervised learning and a large language model. The system comprises three modules, namely a data enhancement module, a semi-supervised joint extraction module and a large-scale language model fine adjustment updating module. The data enhancement module (1) is used for implementing a double enhancement strategy on unmarked data, weak enhancement retains syntactic structures and topological features of entities and relationships, and strong enhancement adopts a large language model which is finely adjusted in advance to generate semantic equivalent disturbance; (2) a semi-supervised joint extraction module which obtains a preliminary extraction model through training of marked data, and then generates prediction on unmarked data based on the preliminary extraction model; (3) a large-scale language model fine-tuning updating module which realizes parameter-efficient large-scale language model fine-tuning by using low-rank self-adaption and dynamically adjusts through a semi-supervised extraction result; and (4) fusing prediction results of the three modules to obtain a final joint extraction model. According to the method, semi-supervised learning, a large language model and joint entity relation extraction are creatively combined, the problem of scarcity of artificial label data in the professional field of an existing joint entity relation extraction method is solved, more efficient information extraction is provided, and therefore the overall performance of a joint extraction model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and specifically relates to a joint entity and relationship extraction method, in particular a joint entity and relationship extraction method that combines semi-supervised learning with large language model enhancement, aiming to improve the accuracy and efficiency in triple extraction tasks. Background Art

[0002] Joint entity and relation extraction aims to extract entities and relations from text content simultaneously and integrate them into relation triplets (subject, relation, object), which is a basic task in natural language processing. These extracted triplets can provide valuable support for multiple research fields, such as knowledge graph construction [1], [2], question answering system development [3], [4] and text summarization [5], [6]. Joint extraction methods significantly improve model extraction performance and show superior ability in resolving triple overlap. These traditional joint extraction methods are all built on the basis of supervised learning, and model performance depends largely on the quality and quantity of labeled training data. However, in actual professional fields, manually annotating professional domain data is a very time-consuming and costly task. This is especially evident in cases involving entity overlap.

[0003] To alleviate the reliance on insufficient labeled data, semi-supervised learning provides a viable solution. It bridges the gap between supervised and unsupervised learning by training models using abundant unlabeled data and minimal labeled examples [7], [8], [9]. By coordinating these learning strategies, semi-supervised learning leverages limited labeled data and abundant unlabeled resources, thereby reducing the reliance on labeled datasets and minimizing the resource-intensive requirements of manual labeling. Traditional semi-supervised learning first uses supervised learning on labeled data to build a baseline prediction model, and then this pre-trained model generates pseudo labels for unlabeled instances. This framework systematically applies consistency regularization during the iterative optimization process, achieving gradual performance improvements by using unlabeled training samples. This reduces the reliance on labeled data.

[0004] However, traditional semi-supervised frameworks, especially those designed for text sequences, have inherent limitations. These methods typically rely on data augmentation, such as perturbing input samples to generate pseudo labels. Traditional augmentation techniques face key shortcomings, including limited generalization capabilities for complex language patterns and susceptibility to noise propagation

[10] . First, traditional text augmentation methods typically rely on simple data transformations, such as synonym replacement, insertion, and deletion. These transformations have limited effectiveness when processing complex natural language text. Second, these methods have difficulty capturing the complexity and contextual dependencies of language, resulting in low-quality manual labels that negatively impact model performance.

[0005] The emergence of large language models has significantly alleviated these challenges. Large language models are pre-trained models based on deep learning that demonstrate sophisticated language understanding and generation capabilities through extensive training on diverse text corpora

[11] ,

[12] . Compared with traditional augmentation techniques, large language models leverage their contextual understanding and semantic processing capabilities to generate augmented data with significantly higher quality and semantic coherence. Summary of the Invention

[0006] To address the aforementioned issues, the present invention proposes a method for joint entity and relationship extraction under low-resource conditions using a large language model augmented by semi-supervised learning. This method comprises a data augmentation module, a semi-supervised joint extraction module, and a large language model fine-tuning and updating module. The large language model is incorporated into the data augmentation module. Specifically, the data augmentation module aims to generate two augmented texts of varying strengths from unlabeled data, focusing on rewriting the text (strong augmentation) with the same semantics as the original content but in a different form using a large language model. The semi-supervised joint extraction module first learns to extract triple information from the annotated dataset through model training to obtain a preliminary extraction model. The preliminary model is then used to pseudo-label the weakly augmented unlabeled data. The model then calculates the loss between the predicted results and the pseudo-labels for the strongly augmented data to produce the final extraction model. The large language model fine-tuning and updating module dynamically adjusts the fine-tuning of the large language model based on the results of the unsupervised training phase during iterative training of the extraction model, and applies the latest fine-tuned large language model to the next round of data augmentation.

[0007] The detailed construction steps of the present invention are:

[0008] 1) Using large language models to enhance unlabeled data;

[0009] 2) Using the labeled dataset to train a preliminary joint entity relationship extraction model;

[0010] 3) Select high-confidence labels as pseudo labels for unlabeled data;

[0011] 4) Further train the extraction model using unlabeled datasets;

[0012] 5) Screening fine-tuning data for large language models based on the loss of unlabeled data training;

[0013] 6) Use the fine-tuning data to fine-tune and update the parameters of the large language model;

[0014] 7) Repeat steps 1) to 5) for iterative training;

[0015] 8) The final high-confidence labeled training data is used to train a joint entity relationship extraction model.

[0016] The main contributions of the large-scale language model enhanced semi-supervised joint extraction method of the present invention are as follows:

[0017] 1. To address the limited labeled data problem in joint entity and relation extraction, this paper proposes a general semi-supervised learning framework. By leveraging a large language model with semi-supervised learning to generate high-quality pseudo-labels, this approach enhances the diversity and complexity of training data, making it compatible with existing joint extraction methods. This approach demonstrates broad adaptability to various joint extraction methods, ensuring robust performance in low-resource scenarios.

[0018] 2. We introduce a novel approach that systematically combines large language models with semi-supervised learning to optimize data augmentation. This approach leverages a large language model to generate semantically diverse synthetic text while employing a semi-supervised loss to filter and optimize pseudo-labels to ensure data quality and consistency. This two-way collaboration allows the large language model to enrich the semi-supervised framework with diverse training samples, while semi-supervised learning guides the fine-tuning of the large language model through the selection of high-quality corpus, enhancing the generalization and adaptability of the extraction model while also improving computational efficiency.

[0019] 3. Experimental results on four public benchmark datasets demonstrate that our method outperforms existing mainstream methods in entity and relation extraction tasks. In particular, when minimal labeled data is available, our method significantly improves accuracy, highlighting its applicability in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the overall framework diagram of the method proposed in the present invention.

[0021] Figure 2 It is the semi-supervised learning framework diagram in the present invention,

[0022] Figure 3(a) and Figure 3(b) are the ablation experiment results of semi-supervised learning on four benchmark datasets.

[0023] Figure 4 This is the ablation experiment result of the present invention on the enhancement of the large language model.

[0024] Figure 5 This is a visualization result of the entity overlap pattern on the WebNLG test set of the present invention. DETAILED DESCRIPTION

[0025] The following further explains this solution with the help of the accompanying drawings and specific implementation methods. The first part summarizes the entire framework method; the second part introduces the data enhancement module, semi-supervised joint extraction module and large language model fine-tuning update module in detail, and explains the training process in detail; the third part covers the experimental details, including the benchmark dataset used, the initialization of experimental parameters, the presentation of results, ablation experiments and the visualization analysis of results; the fourth part provides concluding comments, summarizing the findings and contributions of this work.

[0026] 1. Project Overview

[0027] For a given text content, the goal of joint entity relationship extraction is to extract entity pairs and the relationships between them in the text, and integrate them into relationship triples (subject, relationship, object). In the present invention, we hope to learn an effective and robust joint entity relationship extraction model using unlabeled datasets through a semi-supervised learning strategy when there is no large amount of manually annotated data in a specific professional field. To achieve this goal, the present invention proposes a new joint entity relationship extraction method, which uses a large language model to enhance the unlabeled data in semi-supervised learning, and uses a pseudo-label mechanism to obtain sufficient and high-quality pseudo-labeled samples.

[0028] Figure 1 The overall framework of the method proposed in this paper is presented, which includes seven main steps: (1) using a fine-tuned large language model for data augmentation; (2) using the original labeled dataset to train a preliminary entity relationship extraction model; (3) using the preliminary trained extraction model to pseudo-label the unlabeled data; (4) screening high-confidence labels; (5) using the re-labeled training data to train the model; (6) using the fine-tuning data to fine-tune and update the parameters of the large language model; (7) using the final high-confidence labeled training data to train the joint entity relationship extraction model.

[0029] This invention systematically combines the strengths of large language models by applying their powerful text generation capabilities to semi-supervised learning. This two-way collaborative approach, in which a large language model is used to enhance semi-supervised learning while the semi-supervised learning results guide the fine-tuning of the large language model, enhances the model's generalization and adaptability while improving computational efficiency.

[0030] 2. System Model

[0031] 2.1 Data Augmentation Module

[0032] The data augmentation module rewrites natural language sentences without changing the semantics of the original sentences by applying a powerful large language model, mainly changing the expression form and format. During the initialization process, the original training dataset D is divided into annotated subsets D by stratified sampling. L and the unlabeled subset DU ,, the ratio is r∈(0,1). For a classification task with L classes, the labeled subset is defined as where x i represents the i-th training instance, y i Indicates its true label. The corresponding subset is not labeled Contains μN samples, where μ∈N + Control the size ratio of labeled data to unlabeled data.

[0033] For unlabeled instances, we use weak augmentation A(·) and strong augmentation S(·). Weak augmentation is applied to the original unlabeled data to maintain semantic integrity and avoid complex perturbations of the core content. These minimally perturbed samples are filtered based on confidence to achieve reliable pseudo-label generation. Strong augmentation uses a fine-tuned large language model for context-aware text rewriting. Although pre-trained general language models perform well in text generation, they may change key entities and relations in domain-specific extraction tasks. To alleviate the problem of domain-specific rewriting of general models, we fine-tune the large language model on a paraphrase corpus containing 12,000 entity relations, constraining it to maintain: (1) lexical invariance: retaining the original entity and relation types; (2) structural diversity: varying syntax and discourse patterns; and (3) cross-lingual consistency: semantic equivalence between translations. The resulting strong augmentation S(x i ) preserves task-relevant information while introducing synthetic variability.

[0034] To fully leverage the reasoning capabilities of large language models, this paper employs a rigorous, structured prompt engineering approach to guide the model's output towards task-specific goals. Effective prompt design systematically aligns the behavior of large language models with augmentation goals by integrating thought-chain reasoning with contextual constraints, while maintaining semantic fidelity. Our prompts differ from traditional few-shot paradigms through two key mechanisms: minimizing example dependency bias and enforcing entity relationship invariance.

[0035] 2.2 Semi-supervised joint extraction

[0036] In the semi-supervised joint extraction framework, the main goal is to model the relationship between entities and their associated relations in a sentence. This process is usually achieved by decomposing the problem into two main steps: identifying entity pairs and determining the relationship between them. Given a sentence x i and its annotated triple set T i ={(s, r, o)}, the objective function of the joint extraction model aims to optimize the end-to-end probability of predicting these triplets through hierarchical probability decomposition:

[0037]

[0038] Take extracting relations from sentences as an example. p(r|x i ) means from sentence x i The probability of determining the relation r in p(s, o|x i , r) represents the probability of predicting the head-tail entity pair under the constraint of relation r. Specifically, the head entity is first determined under relation r, and then the tail entity is predicted. This formula achieves its global optimization goal by maximizing the joint likelihood of all annotated samples. Its core advantage lies in dynamically adjusting parameters by leveraging entity and relation interaction information, significantly reducing error propagation. This enables the model to more accurately handle overlapping triples, thereby improving model efficiency and robustness.

[0039] In the semi-supervised joint extraction framework of the present invention, the learning objective integrates supervised and unsupervised loss components. For labeled instances, the model calculates the supervised component of the standard cross entropy loss, where y m (p(s, r, o)|x i ) represents the predicted class distribution, and q represents the target distribution of the labeled data. The supervised cross entropy loss of the labeled data is expressed as:

[0040]

[0041] For the unlabeled instance x i Generate pseudo labels. First, the model predicts a weakly enhanced version of the input A(x i )’s class distribution:

[0042] q b =y m (p|A(x j )), (3)

[0043] Pseudo-labeling Determined by choosing the class with the largest predicted probability: This pseudo label is used to calculate the unsupervised cross entropy loss component. The generation of pseudo labels can be expressed as:

[0044]

[0045] The threshold τ is used to filter the predictions to select reliable pseudo labels. When the maximum predicted class probability exceeds τ, the corresponding class is considered reliable and is selected as the pseudo label for the subsequent training phase. Under consistency regularization, the model must maintain consistency in predictions on perturbed versions of the same input. Therefore, the prediction y obtained from the strongly augmented sample is m (p|S(x i )) and the pseudo labels obtained by the previous step For comparison, the unsupervised loss for unlabeled data is defined as:

[0046]

[0047] For a given training set D L , the total loss function L that guides the model during training total It consists of two parts: supervised training part L s and the unsupervised training part L u The training goal is to minimize the loss function L total Through this iterative semi-supervised learning approach, the performance of supervised extraction models is gradually enhanced because it utilizes both labeled data and a large amount of unlabeled data to optimize model predictions. Furthermore, the integration of a large language model significantly improves the performance of the entity relationship extraction framework. The large language model's powerful contextual understanding and generative capabilities enable accurate and robust extraction of entities and relationships even in complex or domain-specific scenarios, significantly reducing reliance on labeled data.

[0048] 2.3 Large Language Model Fine-tuning Update

[0049] Leveraging and transferring knowledge from large pre-trained models for downstream tasks has become a standard approach, and fine-tuning is crucial to unleashing the power and creativity of large language models and has been widely adopted. Leveraging knowledge from large pre-trained models for downstream tasks is standard practice. Fine-tuning remains key to realizing the full potential of large language models, making it a widely adopted technique. Several factors influence the performance achieved through fine-tuning of large language models, including: (1) pre-training conditions, such as model size and the quality and scope of pre-training data; and (2) fine-tuning conditions, such as the specific downstream task and the choice of fine-tuning method. Understanding these factors is crucial to optimizing the performance of a model on a specific task.

[0050] During the fine-tuning phase of large language models, efficient parameter fine-tuning is typically used. Due to hardware limitations, we adopted the LoRA fine-tuning approach. LoRA achieves efficient parameter fine-tuning of large language models within hardware constraints by performing a low-rank decomposition of weight updates. This approach freezes the pre-trained parameters and optimizes only the low-rank residual matrix. This approach preserves the model's original generalization capabilities while enabling scalable adaptation to downstream tasks. The inherent low-dimensional manifold assumption behind LoRA ensures efficient task-specific optimization without catastrophic forgetting, making it particularly suitable for resource-constrained multi-task learning environments.

[0051] The core principle of LoRA involves introducing a parallel bypass next to the weights of the original pre-trained language model, using low-rank approximation through dimensionality reduction and expansion to capture the intrinsic rank structure. LoRA adopts a down-projection matrix A and an up-projection matrix B to approximate the weight update. During training, A is initialized with random Gaussian values, while B is initialized to a zero matrix. It is worth noting that the pre-trained weights W0 remain frozen and only A and B are updated. After training, LoRA calculates the product BA and adds it to W0 to obtain the fine-tuned model weights. In our framework, fine-tuning the pre-trained large language model design updates its parameters as follows:

[0052] W=W0+ΔW, (6)

[0053] Where W0 represents the parameters initialized by the pre-trained model and ΔW represents the parameters that need to be updated. In LoRA, fine-tuning only involves updating ΔW. Specifically, its update can be expressed as:

[0054] W0+ΔW=W0+BA, (7) During fine-tuning, W0 remains fixed, and only A and B are trainable parameters. In the forward propagation, both W0 and ΔW are multiplied by the same input x, and the results are weighted.

[0055] During the training phase, LoRA fine-tuning is applied to the base large language model, a computationally efficient method to improve performance. The adapted LoRA weights are then integrated with the original model parameters, and the combined weights are saved as a new checkpoint. This mechanism allows the fused model to be loaded using standard methods during inference. The fine-tuning process leverages an unlabeled dataset augmented with the model itself. In the first training iteration, the model processes a large amount of augmented data for the joint extraction task while computing an unsupervised cross-entropy loss. This loss serves as a filtering criterion, eliminating low-quality samples (e.g., those with excessively high loss values ​​or insufficient length) and retaining only the high-confidence outputs of the large language model. In subsequent iterations, the refined model is used for data augmentation, achieving iterative optimization of the framework. The semi-supervised framework's loss function facilitates fine-tuning of the large language model, enabling it to produce outputs that are more consistent with the desired results. This iterative process enhances the model's ability to handle limited labeled data, resulting in more accurate and robust performance in entity recognition and relation extraction tasks.

[0056] 2.3 Framework Self-Training Process

[0057] Leveraging large language models for text data augmentation and facilitating semi-supervised learning training. We employ consistency regularization and pseudo-labeling mechanisms commonly used in semi-supervised learning methods, which have been shown to perform well on image classification datasets. This approach first trains a preliminary extraction model on the original labeled instances. This preliminary extraction model is then used to filter and relabel unlabeled instances, and the final extraction model is then trained on a new, refined dataset.

[0058] In this work, we combine a large language model with semi-supervised learning as the overall training framework. Algorithm 1 shows an overview of the proposed framework. First, the input of the framework is a small amount of labeled data and a large amount of unlabeled data. The large language model is used to perform data augmentation on the unlabeled data, and the labeled data is used to train a preliminary extraction model. Then, the trained preliminary extraction model is used to predict the unlabeled data and assign pseudo labels to obtain more high-confidence labeled data sets. Next, we train the model based on this more labeled data, and at the end of each round of training, we determine whether the large language model needs to be fine-tuned. If necessary, the filtered data is used to fine-tune the large language model and the update is applied to the next round of data augmentation. Finally, iterative training is performed to obtain the final entity relationship extraction model.

[0059] Algorithm 1: Semi-supervised entity relationship extraction method enhanced with large language model Data: dataset D, pre-trained extraction model M, hyperparameter λ, threshold τ, iteration period E, fine-tuning interval n, split ratio r, initial large language model LLM0, enhancement methods A(·) and S(·).

[0060] Result: Final entity relationship extraction model M ′ .

[0061]

[0062]

[0063] 3 Experimental design

[0064] 3.1 Dataset

[0065] In the experiments on semi-supervised entity relation extraction methods enhanced with large language models, four widely used public benchmark datasets are introduced for evaluation. These datasets include WebNLG

[13] , ADE

[14] , SCIERC

[15] , and CONLL04

[16] . The WebNLG dataset was originally designed for natural language generation tasks and was later used for relation extraction tasks. CONLL04 is a corpus for English entity relation extraction tasks, containing phrase-level entity and relation information from various fields, such as corpus web pages and news. SCIERC consists of 500 paper abstracts in the field of artificial intelligence and was originally curated for building scientific knowledge graphs. ADE contains medical descriptions of adverse drug events. Table 1 provides the statistics of these datasets.

[0066] Table 1: Datasets used

[0067]

[0068] 3.2 Evaluating paleobenches

[0069] In order to evaluate the performance of the present invention, we selected three state-of-the-art algorithms as comparison models. The joint extraction models include PFN

[17] , OneRel

[18] , and SpERT

[19] . PFN adopts a dual-channel encoding architecture to achieve decoupled learning of entity and relation features. Through the gated cross-task information filtering mechanism, the semantic conflict in joint modeling is effectively alleviated. OneRel innovatively constructs a unified reasoning paradigm within a single module, using the relationship-aware attention matrix to simultaneously model entity pairs and their semantic associations. The global constraint mechanism is adopted to significantly reduce the error propagation inherent in traditional cascade methods. SpERT proposes a span enumeration-joint optimization strategy based on the Transformer pre-training framework. Its context-sensitive span scoring system, combined with dynamic negative sampling technology, shows superior generalization ability in low-resource scenarios. Standard precision, recall and F1 scores are used to evaluate performance, as shown below:

[0070]

[0071] where c represents the relation category, and TP, FP, and FN represent true positives, false positives, and false negatives, respectively. In named entity recognition tasks, an entity is considered correct only if both its type and boundary are accurately identified. In the context of consistency regularization, a triplet is considered valid if all three components—the entity type, boundary, and the type of relation between entities—are correctly predicted. Model parameters are selected based on validation set performance, and final results are reported on the test set.

[0072] 3.3 Training parameter setting

[0073] To validate the framework's effectiveness in label-scarce settings, we used a significantly smaller annotation set than in previous studies for most experimental conditions. Given the inherent heterogeneity in the distribution of entity and relation types in the dataset, we implemented different parameter configurations, as described in Table 2. The embedding layer was initialized using a pretrained language model. The training protocol used the Adam optimizer with a linear learning rate warmup over 15 epochs. A phased parameter optimization strategy was implemented, fine-tuning the base large language model every 8 epochs, followed by a knowledge-enhanced reprojection operation. To ensure the stability of joint extraction training, a gradient clipping threshold of 1.0 was used throughout the optimization process to constrain parameter updates.

[0074] Table 2: Hyperparameter settings

[0075]

[0076] During model optimization, the ADE dataset underwent two training set augmentation cycles, while the other three datasets underwent three augmentation iterations within 15 training epochs. Due to the significant degradation of model performance after three augmentations, model fine-tuning was limited to a single instance based on two technical constraints: (1) the risk of parameter inflation from repeated fine-tuning significantly increases computational resource requirements; and (2) multi-stage optimization triggers catastrophic forgetting, defined as the gradual degradation of the model's performance on the original task when adapting to a new target.

[0077] 3.4 Results

[0078] Tables 3 and 4 show the prediction performance of our proposed method and the baseline model on four datasets. Table 2 includes the WebNLG and SCIERC datasets, and Table 3 includes the ADE and CONLL04 datasets. The performance parameters in the tables are presented as percentages, with bold indicating the best results.

[0079] Table 3: Results of the model on WebNLG and SCIERC datasets

[0080]

[0081]

[0082] Table 4: Experimental results of the model on ADE and CONLL04 datasets

[0083]

[0084]

[0085] It can be seen from Tables 3 and 4 that on all four mentioned datasets, compared with other state-of-the-art methods, the relation extraction model trained with the large language model-enhanced semi-supervised joint extraction framework proposed in this paper performs better in all indicators.

[0086] SemiER-LLM consistently demonstrates strong performance against independent baseline methods on all four datasets, especially when labeled data is scarce. The framework demonstrates considerable data efficiency; for example, on WebNLG and SCIERC, SemiER-LLM PFN Using only 10% of the labeled data, it achieves comparable or higher F1 scores (74.4 and 16.8, respectively) than the PFN baseline trained on 20% of the labeled data (74.9 and 20.2, respectively). The performance improvement is generally more significant when increasing the labeled data from 10% to 20% than when increasing it from 20% to 30%. Under different experimental conditions, the framework generally maintains a good balance between precision and recall in both entity recognition and relation extraction tasks.

[0087] The framework demonstrates strong generalization capabilities, providing robust and improved results across several diverse benchmark datasets compared to other models. While generally effective, the performance gains on the highly specialized ADE dataset are more modest. This particular case appears to be influenced by two key factors: first, the inherent complexity of the ADE task, which involves nuanced biomedical language and potentially fuzzy entity boundaries; and second, potential limitations of the underlying LLMs used in these experiments. These specialized models, while capable, are relatively small and may lack the deep, specialized knowledge required to achieve top performance in niche domains such as adverse drug event reporting. Despite this domain-specific challenge, the overall results confirm the broad applicability and effectiveness of the framework across a range of datasets.

[0088] 3.5 Ablation Experiment

[0089] To better illustrate the contributions of the key components of our proposed method, we conducted three ablation experiments. The first validated the effectiveness of fine-tuning a large language model; the second validated the effectiveness of semi-supervised learning by comparing it with extraction using a pure large language model; and the third ablation experiment compared the effects of traditional data augmentation with augmentation using a large language model.

[0090] First, we validate the contribution of large language models to data augmentation. Table 5 presents a comparative performance analysis across model configurations. The unmodified Qwen-7B-chat baseline exhibits significant limitations in text rewriting fidelity, particularly by generating semantically inconsistent output, which impairs performance on downstream tasks. Quantitative evaluation shows that on the ADE 10% split, the baseline model achieves an F1 score of 79.5 for NER and 63.7 for RE, respectively, while the fine-tuned variant (FT-Qwen-7B-chat) demonstrates measurable improvements (NER: 83.0; Triple: 72.4), with balanced gains in both precision and recall. Furthermore, the performance gains of the model slow down with increasing amounts of labeled data. In contrast, the ChatGLM3-6B model exhibits a different performance pattern, with the original pre-trained version outperforming its fine-tuned counterpart on 10% of the labeled data. This counterintuitive result may stem from the challenges of adapting to the specialized ADE domain with extremely limited (10%) data. The fine-tuning process could have overfit to this small sample or potentially destroyed the model’s strong pre-trained generalization capabilities, failing to successfully instill reliable task-specific knowledge. Therefore, in this low-resource setting, the broad capabilities of the original model proved more effective than a narrow, potentially imperfect fine-tuned version.

[0091] Next, we validate the role of semi-supervised learning, focusing on comparing the performance of pure LLM extraction with our framework. Figures 3(a) and 3(b) show experimental results on several benchmark datasets. The results demonstrate that pure LLMs have significant limitations in cross-domain generalization and entity relationship capture. By introducing semi-supervised constraints, the proposed framework effectively improves the stability and effectiveness of knowledge extraction. In the entity recognition task, our framework achieves an F1 score of 95.5 on the WebNLG dataset, surpassing the best-performing baseline model, Grok3, which achieves 80.0. On the more challenging SCIERC and ADE datasets, it achieves F1 scores of 64.4 and 87.3, respectively. However, the performance of LLMs fluctuates significantly across datasets, reflecting the limitations of pure LLMs in generalization and stability when faced with varying domains and data characteristics. In the more complex triple extraction task, our framework consistently outperforms the baseline LLM on the WebNLG, ADE, and CoNLL04 datasets, achieving F1 scores of 87.3, 76.0, and 62.9, respectively. In contrast, ChatGPT-4.0 only achieved F1 scores of 7.7 and 38.5 on WebNLG and ADE, respectively. Even the relatively high-performing models Grok3 and Claude3.5 had F1 scores significantly lower than our framework. These results further highlight the limitations of pure large language models in understanding and extracting complex structured information.

[0092] Finally, we verify the role of large language models in data enhancement. Figure 4 The results in

[15] show that traditional enhancement methods have problems with insufficient entity normalization and complex semantics. Specifically, these methods cannot accurately capture the grammatical role of entities in context, resulting in inaccurate links between primary and secondary entities. In contrast, large language models effectively distinguish the grammatical functions of entities through deep semantic parsing and establish logical dependencies between primary entities and supplementary information. Secondly, traditional methods have problems with semantic drift and redundant propagation in relationship extraction, and often deviate from the core semantics due to contextual interference. Large language models can more strictly match the key relations in the input text, ensuring that the generated text is highly consistent with the input semantics. The problem of grammatical compliance in structured text generation is particularly prominent in traditional methods. Large language models can automatically detect and correct punctuation and spelling errors, while ensuring that the generated structured text conforms to grammatical specifications.

[0093] Table 5: Ablation results of fine-tuning large language models on the ADE dataset

[0094]

[0095] 3.6 Results Visualization

[0096] Figure 4 The present invention shows the visualization analysis of the results of different entity overlap patterns extraction on the WebNLG dataset. Figure 4 As shown in , our framework outperforms two baseline models in the overlapping entity recognition task using the same annotated data. Specifically, the extraction model trained using our framework achieves a 17% average F1 score improvement in single-entity overlap scenarios and a 19.18% improvement in triplet extraction tasks in entity-pair overlap scenarios. These experimental results validate our framework's effectiveness in addressing the challenge of entity overlap. The consistent performance improvement across different overlap patterns highlights the robustness of our framework in handling structured information extraction tasks involving overlapping entities.

[0097] 4. Conclusion

[0098] This paper proposes a new method to alleviate the problem of lack of manually annotated data in low-resource scenarios and improve the performance of joint entity and relationship extraction models. This paper proposes a collaborative framework that combines semi-supervised learning with a large language model to perform joint entity-relationship extraction when labeled data is limited. Overall experimental results show that the model in our framework has achieved significant improvements in entity and relationship prediction, especially in low-resource scenarios. Notably, using only 10% labeled data, the performance level of this method is comparable to that of traditional supervised settings, which typically require 20-30% labeled data. In addition, the framework does not rely on a single entity-relationship extraction model, as most existing extraction models can be seamlessly integrated, demonstrating its excellent versatility and adaptability. The simplicity and efficiency of the framework make it highly adaptable and have development potential.

[0099] References:

[0100] [1] Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin MMarlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics. 74-84.

[0101] [2] Jiapu Wang, Boyue Wang, Junbin Gao, YongliHu, and BaocaiYin. 2023. Multi-concept representation learning for knowledge graphcompletion. Association for Computing Machinery Transactions on KnowledgeDiscovery from Data 17, 1(2023), 1-19.

[0102] [3]Dennis Diefenbach,Vanessa Lopez,Kamal Singh,and PierreMaret.2018.Core techniques of question answering systems over knowledgebases:a survey.Knowledge and Information systems 55(2018),529-569.

[0103] [4]Wen-tau Yih,Ming-Wei Chang,Xiaodong He,andJianfengGao.2015.Semantic Parsing via Staged Query Graph Generation:QuestionAnswering with Knowledge Base.In Proceedings of the 53rd Annual Meeting ofthe Association for Computational Linguistics and the 7th InternationalJointConference on Natural Language Processing.1321-1331.

[0104] [5]Yanguang Chen,Yuanyuan Sun,Zhihao Yang,and Hongfei Lin.2020.JointEntity and Relation Extraction for Legal Documents with Legal FeatureEnhancement.In Proceedings of the 28th International Conference onComputational Linguistics.1561-1571.

[0105] [6]Vishal Gupta and Gurpreet Singh Lehal.2010.A survey of textsummarization extractive techniques.Journal of emerging technologies in webintelligence 2,3(2010),258-268.

[0106] [7]David Berthelot,Nicholas Carlini,Ian Goodfellow,Avital Oliver,Nicolas Papernot,and Colin Raffel.2019.MixMatch:a holistic approach to semi-supervised learning.In Proceedings of the 33rd International Conference onNeural Information Processing Systems.5049-5059.

[0107] [8]Kihyuk Sohn,David Berthelot,Nicholas Carlini,Zizhao Zhang,HanZhang,Colin A Raffel,Ekin Dogus Cubuk,Alexey Kurakin,and Chun-LiangLi.2020.Fixmatch:Simplifying semi-supervised learning with consistency andconfidence.Advances in neural information processing systems 33(2020),596-608.

[0108] [9]Xiaohua Zhai,Avital Oliver,Alexander Kolesnikov,and LucasBeyer.2019.S4L:Self-Supervised Semi-Supervised Learning.In Proceedings of the2019 Institute of Electrical and Electronics Engineers / Computer VisionFoundation International Conference on Computer Vision.1476-1485.

[0109]

[10] Jacob Devlin,Ming-Wei Chang,Kenton Lee,and KristinaToutanova.2019.BERT:Pre-training of Deep Bidirectional Transformers forLanguage Understanding.In Proceedings of the 2019 Conference of the NorthAmericarn Chapter of the Association for Computational Linguistics:HumanLanguage Technologies.4171-4186.

[0110]

[11] Yupeng Chang,Xu Wang,Jindong Wang,Yuan Wu,Linyi Yang,Kaijie Zhu,Hao Chen,Xiaoyuan Yi,Cunxiang Wang,Yidong Wang,et al.2024.A survey onevaluation of large language models.Association for Computing Machinerytransactions on intelligent systems and technology 15,3(2024),1-45.

[0111]

[12] Mohaimenul Azam Khan Raiaan,Md Saddam Hossain Mukta,Kaniz Fatema,Nur Mohammad Fahad,Sadman Sakib,Most Marufatul Jannat Mim,Jubaer Ahmad,Mohammed Eunus Ali,and Sami Azam.2024.A review on large language models:Architectures,applications,taxonomies,open issues and challenges.Institute ofElectrical and Electronics Engineers access 12(2024),26839-26874.

[0112]

[13] Xiangrong Zeng,Daojian Zeng,Shizhu He,Kang Liu,and JunZhao.2018.Extracting relational facts by an end-to-end neural model with copymechanism.In Proceedings of the 56th Annual Meeting of the Association forComputational Linguistics,pages 506-514.

[0113]

[14] Harsha Gurulingappa,Abdul Mateen Rajput,Angus Roberts,JulianeFluck,Martin Hofmann-Apitius,and Luca Toldo.2012.Development of a benchmarkcorpus to support the automatic extraction of drugrelated adverse effectsfrom medical case reports.Journal of biomedical informatics,45(5):885-892.

[0114]

[15] Yi Luan ，Luheng He,Mari Ostendorf,and HannanehHajishirzi.2018.Multi-task identification of entities,relations,andcoreference for scientific knowledge graph construction.In Proceedings of the2018 Conference on Empirical Methods in Natural Language Processing,pages3219-3232,Brussels,Belgium.Association for Computational Linguistics.

[0115]

[16] X.Li,F.Yin,Z.Sun,X.Li,A.Yuan,D.Chai,M.Zhou,and J.Li,“Entity-relation extraction as multi-turn question answering,”in Proceedings of the57th Annual Meeting of the Association for Computational Linguistics,pp.1340-1350,2019.

[0116]

[17] Zhiheng Yan,Chong Zhang,Jinlan Fu,Qi Zhang,and Zhongyu Wei.2021.APartition Filter Network for Joint Entity and Relation Extraction.InProceedings of the 2021 Conference on Empirical Methods in Natural LanguageProcessing.185-197.

[0117]

[18] Yu-Ming Shang,Heyan Huang,and Xianling Mao.2022.OneRel:JointEntity and Relation Extraction with One Module in One Step.Association forthe Advancement of Artificial Intelligence Conference on ArtificialIntelligence 36,10(2022),11285-11293.

[0118]

[19] Markus Eberts and Adrian Ulges.2020.Span-based joint entity andrelation extraction with transformer pre-training.In Proceedings of theEuropean Conference on Artificial Intelligence 2020.2006-2013.

Claims

1. A joint entity relationship extraction method based on semi-supervised learning and a large language model. The goal of joint entity relationship extraction is to extract entity pairs and their relationships in the text and integrate them into relationship triples (subject, relationship, object). The following are the steps to build a large language model-enhanced semi-supervised joint extraction model: 1) Data augmentation using a fine-tuned large language model; 2) Use the original annotated dataset to train a preliminary entity relationship extraction model; 3) Use the preliminarily trained extraction model to pseudo-label the unlabeled data; 4) Filter high-confidence labels; 5) Train the model using the relabeled training data; 6) Use the fine-tuning data to fine-tune and update the parameters of the large language model; 7) Use the final high-confidence labeled training data to train a joint entity relationship extraction model. In step 1), the large language model is a neural network model with large-scale parameters. During data augmentation, a large language model is applied to rewrite natural language sentences without changing the semantics of the original sentences. This primarily involves changing the form and format of the expressions. For unlabeled instances, weak augmentation A(·) and strong augmentation S(.) are used. Weak augmentation is applied to the original unlabeled data to maintain semantic integrity and avoid complex perturbations of the core content. These minimally perturbed samples are filtered based on confidence to generate reliable pseudo-labels. Strong augmentation utilizes a fine-tuned large language model for context-aware text rewriting. Specifically: Assume x i ∈D U represents unlabeled input data, A(x i ) and S(x i ) represent the weakly enhanced version and the strongly enhanced version of this unlabeled data respectively. In step 2), when the entity relationship extraction model is trained through semi-supervised learning, the extraction model is trained based on the input data (x, y), specifically: Assume x i ∈D U Represents the input data, and y represents its unique true label. When using semi-supervised learning to train a model, the goal is to optimize the distance between the model prediction results and the true value, that is, close to The cross entropy loss function of the supervised part of semi-supervised learning is expressed as: where y m (p(s, r, o)|x i ) represents the model's prediction result for the input data, and the model's prediction result is calculated with its true label by cross entropy loss. In step 3), in order to utilize the large amount of unlabeled instance data in the original dataset, after supervised training, the weakly enhanced data is extracted using the preliminarily trained extraction model; and a fixed threshold method is used, specifically: Among them, τ represents the threshold constant; τ varies for data sets with different numbers of categories; only when the probability predicted by the model is greater than or equal to the set threshold is it considered a pseudo label. In step 4), after pseudo-labeling the unlabeled data in step 3), a portion of the unlabeled samples is used as labeled data. If the probability of the label obtained after model prediction is less than a predetermined threshold, it is considered that the model cannot extract the true label of this sample in the current iteration; In the next iteration, the pseudo-labeling operation is still performed on the sample. In step 5), the more highly confident labeled data obtained in step 4) are combined with the original labeled data to form a larger training set, and the extraction model is further trained. The cross entropy loss function of the unsupervised part of semi-supervised learning is expressed as: where y m (p|S(x j )) is the prediction result of the model for the strongly enhanced sample, It is the pseudo label assigned by the model, and μN is the number of unlabeled samples. In step 6), the results of the unsupervised training guide the fine-tuning of the large language model, primarily using the cross-entropy loss of the unsupervised portion as a filtering criterion to screen the data augmented by the large language model. Only high-quality augmented sentences are used in the fine-tuning data, retaining only the high-quality output of the large language model. The loss function of the semi-supervised framework facilitates the fine-tuning of the large language model, enabling it to generate output that is more consistent with the expected results. This iterative process enhances the model's ability to handle limited annotated data, thereby achieving more accurate and robust performance in entity recognition and relationship extraction tasks.

2. The method for extracting joint entity relationships based on semi-supervised learning and enhanced by a large language model according to claim 1, characterized in that In step 1), the relationship extraction network model is highly integrated.

3. The method for extracting joint entity relationships based on semi-supervised learning and enhanced by a large language model according to claim 1, characterized in that In step 1), the objective function of the joint entity relation extraction model aims to optimize the end-to-end probability of predicting these triples through hierarchical probability decomposition.

Images