Chinese small sample entity relation joint extraction method and system based on large language model
By combining large language models and comparison learning technology in a small sample data environment, data scarcity and overfitting problems are solved, and entity relationship extraction performance and generalization capabilities are improved in the environment of labeled data scarcity, suitable for professionals Small sample data for the field.
Patent Information
- Application Number
- CN202410951777.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-12
AI Technical Summary
In the professional fields where data is scarce, the existing entity relationship extraction technology performs poorly, mainly due to data scarcity, overfitting problems and category imbalance, resulting in a significant reduction in the performance of the model in small sample scenarios.
The Chinese small sample entity relationship joint extraction method based on large language models is adopted to expand the sample size through data augmentation technology, and a comparative learning fusion entity pairs and relationship characteristics are used in the relationship prediction module to generate entity relationship triples to improve identification performance.
It effectively improves the model performance in the environment of scarce labeled data, reduces the risk of model overfitting, enhances generalization ability, and can handle diversified and ultra-long text data, especially suitable for small sample data environments in professional fields.
Smart Images

Figure CN118940834A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method and system for jointly extracting Chinese small sample entity relationships based on a large language model. Background Art
[0002] In the field of natural language processing, entity relationship extraction is a key technology for identifying entities and their relationships from unstructured text, and is also the basis for building knowledge graphs. Existing entity relationship extraction technologies perform well in environments with sufficient data, but their performance is significantly reduced in professional fields with less labeled data.
[0003] This is mainly due to:
[0004] 1. Data scarcity: In specific fields or small sample scenarios, it is difficult to obtain sufficient labeled data, making traditional model training difficult.
[0005] 2. Overfitting problem: Small sample data can easily lead to model overfitting and poor generalization ability.
[0006] 3. Category imbalance: In the process of identifying entity relationships, samples of certain relationship categories are extremely scarce, which affects the learning efficiency and effect of the model. Summary of the invention
[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Chinese small sample entity relationship joint extraction method and system based on a large language model, which utilizes the advantages of large language models in entity labeling and combines them with small sample relationship extraction methods to improve recognition performance in an environment where labeled data is scarce.
[0008] In order to achieve the above-mentioned purpose, the present invention proposes a method and system for joint extraction of Chinese small sample entity relations based on a large language model. The method expands the sample size for model fine-tuning through data enhancement technology, thereby improving the performance of large language models in entity pair recognition. In the relationship prediction module, contrastive learning is used to achieve effective fusion and learning of entity pairs and relationship features, thereby completing the extraction of entity relations and generating entity relationship triplets. These triplets can be used to construct knowledge graphs in professional fields, providing strong support for related research and applications.
[0009] The technical solution adopted by the present invention is specifically as follows:
[0010] A Chinese small sample entity relationship joint extraction method based on a large language model, comprising:
[0011] Step 1: After data enhancement, the Chinese small sample data is divided into a support set and a query set;
[0012] Step 2: for the support set, sample a number of texts and triples; for the texts in the query set, use a large language model to perform entity pair recognition on the texts to obtain texts with entity pair identifiers;
[0013] Step 3, for the text and the triple, using a pre-trained relation encoder to obtain a relation feature representation, using a pre-trained text encoder to obtain an entity pair feature representation, and fusing the relation feature representation with the entity pair feature representation to obtain a fused feature representation;
[0014] Step 4, for the text with entity pair identifiers, using the pre-trained text encoder to obtain its entity pair feature representation;
[0015] Step 5, calculate the similarity between the fused feature representation and the entity pair feature representation obtained in step 4, take the relationship with the highest similarity as the relationship of the corresponding text in the query set, and finally obtain the entity relationship triplet.
[0016] The present invention also provides a system for implementing the method for jointly extracting Chinese small sample entity relations based on a large language model, comprising:
[0017] The data enhancement module is used to enhance the input Chinese small sample data and divide the enhanced data into a support set and a query set;
[0018] A sampling module, sampling the support set to obtain a number of texts and triples;
[0019] An entity pair recognition module, for the text in the query set, performs entity pair recognition on the text using a large language model to obtain text with entity pair identifiers;
[0020] The relationship prediction module includes a pre-trained relationship encoder, a pre-trained text encoder, a fusion module and a similarity calculation module. The text and the triple are represented by a relationship feature through the pre-trained relationship encoder, and an entity pair feature representation is obtained through the pre-trained text encoder. The fusion module fuses the relationship feature representation and the entity pair feature representation to obtain a fused feature representation; the text with the entity pair identifier obtains its entity pair feature representation through the pre-trained text encoder, and the similarity calculation module calculates the similarity between the fused feature representation and the entity pair feature representation of the text with the entity pair identifier, takes the relationship with the highest similarity as the relationship of the corresponding text in the query set, and finally obtains the entity relationship triple.
[0021] Compared with the prior art, the beneficial effects of the present invention include:
[0022] 1. This paper effectively improves the model performance in an environment with scarce labeled data by integrating a large language model with a small sample comparative learning strategy.
[0023] 2. The present invention adopts a self-verification strategy and a dual encoder architecture to effectively reduce the risk of model overfitting and enhance the generalization ability of the model when processing new data.
[0024] 3. The method and system proposed in the present invention can effectively process diverse and ultra-long text data, and are particularly suitable for small sample data environments in professional fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is the system framework of the present invention.
[0026] Figure 2 This is an example of a data augmentation method.
[0027] Figure 3 Identify examples for entity pairs.
[0028] Figure 4 This is an example of an entity self-authentication strategy.
[0029] Figure 5 It is a dual encoder pre-training method.
[0030] Figure 6 It is the entity relationship joint extraction process. DETAILED DESCRIPTION
[0031] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.
[0032] The present invention is a Chinese small sample entity relationship joint extraction method (LER-SSC) based on a large language model and a system, which combines a large language model with a small sample learning technology. Figure 1 The overall architecture of the system is presented. The framework of the system consists of three core components: data enhancement module, entity pair recognition module and relationship prediction module.
[0033] First, the small sample data is processed by the data enhancement module to obtain a richer data set, which is divided into a support set S and a query set Q. The input data is extracted into an N-way-K-shot small sample form.
[0034] Then, the entity pair recognition module generates entity pair identifiers through a large language model and performs entity self-verification to obtain entity pairs.
[0035] Next, the relationship prediction module uses two pre-trained encoders (relation encoder and text encoder) to build a shared representation space based on the concept of multi-view contrastive learning. By evaluating the similarity between the relationship and entity pair embedding vectors in this space, relationship prediction is achieved, and finally the entity relationship triples are obtained.
[0036] According to the above basic process, the implementation steps of the present invention can be further described as follows in combination with the above modules:
[0037] 1. Data Enhancement Module
[0038] The present invention performs data enhancement on the Chinese small sample data and divides it into a support set and a query set. Specifically, in order to address the problem of scarce labeled data, the data enhancement module can use data enhancement methods such as large language models to expand the sample data to improve the performance of subsequent model fine-tuning.
[0039] The richness of training data plays a decisive role in the performance of the model. In order to overcome the problem of insufficient training data, the present invention uses a large language model to perform text data enhancement to expand the training data set, thereby enhancing the generalization ability of the model, which mainly involves entity pair labeling of the original training text and using a large language model to perform text data enhancement, as follows:
[0040] (1) Symbols for marking entity pairs in text. Special tags “([Sub,\Sub,Obj,\Obj])” are used to highlight the location of entity pairs in the training text. For example, the original text “Company A was founded by Zhang XX.” Adding special tags would be: “[Sub]Company A[\Sub] was founded by [Obj]Zhang XX Large Language Model[\Obj].”
[0041] (2) Enhance text data and give full play to the excellent performance of large language models in text generation. The main methods are as follows: Figure 2 As shown, where:
[0042] Entity pair symbol replacement: uniformly use "entity A" and "entity B" to replace the head entity and tail entity in the text. This strategy helps the model understand the syntactic structure of the text more deeply.
[0043] Text reorganization, which uses large language models to adjust the order of sentences or paragraphs while maintaining the original meaning of the text;
[0044] Random deletion: By setting a certain probability P, randomly delete words in the text to encourage the model to infer the deleted information from the remaining text context;
[0045] Text generation, based on a given text, uses a large language model to recreate new text containing different entity pairs.
[0046] These enhanced text data can not only improve the model's adaptability to changes in sentence structure, but also help the model learn and master multiple ways of expressing the same information, thereby enhancing the model's generalization ability for unknown data.
[0047] 2. Entity Pair Recognition Module
[0048] Although large language models have achieved the best performance in various natural language processing tasks, their performance in the field of entity recognition is still inferior to supervised baseline models. This is mainly due to the essential differences between named entity recognition models and large language models: named entity recognition is a sequence labeling task that requires accurate identification and labeling of entities in text; while large language models mainly focus on text generation, and their original design intention is not to focus on this kind of refined labeling work. To this end, the present invention converts the sequence labeling task into a generation task that is easy for large language models to handle, thereby effectively bridging the gap between the two.
[0049] The present invention uses a large language model to perform entity pair recognition on the text in the query set to obtain text with entity pair identifiers. This can be achieved by building prompts. The strategy can be divided into three main stages:
[0050] First, for the input text X, a specific prompt is constructed for it, denoted as Prompt(X); then, this constructed prompt is provided to a large language model to obtain a generated text sequence; finally, the text sequence W = {w1, w2, ..., wn} is converted into a set of corresponding entity tags to obtain the final result.
[0051] Figure 3 What is shown is an example of building a specific prompt for the input text. Its structure can be divided into task description and text input, where:
[0052] The task description provides a macro-task concept for the large language model, telling the large language model what examples to refer to when generating output. It clarifies that the task is to add tags to the text. It generally contains three fixed sentences, which are:
[0053] “You are a great linguist”: tells large language models that they need to apply linguistic knowledge when generating output;
[0054] “Please add special tags to the text [sub][ / sub][obj][ / obj]”: For large language models, the specific task is to add tags to the text [sub][ / sub][obj][ / obj];
[0055] Text input, such as Figure 3 as shown in the bottom portion of the .
[0056] Output: The large language model generates tags for the input text to identify the entities in the text. For example, the text to be processed "Li Simin works at China Telecom" is provided to the large language model to generate the tagged text: "[Obj]Large language model Li Simin Large language model[\Obj]Large language model in Large language model[Sub]Large language model China Telecom Large language model[\Sub]Large language model works". In this example, the head and tail entity pairs "China Telecom" and "Li Simin" are successfully identified.
[0057] A large number of studies have shown that LLM has hallucination or over-prediction problems when processing information. In order to correct this error, the present invention adopts a self-verification strategy, which requires LLM to use the self-verification strategy to perform entity self-verification on the generated entity pair identification after marking the entity pair, and further confirm whether the entity is correct. The confirmation method is to answer a simple yes or no question.
[0058] Self-verification strategy design Figure 4 As shown, the process is as follows:
[0059] The prompt begins with a task description: that is, the task is described as verifying whether the head and tail entity pairs in the text are correct. In order to improve the accuracy of self-verification, some demonstration examples are provided. In the demonstration examples, the answer is yes or no. These examples are Figure 4 Each example consists of three parts: first, the given text, such as "Company A was founded by Zhang XX." Then, the question is asked: "Are Company A and Zhang XX the head and tail entities in the text respectively? Answer "yes" or "no"; finally, the answer is "yes" or "no". In the small sample scenario, multiple such demonstration examples are integrated into the prompt. After the demonstration, there are examples to be tested, which are then input into the large language model to obtain its output judgment.
[0060] In view of data security considerations and the control of actual application costs, the present invention adopts a fine-tuning approach to improve the performance of large open source language models in entity pair recognition tasks, so that they can better adapt to specific task requirements. In addition, due to the large number of parameters of large language models, full parameter training is not only costly, but also difficult to widely use. Therefore, the present invention provides the enhanced training set to the large open source language model, and uses the LoRA method for fine-tuning training, so that the large language model can better adapt to and understand this specific data set.
[0061] The entity pair recognition module of the present invention uses a large language model to convert the text sequence labeling task into a generation task, and can improve the accuracy of labeling by building prompts and using self-verification strategies. In order to match the feature extraction of the relationship prediction module, a sampling module is also required to sample a number of texts and triples from the support set. The sampling module can be a separate module or a module integrated with the entity pair recognition module.
[0062] 3. Relationship Prediction Module
[0063] The method of the relationship prediction module mainly includes dual encoder pre-training, and uses the pre-trained dual encoder to perform feature extraction and relationship prediction, and may further include model fine-tuning.
[0064] The pre-trained dual encoder is used for feature extraction and relationship prediction. The main method is: for the text and triples, the pre-trained relationship encoder is used to obtain the relationship feature representation, the pre-trained text encoder is used to obtain the entity pair feature representation, and the relationship feature representation is fused with the entity pair feature representation to obtain a fused feature representation. For the text with entity pair identifiers, the pre-trained text encoder is used to obtain its entity pair feature representation. The similarity between the fused feature representation and the entity pair feature representation of the text with entity pair identifiers is calculated, and the relationship with the highest similarity is taken as the relationship of the corresponding text in the query set, and finally the entity relationship triple is obtained.
[0065] In the pre-training stage, supervised corpus data (typically Wikipedia corpus data set) is mainly used to perform SCL (symmetric contrast learning) and MLM (masked pre-training) on the BERT model to obtain a pre-trained relational encoder and a pre-trained text encoder to form a dual encoder structure, where the corpus data set consists of a relational dictionary and a text collection. In the fine-tuning stage, small sample data is mainly used to fine-tune the model parameters.
[0066] The method in the pre-training stage is described as follows:
[0067] 1. Dual encoder pre-training based on BERT
[0068] By pre-training the BERT model, we build a text encoder that focuses on learning text entity pair features and a relation encoder that focuses on learning relation features.
[0069] (1) Pre-training dataset
[0070] A large dataset is constructed using the Wikipedia corpus for the pre-training task of this module. The dataset consists of two parts: one is the relational dictionary R, which contains R 1 ,R 2 ,....,R metc., and a text set T, including T 1 ,T 2 ,....,T n , each text contains specific information. Where m represents the number of relationship categories, and n represents the number of texts.
[0071] (2) Pre-training method
[0072] The purpose of pre-training is to build a dedicated encoder for different feature learning based on the BERT model, which specifically includes two tasks: symmetric contrastive learning (SCL) and masked language modeling (MLM). Figure 5 shown.
[0073] Small batches of data containing n texts and m relations are used in training. The goal is to accurately predict these n×m possible actual combinations (text-relationships). To achieve this goal, a contrastive learning mechanism is used in the model (i.e., the dual encoder structure) to build a robust and unified representation space by collaboratively training the text encoder and the relationship encoder. The training goal is to increase the embedding cosine similarity of the n correct combinations to the maximum value, while ensuring that the embedding cosine similarity of the n×mn incorrect combinations is reduced to the minimum value. In order to improve the contrast effect, a symmetric contrast target is adopted in the embodiment of the present invention, that is, contrastive learning includes two parts: text-based contrastive learning (SCLt) and relationship-based contrastive learning (SCLr). This symmetric contrast target design is intended to ensure that the training framework can accurately capture the deep connection between text and relationship, thereby optimizing the performance of downstream small sample relationship prediction tasks.
[0074] In text-based contrastive learning (SCLt), each individual text t i is selected as a reference point, and positive and negative samples are screened from the relationship label set. Specifically, for the determined positive pair (t i ,r i ) in the text t i , t i Paired with all other remaining relations to form negative pairs (t i ,r i ), where r i t i The matching relationship, r j Indicates the difference between r i In this way, a text-centric contrastive learning framework can be constructed, which aims to optimize the representation ability of the model by distinguishing positive and negative samples. The training loss is expressed as:
[0075]
[0076] in, express and The cosine similarity between them, τ represents the temperature.
[0077] Accordingly, in relation-based contrastive learning, each independent relation is selected as a reference point, and positive and negative samples are screened from the text collection. Specifically, for a given relation r i , the set A contains all relations r i The positive sample of the present invention is the present invention, while the negative sample is the relationship is not r i In this way, a relation-centric contrastive learning framework can be constructed, which aims to optimize the representation ability of the model by distinguishing positive and negative samples. The training loss is expressed as:
[0078]
[0079] For any relational benchmark, all positive texts in a mini-batch will have a positive impact on the molecule. Therefore, the loss function L SCLr This will encourage the encoder to align all pairs of representations tightly, thereby building a more stable clustering of the representation space, which will make the encoder more stable when updating.
[0080] In these two contrastive learning tasks, there is a complementary relationship between each other. The final contrastive training loss calculation of the framework is based on the combination of these two losses, expressed as: L SCL =L SCLt +L SCLr .
[0081] In order to maintain BERT's language understanding ability and alleviate the problem of catastrophic forgetting, the present invention introduces masked language modeling (MLM) in the pre-trained relational encoder and the pre-trained text encoder. The core idea of the MLM objective is to randomly mask certain tokens or words in the text, and then predict these masked tokens based on their surrounding context. In the dual encoder architecture of the present invention, two training loss functions are set for the MLM task, namely L MLMt and L MLMr These two loss functions are used to train the MLM task for text encoder and relation encoder respectively.
[0082] L MLMt and L MLMr The formulas are as follows:
[0083]
[0084] Where m(Xt) represents the masked word in the text encoder input sentence, and Xt-m(Xt) represents the remaining words in the text encoder input sentence after removing the masked words. Represents the word at the masked position of the encoder output. Represents the probability that the text encoder correctly outputs the masked word.
[0085]
[0086] Where m(Xr) represents the masked word in the input sentence of the relational encoder, and Xr-m(Xr) represents the remaining words in the input sentence of the relational encoder after removing the masked words. Represents the word at the masked position of the encoder output. Represents the probability that the relation encoder correctly outputs the masked word.
[0087] The loss function of the masked language model is finally defined as follows: L MLM =L MLMt +L MLMr .
[0088] In order to balance L SCL and L MLM The two losses are calculated in a weighted manner. That is, the total loss of the dual encoder architecture of the text encoder and the relation encoder is expressed as: L = α × L SCL +β×L MLM , where α and β are L SCL and L MLM The weight coefficient of .
[0089] By using supervised Wikipedia corpus data to pre-train the BERT-based dual encoder, we construct feature learning from different perspectives of the data, thereby obtaining a text encoder that focuses on learning text entity pairs and a relation encoder that focuses on learning relation features. The two correspond to each other and are completely independent. Next, in order to further improve the prediction accuracy, we need to use small sample data {t n ,r m} Fine-tune the parameters of the dual encoder architecture. The training data includes the text set T and the predefined relation dictionary R. Given a training example mini-batch sample {t n ,r m}, where t n From the text set T, r m From the predefined relation dictionary R, the fine-tuning method is described as follows:
[0090] (1) Text entity pair encoding
[0091] In entity pair recognition, the present invention uses specific markers to highlight the location of entities in the input text. These markers are also used in the representation of text feature vectors. The feature vector corresponding to the input sequence is represented as:
[0092]
[0093] Among them, l S represents the length of the input sequence, T i f Represents the vector obtained after a token of the input sequence passes through the text encoder, represents the text entity pair feature vector obtained after the input sequence passes through the text encoder, b and e correspond to the positions of special tags [Sub] and [Obj] respectively, and ⊕ represents the connection operation; The embedding vector representing the start token of the input sequence, represents the embedding vector of the jth token in the input sequence.
[0094] (2) Relational Coding
[0095] The goal of the relation encoder is to generate a i Generate relation feature vector
[0096]
[0097] Among them, l R Indicates the length of the relation dictionary, represents the embedding vector of the jth token in the input sequence, The embedding vector representing the start token of the input sequence, Represents the vector obtained after a token of the input sequence passes through the relational encoder, represents the relational feature vector obtained after the input sequence passes through the relational encoder, d is the dimension of the embedding vector, YesR i The average of all token embedding vectors in , ⊕ represents the concatenation operation.
[0098] (3) Fusion feature representation
[0099] In order to minimize the introduction of additional linear layers (or parameters), the text feature representation is directly concatenated and relational feature representation Fuse together to obtain support instance S i The final representation P i final :
[0100]
[0101] In the final relationship prediction, the present invention calculates the relationship between instance q and each supporting instance The similarity between them is calculated using vector cosine similarity, and the index of the maximum value is selected as the relationship prediction result:
[0102]
[0103] Use cross entropy (CE) loss as the loss function: L CE = -log(z y ), where y is the relationship label, z y The prediction is the estimated probability of relation y.
[0104] Therefore, the relationship prediction module of the present invention adopts a dual encoder pre-training method, including symmetric contrast learning and mask pre-training, to construct a shared representation space, realize the effective fusion of entity pair features and relationship features and independent feature learning, and use small sample data to fine-tune the model parameters to improve the accuracy of relationship prediction.
[0105] According to the above process, the present invention uses the LER-SSC model to perform the text entity relationship joint extraction process. This task can be completed in two stages. In the first stage, the model is fine-tuned using the training set, mainly fine-tuning the open source large language model and the dual encoder (including text encoder and relationship encoder) in the relationship prediction module. After completing the fine-tuning training of the model, it enters the second stage, that is, the joint extraction of entity relationships. The overall process is as follows: Figure 6 The detailed steps are as follows:
[0106] 1. Select a method to tag the entity pairs of text A. You can choose to call the API interface of LLM models such as GPT for processing, but you need to provide LLM with example samples from the training set; or directly use the local chatGLM2-6B model that has been fine-tuned and trained.
[0107] 2. Use the selected large language model to perform entity pair tagging on text A and generate text A′ with entity pair tags.
[0108] 3. Use the text encoder to vectorize the text A′ and obtain the text vector representation E_A.
[0109] 4. For the labeled samples in the support set, use the text encoder and relationship encoder to convert their text and relationship into vector representations respectively, and obtain the text vector representation E_B and the relationship vector representation E_R.
[0110] 5. Concatenate E_B and E_R to form a fused vector representation E_C.
[0111] 6. Calculate the similarity between E_A and E_C.
[0112] 7. Repeat steps 4-6 until the vector similarity calculation is completed between the texts with different relations in the support set and text A′.
[0113] 8. Select the text with the highest vector similarity to text A′, and determine that the relationship P contained in the text is the relationship in text A′.
[0114] 9. Finally, combine the marked entity pairs (S, P) in the text A′ with the determined relation P and output the triple (S, P, O).
Claims
1. A Chinese small sample entity relationship joint extraction method based on a large language model, characterized in that: include: Step 1: After data enhancement, the Chinese small sample data is divided into a support set and a query set; Step 2: for the support set, sample a number of texts and triples; for the texts in the query set, use a large language model to perform entity pair recognition on the texts to obtain texts with entity pair identifiers; Step 3, for the text and triples, using a pre-trained relation encoder to obtain a relation feature representation, using a pre-trained text encoder to obtain an entity pair feature representation, and fusing the relation feature representation with the entity pair feature representation to obtain a fused feature representation; Step 4, for the text with entity pair identifiers, using the pre-trained text encoder to obtain its entity pair feature representation; Step 5, calculate the similarity between the fused feature representation and the entity pair feature representation obtained in step 4, take the relationship with the highest similarity as the relationship of the corresponding text in the query set, and finally obtain the entity relationship triplet.
2. According to the method for joint extraction of Chinese small sample entity relations based on a large language model according to claim 1, it is characterized in that: The step 1 uses a large language model data enhancement method to perform data enhancement on the Chinese small sample data, including: Entity pair symbol replacement, using entity A and entity B to replace the head entity and tail entity in the text respectively; Text reorganization, which uses a large language model to adjust the order of sentences or paragraphs while maintaining the original meaning of the text; Random deletion, set the probability P and randomly delete words in the text; Text generation, based on a given text, uses a large language model to recreate new text containing different entity pairs.
3. According to the method for joint extraction of Chinese small sample entity relations based on a large language model according to claim 1, it is characterized in that: The step 2, using a large language model to perform entity pair recognition on the text in the query set to obtain text with entity pair identifiers, includes: First, construct a specific prompt for the input text; Then, the specific prompt is fed into a large language model to obtain a generated text sequence; Finally, the text sequence is converted into a set of corresponding entity labels to obtain the final result.
4. The Chinese small sample entity relationship joint extraction method based on a large language model according to claim 3 is characterized in that: The method for constructing a specific prompt for the input text is as follows: Task description, which tells the large language model what examples to refer to when generating output, and makes it clear to the large language model that the task is to add tags to the text; Text input; Output: A large language model generates tags for the input text to identify entities in the text.
5. The Chinese small sample entity relationship joint extraction method based on a large language model according to claim 1, 2, 3 or 4, characterized in that: In step 2, the self-verification strategy is used to perform entity self-verification on the generated entity pair identifier, and the method is as follows: The task is described as verifying whether the head-tail entity pairs in the text are correct; Provide demonstration examples, answer yes or no; Feed the test example into a large language model to get its output verdict.
6. The Chinese small sample entity relationship joint extraction method based on a large language model according to claim 1 is characterized in that: The pre-trained relation encoder and the pre-trained text encoder are obtained by performing symmetric contrast learning and mask pre-training on the BERT model using a supervised corpus dataset; The corpus dataset consists of a relational dictionary and a text collection, and the pre-training method is as follows: A small batch of data containing n texts and m relations is used for training. A contrastive learning mechanism is applied in the model. By co-training the text encoder and the relation encoder, a robust and unified representation space is constructed to maximize the embedding cosine similarity of n correct combinations and minimize the embedding cosine similarity of n×mn incorrect combinations as the training target, thereby accurately predicting n×m possible actual text-relation combinations. The contrastive learning includes text-based contrastive learning and relationship-based contrastive learning; in text-based contrastive learning, each independent text is selected as a reference point, and positive samples and negative samples are screened from a relationship tag set to construct a text-centric contrastive learning framework; in relationship-based contrastive learning, each independent relationship is selected as a reference point, and positive samples and negative samples are screened from a text set to construct a relationship-centric contrastive learning framework.
7. The Chinese small sample entity relationship joint extraction method based on a large language model according to claim 6 is characterized in that: In the text-based contrastive learning, each independent text is selected as a reference point, and positive and negative samples are screened from the relationship tag set. i , r i ) in the text t i , t i Paired with all other remaining relations to form negative pairs (t i , r j ), where r i t i The matching relationship, r j Indicates the difference between r i Any of the other m-1 relations except ; the training loss is expressed as: in, Table and The cosine similarity between , τ represents the temperature; In relation-based contrastive learning, each independent relation is selected as a reference point, and positive and negative samples are screened from the text collection. i , the set A contains all relations r i The positive sample of the present invention is the present invention, while the negative sample is the relationship is not r i The text composition of; the training loss is expressed as: The final contrastive training loss is expressed as: L SCL =L SCLt +L SCLr .
8. The Chinese small sample entity relationship joint extraction method based on a large language model according to claim 7 is characterized in that: Introduce masked language modeling in the pre-trained relation encoder and the pre-trained text encoder, and set the training loss function L MLMt and L MLMr The masked language modeling tasks of the text encoder and relation encoder are trained separately, L MLMt and L MLMr The formulas are as follows: Where m(Xt) represents the masked word in the text encoder input sentence, and Xt-m(Xt) represents the remaining words in the text encoder input sentence after removing the masked words. Represents the word at the masked position of the encoder output, Represents the probability that the text encoder correctly outputs the masked word; Where m(Xr) represents the masked word in the input sentence of the relational encoder, and Xr-m(Xr) represents the remaining words after removing the masked words from the input sentence of the relational encoder. Represents the word at the masked position of the encoder output, Represents the probability that the relation encoder correctly outputs the masked word; The final loss function of the masked language model is defined as follows: L MLM =L MLMt +L MLMr ; The total loss of the dual encoder architecture of the text encoder and the relation encoder is expressed as: L = α × L SCL +β×L MLM , where α and β are L SCL and L MLM The weight coefficient of .
9. The Chinese small sample entity relationship joint extraction method based on a large language model according to claim 7 or 8, characterized in that: Using small sample data {t n , r m Fine-tune the parameters of the dual encoder architecture to improve the accuracy of relationship prediction, where t n From the text set T, r m From the predefined relation dictionary R, the fine-tuning method is as follows: (1) Text entity pair encoding The feature vector corresponding to the input sequence is represented as: Among them, l S represents the length of the input sequence, Represents the vector obtained after a token of the input sequence passes through the text encoder, represents the text entity pair feature vector obtained after the input sequence passes through the text encoder, b and e correspond to the positions of the special tags [Sub] and [Obj] respectively, Represents a connection operation; The embedding vector representing the start token of the input sequence, represents the embedding vector of the jth token in the input sequence; (2) Relational Coding The goal of the relation encoder is to generate a i Generate relationship feature vector Among them, l R Indicates the length of the relation dictionary, represents the embedding vector of the jth token in the input sequence, The embedding vector representing the start token of the input sequence, Represents the vector obtained after a token of the input sequence passes through the relational encoder, represents the relational feature vector obtained after the input sequence passes through the relational encoder, d is the dimension of the embedding vector, YesR i The average of all token embedding vectors in , Represents a connection operation; (3) Fusion feature representation The text features are represented by direct cascade operations and relational feature representation Fuse together to obtain support instance S i The final expression 10. A Chinese small sample entity relationship joint extraction system based on a large language model, used to implement the Chinese small sample entity relationship joint extraction method based on a large language model as described in claim 1, characterized in that: include: The data enhancement module is used to enhance the input Chinese small sample data and divide the enhanced data into a support set and a query set; A sampling module, sampling the support set to obtain a number of texts and triples; An entity pair recognition module, for the text in the query set, performs entity pair recognition on the text using a large language model to obtain text with entity pair identifiers; The relationship prediction module includes a pre-trained relationship encoder, a pre-trained text encoder, a fusion module and a similarity calculation module. The text and the triple are represented by a relationship feature through the pre-trained relationship encoder, and an entity pair feature representation is obtained through the pre-trained text encoder. The fusion module fuses the relationship feature representation and the entity pair feature representation to obtain a fused feature representation; the text with the entity pair identifier obtains its entity pair feature representation through the pre-trained text encoder, and the similarity calculation module calculates the similarity between the fused feature representation and the entity pair feature representation of the text with the entity pair identifier, takes the relationship with the highest similarity as the relationship of the corresponding text in the query set, and finally obtains the entity relationship triple.
Citation Information
Patent Citations
Entity relationship extraction method and device, electronic equipment and storage medium
CN113221566A
Small sample medical entity identification method based on boundary and mutual information enhancement
CN116976350A
Double-encoder pre-training small sample relation extraction method based on comparative learning
CN117131873A
Large model medical training data generation method and system based on multistage semantics
CN117637175A
Cited By
General scientific research entity corpus construction method and device based on scientific and technical literature and medium
CN121350242A
General scientific research entity corpus construction method and device based on scientific literature, and medium
CN121350242B
