An entity relationship joint extraction method and device

Abstract

The present application relates to the field of natural language processing, and particularly relates to an entity relationship joint extraction method and device. Mainly comprising: determining an entity category library and a relationship category library according to a task data set, and composing a training prompt template based on the entity category library and the relationship category library; using a prompt trainer to generate initial embedding parameters based on the training prompt template; inputting the initial embedding parameters into the prompt trainer, and using the prompt trainer to train a large language model; normalizing and exponentiating the output features of the large language model according to the entity category library and the relationship category library, and then updating the prompt trainer through back propagation until a trained prompt trainer is obtained, and continuous embedding parameters are obtained; using the trained prompt trainer and the continuous embedding parameters to perform entity relationship joint extraction on unlabeled text to obtain entity relationships in the unlabeled text. The present application can fine-tune through the prompt trainer, and more efficiently and accurately complete the entity relationship extraction task in the large language model.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a method and apparatus for joint extraction of entity relations. Background Technology

[0002] Natural Language Processing (NLP), a core technology in text mining, primarily studies how to enable machines to better understand human language. Since 2018, the concept of transfer learning has been widely applied in NLP, leading to the development of many pre-trained language models, such as Generative Pre-Trained Transformer (GPT), Bidirectional Encoder Representation from Transformers (BERT), and Generalized Autoregressive Pretraining for Language Understanding (XLNET). Using pre-trained language models can reduce later training costs, accelerate model convergence in downstream tasks, and significantly improve the accuracy of downstream tasks.

[0003] With the improvement of computer performance, the number of parameters in pre-trained language models has increased from 110 million (Bert-base) in 2018 to 5.3 trillion (Megatron-Turing) in 2021. These pre-trained language models with such a large number of parameters are called Large Language Models (LLMs), designed to understand and generate human language. They are trained on massive amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. However, while the increase in the number of parameters in large language models brings benefits, it also reveals many hidden dangers. The huge number of parameters leads to a decrease in the transferability of large language models; even when using large datasets for fine-tuning, it cannot be guaranteed that large language models can quickly learn fine-tuning samples. Furthermore, traditional fine-tuning methods require full model fine-tuning for each downstream task and storing a copy of the model samples for that task, placing enormous pressure on storage resources.

[0004] Therefore, how to overcome the shortcomings of existing technologies and solve the problem that existing large language models are not easy to fine-tune during migration is a problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention solves the problem that existing large language models are not easy to fine-tune during migration.

[0006] The embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, the present invention provides a method for joint entity relation extraction, specifically comprising: determining an entity category library and a relation category library based on a task dataset; forming a training prompt template based on the entity category library and the relation category library; using a prompt trainer to generate initial embedding parameters based on the training prompt template; wherein the data in the task dataset is labeled text containing at least one sentence, and the labels in the labeled text are used to represent the relations between entities in the sentence; passing the initial embedding parameters into the prompt trainer, using the prompt trainer to train a large language model, and after normalizing the output features of the large language model according to the entity category library and the relation category library, updating the prompt trainer through backpropagation until a trained prompt trainer is obtained and continuous embedding parameters are obtained; using the trained prompt trainer and the continuous embedding parameters to perform joint entity relation extraction on unlabeled text to obtain entity relations in the unlabeled text.

[0008] Preferably, the step of determining the entity category library and relation category library based on the task dataset, and forming a training prompt template based on the entity category library and relation category library, specifically includes: obtaining sentences from the task dataset; extracting the head entity and tail entity from each sentence according to the label corresponding to the sentence, and establishing corresponding sub-prompt, entity category library and relation category library; selecting a specified token related to the current extraction task as an anchor token based on the entity category library and relation category library; and the prompt generator forming a training prompt template for the large language model based on the sub-prompt and anchor token.

[0009] Preferably, the establishment of the corresponding sub-prompt, entity category library, and relationship category library specifically includes: specifying a first tag corresponding to the head entity, and forming a head entity token group by combining the head entity and the first tag; specifying a second tag corresponding to the tail entity, and forming a tail entity token group by combining the tail entity and the second tag; adding a relationship text and a third tag, and forming an inter-entity relationship token group by combining the relationship text and the third tag; determining the category to which the head entity, tail entity, and inter-entity relationship belong based on the head entity token group, tail entity token group, and inter-entity relationship token group, and formulating the entity category library and relationship category library based on the category to which they belong.

[0010] Preferably, the prompt generator is based on sub-prompts and anchor tokens to form the training prompt template of the large language model. Specifically, it includes: obtaining the first sentence in the task dataset; the prompt generator replacing the tokens in the first sentence with the corresponding head entity tokens or tail entity tokens; using the replaced first sentence as the second sentence; concatenating the entity relationship tokens after the corresponding first sentence; and generating the training prompt template for the task together with the anchor tokens and trainable consecutive tokens.

[0011] Preferably, the step of generating initial embedding parameters based on the training prompt template using the prompt trainer specifically includes: obtaining all tokens in the sentence except for the head entity, tail entity, and anchor token; using the obtained tokens as trainable consecutive tokens in the training prompt template; forming a token sequence from the trainable consecutive tokens in the training prompt template; and using the embedding obtained from training the large language model as the initial embedding parameter for each token in the token sequence.

[0012] Preferably, the step of passing the initial embedding parameters into the prompt trainer and using the prompt trainer to train the large language model specifically includes: passing the initial embedding parameters into the prompt trainer, the prompt trainer obtaining the global feature vector of the token sequence; inputting the global feature vector into the large language model, the large language model encoding the token sequence based on the global feature vector to output the corresponding context-aware representation.

[0013] Preferably, the step of normalizing the output features of the large language model according to the entity category library and the relation category library, and then updating the prompt trainer through backpropagation until a trained prompt trainer is obtained and continuous embedding parameters are obtained, specifically includes: performing normalization exponential processing on the first label, second label, and third label output by the large language model to obtain the probability distribution of each label; comparing the probability distribution of the labels with the entity category library and the relation category library, calculating the loss gradient of the large language model using the backpropagation algorithm, and updating the model parameters of the large language model according to the loss gradient; acquiring data in the task dataset that matches the current task, and using the acquired data to train the prompt trainer for at least one round to obtain a prompt trainer for extracting the current task and obtain the corresponding continuous embedding parameters.

[0014] Preferably, the step of using the trained prompt trainer and continuous embedding parameters to jointly extract entity relations from unlabeled text to obtain entity relations in unlabeled text specifically includes: obtaining sentences from unlabeled text; establishing an extraction prompt template based on the obtained sentences; generating extraction embedding parameters based on the extraction prompt template using the prompt trainer; inputting the extraction embedding parameters into the prompt trainer; and inputting the global feature vector output by the prompt trainer into the large language model to obtain entity relations in unlabeled text.

[0015] Secondly, the present invention provides an entity relation joint extraction apparatus, specifically comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the entity relation joint extraction method in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect.

[0017] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in memory, performing the method as described in the first aspect.

[0018] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the method as described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: the entity relation joint extraction method based on prompt fine-tuning of a large language model can be fine-tuned through a prompt trainer and generate prompt templates for prompting, thereby completing the entity relation extraction task more efficiently and accurately in a large language model. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1 A flowchart of an entity relation joint extraction method provided in an embodiment of the present invention;

[0022] Figure 2 Flowchart of another entity relation joint extraction method provided in an embodiment of the present invention;

[0023] Figure 3 Flowchart of another entity relation joint extraction method provided in an embodiment of the present invention;

[0024] Figure 4 Flowchart of another entity relation joint extraction method provided in an embodiment of the present invention;

[0025] Figure 5 Flowchart of another entity relation joint extraction method provided in an embodiment of the present invention;

[0026] Figure 6 Flowchart of another entity relation joint extraction method provided in an embodiment of the present invention;

[0027] Figure 7 Flowchart of another entity relation joint extraction method provided in an embodiment of the present invention;

[0028] Figure 8 This is a schematic diagram illustrating the process of creating a Prompt template in a specific scenario of the present invention.

[0029] Figure 9 This is a schematic diagram illustrating the execution process of an entity relationship extraction task in a specific scenario of the present invention;

[0030] Figure 10 This is a schematic diagram of an entity relationship joint extraction device provided in an embodiment of the present invention;

[0031] The accompanying figure is labeled as follows:

[0032] 11: Processor; 12: Memory. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0035] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] Example 1:

[0037] To more efficiently utilize large language models for joint extraction of domain-specific entity relations, this embodiment presents a joint entity relation extraction method in which the large language model is trained based on a prompt fine-tuning method.

[0038] Prompt-tuning, pioneered by GPT-3 and Pre-trained Embedding Models (PET), is a novel fine-tuning paradigm based on pre-trained language models. It aims to avoid introducing extra parameters by adding templates, allowing language models to achieve ideal results in few-shot or zero-shot scenarios. The prompt-tuning method can transform different downstream tasks into forms commonly used in the pre-training of large language models using prompt templates, reducing the discrepancy between training data during pre-training and fine-tuning, and improving the performance of large language models on downstream tasks.

[0039] like Figure 1 As shown, the specific steps of the entity relationship joint extraction method provided in this embodiment of the invention are as follows:

[0040] Step 101: Determine the entity category library and relation category library based on the task dataset, and combine the entity category library and relation category library to form a training prompt template; use the prompt trainer to generate initial embedding parameters based on the training prompt template.

[0041] Entity relation extraction is a crucial step in natural language processing (NLP) tasks. It primarily identifies entity objects and semantic relationships between entities from unstructured text data, representing a method of structuring text data into knowledge represented by abstract relations. In NLP, tokens are typically used as the smallest semantic unit and the basic elements of the processing. Specifically, a single word or phrase with independent and complete semantic meaning in the text can be considered a token, and entity relation extraction is performed on a token-by-token basis.

[0042] To train a large language model for entity relation extraction, the first step is to select the task dataset required for each entity relation extraction task from labeled text. Specifically, the task dataset consists of labeled text containing at least one sentence, where the labels represent the relationships between entities within the sentence.

[0043] To fine-tune the parameters of a large language model using the prompt fine-tuning method, it is also necessary to use the task dataset to build sub-prompts, determine anchor tokens, entity category libraries, and relation category libraries, and then combine them into a prompt template for the large language model using a prompt generator.

[0044] After obtaining the prompt template, the prompt template is then used by the prompt trainer to generate a trainable input embedding. In this embodiment, the trainable input embedding generated based on the original task dataset is called the initial mapping parameter.

[0045] Step 102: Pass the initial embedding parameters into the prompt trainer, use the prompt trainer to train the large language model, and after normalizing the output features of the large language model according to the entity category library and the relation category library, update the prompt trainer through backpropagation until the trained prompt trainer is obtained and the continuous embedding parameters are obtained.

[0046] The input embedding obtained in step 101 is fed into the prompt trainer and then passed to the large language model for training. The output features from the large language model are normalized exponentially (softmax) based on the entity category library and relation category library. Softmax is an activation function that normalizes a numerical vector into a probability distribution vector, where the sum of all probabilities in the probability distribution vector is 1. Softmax can be used as the last layer of a neural network for multi-class classification problems. Its output value is the probability that the feature values ​​of the input data conform to each category, where the category with the highest probability can be considered the category to which the input data belongs, but the probability of the input data belonging to other categories is not 0.

[0047] By updating the prompt trainer through backpropagation, after multiple rounds of training, a prompt trainer for joint extraction of entity relations and the embedding parameters of trainable consecutive tokens are completed. In this embodiment, the embedding parameters of trainable consecutive tokens are referred to as consecutive embedding parameters.

[0048] Step 103: Use the trained prompt trainer and continuous embedding parameters to jointly extract entity relations from the unlabeled text to obtain entity relations in the unlabeled text.

[0049] After training using m specific entity relation extraction tasks, m trained prompt trainers can be obtained, along with the continuous embedding parameters used by each prompt trainer. By using the trained prompt trainers to perform the embedding parameter acquisition and entity extraction process in step 102, entity relation triples that meet specific requirements can be efficiently and accurately extracted from unlabeled text (also known as raw text). These entity relation triples represent the relationships between different entities in the text. Specifically, an entity relation triple can be represented as: "Head entity - Relationship between head and tail entities - Tail entity".

[0050] After steps 101-103 provided in this embodiment, the prompt fine-tuning method can be used to quickly complete the training of the large language model to meet the needs of the entity relation extraction task.

[0051] like Figure 2 As shown, the entity category library and relation category library can be determined based on the task dataset in the following way, and the training prompt template can be formed by combining the entity category library and relation category library.

[0052] Step 201: Obtain sentences from the task dataset, extract the head and tail entities from each sentence according to the corresponding labels, and build the corresponding sub-prompt, entity category library and relation category library.

[0053] Based on the specific entity relations to be extracted, determine the head and tail entities in the sentence that need to be jointly extracted from the entity relations. Based on the categories of head entities, tail entities, and relations in all sentences, establish a sub-prompt, an entity category library, and a relation category library. The sub-prompt contains the sub-prompt tokens of the head and tail entities in all sentences, the entity category library contains the categories of all head and tail entities in all sentences, and the relation category library contains the categories of all relations in all sentences.

[0054] Step 202: Select a specified token related to the current extraction task as the anchor token.

[0055] To improve the efficiency of entity extraction, additional representative tokens related to the current entity relationship extraction task can be selected as anchor tokens. For example, when the entities to be extracted are mechanical designers and mechanical models, the mechanical category can be used as the anchor token.

[0056] Step 203: The prompt generator uses sub-prompts and anchor tokens to form the training prompt template for the large language model.

[0057] After obtaining the entity category library, the relationship category library, and the anchor token, the obtained information can be used to form a prompt template according to the requirements of the prompt fine-tuning method. In this embodiment, the prompt template obtained at this time is called the training prompt template.

[0058] After steps 201-203 provided in this embodiment, the training prompt template for training large language models can be obtained.

[0059] In practice, the entity category library and relationship category library can be obtained in the following ways.

[0060] (1) Specify the first tag corresponding to the header entity, and form a header entity token group, i.e., "header entity sub-prompt tokens", by combining the header entity and the first tag. In this embodiment, [MASK1] is used to represent the first tag, and each "header entity sub-prompt token" is "header entity [MASK1]".

[0061] (2) Specify the second tag corresponding to the tail entity, and combine the tail entity and the second tag into a tail entity token group, namely "tail entity sub-prompt tokens". In this embodiment, [MASK2] is used to represent the second tag, and each "tail entity sub-prompt token" is "tail entity [MASK2]".

[0062] (3) Add relation text and third tag, and form an entity relationship token group, namely "relation text sub-prompt tokens". In this embodiment, the fixed text "The relation is" is used as the relation text, and [MASK3] is used to represent the third tag. The entity relationship sub-prompt tokens are fixed as "Therelation is[MASK3]".

[0063] After obtaining the above sub-prompt, the possible categories of entities and relationships are determined based on task requirements and prior knowledge. Based on the head entity token group, tail entity token group, and inter-entity relationship token group, the categories to which the head entity, tail entity, and inter-entity relationship belong are determined. Based on the categories, entity category libraries and relationship category libraries are formulated to assist in the final verification of the task effect of the prompt template on the joint extraction of entity relationships.

[0064] like Figure 3 As shown, the prompt generator can be used to compose the training prompt template for a large language model based on sub-prompts and anchor tokens in the following way.

[0065] Step 301: Obtain the first sentence from the task dataset. The prompt generator replaces the token in the first sentence with the corresponding header entity token or tail entity token, and uses the replaced first sentence as the second sentence.

[0066] For ease of description, in this embodiment, the sentence in the original text of the task dataset is referred to as the first sentence. The sentence generated by the prompt generator after replacing the tokens in the first sentence with the obtained head entity sub-prompt tokens and tail entity sub-prompt tokens is referred to as the second sentence.

[0067] Step 302: Concatenate the entity relationship token after the corresponding first sentence, and generate the training prompt template for this task together with the anchor token and trainable consecutive tokens.

[0068] To represent the relationships between entities, it is also necessary to concatenate entity relationship sub-prompt tokens after the second sentence. The second sentence, anchor token, and trainable consecutive tokens are then used to generate the large language model prompt template for this task.

[0069] After steps 301-302 provided in this embodiment, the training prompt template can be obtained.

[0070] To facilitate processing, after obtaining the training prompt template, a word segmentation tool is used to segment the training prompt template to obtain all tokens in the sentence. Each token is mapped to a unique integer identifier, and the integer identifiers of all tokens are combined into an input ids array. The position information of all tokens is also stored as position ids. Tokens that need attention in the entity extraction task are used as attention masks. Other necessary information or corresponding data preprocessing can also be obtained as needed. In specific implementation, the pre-trained language model used for word segmentation can be selected as needed, such as BERT, BERT Whole Word Masking (BERT-wwm), A PRE-TRAINED LANGUAGE MODEL WITH MULTI-GRAINED TOKENIZATION (AMBERT), or Lattice-BERT (Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models).

[0071] like Figure 4 As shown, after obtaining the training prompt template, the initial embedding parameters can be generated based on the training prompt template using the prompt trainer in the following way.

[0072] Step 401: Obtain all tokens in the sentence except for the head entity, tail entity, and anchor token, and use the obtained tokens as trainable consecutive tokens in the training prompt template.

[0073] In the method provided in this embodiment, the trainable consecutive tokens are all tokens in the first sentence except for the head entity, tail entity, and anchor token.

[0074] Step 402: Combine the trainable consecutive tokens in the training prompt template into a token sequence, and use the embedding obtained from training the large language model as the initial embedding parameter for each token in the token sequence.

[0075] Obtain the embedding result obtained from the training of the pre-trained large language model M, form a new token sequence h from the continuously learnable tokens in the prompt template, and set the initial embedding of each token in the sequence h to the embedding result obtained from the training of the large language model M.

[0076] After steps 401-402 provided in this embodiment, the initial embedding parameters can be generated.

[0077] The initial embedding parameters are input into the prompt trainer, and then passed to the large language model for training. After softmaxing the output features based on the entity category library and the relation category library, the prompt trainer is updated through backpropagation. After multiple rounds of training, the prompt trainer for joint entity and relation extraction and the embedding parameters for trainable continuous tokens can be completed.

[0078] like Figure 5 As shown, the initial embedding parameters can be passed to the prompt trainer in the following way, and the prompt trainer can be used to train the large language model.

[0079] Step 501: Pass the initial embedding parameters into the prompt trainer, and the prompt trainer obtains the global feature vector of the token sequence.

[0080] The prompt trainer can be trained using a language model, such as a Bi-directional Long Short-Term Memory (Bi-LSTM) network or LSTM. By inputting the initial embedding parameters into the trained prompt trainer, a global feature vector for each continuously learnable token can be obtained.

[0081] Step 502: Input the global feature vector into the large language model. The large language model encodes the token sequence based on the global feature vector to output the corresponding context-aware representation.

[0082] The global feature vector output by the prompt trainer is input into the large language model to encode the input sequence. The large language model will generate a corresponding context-aware representation based on the input token sequence. In practice, a large pre-trained language model suitable for joint entity relation extraction, such as BERT or GPT-3, can be selected as needed.

[0083] After steps 501-502 provided in this embodiment, the large language model can be trained using the prompt trainer.

[0084] like Figure 6 As shown, the output features of a large language model can be normalized to an index based on an entity category library and a relation category library, and then the prompt trainer can be updated through backpropagation until the trained prompt trainer is obtained, and continuous embedding parameters are obtained.

[0085] Step 601: Perform normalization exponential processing on the first, second, and third labels output by the large language model to obtain the probability distribution of each label.

[0086] The labels [MASK1], [MASK2], and [MASK3] output by the large language model are subjected to a softmax operation to obtain the probability distribution of each label.

[0087] Step 602: Compare the probability distribution of the tags with the entity category library and the relation category library, use the backpropagation algorithm to calculate the loss gradient of the large language model, and update the model parameters of the large language model according to the loss gradient.

[0088] The probability distribution of the labels is compared with the real entity category library and relation category library. The backpropagation algorithm is used to calculate the loss gradient, and the parameters of the Bi-LSTM model are updated according to the gradient.

[0089] In practice, the embedding parameters of the trainable consecutive tokens in the token sequence h are also retained as the initial values ​​of the trainable consecutive tokens in the token sequence h for the next training.

[0090] Step 603: Obtain data in the task dataset that matches the current task, use the obtained data to train the prompt trainer for at least one round to obtain a prompt trainer for extracting the current task, and obtain the corresponding continuous embedding parameters.

[0091] The task dataset can contain a variety of different entity relation extraction tasks. After training with data from each type of task, a prompt trainer suitable for that type of entity relation extraction task can be obtained, along with the embedding parameters of trainable continuous tokens after multiple rounds of data training. In this embodiment, the embedding parameters obtained at this time are referred to as continuous embedding parameters.

[0092] After steps 601-602 provided in this embodiment, the trained prompt trainer can be obtained, and continuous embedding parameters can be obtained.

[0093] After training with m specific tasks, m prompt trainers can be obtained. Using these prompt trainers, along with the corresponding continuous embedding parameters, entity relation triples for the corresponding task can be extracted efficiently and accurately from unlabeled raw text.

[0094] like Figure 7 As shown, entity relations can be jointly extracted from unlabeled text using the trained prompt trainer and continuous embedding parameters in the following way to obtain entity relations in unlabeled text.

[0095] Step 701: Obtain sentences from the unlabeled text and create a prompt extraction template based on the obtained sentences.

[0096] After inputting the raw text sentence, repeat step 101, treating each token in the raw text as a head entity and forming "head entity sub-prompt tokens" with "[MASK1]". Then, sequentially select another token as a tail entity and form "tail entity sub-prompt tokens" with "[MASK2]". Then, each time, add "Therelation is[MASK3]" to the end of the sentence as a sub-prompt token for the relationship between entities. Replace all other words with trainable consecutive tokens to obtain the large language model prompt template.

[0097] Step 702: Using the prompt trainer, generate extraction embedding parameters based on the extracted prompt template.

[0098] Repeat step 402, but set the initial embedding of the token to the embedding parameters of the trainable consecutive tokens after multiple rounds of training on the current task.

[0099] Step 703: Pass the extracted embedding parameters into the prompt trainer, and pass the global feature vector output by the prompt trainer into the large language model to obtain entity relationships in the unlabeled text.

[0100] Repeat steps 501, 502, and 601 to obtain the probability distribution of each label [MASK1], [MASK2], and [MASK3]. If the distribution does not meet the requirements of the current task, it is determined that it is not an entity relation triplet to be extracted. The large language model can quickly complete this process, thereby extracting entity relation triplets from the raw text that meet the specific task requirements.

[0101] After steps 701-703 provided in this embodiment, the entity relation extraction task can be completed using the large language model trained by the prompt fine-tuning method.

[0102] The entity relation joint extraction method provided in this embodiment proposes an entity relation joint extraction method based on prompt fine-tuning of a large language model. It uses a large language model to perform joint extraction of entity relations in a specific domain. During the extraction process, the prompt trainer trained by the large language model and the embedding parameters of trainable continuous tokens can be reused in subsequent identical tasks, thereby significantly improving the efficiency of the entity relation joint extraction task.

[0103] Example 2:

[0104] Based on the entity relationship joint extraction method provided in Embodiment 1, it can be implemented in some specific embodiments through the specific implementation methods described in this embodiment. It is understood that the specific implementation methods provided in this embodiment are only used to illustrate the specific implementation process of the method in Embodiment 1 in certain specific scenarios and are not intended to limit the scope of protection.

[0105] In the scenario of this embodiment, a prompt template that can extract entity relation triples related to mechanical manufacturing and design needs to be trained for fine-tuning of the large language model. Sentences with text content similar to the task requirements can be selected from labeled text and added to the same task dataset.

[0106] In this embodiment, the categories in the head entity category library include: Person, Organization, and Equipment, etc.; the categories in the tail entity category library include: Person, Organization, and Equipment, etc.; and the categories in the relationship category library include: "design", "designed by", and "reform", etc.

[0107] In this embodiment, the task dataset contains a labeled text: "[CLS]...Wu is the designer of the C919 plane...[SEP]". Here, [CLS] indicates the start of input, [SEP] indicates the end of input, and "Wu is the designer of the C919 plane." is a sentence in the text. Each word in the sentence serves as a token. In this embodiment, this sentence is used as the first sentence.

[0108] When training a large language model, step 201 is followed to determine the head and tail entities in the sentence that require joint entity relation extraction. In this scenario, when designing the prompt template for mechanical designers, entity extraction is performed based on the "person-design-equipment" entity relation triple as a template, according to the text's tags. Figure 8 For example, the head entity is 2 "Wu" and the tail entity is 4 "C919". Using this method, after training with m sets of specific tasks, different entity relationships are extracted to form m task datasets based on task requirements. Each set contains data similar to a particular task category.

[0109] The head entity sub-prompt tokens include: 1 "[MASK1]" and 2 "Wu", which are of the category Person; the tail entity sub-prompt tokens include: 3 "[MASK2]" and 4 "C919"; the anchor token includes: 5 "plane"; the entity relationship sub-prompt tokens include: 6 "The relation is" and 7 "[MASK3]"; the remaining tokens are trainable consecutive tokens, numbered 0.

[0110] Based on the extracted entities, a sub-prompt database and an entity category database and a relationship category database are created. Figure 8 For example, the head entity 2 "Wu" and 1 "[MASK1]" form the "head entity sub-prompt tokens", the tail entity 4 "C919" and 3 "[MASK2]" form the "tail entity sub-prompt tokens", and "The relation is [MASK3]" is added as the sub-prompt tokens for the relationship between entities.

[0111] Then, based on task requirements and prior knowledge, the possible categories of entities and relationships are determined, thereby establishing entity category libraries and relationship category libraries to assist in the final verification of the prompt template's effectiveness in jointly extracting entities and relationships. In this scenario, the head entity category is Person, the tail entity category is Equipment, and the relationship category is "design".

[0112] Following step 202, determine the anchor token. Additional representative tokens related to the current entity relationship extraction task can be selected as anchor tokens. In the first sentence of this embodiment, the token "warplane," which is related to the type of mechanical equipment, is selected as the anchor token during the training process of this prompt template, improving the template's fine-tuning effect in this direction within the large language model.

[0113] Following step 203, the prompt generator constructs the prompt template for the large language model. The prompt generator replaces the corresponding tokens in the first sentence with the head entity sub-prompt tokens and tail entity sub-prompt tokens obtained in step 201, then appends the entity relationship sub-prompt tokens to the end of the sentence. These, along with the anchor token and trainable consecutive tokens, generate the training prompt template for the large language model of this task. For example... Figure 9 As shown, in the prompt template, tokens numbered 1-7 are... Figure 8 The content is consistent; numbers 01, 02, and 03 each correspond to a set of trainable consecutive tokens, and their contents are the corresponding tokens in the first sentence, and... Figure 8 The position corresponding to number 0 in the middle.

[0114] On the other hand, when performing actual entity extraction tasks, in steps 201 and 202, since the raw text does not contain labels, it is impossible to directly determine the tokens corresponding to the head and tail entities. Therefore, each token in the raw text needs to be treated as a head entity, and the head entity is combined with "[MASK1]" to form "head entity sub-prompt tokens"; then, another token is selected sequentially as the tail entity, and the selected tail entity is combined with "[MASK2]" to form "tail entity sub-prompt tokens". Furthermore, "The relation is [MASK3]" is added to the end of each sentence as a sub-prompt token for the relationship between entities. Finally, all other words are replaced with trainable consecutive tokens to obtain the extraction prompt template for the large language model.

[0115] The prompt template was segmented using BERT's word segmentation tool to obtain input IDs, position IDs, attention masks, etc. Each group of trainable consecutive tokens was segmented as: 01"[P0]...[P i ]”、02“[P i+1 ...[P n ]”、03“[P n+2 ...[P m ]”; where each [P num ] represents a token from a series of trainable tokens, and the index num is the integer identifier corresponding to the token, that is, the mapping value of the token in the input ids array.

[0116] When training a large language model, following step 402, the trainable consecutive tokens in the prompt template are combined into a new token sequence h. For example... Figure 9 As shown, the trainable consecutive tokens numbered 01 correspond to the sequence h: h0...hi of number 04; the trainable consecutive tokens numbered 02 correspond to the sequence h: hi+1...hn of number 05; and the trainable consecutive tokens numbered 03 correspond to the sequence h: hi+1...hn of number 06. The initial embedding of each token in each sequence h is set to the embedding result trained by the pre-trained language model M, thereby obtaining the initial embedding parameters.

[0117] At the same time, such as Figure 9 As shown, in the input embedding, the head entity tokens in numbers 1 and 2, the tail entities in numbers 3 and 4, the anchor token in number 5, and the third tag in number 7 are converted into the corresponding e(“text”), where “text” is the corresponding text content in the prompt template. For example, e(1) is “e([MASK1])”, e(2) is “e(Wu)”, and so on. Numbers 61, 62, and 63 are the tokens obtained after segmenting and converting the relation text in number 6 in the prompt template, namely: 61 “e(The)”, 62 “e(relation)”, and 63 “e(is)”.

[0118] On the other hand, during the actual training process, the embedding parameters of the trainable consecutive tokens in the token sequence h are also retained as the initial values ​​for the trainable consecutive tokens in the token sequence h during the next training. When performing the actual entity extraction task, in the process corresponding to step 402, the initial embedding of each token in each sequence h is set to the embedding parameter result of the trainable consecutive tokens after multiple rounds of data training in the current task, so as to improve the accuracy of the extraction results in the entity extraction task.

[0119] Choose a large pre-trained language model suitable for joint entity relation extraction, such as BERT or GPT-3.

[0120] Following step 501, the generated trainable input embedding is input into the prompt trainer. The prompt trainer can be trained using a language model; in this embodiment, a bidirectional long short-term memory network (Bi-LSTM) is used. After input, a global feature vector of the sequence of each continuously learnable token is obtained. This vector is then input into the selected large language model to encode the sequence of all tokens in the input embedding. The large language model will then generate a corresponding context-aware representation based on the input token sequence.

[0121] Following step 601, perform a softmax operation on the labels [MASK1], [MASK2], and [MASK3] output by the model to obtain the probability distribution of each label.

[0122] Following step 602, based on the probability distribution of the labels and the comparison with the real entity category library and relation category library, the backpropagation algorithm is used to calculate the loss gradient, and the parameters of the Bi-LSTM model are updated according to the gradient.

[0123] According to step 603, after each group in the task dataset contains data similar to a certain type of task, it has been trained through the previous steps to obtain a prompt trainer suitable for extracting the current task, and the embedding parameters of trainable continuous tokens after multiple rounds of data training.

[0124] Entity extraction is performed using the trained prompt trainer and embedding parameters to obtain the probability distribution of each label [MASK1], [MASK2], and [MASK3] in different categories. If it does not meet the requirements of the current specific task, it is determined that it is not an entity relation triplet to be extracted.

[0125] As can be seen from the specific implementation process in this embodiment, the entity relation joint extraction method provided in Embodiment 1 enables the large language model to quickly complete the training process, thereby extracting entity relation triples required for specific tasks in the raw text and completing the required entity relation extraction task.

[0126] Example 3:

[0127] Based on the entity relation joint extraction method provided in Embodiments 1 and 2 above, the present invention also provides an entity relation joint extraction device that can be used to implement the above method, such as... Figure 10 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The entity relationship extraction device of this embodiment includes one or more processors 11 and a memory 12. Figure 10 Take a processor 11 as an example.

[0128] Processor 11 and memory 12 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0129] The memory 12 serves as a non-volatile computer-readable storage medium for the entity relation extraction method. It can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the entity relation extraction methods in Embodiments 1 and 2. The processor 11 executes various functional applications and data processing of the entity relation extraction device by running the non-volatile software programs, instructions, and modules stored in the memory 12, thereby implementing the entity relation extraction methods of Embodiments 1 and 2.

[0130] Memory 12 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 12 may optionally include memory remotely located relative to processor 11, which can be connected to processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The program instructions / modules are stored in memory 12. When executed by one or more processors 11, they perform the entity relation joint extraction method described in Embodiments 1 and 2 above. For example, they perform the methods described above. Figures 1 to 9 The various processes shown.

[0132] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 10One of the processors 11 can enable the above-described one or more processors to execute the entity relation joint extraction method in Embodiment 1 or Embodiment 2, for example, to perform the above-described... Figures 1 to 7 The steps shown can also be implemented. Figure 8 and Figure 9 The various modules and units mentioned above.

[0133] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0134] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for joint extraction of entity relations, characterized in that, include: Based on the task dataset, determine the entity category library and the relation category library, and compose the training prompt template based on the entity category library and the relation category library; The prompt trainer is used to generate initial embedding parameters based on the training prompt template; wherein, the data in the task dataset is labeled text containing at least one sentence, and the labels in the labeled text are used to represent the relationships between entities in the sentence; The initial embedding parameters are passed into the prompt trainer. The prompt trainer is used to train the large language model. The output features of the large language model are normalized exponentially according to the entity category library and the relation category library. The prompt trainer is then updated through backpropagation until the trained prompt trainer is obtained and continuous embedding parameters are obtained. The trained prompt trainer and continuous embedding parameters are used to jointly extract entity relations from unlabeled text to obtain entity relations in unlabeled text. The process of determining the entity category library and relation category library based on the task dataset, and constructing a training prompt template based on the entity category library and relation category library, specifically includes: obtaining sentences from the task dataset; extracting the head entity and tail entity from each sentence according to the corresponding label, and establishing corresponding sub-prompt, entity category library, and relation category library; selecting a specified token related to the current extraction task as an anchor token based on the entity category library and relation category library; and the prompt generator constructing a training prompt template for the large language model based on the sub-prompt and anchor token. The construction of the corresponding sub-prompt, entity category library, and relation category library includes: specifying the first label corresponding to the head entity, and forming a head entity token group with the head entity and the first label; specifying the second label corresponding to the tail entity, and forming a tail entity token group with the tail entity and the second label; adding relation text and a third label, and forming an inter-entity relation token group with the relation text and the third label; determining the category to which the head entity, tail entity, and inter-entity relation belong based on the head entity token group, tail entity token group, and inter-entity relation token group, and constructing the entity category library and relation category library based on the belonging category. The prompt generator, based on sub-prompts and anchor tokens, constructs a training prompt template for the large language model. This process includes: obtaining the first sentence from the task dataset; replacing the tokens in the first sentence with the corresponding head entity tokens or tail entity tokens; using the replaced first sentence as the second sentence; concatenating the entity relationship tokens after the corresponding first sentence; and generating the training prompt template for the task along with the anchor tokens and trainable consecutive tokens.

2. The entity relation joint extraction method according to claim 1, characterized in that, The process of using the prompt trainer to generate initial embedding parameters based on the training prompt template specifically includes: Get all tokens in the sentence except for the head entity, tail entity and anchor token, and use the obtained tokens as trainable consecutive tokens in the training prompt template; The trainable consecutive tokens in the training prompt template are grouped into a token sequence, and the embedding obtained from training the large language model is used as the initial embedding parameter for each token in the token sequence.

3. The entity relation joint extraction method according to claim 2, characterized in that, The step of passing the initial embedding parameters into the prompt trainer and using the prompt trainer to train the large language model specifically includes: The initial embedding parameter is passed into the prompt trainer, which then obtains the global feature vector of the token sequence. The global feature vector is input into the large language model, which encodes the token sequence based on the global feature vector to output the corresponding context-aware representation.

4. The entity relation joint extraction method according to claim 1, characterized in that, The output features of the large language model are normalized exponentially based on entity and relation category libraries, and then the prompt trainer is updated through backpropagation until the trained prompt trainer is obtained, and continuous embedding parameters are obtained. Specifically, this includes: The first, second, and third labels output by the large language model are normalized exponentially to obtain the probability distribution of each label. The probability distribution of the tags is compared with the entity category library and the relation category library. The backpropagation algorithm is used to calculate the loss gradient of the large language model, and the model parameters of the large language model are updated according to the loss gradient. Obtain data from the task dataset that matches the current task, use the obtained data to train the prompt trainer for at least one round to obtain a prompt trainer for extracting the current task, and obtain the corresponding continuous embedding parameters.

5. The entity relation joint extraction method according to claim 2, characterized in that, The step of using the trained prompt trainer and continuous embedding parameters to jointly extract entity relations from unlabeled text to obtain entity relations in unlabeled text specifically includes: Extract sentences from unlabeled text and create a prompt template based on the extracted sentences; The prompt trainer generates extraction embedding parameters based on the extracted prompt template. The extracted embedding parameters are passed to the prompt trainer, and the global feature vector output by the prompt trainer is passed to the large language model to obtain entity relationships in the unlabeled text.

6. An entity relationship joint extraction device, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the entity relation joint extraction method according to any one of claims 1-5.

7. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the entity relation joint extraction method as described in any one of claims 1-5.

Images