Explicit-implicit relationship collaborative extraction method and device based on large model and pre-training model

Abstract

The invention discloses an explicit-implicit relationship collaborative extraction method and device based on a large model and a pre-training model, and the method comprises the steps: 1, obtaining a word vector of a text through a pre-training language model, building a word-level table labeling frame based on the word vector of the text, and building a table labeling frame based on the word level; obtaining an explicit entity relationship triple in the text; wherein the pre-training language model is BERT; 2, acquiring implicit knowledge in the text from the GPT model as an external knowledge source; 3, performing instruction fine tuning by fusing the open source large language model with implicit knowledge and prompts, and performing reasoning through texts and implicit knowledge to obtain an implicit entity relationship triple; and 4, systematically integrating the explicit and implicit entity relationship triples obtained from the text by the open source large language model fused with implicit information fine adjustment and the pre-training language model, and finally obtaining a combined triad.

Description

Technical Field

[0001] The present invention belongs to the technical field of text information processing, and specifically relates to a method and device for collaborative extraction of explicit and implicit relations based on a large model and a pre-trained model. Background Art

[0002] Entity relation triple extraction has always been an important part of information extraction. The field has developed to the point where deep learning methods can automatically and efficiently extract task-related features from sentences. Recent research has focused on improving task performance and efficiency through various innovative strategies. However, existing methods still have some defects. The complex model structure and large-scale training process require a lot of computing resources, which limits their application in resource-constrained environments. At the same time, deep learning models are often regarded as black boxes, and it is difficult to explain their decision-making process, which is a challenge for some applications that require interpretability.

[0003] In recent years, methods in this field have been able to automatically and efficiently extract features related to specific tasks from text sentences. Current research focuses on improving the performance and efficiency of tasks through a variety of innovative strategies. In deep learning, there are two types of relationship extraction methods, namely Pipe-Line and joint methods. The entity relationship extraction task based on the Pipe-Line framework processes the two subtasks of entity recognition and relationship extraction separately, and connects them into a process to complete the entire entity relationship extraction task.

[0004] Cai et al. proposed a new solution to perform one-pass encoding on the input paragraph to complete multiple entity relationship extraction tasks. This solution can solve the efficiency problem of multiple passes with only one encoding input, greatly increasing efficiency and scalability. By modifying BERT, including introducing a structured prediction layer to predict multiple relations between different entity pairs, and making the self-attention layer aware of the location of all entities in the input paragraph. Wang et al. This paper introduces a model called bidirectional recurrent convolutional neural network (BCRNN) for relationship classification. The BCRNN model contains two RCNNs that learn features forward and backward along the shortest dependency path (SDP) to capture the features of local substructures and perform reverse capture at the same time. A dual-channel recurrent neural network with LSTM units is used to exploit the information of words and dependencies.

[0005] The joint learning method combines the two tasks of entity recognition and relation extraction. At the same time, different relation types can exist in the same sentence, and there can be multiple relation types between the same entities. Different relations may also contain some hidden information. With the advancement of technology, more innovative methods have been proposed to solve the complex problems in triple extraction. The current joint extraction model designs different extraction strategies or complex annotation schemes to solve such problems. Huang et al. designed a sliding window filling strategy to fill the matrix to shorten the distance between entity pairs, thereby promoting their interaction. The perception representation module and an entity enhancement representation module predict and combine different features of directed entity pairs, and predict potential entity pairs through binary classifiers, map these features to each relation category, and enhance the interaction between entities in each relation. Wei et al. proposed a cascade pointer network method with a cascade binary classification framework to effectively handle the overlap between multi-relation triplets. This method first identifies the subject of the sentence, then uses a predicate-specific classifier to detect all possible objects related to the subject, and then obtains entity-relation triples through relation mapping, which greatly improves the effect of entity relationship extraction. Tang et al. proposed a unified interaction method to model entity-entity interaction and entity-relationship interaction by encoding the representation of entities and relations in a connected natural language sequence and modeling the interaction uniformly through Interaction Map. They extracted structured triples from raw text in an end-to-end manner, and realized the extraction of entities and relations in complex scenarios with rich intrinsic correlations.

[0006] Various large language models (LLMs) have been shown to have the ability to significantly exploit the labeled semantic information in natural language sequences of different lengths. Li et al. formally defined the emergent ability of large language models (LLMs) as "the ability that does not exist in small models but arises in large models", which is one of the most significant features that distinguish LLMs from previous large-scale pre-trained language models (PLMs). LLMs achieve outstanding performance in reasoning tasks. Specifically, LLMs have mastered rich semantic relationships and world knowledge through pre-training on a large amount of text data, thus showing accuracy and flexibility that surpass traditional models in various types of tasks such as mathematical reasoning, common sense reasoning, and situational reasoning. These models can not only effectively understand and generate natural language, but also perform logical deduction and knowledge reasoning in complex reasoning scenarios, reflecting their deep language understanding ability and powerful computing power.

[0007] Zhang et al. introduced a framework to treat the relation extraction task as a multiple-choice question answering (QA) system, allowing LLMs to select options that represent the correct relation type. By aligning the relation extraction task with the QA task, large language models can better learn the relation extraction task. The QA4RE framework has achieved significant improvements on multiple relation extraction datasets and LLMs of different sizes, solving the problem that large language models perform poorly in zero-shot relation extraction tasks and improving the performance of large language models in relation extraction tasks. Wan et al. introduced task-aware representations into retrieval and enriched the demonstrations with reasoning logic induced by gold labels. First, task-aware retrieval is used, which is achieved by emphasizing the representation of entity and relation information. This is achieved through sentence embeddings of entity hints and fine-tuned relation representations. Second, GPT-RE enriches the demonstrations with reasoning induced by gold labels, enabling LLMs to induce reasoning processes and align relations with diverse expressions. Improve the low correlation between entities and relations and the problem of inability to explain input-label mapping. Pawan et al. leveraged the potential of GPT+ICL in financial relation extraction and adopted two retrieval strategies to find the top K contextual learning examples from the training data that are relevant to a given test example. ICL provides a core capability to infer new tasks from context, defining the process of generating outputs given a new test input text. Li et al. used semantic enhancement and synergy of large language models and small pre-trained language models to enhance relation triple extraction in specific domains. It performs well in processing text structures in the specific domain of news and extracts relation triplets from various news texts more accurately than other models. The introduction of the relational semantic enhancement module is crucial to improving the efficiency of semantic interaction, while the proposed voting strategy and data enhancement strategy also effectively enhance the model performance.

[0008] Currently, existing relation triple extraction models generally only focus on explicit relation triples directly expressed in the text, usually ignoring some indirect or implicit information that may exist in the text. This information does not directly provide the relationship between entities, but plays an important role in understanding the inference of relationships. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention proposes a method and device for collaborative extraction of explicit and implicit relationships based on a large model and a pre-trained model, wherein explicit triples refer to entity relationships that are clearly stated in the text, while implicit triples refer to relationships that can only be identified through reasoning or contextual understanding. The method of using a large language model to obtain implicit triples and a pre-trained model to obtain explicit triples converts unstructured natural language text into structured information, making full use of the inherent advantages of the table filling method, making the recognition process of overlapping relationships more efficient and accurate.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] The method for collaboratively extracting explicit and implicit relations based on a large model and a pre-trained model includes the following steps:

[0012] Step 1: Use the pre-trained language model to obtain the word vector of the text, build a word-level table annotation framework based on the word vector of the text, and obtain the explicit entity relationship triples in the text based on the word-level table annotation framework; the pre-trained language model is BERT;

[0013] Step 2: Obtain implicit knowledge in the text from the GPT model as an external knowledge source;

[0014] Step 3: Fine-tune the instructions by integrating implicit knowledge and prompts into the open source large language model, and obtain implicit entity relationship triples through reasoning through text and implicit knowledge;

[0015] Step 4: Systematically integrate the explicit and implicit entity relationship triplets obtained from the text by the open source large language model fine-tuned with implicit information and the pre-trained language model, and finally obtain the combined triples.

[0016] Preferably, the BERT model includes an encoding layer, a pooling layer, a table filling layer, and a decoding layer; based on a word-level table annotation framework, a method for obtaining explicit entity relationship triples in a text includes:

[0017] Perform word segmentation, stop word removal and normalization preprocessing on the input text;

[0018] Using the encoder of the BERT model, the preprocessed text is input into the encoding layer to obtain the context-dependent word vector representation of each word;

[0019] In the pooling layer, the maximum pooling technique is applied to extract the morpheme feature with the largest value as the representative vector of the word, and generate a table dimension consistent with the text length;

[0020] According to the set annotation framework, the table is filled with word-level vector representations and the categories of explicit entities and relations are preset;

[0021] The filled information is passed to the decoding layer to extract the explicit entities and corresponding relationships in the text to form a triple (entity 1, entity 2, relationship).

[0022] Preferably, using the encoder of the BERT model, the preprocessed text is input into the encoding layer, and the method of obtaining the context-related word vector representation of each word includes:

[0023] The input is a sequence of words w1, w2, .., w N , BERT outputs a vector representation of each input word, as shown in the formula:

[0024]

[0025] Among them, Q and K are the query vector and key vector of the token in the sentence respectively. represents the scaling factor, W is the word embedding output by training BERT, W∈R 768 ; After extracting word vectors through BERT, more accurate semantic understanding is achieved and text classification accuracy is improved, while reducing dependence on training data;

[0026] According to the set annotation framework, the methods of filling the table with word-level vector representation include:

[0027] The 2D table matrix T∈R concatenated using word embeddings N*N , where N represents the length of the sentence; the row and column of each cell in the table represent the i-th word and j-th word in the sentence respectively, and the concatenation operation is performed by concatenating two pooled word vectors representing the current word; the upper triangular matrix part is selected and input into the linear layer to obtain the final form of the table, which is expressed as follows:

[0028] T i，j =tanh(FC tabal ([emb i ;emb j ])), (j≥i) where FC tabel () indicates feature extraction by linear layer, [;] indicates concatenation of two features, T i,j is the feature representing the cell in row i and column j in the table, emb i 、emb j The final word embedding representation is represented by the word embedding representation. By introducing an additional feature fusion mechanism, the model is enhanced to capture complex semantic relationships, thereby improving the prediction accuracy and generalization ability of the model.

[0029] The filled information is passed to the decoding layer, and the explicit entities and corresponding relationships in the text are extracted to form a triple, namely entity 1, entity 2, and relationship. The method includes:

[0030] Assign the filled information to the entity recognition layer and the relationship recognition layer.

[0031] h i，j =Softmax(FC(T i，j ))

[0032] T′ i，j = argmax(h i，j )

[0033] Among them, FC() can represent the linear layers of entity, relation head and relation tail respectively, T' i,j Indicates that the model finally predicts that the i-th row and j-th column in the table are entities, relation heads, and relation tails. In addition, the number of linear layers of relation heads and relation tails is the same as the number of predefined relations. i,j Represents the predicted table label; through the entity recognition layer and the relationship recognition layer, accurate recognition and positioning of entity relationship pairs are achieved, achieving the effect of improving the accuracy and efficiency of knowledge graph construction;

[0034] During the decoding process, each relationship head and tail identified in the relationship table are matched with the entities identified in the entity table. The mutual matches identified by these three constitute a triple that meets the preset requirements.

[0035] Preferably, the method of obtaining implicit knowledge in the text as an external knowledge source from the GPT model includes:

[0036] Merge the input text with relevant prompt information;

[0037] Organize the fused text into a format suitable for GPT model processing;

[0038] After submitting the sorted input text to the GPT model, relevant implicit knowledge is obtained;

[0039] Organize the filtered implicit knowledge into a structured JSON file;

[0040] The method of integrating the input text with the relevant prompt information includes:

[0041] Assume that the input text is T and the prompt information is P, then the fused text is represented as:

[0042] F = T + P;

[0043] After submitting the sorted input text to the GPT model, the methods for obtaining relevant implicit knowledge include:

[0044] K=GPT(F)

[0045] Among them, K represents implicit knowledge text; by fusing the input text F with the prompt information P and combining it with the powerful contextual understanding ability of the GPT model, the effect of extracting deep semantics and potential knowledge from the text is achieved; this method can effectively improve the model's understanding of text content and knowledge inference capabilities, thereby achieving more accurate knowledge extraction and application in fields such as text generation, information retrieval, and question-answering systems.

[0046] Preferably, the method of fine-tuning the instruction by fusing implicit knowledge and prompts into the open source large language model and obtaining the implicit entity relationship triples by reasoning through text and implicit knowledge includes:

[0047] Create training samples that contain implicit knowledge, label the corresponding inputs and outputs, and determine the learning rate, batch size, and number of training rounds for fine-tuning;

[0048] During fine-tuning, the extracted implicit knowledge is combined with the input text to preset prompt instructions for the open source large language model;

[0049] Run the fine-tuning process on the training samples, monitor the loss function and accuracy indicators, and ensure that the model gradually converges;

[0050] After fine-tuning is completed, implicit entity relationship triples are extracted from the fine-tuned model;

[0051] The method of creating training samples containing implicit knowledge includes:

[0052] D=(X1,Y1)(X2,Y2),...,(X n , Y n )

[0053] Where D represents the set of training samples containing implicit knowledge, X n represents the input of the nth training sample, Y n Represents the output of the nth training sample;

[0054] During fine-tuning, methods for combining the extracted implicit knowledge with the input text include:

[0055] X′=X+K, where X represents the input of the training sample, and X' is the text after the implicit knowledge is combined with the input text. By integrating the implicit knowledge, the model can better understand the contextual information of the text and combine the implicit knowledge. The model can use additional background information in the reasoning process, thereby enhancing the accuracy of knowledge reasoning;

[0056] The prompt command prompt is preset for the open source large language model. The input of the open source large language model is represented as:

[0057] I=prompt+X′.

[0058] Preferably, the method of systematically integrating the explicit and implicit entity relationship triples obtained from the text by the open source large language model fine-tuned by integrating implicit information and the pre-trained language model to finally obtain the combined triples includes:

[0059] Formulate integration rules, perform deduplication operations on explicit and implicit entity relationship triples, determine whether there is repeated or contradictory information, and integrate explicit and implicit entity relationship triples; through deduplication, cross-validation and conflict resolution, the accuracy and completeness of the knowledge base can be significantly improved; this method not only eliminates data redundancy, but also ensures the consistency and logical self-consistency of knowledge; in addition, such integration enhances the model's ability to understand complex texts and improves the performance of the model;

[0060] According to the integration strategy, the explicit and implicit triples are combined to form the final set of combined triples;

[0061] Among them, the method of formulating integration rules, performing deduplication operations on explicit and implicit triples, and judging whether there is repeated or contradictory information includes:

[0062] E′=set(E)

[0063] I′=set(I);

[0064] Among them, E' and I' are the explicit triples and implicit triples removed by the set, E is the extracted explicit triple, and I is the extracted implicit triple;

[0065] According to the integration strategy, the method of combining explicit and implicit triples to form the final combined triple set includes:

[0066] output=E′+I′

[0067] Among them, output represents the final set of combined triples.

[0068] The present invention also provides an explicit and implicit relationship collaborative extraction device based on a large model and a pre-trained model, comprising: an explicit triple acquisition module, a knowledge source acquisition module, an implicit triple acquisition module and a combination module;

[0069] The explicit triple acquisition module is used to use a pre-trained language model to obtain word vectors of the text, build a word-level table annotation framework based on the word vectors of the text, and obtain explicit entity relationship triples in the text based on the word-level table annotation framework; wherein the pre-trained language model is BERT;

[0070] The knowledge source acquisition module is used to obtain implicit knowledge in the text from the GPT model as an external knowledge source;

[0071] The implicit triple acquisition module is used to fine-tune the instructions by fusing implicit knowledge and prompts into the open source large language model, and obtain implicit entity relationship triples by reasoning through text and implicit knowledge;

[0072] The combination module is used to systematically integrate the explicit and implicit entity relationship triples obtained from the text by the open source large language model fine-tuned by integrating implicit information and the pre-trained language model, and finally obtain the combined triples.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] The present invention adopts a method of obtaining implicit triples by using a large language model and obtaining explicit triples by using a pre-trained model to convert unstructured natural language text into structured information.

[0075] The text explicit vocabulary features identify possible entities and relations, and based on this, all possible candidate triples contained in the sentence are enumerated. The present invention makes full use of the inherent advantages of the table filling method, making the identification process of overlapping relations more efficient and accurate.

[0076] The present invention significantly improves the performance of large language models by using implicit information. Implicit information can effectively capture relational features and improve the reasoning ability of LLM in complex relational sentences. When the complexity of the relationship is high, the LLM that does not accept implicit information shows a performance bottleneck. As the number of triples increases, the recall rate of the LLM that integrates implicit information is significantly improved, at least twice that of the non-integrated one, indicating that implicit information enhances the model's ability to capture potential relationships. In the NYT dataset, the Vicuna series models improved by 6.47% in F1 value compared to ordinary LLM, and llama2-13B and vicuna-13B in WebNLG improved by 2.9% and 1.35% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0078] Figure 1 It is a schematic diagram of a specific implementation process of the method for collaboratively extracting explicit and implicit relations based on a large model and a pre-trained model according to an embodiment of the present invention;

[0079] Figure 2 It is a schematic diagram of evaluating the results of different large-scale language models in pure implicit triples of NYT dataset according to an embodiment of the present invention;

[0080] Figure 3 It is a schematic diagram of evaluating the results of different large-scale language models in pure implicit triples of the WebNLG dataset according to an embodiment of the present invention;

[0081] Figure 4 Schematic diagram of a collaborative extraction method of explicit and implicit relationships based on a large model and a pre-trained model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0084] Embodiment 1

[0085] like Figure 4 The present invention provides a method for collaboratively extracting explicit and implicit relationships based on a large model and a pre-trained model, comprising the following steps:

[0086] Step 1: Use the pre-trained language model to obtain the word vector of the text, build a word-level table annotation framework based on the word vector of the text, and obtain the explicit entity relationship triples in the text based on the word-level table annotation framework; the pre-trained language model is BERT;

[0087] Step 2: Obtain implicit knowledge in the text from the GPT model as an external knowledge source;

[0088] Step 3: Fine-tune the instructions by integrating implicit knowledge and prompts into the open source large language model, and obtain implicit entity relationship triples through reasoning through text and implicit knowledge;

[0089] Step 4: Systematically integrate the explicit and implicit entity relationship triplets obtained from the text by the open source large language model fine-tuned with implicit information and the pre-trained language model, and finally obtain the combined triples.

[0090] Step 1: Use the pre-trained language model to obtain the word vectors of the text and build a word-level table annotation framework. The following are the specific operations. First, select BERT as the base model to ensure that it is suitable for the target dataset. Then, the input text is segmented, stop words are removed, and normalized to ensure that the model can process efficiently. Next, use the encoder of the BERT model to input the preprocessed text into the encoding layer to obtain the context-related word vector representation of each word.

[0091] The input is a sequence of words w1, w2, ..., w N , BERT outputs a vector representation of each input word, and its formula is as follows:

[0092]

[0093] Among them, Q and K are the query vector and key vector of the token in the sentence respectively. represents the scaling factor, W is the word embedding output by training BERT, W∈R 768 , and the word embedding output by training BERT. After extracting word vectors through BERT, more accurate semantic understanding is achieved and text classification accuracy is improved, while reducing the dependence on training data. This method effectively captures the explicit contextual relationships in the text, thereby obtaining deep semantic information in the sentence. Based on this embedded representation, a table filling method is constructed to achieve the recognition of entities and relationships in complex texts.

[0094] In order to overcome the problem that the word segmentation method divides the words that do not appear in the vocabulary into multiple morphemes according to the words that appear in the vocabulary, it will cause the feature length to be changed from the number of words in the text to the number of morphemes, resulting in inconsistency between the feature length and the text length, which will cause the table to be too large. In order to solve this problem, the present invention uses the maximum pooling technology to extract the one with the largest value in the morpheme feature as the representative vector of the word, and generates a table dimension consistent with the text length, thereby reducing the redundancy of the table. This has a huge impact on the subsequent table filling, and the maximum pooling method is used to establish the table, and its formula is as follows:

[0095] emb i =maxpooling(W i：start , ..., W i：end )

[0096] Among them, W i:start ,...,W i:end Indicates multiple subwords of a word generated by the BPE algorithm of the BERT model, maxpooling() represents the maximum pooling function, emb i Represents the final word embedding representation.

[0097] The table is filled with word-level vector representations according to the set annotation framework, which should define explicit entity and relation categories. The two-dimensional table matrix T∈R is concatenated using word embeddings. N*N , where N represents the length of the sentence. The row and column of each cell in the table represent the i-th word and j-th word in the sentence respectively, and the concatenation operation is performed by concatenating two pooled word vectors representing the current word. Select the upper triangular matrix part and input it into the linear layer to get the final form of the table. The formula is as follows:

[0098] T i，j =tanh(FC tabel ([emb i ;emb j ])), (j≥i)

[0099] Among them, FC tabel () indicates feature extraction by linear layer, [;] indicates concatenation of two features, T i,j is the feature representing the cell in the i-th row and j-th column of the table. By introducing an additional feature fusion mechanism, the model is enhanced to capture complex semantic relationships, thereby improving the prediction accuracy and generalization ability of the model.

[0100] Finally, the filled information is passed to the decoding layer to extract the explicit entities and their relationships in the text to form a triple (entity 1, entity 2, relationship), such as (Apple, founder, Steve Jobs). It is assigned to the entity recognition layer and the relationship recognition layer. The formula is as follows

[0101] h i，j =Softmax(FC(T i，j ))

[0102] T′ i，j = argmax(h i，j )

[0103] Among them, FC() can represent the linear layers of entity, relation head and relation tail respectively, T' i,j Indicates that the model finally predicts that the i-th row and j-th column in the table are entities, relation heads, and relation tails. In addition, the number of linear layers of relation heads and relation tails is the same as the number of predefined relations. i,j Represents the predicted table label. Through the entity recognition layer and the relationship recognition layer, the entity relationship pairs are accurately identified and located, achieving the effect of improving the accuracy and efficiency of knowledge graph construction. In the decoding process, the head and tail of each relationship identified in the relationship table are matched with the entities identified in the entity table. Only when the three are matched can a qualified triple be formed.

[0104] Step 2: Obtain implicit knowledge in the text from GPT (Generative Pre-trained Transformer) as an external knowledge source. First, you need to prepare the input text and fuse it with relevant prompt information to guide the model to generate the required implicit knowledge. Let the input text be T and the prompt information be P, then the fused text can be expressed as:

[0105] F=T+P

[0106] Organizing the fused text into a format suitable for model processing can improve the efficiency of information processing and enhance the relevance of the generated text. After submitting the organized input text to the GPT model, relevant implicit knowledge can be obtained. Inputting the fused text F into the GPT model can obtain the implicit knowledge text K, which can be expressed by the equation:

[0107] K=GPT(F)

[0108] Finally, the implicit knowledge filtered out by the implicit knowledge text K is stored as a structured json file. By fusing the input text F with the prompt information P and combining the powerful contextual understanding ability of the GPT model, the effect of extracting deep semantics and potential knowledge from the text is achieved. This method can effectively improve the model's understanding of text content and knowledge inference capabilities, thereby achieving more accurate knowledge extraction and application in fields such as text generation, information retrieval, and question-answering systems. This method can not only be used directly for current tasks, but also serve as the basis for subsequent reasoning to provide support for in-depth analysis and decision-making. Such integration and systematization will make the application of implicit knowledge more efficient and accurate.

[0109] Step 3: Fine-tune the instructions by integrating implicit information and prompts into the open source large language model. First, create training samples containing implicit knowledge and label the corresponding input X and output Y. Suppose the training sample set is:

[0110] D=(X1,Y1)(X2,Y2),...,(X n , Y n )

[0111] Determine the learning rate η, batch size B, and number of training rounds epoch for fine-tuning. During the fine-tuning process, the implicit knowledge extracted in the previous step is combined with the input text to provide the model with richer contextual information.

[0112] X′=X+K

[0113] Among them, X is the input text, K is the implicit knowledge text, and X' is the text after the implicit knowledge is combined with the input text. By integrating implicit knowledge, the model can better understand the contextual information of the text and combine implicit knowledge. The model can use additional background information in the reasoning process, thereby enhancing the accuracy of knowledge reasoning.

[0114] Design clear prompts for large language models to help the model understand the expected output and make effective inferences. Let the prompt be P, then the input of the model can be expressed as:

[0115] I=prompt+X′

[0116] Then, the fine-tuning process is run on the prepared dataset D, and indicators such as loss function and accuracy are monitored to ensure that the model gradually converges. After fine-tuning is completed, implicit entity relationship triples are extracted from the fine-tuned model, for example, (JoeBuck, St.Louis Cardinals, children).

[0117] Step 4: Systematically integrate the large language model fine-tuned with implicit information with the triplets of the pre-trained language model. First, extract the explicit triples E and implicit triples I from steps 1 and 3 respectively to form two independent sets of triples.

[0118] Formulate integration rules to remove duplicates from explicit and implicit triples and determine whether there is duplicate or contradictory information, which can be expressed as:

[0119] E′=set(E)

[0120] I′=set(I)

[0121] Among them, E' and I' are the explicit triples and implicit triples respectively through set deduplication, E is the explicit triple extracted in step 1, and I is the implicit triple extracted in step 3. Integrating explicit and implicit entity relationship triples; through deduplication, cross-validation and conflict resolution, the accuracy and completeness of the knowledge base can be significantly improved; this method not only eliminates data redundancy, but also ensures the consistency and logical self-consistency of knowledge; in addition, such integration enhances the model's ability to understand complex texts and improves the performance of the model.

[0122] In addition, it is ensured that the triples output by the two models are semantically consistent, so that the semantic information expressed is logically self-consistent and non-contradictory. According to the integration strategy, the explicit and implicit triples are combined to form the final combined triple set output.

[0123] output=E′+I′

[0124] Finally, the combined triples are verified to ensure that they are logically reasonable and meet the expected task objectives. Figure 1 The framework for combining large and small model relation extraction is shown. The input layer receives raw data and performs preprocessing, which is then passed to the small table filling model and the large language model for calculation. The small model obtains explicit triples, which is limited in complexity; while the large language model consumes more resources, it can obtain triples based on implicit information. In the rule fusion layer, the outputs of the two models are integrated through predefined rules.

[0125] In the present invention, in order to train the model and evaluate the performance of the relation triple extraction task, a cross entropy loss function is used, and its formula is as follows:

[0126]

[0127] Among them, g i,j represents the actual table label, indicating the real semantic connection between elements in the sentence, h i,j Represents the predicted table labels.

[0128] This paper uses LLM's deep semantic understanding and contextual reasoning capabilities to effectively extract implicit knowledge from texts and fine-tune the performance of LLM in this field, thereby helping to improve the accuracy and effectiveness of relation extraction. The effectiveness of prompts has an important impact on model performance. Therefore, optimizing prompt design may be one of the key factors to improve the performance of LLMs on specific tasks. The prompts for complex relation extraction are as follows:

[0129] Task description: Provides a set of predefined relation classes R and a description of the RE task. Relation R belongs to the predefined class, and explicitly requires the model to output triples of relations in the set, and the format of the model output triples is fixed. Because it is a complex relation triple extraction, multiple triples need to be output.

[0130] Implicit information: OpenAI API is used to analyze implicit information through sentences and entities. Accessed through the gpt-3.5-turbo-0301 model, it can efficiently understand the text and extract key implicit information based on common sense knowledge obtained through training on a large amount of text data. The acquired implicit information is integrated into the prompt, providing effective information for LLM to extract relations and build a new prompt.

[0131] Text input: Provides the context input of the dataset, expressed using .

[0132] Given an input text C and the above description combined with the implicit information prompt, the goal is to generate a text sequence Y of relation triples extracted based on implicit information. The probability of generating a relation triple sequence Y with the input text C and the prompt as conditions is denoted as ρLLM(y|C,x). This probability can be decomposed into the product of the conditional probabilities of each token Y,n in the output sequence Y, where N represents the number of tokens in the output sequence, as shown in the following formula:

[0133]

[0134] In complex relation extraction, a labeled sentence usually contains multiple related triples, which represent different entities in the sentence and the relationships between them. In order to effectively use this information, it is necessary to integrate multiple triples from two different models through a specific method. To this end, the present invention effectively integrates the elimination of redundant triples and semantic consistency rules, fully utilizes the advantages of each model, and improves the overall reasoning ability and accuracy.

[0135] Eliminate redundant triples: When merging triples, identify and eliminate redundant information to avoid duplication and improve the simplicity and clarity of the results. Use fuzzy matching and similarity calculation to compare the similarity between entities and relationships. Combined with the algorithm, you can automatically identify and exclude triples with high similarity.

[0136] Semantic consistency rule: Ensure that the triples output by the two models are semantically consistent, so that the semantic information expressed is logically self-consistent and not contradictory. When faced with such contradictory triples, choose to retain the results generated by LLM because LLM can integrate multiple sources of information and provide a relatively comprehensive perspective.

[0137] Embodiment 2

[0138] This embodiment is carried out in a hardware environment with a graphics card NVIDIA A40 and a video memory of 48G*5. The system is Ubuntu 20.04, the development language is Python 3.11, and the deep learning framework is Pytorch 2.1.1.

[0139] In the present invention, the relation extraction datasets NYT and WebNLG are used. These complex relation extraction datasets that are publicly available for download are challenging for many extraction methods.

[0140] The NYT dataset contains 66k sentences, which is generated by aligning the relations in Freebase with the New York Times (NYT) corpus. It contains 24 relation tags, and there are multiple triples in the text. The WebNLG corpus consists of triples describing facts (entities and relations between entities) and corresponding facts in the form of natural language text. The corpus contains up to 7 triples, each with one or more reference texts, and the distribution of the data is shown in Table 1.

[0141] Table 1. Data distribution table

[0142]

[0143] In order to better evaluate the effectiveness of the method in complex scenarios, this study uses exact match (EM) to evaluate the reasoning results of the model in the experiment. The extracted relation triple is considered correct only when it is completely consistent with the true relation triple. This consistency requirement covers the subject and object, while also ensuring that the identified relationship is accurate. In addition, all baseline models were subjected to standard micro-precision, recall, and F1 value statistical analysis to evaluate their performance.

[0144] During the large model training process, the specific parameter settings are shown in Table 2.

[0145] Table 2. Lora parameter table

[0146]

[0147] This framework is experimented on multiple datasets, and the experimental results are shown in Table 3. w / o II means that the large model in the framework does not use implicit information, w / o RF means that the large model only uses implicit information without using the framework, and w / o all means that the framework is not used and the large model has no implicit information.

[0148] As can be seen from Table 3, the proposed method and device for collaborative extraction of explicit and implicit relations based on large models and pre-trained models were evaluated on the parts of the NYT and Webnlg test sets where the number of implicit triples exceeded 60%.

[0149] The implicit information generated by GPT is integrated into the prompt of LLM to obtain common sense knowledge between entities that is not explicit in the text, and then determine the non-explicit relationship fragments in the text to help LLM perform relationship extraction tasks. Compared with the prompt without implicit information in entity relationship extraction, vicuna-13 achieved the best results with the help of implicit information and framework in the complex relationship extraction triple task, and the improvement effect was also the greatest. Other large models also achieved different degrees of improvement through implicit information and framework. Compared with the part that does not use the framework and the large model does not contain implicit information, the smallest improvement in the NYT dataset is 29.05%, and the smallest improvement in WebNLG is 16.21%. Implicit information is significantly helpful in identifying the relationship between entities, giving full play to the reasoning ability of the large language model (LLM), so the effectiveness of implicit information in relationship extraction tasks is certain.

[0150] Comparing the use of implicit information under the same framework, it can be found that the effect is equally significant. There has been a significant improvement in each large language model, with the smallest improvement of 5.12% in the NYT dataset and 5.64% in WebNLG. Compared with the performance of the large language model with ordinary prompts and the LLM with implicit information. In the NYT dataset, the Vicuna series models can make full use of the implicit information provided when processing tasks, thereby improving the F1 value by 6.47% and 0.88% over ordinary LLMs that rely on traditional prompts. In the WebNLG dataset, llama2-13B and vicuna-13B also showed this phenomenon, with improvements of 2.9% and 1.35%.

[0151] Specifically, when the number of parameters of a large model reaches a certain level, it will have a better effect in complex situations. By optimizing its interpretation and application of implicit information, it shows a higher depth of understanding and reasoning ability. This phenomenon shows that the effective use of implicit information not only improves the generation quality of the model, but also enhances its adaptability and flexibility in specific situations. In addition, the experiment also revealed the advantages of these models in understanding complex contexts and capturing potential connections.

[0152] Table 3. Experimental results

[0153]

[0154]

[0155] In order to verify whether the large model can really provide help when facing texts with multiple entity relationship triples with the help of implicit information, we classified them according to the difference in the number of triples to determine the efficiency improvement brought by implicit information in facing different complex situations. Table 4 takes vicuna-13B as an example, and it can be found that: (1) LLM achieved better performance in facing sentences with different degrees of complex relations after incorporating implicit information. This means that implicit information can capture the characteristics of relations and thus help LLM to reason. (2) The F1 of LLM without implicit information is roughly the same when the triples are 3 to 4 and greater than or equal to 5. This shows that when the relations in the data are complex to a certain extent, LLM is limited in its ability to extract relations, resulting in a performance bottleneck. Therefore, as the complexity of relations increases, the role of implicit information becomes more significant. (3) As the number of triples increases, the precision of LLM with implicit information does not change much, and the recall rate is at least twice that of LLM without implicit information. This shows that implicit information enables the model to capture potential relationships or patterns more effectively, thereby significantly enhancing the recall rate.

[0156] Table 4. Effects of vicuna-13B at different complexity on NYT data

[0157]

[0158] In the case of purely implicit triples, the improvement brought by implicit information is Figure 2 , Figure 3 It can be seen that: (1) In the absence of implicit information, the fine-tuned LLM may miss key contextual clues when performing relation extraction, resulting in inaccurate understanding and extraction of relations. Therefore, by introducing implicit information, the model can understand the text content more comprehensively and improve its performance in complex relation extraction, proving the importance of implicit information in relation extraction tasks. (2) When various fine-tuned LLMs use implicit information to face pure implicit triples, the results obtained are generally higher recall rates than precision rates, which shows that the model can capture as many real positive examples as possible in the implicit triple field. The reason for this phenomenon is that with the addition of implicit information, LLM obtains sufficient information to perform entity relationship triple extraction tasks, but it is precisely because of the implicit information that some of the extracted triples are logically correct but do not appear in the golden label. Such triples are false positives.

[0159] Table 5 Performance of different LLMs on NYT data where all triples require implicit information. The data is roughly classified according to the number of triples, where II stands for implicit information.

[0160]

[0161] Embodiment 3

[0162] The present invention also provides an explicit and implicit relationship collaborative extraction device based on a large model and a pre-trained model, comprising: an explicit triple acquisition module, a knowledge source acquisition module, an implicit triple acquisition module and a combination module;

[0163] The explicit triple acquisition module is used to obtain the word vector of the text using the pre-trained language model, build a word-level table annotation framework based on the word vector of the text, and obtain the explicit entity relationship triples in the text based on the word-level table annotation framework; wherein the pre-trained language model is BERT;

[0164] The knowledge source acquisition module is used to obtain implicit knowledge in the text from the GPT model as an external knowledge source;

[0165] The implicit triple acquisition module is used to fine-tune instructions by integrating implicit knowledge and prompts into the open source large language model, and to obtain implicit entity relationship triples through reasoning based on text and implicit knowledge;

[0166] The combination module is used to systematically integrate the explicit and implicit entity relationship triplets obtained from the text by the open source large language model fine-tuned with implicit information and the pre-trained language model, and finally obtain the combined triples.

[0167] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A collaborative extraction method of explicit and implicit relations based on a large model and a pre-trained model, characterized in that: The following steps are involved: Step 1: Use the pre-trained language model to obtain the word vector of the text, build a word-level table annotation framework based on the word vector of the text, and obtain the explicit entity relationship triples in the text based on the word-level table annotation framework; the pre-trained language model is BERT; Step 2: Obtain implicit knowledge in the text from the GPT model as an external knowledge source; Step 3: Fine-tune the instructions by integrating implicit knowledge and prompts into the open source large language model, and obtain implicit entity relationship triples through reasoning through text and implicit knowledge; Step 4: Systematically integrate the explicit and implicit entity relationship triplets obtained from the text by the open source large language model fine-tuned with implicit information and the pre-trained language model, and finally obtain the combined triples.

2. The method for collaborative extraction of explicit and implicit relations based on a large model and a pre-trained model according to claim 1 is characterized in that: The BERT model consists of an encoding layer, a pooling layer, a table filling layer, and a decoding layer; Based on the word-level table annotation framework, the methods for obtaining explicit entity relationship triples in the text include: Perform word segmentation, stop word removal and normalization preprocessing on the input text; Using the encoder of the BERT model, the preprocessed text is input into the encoding layer to obtain the context-dependent word vector representation of each word; In the pooling layer, the maximum pooling technique is applied to extract the morpheme feature with the largest value as the representative vector of the word, and generate a table dimension consistent with the text length; According to the set annotation framework, the table is filled with word-level vector representations and the categories of explicit entities and relations are preset; The filled information is passed to the decoding layer to extract the explicit entities and corresponding relationships in the text to form a triple (entity 1, entity 2, relationship).

3. The method for collaborative extraction of explicit and implicit relationships based on a large model and a pre-trained model according to claim 2 is characterized in that: Using the encoder of the BERT model, the preprocessed text is input into the encoding layer. The method of obtaining the context-related word vector representation of each word includes: The input is a sequence of words w1,w,2,...,w N , BERT outputs a vector representation of each input word, as shown in the formula: Among them, Q and K are the query vector and key vector of the token in the sentence respectively. represents the scaling factor, W is the word embedding output by training BERT, W∈R 768 ; According to the set annotation framework, the methods of filling the table with word-level vector representation include: The 2D table matrix T∈R concatenated using word embeddings N*N , where N represents the length of the sentence; the row and column of each cell in the table represent the i-th word and j-th word in the sentence respectively, and the concatenation operation is performed by concatenating two pooled word vectors representing the current word; the upper triangular matrix part is selected and input into the linear layer to obtain the final form of the table, which is expressed as follows: T ij =tanh(FC tabel ([emb i ;emb j ])),(j≥i) Among them, FC tabel () indicates feature extraction by linear layer, [;] indicates concatenation of two features, T i，j is the feature representing the cell in row i and column j in the table, emb i 、emb j Represents the final word embedding representation; The filled information is passed to the decoding layer to extract the explicit entities and corresponding relationships in the text to form a triple, namely entity 1, entity 2, and relationship. The method includes: Assign the filled information to the entity recognition layer and the relationship recognition layer. h ij =Softmax(FC(T i，j )) T′ i，j =argmax(h i，j ) Among them, FC() can represent the linear layers of entity, relation head and relation tail respectively, T′ i，j Indicates that the model finally predicts that the i-th row and j-th column in the table are entities, relation heads, and relation tails. In addition, the number of linear layers of relation heads and relation tails is the same as the number of predefined relations. i，j Represents the predicted table label; During the decoding process, each relationship head and tail identified in the relationship table are matched with the entities identified in the entity table. The mutual matches identified by these three constitute a triple that meets the preset requirements.

4. The method for collaborative extraction of explicit and implicit relations based on a large model and a pre-trained model according to claim 1 is characterized in that: Methods for obtaining implicit knowledge in text from the GPT model as an external knowledge source include: Merge the input text with relevant prompt information; Organize the fused text into a format suitable for GPT model processing; After submitting the sorted input text to the GPT model, relevant implicit knowledge is obtained; Organize the selected implicit knowledge into a systematic knowledge base; The method of integrating the input text with the relevant prompt information includes: Assume that the input text is T and the prompt information is P, then the fused text is represented as: F = T + P; After submitting the sorted input text to the GPT model, the methods for obtaining relevant implicit knowledge include: K=GPT(F) Among them, K represents implicit knowledge text.

5. The method for collaborative extraction of explicit and implicit relations based on a large model and a pre-trained model according to claim 4 is characterized in that: The method of fine-tuning the instructions by integrating implicit knowledge and prompts into the open source large language model and obtaining implicit entity relationship triples through reasoning through text and implicit knowledge includes: Create training samples that contain implicit knowledge, label the corresponding inputs and outputs, and determine the learning rate, batch size, and number of training rounds for fine-tuning; During fine-tuning, the extracted implicit knowledge is combined with the input text to preset prompt instructions for the open source large language model; Run the fine-tuning process on the training samples, monitor the loss function and accuracy indicators, and ensure that the model gradually converges; After fine-tuning is completed, implicit entity relationship triples are extracted from the fine-tuned model; The method of creating training samples containing implicit knowledge includes: D=(X1,Y1)(X2,Y2),…,(X n ,Y n ) Where D represents the set of training samples containing implicit knowledge, X n represents the input of the nth training sample, Y n Represents the output of the nth training sample; During fine-tuning, methods for combining the extracted implicit knowledge with the input text include: X′=X+K Among them, X represents the input of the training sample, and X′ is the text after the implicit knowledge is combined with the input text; The prompt command prompt is preset for the open source large language model. The input of the open source large language model is represented as: I=prompt+X′.

6. The method for collaborative extraction of explicit and implicit relations based on a large model and a pre-trained model according to claim 1, characterized in that: The method of systematically integrating the explicit and implicit entity relationship triples obtained from the text by the open source large language model fine-tuned with implicit information and the pre-trained language model to finally obtain the combined triples includes: Formulate integration rules to deduplicate explicit and implicit entity relationship triples and determine whether there is duplicate or contradictory information; According to the integration strategy, the explicit and implicit triples are combined to form the final set of combined triples; Among them, the method of formulating integration rules, performing deduplication operations on explicit and implicit triples, and judging whether there is repeated or contradictory information includes: E′=set(E) I′=set(I); Among them, E′ and I′ are the explicit triples and implicit triples removed by set, E is the extracted explicit triple, and I is the extracted implicit triple; According to the integration strategy, the method of combining explicit and implicit triples to form the final combined triple set includes: output=E′+I′ Among them, output represents the final set of combined triples.

7. A collaborative extraction device for explicit and implicit relations based on a large model and a pre-trained model, characterized in that: include: Explicit triple acquisition module, knowledge source acquisition module, implicit triple acquisition module and combination module; The explicit triple acquisition module is used to use a pre-trained language model to obtain word vectors of the text, build a word-level table annotation framework based on the word vectors of the text, and obtain explicit entity relationship triples in the text based on the word-level table annotation framework; wherein the pre-trained language model is BERT; The knowledge source acquisition module is used to obtain implicit knowledge in the text from the GPT model as an external knowledge source; The implicit triple acquisition module is used to fine-tune the instructions by fusing implicit knowledge and prompts into the open source large language model, and obtain implicit entity relationship triples by reasoning through text and implicit knowledge; The combination module is used to systematically integrate the explicit and implicit entity relationship triples obtained from the text by the open source large language model fine-tuned by integrating implicit information and the pre-trained language model, and finally obtain the combined triples.

Images