Large language model knowledge injection method based on domain data matching

Abstract

The invention relates to a large language model knowledge injection method based on domain data matching, and belongs to the technical field of large language model, domain knowledge and natural language processing. According to the method, a self-supervised continuous pre-training framework is designed, and the framework can utilize selected knowledge to perform knowledge enhancement on LLMs under the condition of not increasing extra manual annotation cost; a knowledge screening mechanism based on homogeneity measurement is provided and used for selecting high-quality knowledge related to a target task from a large-scale knowledge graph, and the domain perception of LLMs is effectively improved. Wide experiments are carried out on NER tasks in a plurality of low-resource vertical fields, the effectiveness of the method is verified by experimental results, and particularly, the method has excellent performance in the aspect of improving the recognition capability of the model on complex entity types.

Description

Technical Field

[0001] The present invention belongs to the technical fields of large language models, domain knowledge, and natural language processing, and specifically relates to a large language model knowledge injection method based on domain data matching. Background Art

[0002] Continuous pre-training on large language models has been widely used to extend the model's basic understanding of specific downstream domains (e.g., mathematics and code). For continuous pre-training of large language models on specific domain knowledge, an important question is how to choose the best mixing ratio between the general corpus and the downstream domain corpus. Existing methods usually use manual grid search for a set of mixing ratios, which consumes a lot of GPU training overhead. In addition, we cannot guarantee that the selected ratio is optimal for a specific domain. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] Considering the complexity of domain knowledge in real network environments, current LLMs often perform poorly when processing NER tasks in vertical fields or low-resource languages ​​due to lack of sufficient knowledge. Therefore, the present invention proposes a large language model knowledge injection method.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention provides a large language model knowledge injection method based on domain data matching, comprising the following steps:

[0007] Step 1: Knowledge data construction:

[0008] Firstly, the non-real knowledge ontology skeleton set T_uos of large language models (LLMs) is obtained by the OntoFact method, a framework for factual detection based on ontology. Then, the non-fact rate UR of each ontology-level triple in T_uos obtained by the OntoFact method is used to sort the ontology-level triples from high to low using the OntoInjection framework of ontology, and the first n ontology-level triples are selected. Finally, OntoInjection obtains the corresponding triple text and natural language text dataset from the knowledge base based on the instance-level triples of non-real knowledge related to these n ontology-level triples.

[0009] Step 2: Ontology-driven knowledge filtering:

[0010] Using OntoInjection, the crawled text is filtered based on the instance-level triples extracted from the dataset of the triple text and the natural language text, so as to screen out the natural language text that has the potential to enhance the domain knowledge of the large language model (LLMs), that is, to obtain a screened natural language text dataset;

[0011] Step 3: Knowledge Injection:

[0012] The natural language text dataset obtained after screening in step 2 is used to inject knowledge into LLMs, wherein the injection of LLMs is achieved through continuous pre-training to obtain a large language model with stronger domain perception capabilities.

[0013] The present invention also provides a system for implementing the method.

[0014] The present invention also provides a method for evaluating the effectiveness of the method.

[0015] (III) Beneficial effects

[0016] The present invention provides a large language model knowledge injection method based on domain data matching, the main advantages of which are reflected in the following aspects:

[0017] 1. We design a self-supervised continuous pre-training framework that can enhance LLMs with selected knowledge without incurring additional manual annotation costs.

[0018] 2. A knowledge screening mechanism based on homogeneity measurement is proposed to select high-quality knowledge related to the target task from large-scale knowledge graphs, which effectively improves the domain perception of LLMs;

[0019] 3. Extensive experiments were conducted on NER tasks in multiple low-resource vertical fields. The experimental results verified the effectiveness of this method, especially in improving the model's ability to recognize complex entity types. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a logic flow chart of the present invention;

[0021] Figure 2 , Figure 3 , Figure 4 The specific framework diagrams of the three steps of the method of the present invention are shown in sequence. DETAILED DESCRIPTION

[0022] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0023] The present invention proposes a large language model knowledge injection method based on domain data matching, aiming to inject rich and accurate knowledge into large language models (LLMs) through a carefully designed knowledge injection strategy. The present invention includes three steps: knowledge data construction, ontology-driven knowledge filtering, and knowledge injection. The focus of knowledge data construction is to carefully design data that is beneficial to the continuous pre-training of LLMs. First, the non-real knowledge ontology skeleton set T_uos of LLMs is obtained through the ontology-based factual detection framework (OntoFact) method. Then, based on the non-fact rate (UR) of each ontology-level triple output by the OntoFact module, the ontology-based knowledge injection framework (OntoInjection) sorts them from high to low and selects n ontology-level triples. Finally, OntoInjection obtains the corresponding triple text and natural language text from the knowledge base according to the instance-level triples of non-real knowledge related to the n ontology-level triples. Ontology-driven knowledge filtering, given instance-level triples (h, r, t) derived from natural language texts (NLTs), we first summarize them into ontology-level triples (C h ,r,C t ), proposed a quantitative metric, the homogeneity metric score, which considers the density of knowledge in NLTs and the knowledge missing rate relative to LLMs. Knowledge injection injects knowledge into LLMs through the selected dataset, focusing on injecting knowledge into LLMs through continuous pre-training.

[0024] In this paper, we design a self-supervised continuous pre-training framework, which can enhance LLMs with selected knowledge without incurring additional manual annotation costs.

[0025] In addition, the present invention proposes a knowledge screening mechanism based on homogeneity measurement, which is used to select high-quality knowledge related to the target task from a large-scale knowledge graph, effectively improving the domain perception of LLMs. The present invention has conducted extensive experiments on NER tasks in multiple low-resource vertical fields, and the experimental results verify the effectiveness of the method, especially in improving the model's ability to recognize complex entity types. Therefore, the present invention has broad application prospects and promotion scope.

[0026] As a specific embodiment of the present invention, the present invention provides a logic flow chart as follows Figure 1 The model flow chart is shown in Figure 2 , 3 , a large language model knowledge injection method based on domain data matching shown in 4, the specific steps are:

[0027] Step 1: Knowledge data construction:

[0028] refer to Figure 1 First, the non-real knowledge ontology skeleton set T_uos of LLMs is obtained by the OntoFact method. Then, based on the non-fact rate UR of each ontology-level triple in T_uos obtained by the OntoFact method, the ontology-based knowledge injection framework OntoInjection sorts the ontology-level triples from high to low and selects the first n ontology-level triples. Finally, OntoInjection obtains the corresponding triple texts and natural language texts NLTs from the knowledge base (such as Wikipedia) according to the instance-level triples of non-real knowledge related to these n ontology-level triples.

[0029] Step 2: Ontology-driven knowledge filtering:

[0030] refer to Figure 2 In view of the diversity of natural language texts, OntoInjection needs to use instance-level triples extracted from triple texts and natural language texts to filter crawled texts, so as to screen out natural language texts NLTs that have the potential to enhance the domain knowledge of large language models (LLMs) and obtain filtered natural language texts. To this end, the present invention proposes an ontology-driven knowledge filtering strategy. Specifically, given instance-level triples (h, r, t) derived from NLTs, they are first summarized into ontology-level triples (C h ,r,C t ), where g refers to the head entity, r refers to the relationship, t refers to the tail entity, and C h Refers to the entity corresponding to the head entity, C t Refers to the ontology corresponding to the tail entity. In order to measure the quality of NLTs, this paper proposes a quantitative metric - homogeneity metric score, which considers the density of knowledge in NLTs and the missing knowledge completion rate relative to LLMs. Each ontology-level triple (C h ,r,C t ) is calculated as follows:

[0031]

[0032] Where L represents the length of NLT, N represents the number of ontology-level triples mapped to instance-level triples derived from the current NLT, and |Xi| represents the length of the i-th ontology-level triple in NLT.

[0033] In addition, using the ontology-based fact detection framework OntoFact, each ontology-level triple (C h ,r,C t ) of UR((C h ,r,C t)), assign a missing knowledge completion rate relative to LLMs to the current NLT, which is expressed as follows:

[0034]

[0035] represents the ith ontology-level triplet, with a total of N ontology-level triplet;

[0036] In summary, the homogeneity measure score can be expressed as follows:

[0037] score=α×ρ+β×τ

[0038] Among them, α and β are hyperparameters used to balance the weights of knowledge density and missing knowledge completion rate in NLTs. Using the above metrics, the relevant NLTs are ranked from high to low scores and knowledge injection is prioritized to enhance the domain awareness of native LLMs.

[0039] In this way, OntoInjection is able to not only evaluate the knowledge richness and the ability to complete missing knowledge in NLTs, but also optimize the knowledge selection and injection process based on these metrics, thereby improving the accuracy and reliability of LLMs in processing real-world knowledge.

[0040] Step 3: Knowledge Injection:

[0041] refer to Figure 3 , knowledge injection is performed into LLMs through the screened NLTs dataset, focusing on the injection of LLMs through continuous pre-training. Let D represent the obtained screened NLTs dataset, and use the parameter-efficient fine-tuning method to optimize the following knowledge injection objective function L(θ LLMs ), which is expressed as:

[0042]

[0043] Among them, θ LLMs Indicates the LLMs parameters to be updated, x 1 ,x 2 ,…,x L represents a piece of text in the dataset D, the maximum length of the text is L, represents the probability of word x, Represents the probability of the next word. Through sufficient continuous pre-training, a new large language model LLMsθ′ with updated parameters can be obtained LLMs , hoping to improve its domain perception ability, θ′ LLMs Represents the updated LLMs parameters. Finally, the updated LLMsθ′ is used LLMs Complete the named entity recognition (NER) task.

[0044] In order to verify the effectiveness of knowledge injection in this simple way, the present invention incorporates two types of evaluation scenarios: multiple-choice questions and open-ended questions. In these two types of scenarios, OntoInjection designs simple and complex questions, where simple questions mean that the answers can be read directly from the text without thinking, and complex questions mean that the answers can only be obtained by a certain degree of reasoning in the text.

[0045] OntoInjection can not only evaluate the impact of knowledge injection on the improvement of LLMs' domain perception ability, but also comprehensively test the actual application performance of knowledge injection effects through two different types of evaluation scenarios: multiple choice and open-ended questions. The comparison of simple and complex questions further demonstrates the difference in the model's ability to process information of different complexities, as well as the contribution of knowledge injection to enhancing the model's understanding and reasoning capabilities. This evaluation method helps to fully understand the impact of knowledge injection on improving the authenticity and reliability of LLMs in real application scenarios.

[0046] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A large language model knowledge injection method based on domain data matching, characterized in that: The following steps are involved: Step 1: Knowledge data construction: Firstly, the non-real knowledge ontology skeleton set T_uos of large language models (LLMs) is obtained by the OntoFact method, a framework for factual detection based on ontology. Then, the non-fact rate UR of each ontology-level triple in T_uos obtained by the OntoFact method is used to sort the ontology-level triples from high to low using the OntoInjection framework of ontology, and the first n ontology-level triples are selected. Finally, OntoInjection obtains the corresponding triple text and natural language text dataset from the knowledge base based on the instance-level triples of non-real knowledge related to these n ontology-level triples. Step 2: Ontology-driven knowledge filtering: Using OntoInjection, the crawled text is filtered based on the instance-level triples extracted from the dataset of the triple text and the natural language text, so as to screen out the natural language text that has the potential to enhance the domain knowledge of the large language model (LLMs), that is, to obtain a screened natural language text dataset; Step 3: Knowledge Injection: The natural language text dataset obtained after screening in step 2 is used to inject knowledge into LLMs, wherein the injection of LLMs is achieved through continuous pre-training to obtain a large language model with stronger domain perception capabilities.

2. The method according to claim 2, characterized in that In step 2, natural language text filtering is implemented based on an ontology-driven knowledge filtering strategy.

3. The method according to claim 1, characterized in that In step 2, the method for implementing natural language text filtering based on an ontology-driven knowledge filtering strategy is as follows: Given instance-level triples (h, r, t) derived from natural language text, we first generalize them into ontology-level triples (C h , r, C t ), where h refers to the head entity, r refers to the relationship, t refers to the tail entity, and C h Refers to the entity corresponding to the head entity, C t Refers to the ontology corresponding to the tail entity. The homogeneity metric score is used as a standard for quantitatively measuring natural language texts. The homogeneity metric score is obtained by considering the density of knowledge in the natural language text dataset and the missing knowledge completion rate relative to LLMs. Each ontology-level triple (C h , r, C t ) The calculation formula of the corresponding bulk density ρ is as follows: Where L represents the length of the natural language text, N represents the number of ontology-level triples mapped to the instance-level triples derived from the current natural language text, and |Xi| represents the length of the i-th ontology-level triple in the natural language text; In addition, using each ontology-level triple (C h , r, C t ) of UR((C h , r, C t )), assign a missing knowledge completion rate relative to LLMs to the current NLT, expressed as follows: represents the ith ontology-level triplet, with a total of N ontology-level triplet; The formula for the homogeneity measure score is as follows: score=α×ρ+β×τ Among them, α and β are hyperparameters used to balance the weights of knowledge density and missing knowledge completion rate in natural language text datasets; the homogeneity measurement score is used to rank the relevant natural language texts from high to low scores, and the natural language texts are filtered based on the ranking to obtain the filtered natural language text dataset.

4. The method according to claim 3, characterized in that In step 3, let D represent the obtained filtered natural language text dataset, and use the parameter fine-tuning method to optimize the knowledge injection objective function L(θ LLMs ), expressed as: Among them, θ LLMs Indicates the LLMs parameters to be updated, x1, x2, ..., x L represents a piece of text in the dataset D, the maximum length of the text is L, represents the probability of word x, Represents the probability of the next word; through sufficient continuous pre-training, a new large language model LLMsθ′ with updated parameters is obtained LLMs ,θ′ LLMs Represents the updated LLMs parameters.

5. The method according to claim 4, characterized in that Finally, the LLMsθ′ after parameter update is used LLMs Complete the named entity recognition task.

6. The method according to any one of claims 1 to 5, characterized in that This method is applied in the fields of large language models, domain knowledge, and natural language processing technology.

7. A system for implementing the method according to any one of claims 1 to 6.

8. A method for evaluating the effectiveness of a method as claimed in any one of claims 1 to 6.

9. The method according to claim 8, characterized in that The assessment scenario is a multiple-choice question.

10. The method according to claim 8, characterized in that The evaluation scenario is an open-ended question and answer session.

Images