A data matching method, device, and medium combining external knowledge retrieval to enhance large language models

Abstract

The present invention discloses a data matching method, device, and medium for enhancing a large language model in combination with external knowledge retrieval, comprising: decomposing a data matching task into a set of rules in natural language form; traversing each sample in a data matching data set; for the i-th sample, obtaining a database role entity and its metadata related to a source database in the sample; constructing a first knowledge set using the database role entity as an index and its corresponding metadata as a value; inputting the sample into a large language model to extract keywords, retrieving entities corresponding to the keywords in an external domain knowledge base, and obtaining one-hop relationships of the entities; constructing a second knowledge set using the keywords as an index and its corresponding one-hop relationships as a value; obtaining compressed in-context example data; constructing prompt words based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context example data, and inputting the prompt words into the large language model to obtain data matching results.

Description

Technical Field

[0001] The present invention belongs to the field of data integration, and in particular relates to a data matching method, device, and medium for enhancing a large language model in combination with external knowledge retrieval. Background Art

[0002] Data integration is the process of logically or physically integrating data from different sources and formats. Among them, data matching is a technology that matches data from different sources at a specific granularity. Schema matching and entity matching are important branches that match at the database schema granularity and database record granularity, respectively. Schema matching is the process of identifying the correspondence between two or more database schema elements. The purpose of schema matching is to establish semantic correspondences between elements of different schemas in order to effectively integrate and combine data from different sources. The purpose of entity matching is to identify database records that point to the same real-world entity, even if the records have different representations. Schema matching often appears as an upstream task of entity matching. The common purpose of both is to suggest mapping relationships between different data sources in order to carry out other downstream data integration subtasks.

[0003] When solving data matching tasks based on a large language model, the matching task data—that is, the source database records or patterns to be matched—are submitted to the large language model in natural language format as prompts. The model then outputs the actual matching data pairs or matching results. During the prompt construction process, since large language models can only directly utilize input data in text form, and the metadata for the data matching task exists in the form of database records or structured data, a serialization strategy is required to convert it into text. This allows the large language model to leverage its text understanding and reasoning capabilities to solve data matching tasks.

[0004] Due to the limitations of large language models, using large language models to solve data matching still faces the problems of hallucination, overmatching, and undermatching. To address these issues, currently proposed technologies can be mainly divided into two aspects: retrieval-augmented generation (RAG) and model training. RAG technology, based on existing requests, retrieves relevant text segments externally to serve as a reference for large language model inference. This can alleviate these issues to a certain extent, but requires the availability of relevant corpora. Model training involves a pre-training phase or fine-tuning phase, which alleviates the hallucination problem of large language models by designing model training data. However, this type of method has high overhead in training data collection and model training costs. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a data matching method, device and medium for enhancing a large language model by combining external knowledge retrieval.

[0006] In a first aspect, an embodiment of the present invention provides a data matching method for a large language model enhanced by combining external knowledge retrieval, the method comprising:

[0007] Decompose the data matching task into a set of rules in natural language form;

[0008] Traverse each sample in the data matching dataset; for the i-th sample, obtain the database role entity related to the source database in the i-th sample; traverse the samples in the data matching dataset except the i-th sample to obtain the metadata of the database role entity corresponding to the i-th sample; use the database role entity as an index and its corresponding metadata as a value to construct a first knowledge set;

[0009] Traverse each sample in the data matching dataset; for the i-th sample, input the i-th sample into the large language model to extract keywords; search the external domain knowledge base based on the keywords, obtain the entity corresponding to the keyword in the external domain knowledge base, and search the one-hop relationship of the entity in the external domain knowledge base; use the keyword as the index and the corresponding one-hop relationship as the value to construct the second knowledge set;

[0010] Selecting original in-context example data from the data matching dataset, inputting the original in-context example data into the large language model to obtain compressed in-context example data;

[0011] Constructing prompt words based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context example data;

[0012] Input the prompt word into the large language model to obtain the data matching result.

[0013] In a second aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned data matching method of enhancing a large language model by combining external knowledge retrieval.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned data matching method for enhancing a large language model by combining external knowledge retrieval.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned data matching method of enhancing a large language model by combining external knowledge retrieval.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention provides a data matching method for enhancing large language models by combining external knowledge retrieval. In the external domain knowledge base retrieval scenario, this method can construct high-quality natural language knowledge based on the existing domain knowledge base in the absence of an available text block corpus, and alleviate the hallucination problem of large language models based on the constructed knowledge. At the same time, this method improves the reasoning performance of large language models for data matching tasks by disassembling rules. Compared with traditional data matching task methods, this method achieves precision and recall rates that exceed those of traditional data matching tasks without the need for fine-tuning, thereby improving the accuracy and efficiency of data matching tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is a flow chart of a data matching method for enhancing a large language model by combining external knowledge retrieval, provided by an embodiment of the present invention;

[0020] Figure 2 This is an overall framework diagram for executing a pattern matching task provided by an embodiment of the present invention;

[0021] Figure 3 This is an overall framework diagram for performing entity matching tasks provided by an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0025] like Figure 1 As shown, an embodiment of the present invention provides a data matching method for a large language model enhanced by combining external knowledge retrieval, which specifically includes the following steps:

[0026] Step S1: Decompose the data matching task into a set of rules in natural language form.

[0027] Furthermore, the data matching task is a pattern matching task or an entity matching task, and the rule set is a list consisting of a single set of sufficient conditions. The rule is defined as a sufficient condition rule in the form of p→q, where p is a conditional constraint on whether the target data matches, and q is yes or no.

[0028] For example, for a pattern matching task, the rule set may be set as:

[0029] Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached;

[0030] Rule 2: If the columns of the two schemas have different data types in the database, then the answer is no, otherwise check rule 3;

[0031] Rule 3: If the tables of the two schemas are not semantically identical, then the answer is no, otherwise check rule 4;

[0032] Fourth rule: If the columns of the two patterns do not refer to the same concept, the answer is no, otherwise the answer is yes.

[0033] For example, for an entity matching task, the rule set may be set as:

[0034] Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached;

[0035] Rule 2: If the first entity is an abbreviation of the second entity or vice versa, then the answer is yes, otherwise check rule 3;

[0036] Rule 3: If the first entity is an alias of the second entity or vice versa, then the answer is yes, otherwise check rule 4;

[0037] Rule 4: If the first entity and the second entity refer to the same real-world concept, then the answer is yes, otherwise the answer is no.

[0038] Further, Figure 2 Shows the overall framework diagram for performing pattern matching tasks; Figure 3Shows the overall framework diagram for performing entity matching tasks.

[0039] Step S2, traverse each sample in the data matching dataset; for the i-th sample, obtain the database role entities related to the source database in the i-th sample; traverse the samples in the data matching dataset except the i-th sample to obtain the metadata of the database role entities corresponding to the i-th sample; construct the first knowledge set with the database role entities as indexes and their corresponding metadata as values.

[0040] Specifically, step S2 includes the following sub-steps:

[0041] Step S201, initialize the entity list and the metadata list.

[0042] Step S202, traverse each sample in the data matching dataset Set the initially traversed sample as x i , i = 0, for the i-th sample, obtain the database role entities related to the source database in the i-th sample, and add the database role entities to the entity list.

[0043] Further, the database role entities are tables and columns in the source database.

[0044] When the data matching task is pattern matching, the sample x i in the data matching dataset includes the first pattern and the second pattern to be matched; when the data matching task is entity matching, the sample x i in the data matching dataset includes the first entity and the second entity to be matched.

[0045] Step S203, traverse the samples in the data matching dataset except the i-th sample to obtain the metadata of the database role entities corresponding to the i-th sample, and add it to the metadata list corresponding to the entity.

[0046] Step S204, i = i + 1; if i < N, where N is the length of the data matching dataset , then repeat steps S202 to S203, otherwise execute step S205.

[0047] Step S205, construct the first knowledge set with the entities in the entity list as indexes and the entity corresponding metadata list as values.

[0048] Step S3: Traverse each sample in the data matching dataset; for the i-th sample, input the i-th sample into the large language model to extract keywords; retrieve the external domain knowledge base based on the keywords, obtain the entities corresponding to the keywords in the external domain knowledge base, and retrieve the one-hop relationships of these entities in the external domain knowledge base; construct the second knowledge set with the keywords as the index and their corresponding one-hop relationships as the values.

[0049] Further, the step S3 specifically includes the following steps:

[0050] Step S301: Initialize the knowledge list.

[0051] Step S302: Traverse each sample in the data matching dataset <00,00124>and set the initially traversed sample as x i , i = 0. For the i-th sample, input the i-th sample into the large language model to extract keywords.

[0052] When the data matching task is pattern matching, the sample x i in the data matching dataset includes the first pattern and the second pattern to be matched; when the data matching task is entity matching, the sample x i in the data matching dataset includes the first entity and the second entity to be matched.

[0053] Step S303: Use the keywords as input to retrieve the external domain knowledge base and obtain the entities corresponding to the keywords in the external domain knowledge base.

[0054] Step S304: Based on this entity, retrieve the one-hop relationships of this entity in the external domain knowledge base a second time to obtain the one-hop relationship list.

[0055] Step S305: i = i + 1; if i < N, where N is the length of the data matching dataset , repeat steps S302 to S304, otherwise execute step S306.

[0056] Step S306: Serialize the one-hop relationship list into a natural language format, construct the second knowledge set with the keywords corresponding to the sample as the index and their corresponding one-hop relationships as the values.

[0057] Step S4: Select the original in-context example data from the data matching dataset, input the original in-context example data into the large language model, and obtain the compressed in-context example data.

[0058] Specifically, the step S4 includes the following sub-steps:

[0059] Step S401, select the original in-context example data from the data matching dataset ;

[0060] Step S402, use the original in-context example data to request the large language model to obtain the compressed and restated in-context example data;

[0061] Step S5, construct a prompt based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context example data.

[0062] Specifically, the step S5 includes the following sub-steps:

[0063] Step S501, initialize the enhanced example data list.

[0064] Step S502, traverse all the compressed in-context example data; set the initially traversed compressed in-context example data i = 0; for the i-th compressed in-context example data, refer to the rule set to produce the reasoning and answer based on the rule set ; splice the reasoning and answer at the end of the i-th compressed in-context example data to obtain the enhanced example data corresponding to the i-th compressed in-context example data, and add it to the enhanced example data list.

[0065] Step S503, i = i + 1; if i < N', where N' is the number of compressed in-context example data, then repeat step S502, otherwise enter step S504.

[0066] Step S504, initialize the prompt list, set the current traversal index i = 0;

[0067] Step S505, obtain the current sample with index i in the data matching dataset . The number of knowledge sets is n, and obtain a total of n knowledge lists with index i in the n knowledge sets. Traverse the n knowledge lists. For each knowledge list, splice the task instruction, the rule set, the enhanced example data list, the i-th sample, and the current knowledge list. Traverse each knowledge list to obtain n prompts corresponding to the i-th sample.

[0068] Furthermore, the knowledge set includes several first knowledge sets obtained by repeating step S2 and several second knowledge sets obtained by repeating step S3.

[0069] Step S6: input the prompt into the large language model to obtain a data matching result.

[0070] Specifically, step S6 includes the following sub-steps:

[0071] Step S601: Match the data to the n prompts corresponding to the i-th sample in the data set, input them into the large language model, and obtain n request results;

[0072] Step S602: perform data cleaning on the n request results to obtain n binary classification results;

[0073] Step S603: Perform majority voting on the n binary classification results, and use the classification result with the most votes as the data matching result of the i-th sample.

[0074] Step S604: traverse each sample in the data matching data set to obtain a data matching result corresponding to each sample.

[0075] Furthermore, in this example, the large language model may adopt Llama-3, GLM-4, etc.

[0076] Example 1

[0077] In this example, the pattern matching task on the pattern matching dataset Synthea in the public health field is taken as an example to further illustrate the method proposed in the present invention of combining external knowledge retrieval to enhance the large language model for data matching tasks.

[0078] Step S1, decomposing the pattern matching task in the public health field into a set of rules in natural language form;

[0079] Furthermore, in this example, four rules were designed for the pattern matching task in the public health field, which respectively constrained the column data type, table semantics, and pattern pointing concept of the target data pattern. They include:

[0080] Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached;

[0081] Rule 2: If the columns of the two schemas have different data types in the database, then the answer is no, otherwise check rule 3;

[0082] Rule 3: If the tables of the two schemas are not semantically identical, then the answer is no, otherwise check rule 4;

[0083] Fourth rule: If the columns of the two patterns do not refer to the same concept, the answer is no, otherwise the answer is yes.

[0084] Step S2: Discover the entities corresponding to the table objects in the Synthea database from the Synthea sample of the pattern matching dataset in the public health field, traverse the dataset to retrieve the table description metadata related to the table object entities, and use the discovered table object entities and table description metadata to construct a knowledge set. The format of a single knowledge item is [table name, table description].

[0085] Furthermore, the length N of the test set of the Synthea pattern matching dataset in the public health field used in this example is 2964.

[0086] Step S3: Use the Synthea sample data set in the public health field to request the large language model to extract keywords, use the keywords to query the external medical field knowledge base, and serialize the query results into a natural language format. Use the keywords and the serialized query results to build a knowledge set.

[0087] Furthermore, in this example, the large language model selected is the GPT-3.5 model, the external medical knowledge base is Snomed-CT, and the retrieved one-hop relationship is the parents and children relationship of the corresponding entity in Snomed-CT. The retrieved parents and children relationship is serialized into a natural language format in the form of {children] is a kind of {parents} through the entity name in the corresponding relationship.

[0088] In step S4, the user selects 4 pieces of data from the training set of Synthea, a pattern matching dataset in the public health field, as original in-context example data; uses the selected 4 original in-context example data to request GPT-3.5 to generate a summary of the pattern tables and columns in the given example data as compressed and restated in-context example data.

[0089] Step S5, construct prompts based on the rule set, knowledge set, and in-context example data obtained in the above steps; traverse the selected 4 in-context example data, and based on the 4 rules designed for this example, the user checks the above 4 rules in sequence and produces at most 4 reasoning processes and the final answer produced by the rule at the end of the reasoning, and splices them to the end of the current in-context example data to obtain the current enhanced example data. After the traversal is completed, a total of 4 enhanced example data are obtained. On this basis, the test set of Synthea, a pattern matching dataset in the public health field, is traversed. For the current target data traversed, a total of 3 knowledge lists are obtained, which are derived from Wikidata and the knowledge set constructed in step S2, Wikipedia, and the knowledge set constructed in step S3. Based on the above 3 knowledge lists, 3 prompts for the current target data are spliced ​​in sequence.

[0090] In step S6, traverse the 3N prompts obtained in step S5, and use the current 3 prompts to request GPT-3.5 three times respectively, and obtain 3 request results; use regular expression matching to obtain the binary classification results in the 3 request results, a total of 3 binary classification results; based on the above 3 binary classification results, a majority vote of 3 votes is performed, and the classification result with the most votes is finally output as the method corresponding to the current target data.

[0091] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned data matching method for enhancing a large language model by combining external knowledge retrieval. Figure 4 As shown, a hardware structure diagram of any device with data processing capability is provided for the data matching method of combining external knowledge retrieval to enhance the large language model according to an embodiment of the present invention, except Figure 4 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0092] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the data matching method for enhancing a large language model by combining external knowledge retrieval as described above. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the principles and scope of the present invention. The scope of protection of the present invention shall be based on the claims.

Claims

1. A data matching method for enhancing a large language model by combining external knowledge retrieval, characterized in that: The method comprises: Decompose the data matching task into a set of rules in natural language form; Traverse each sample in the data matching dataset; for the i-th sample, obtain the database role entity related to the source database in the i-th sample; traverse the samples in the data matching dataset except the i-th sample to obtain the metadata of the database role entity corresponding to the i-th sample; use the database role entity as an index and its corresponding metadata as a value to construct a first knowledge set; Traverse each sample in the data matching dataset; for the i-th sample, input the i-th sample into the large language model to extract keywords; search the external domain knowledge base based on the keywords, obtain the entity corresponding to the keyword in the external domain knowledge base, and search the one-hop relationship of the entity in the external domain knowledge base; use the keyword as the index and the corresponding one-hop relationship as the value to construct the second knowledge set; Selecting original in-context example data from the data matching dataset, inputting the original in-context example data into the large language model to obtain compressed in-context example data; Constructing a prompt based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context example data; Input the prompt into the large language model to obtain data matching results; When the data matching task is pattern matching, the rule set includes: Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached; Rule 2: If the columns of the two schemas have different data types in the database, then the answer is no, otherwise check rule 3; Rule 3: If the tables of the two schemas are not semantically identical, then the answer is no, otherwise check rule 4; Rule 4: If the columns of the two patterns do not refer to the same concept, the answer is no, otherwise the answer is yes; When the data matching task is entity matching, the rule set includes: Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached; Rule 2: If the first entity is an abbreviation of the second entity or vice versa, then the answer is yes, otherwise check rule 3; Rule 3: If the first entity is an alias of the second entity or vice versa, then the answer is yes, otherwise check rule 4; Rule 4: If the first entity and the second entity refer to the same real-world concept, then the answer is yes, otherwise the answer is no; The process of constructing prompt words based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context sample data includes: Initialize the enhanced sample data list; Traverse all compressed in-context example data; for the i-th compressed in-context example data, generate inferences and answers based on the rule set; concatenate the inferences and answers to the end of the i-th compressed in-context example data to obtain the enhanced example data corresponding to the i-th compressed in-context example data, and add it to the enhanced example data list; Traverse each sample in the data matching data set; for the i-th sample, the total number of the first knowledge set and the second knowledge set is n, and obtain n knowledge lists corresponding to the i-th sample in the first knowledge set and the second knowledge set; for a certain knowledge list, splice the task instructions, rule set, enhanced example data list, i-th sample, and current knowledge list to obtain the prompt corresponding to the current knowledge list of the i-th sample; traverse each knowledge list to obtain n prompts corresponding to the i-th sample.

2. The data matching method for enhancing a large language model by combining external knowledge retrieval according to claim 1, characterized in that: When the data matching task is pattern matching, the samples in the data matching data set include a first pattern and a second pattern to be matched; When the data matching task is entity matching, the samples in the data matching dataset include a first entity and a second entity to be matched.

3. The data matching method for enhancing a large language model by combining external knowledge retrieval according to claim 1 or 2, characterized in that: The process of building the first knowledge set includes: Initialize entity list and metadata list; Traverse each sample in the data matching dataset; for the i-th sample, obtain the database role entity related to the source database in the i-th sample and add the database role entity to the entity list; Traverse the samples in the data matching dataset except the i-th sample to obtain the metadata of the database role entity corresponding to the i-th sample and add it to the metadata list corresponding to the entity; The first knowledge set is constructed by using the entities in the entity list as indexes and the metadata list corresponding to the entities as values.

4. The data matching method for enhancing a large language model by combining external knowledge retrieval according to claim 1 or 2, characterized in that: The process of building the second knowledge set includes: Traverse each sample in the data matching dataset; for the i-th sample, input the i-th sample into the large language model to extract keywords; Search the external domain knowledge base based on keywords to obtain the entities corresponding to the keywords in the external domain knowledge base; Based on the entity, the one-hop relationship of the entity is retrieved again in the external domain knowledge base to obtain a one-hop relationship list; The one-hop relationship list is serialized into a natural language format, and the keywords corresponding to the samples are used as indexes and the corresponding one-hop relationships are used as values ​​to construct the second knowledge set.

5. The data matching method for enhancing a large language model by combining external knowledge retrieval according to claim 1, characterized in that: The process of inputting the prompt word into the large language model and obtaining the data matching result includes: Traverse the data to match each sample in the dataset; Match the data to the n prompts corresponding to the i-th sample in the dataset, input them into the large language model, and obtain n request results; Perform data cleaning on n request results to obtain n binary classification results; Perform majority voting on the n binary classification results, and take the classification result with the most votes as the data matching result of the i-th sample.

6. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the data matching method of combining external knowledge retrieval to enhance the large language model as described in any one of claims 1 to 5 above.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data matching method combining external knowledge retrieval and enhancing a large language model according to any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the data matching method of any one of claims 1 to 5 combined with external knowledge retrieval to enhance a large language model is implemented.

Images