Data matching method and device for enhancing large language model in combination with external knowledge retrieval, and medium

Abstract

The invention discloses a data matching method and device for enhancing a large language model in combination with external knowledge retrieval, and a medium. The method comprises the following steps: disassembling a data matching task into a rule set in a natural language form; traversing each sample in the data matching data set; for the ith sample, database role entities related to the source database and metadata of the database role entities in the sample are obtained; constructing a first knowledge set by taking the database role entity as an index and the metadata corresponding to the database role entity as a value; inputting the sample into a large language model to extract a keyword, performing retrieval to obtain an entity corresponding to the keyword in an external domain knowledge base, and obtaining a one-hop relationship of the entity; constructing a second knowledge set by taking the keyword as an index and taking the one-hop relationship corresponding to the keyword as a value; obtaining the compressed in-context example data, and obtaining the compressed in-context example data; and on the basis of the rule set, the first knowledge set, the second knowledge set and the compressed in-context example data, constructing cue words, and inputting the cue words into the large language model to obtain a data matching result.

Description

Technical Field

[0001] The present invention belongs to the field of data integration, and in particular relates to a data matching method, device and medium for enhancing a large language model in combination with external knowledge retrieval. Background Art

[0002] Data Integration is the process of logically or physically integrating data from different sources and formats. Among them, Data Matching is a technology that matches data from different sources at a specific granularity. Schema Matching and Entity Matching are important branches that match at the database schema granularity and database record granularity respectively. Schema matching is the process of identifying the correspondence between two or more database schema elements. The purpose of schema matching is to establish semantic correspondence between elements of different schemas in order to effectively integrate and combine data from different sources. The purpose of entity matching is to identify database records that point to the same real-world entity, even if the records are represented in different forms. Schema matching often appears as an upstream task of entity matching. The common purpose of both is to suggest mapping relationships between different data sources in order to carry out other downstream data integration subtasks.

[0003] In solving data matching tasks based on a large language model, it is necessary to request the large language model with the data matching task data, that is, the records or patterns of the source database to be matched, in natural language format as prompts, and obtain the actual matching data pairs or matching results from the output of the large language model. In the process of building prompts, since the large language model can only directly use the input data in text form, and the metadata of the data matching task exists in the form of database records or structured data, a serialization strategy is required to convert it into text form. Based on this, the text understanding and reasoning capabilities of the large language model are used to solve the data matching task.

[0004] Due to the limitation of the large language model's own capabilities, the use of large language models to solve data matching still has the problems of hallucination, overmatching, and undermatching. To solve the above problems, the technologies that have been proposed can be mainly divided into two aspects: retrieval enhanced generation (RAG) and model training. Among them, the retrieval enhanced generation technology is based on the existing request to retrieve relevant text segments from the outside as a reference for large language model reasoning, which can alleviate the above problems to a certain extent, but requires the availability of relevant corpora; model training involves the pre-training stage or fine-tuning stage, and the hallucination problem of the large language model is alleviated by designing model training data. This type of method has a large overhead in training data collection and model training costs. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a data matching method, device and medium for enhancing a large language model in combination with external knowledge retrieval.

[0006] In a first aspect, an embodiment of the present invention provides a data matching method for a large language model enhanced by combining external knowledge retrieval, the method comprising:

[0007] Decompose the data matching task into a set of rules in natural language form;

[0008] Traverse each sample in the data matching data set; for the i-th sample, obtain the database role entity related to the source database in the i-th sample; traverse the samples other than the i-th sample in the data matching data set to obtain the metadata of the database role entity corresponding to the i-th sample; use the database role entity as an index and its corresponding metadata as a value to construct a first knowledge set;

[0009] Traverse each sample in the data matching data set; for the i-th sample, input the i-th sample into the large language model to extract keywords; search the external domain knowledge base based on the keywords, obtain the entity corresponding to the keyword in the external domain knowledge base, and search the one-hop relationship of the entity in the external domain knowledge base; use the keyword as the index and its corresponding one-hop relationship as the value to construct the second knowledge set;

[0010] Select original in-context sample data from the data matching dataset, input the original in-context sample data into the large language model, and obtain compressed in-context sample data;

[0011] Constructing prompt words based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context sample data;

[0012] Input the prompt word into the large language model to obtain the data matching result.

[0013] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned data matching method of enhancing a large language model in combination with external knowledge retrieval.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned data matching method of enhancing a large language model in combination with external knowledge retrieval.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned data matching method of enhancing a large language model in combination with external knowledge retrieval.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention provides a data matching method for enhancing a large language model in combination with external knowledge retrieval. In the external domain knowledge base retrieval scenario, the method can construct high-quality knowledge in natural language form based on the existing domain knowledge base in the absence of an available text block corpus, and alleviate the hallucination problem of the large language model based on the constructed knowledge. At the same time, the method improves the reasoning performance of the large language model for data matching tasks by disassembling rules. Compared with traditional data matching task methods, this method achieves precision and recall rates that exceed traditional data matching tasks without the need for fine-tuning, thereby improving the accuracy and efficiency of data matching tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a flow chart of a data matching method combining external knowledge retrieval to enhance a large language model provided by an embodiment of the present invention;

[0020] Figure 2 It is an overall framework diagram of executing a pattern matching task provided by an embodiment of the present invention;

[0021] Figure 3 It is an overall framework diagram for performing entity matching tasks provided by an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0025] like Figure 1 As shown, the embodiment of the present invention provides a data matching method for a large language model enhanced by combining external knowledge retrieval, which specifically includes the following steps:

[0026] Step S1, decomposing the data matching task into a set of rules in natural language form.

[0027] Furthermore, the data matching task is a pattern matching task or an entity matching task, and the rule set is a list consisting of a single set of sufficient conditions. The rule is defined as a sufficient condition rule in the form of p→q, where p is a conditional constraint on whether the target data matches, and q is yes or no.

[0028] Exemplarily, for a pattern matching task, the rule set may be set as:

[0029] Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached;

[0030] Rule 2: If the columns of the two schemas have different data types in the database, then the answer is no, otherwise check rule 3;

[0031] Rule 3: If the tables of the two schemas are not semantically identical, then the answer is no, otherwise check rule 4;

[0032] Fourth rule: If the columns of the two patterns do not refer to the same concept, the answer is no, otherwise the answer is yes.

[0033] Exemplarily, for an entity matching task, the rule set may be set as:

[0034] Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached;

[0035] Rule 2: If the first entity is a shortened version of the second entity or vice versa, then the answer is yes, otherwise check rule 3;

[0036] Rule 3: If the first entity is an alias of the second entity or vice versa, then the answer is yes, otherwise check rule 4;

[0037] Rule 4: If the first entity and the second entity refer to the same real-world concept, then the answer is yes, otherwise the answer is no.

[0038] Further, Figure 2 The overall framework diagram for executing pattern matching tasks is shown; Figure 3Shows the overall framework diagram for performing entity matching tasks.

[0039] Step S2: Traverse each sample in the data matching dataset; for the i-th sample, obtain the database role entities related to the source database in the i-th sample; traverse the samples in the data matching dataset except the i-th sample to obtain the metadata of the database role entities corresponding to the i-th sample; construct the first knowledge set with the database role entities as the index and their corresponding metadata as the values.

[0040] Specifically, step S2 includes the following sub-steps:

[0041] Step S201: Initialize the entity list and the metadata list.

[0042] Step S202: Traverse each sample in the data matching dataset Set the initially traversed sample as x i , i = 0. For the i-th sample, obtain the database role entities related to the source database in the i-th sample and add the database role entities to the entity list.

[0043] Further, the database role entities are tables and columns in the source database.

[0044] When the data matching task is pattern matching, the sample x i in the data matching dataset includes the first pattern and the second pattern to be matched; when the data matching task is entity matching, the sample x i in the data matching dataset includes the first entity and the second entity to be matched.

[0045] Step S203: Traverse the samples in the data matching dataset except the i-th sample to obtain the metadata of the database role entities corresponding to the i-th sample and add it to the metadata list corresponding to the entity.

[0046] Step S204: i = i + 1; if i < N, where N is the length of the data matching dataset , then repeat steps S202 to S203, otherwise execute step S205.

[0047] Step S205: Construct the first knowledge set with the entities in the entity list as the index and the entity corresponding metadata list as the values.

[0048] Step S3: Traverse each sample in the data matching dataset; for the i-th sample, input the i-th sample into the large language model to extract keywords; retrieve the external domain knowledge base based on the keywords, obtain the entities corresponding to the keywords in the external domain knowledge base, and retrieve the one-hop relationships of these entities in the external domain knowledge base; use the keywords as indices and their corresponding one-hop relationships as values to construct the second knowledge set.

[0049] Further, the step S3 specifically includes the following steps:

[0050] Step S301: Initialize the knowledge list.

[0051] Step S302: Traverse each sample in the data matching dataset and set the initially traversed sample as x i , i = 0. For the i-th sample, input the i-th sample into the large language model to extract keywords.

[0052] When the data matching task is pattern matching, the sample x i in the data matching dataset includes the first pattern and the second pattern to be matched; when the data matching task is entity matching, the sample x i in the data matching dataset includes the first entity and the second entity to be matched.

[0053] Step S303: Use the keywords as input and retrieve the external domain knowledge base to obtain the entities corresponding to the keywords in the external domain knowledge base.

[0054] Step S304: Based on this entity, retrieve the one-hop relationships of this entity in the external domain knowledge base a second time to obtain a list of one-hop relationships.

[0055] Step S305: i = i + 1; if i < N, where N is the length of the data matching dataset , then repeat steps S302 to S304, otherwise execute step S306.

[0056] Step S306: Serialize the list of one-hop relationships into a natural language format, use the keywords corresponding to the sample as indices, and their corresponding one-hop relationships as values to construct the second knowledge set.

[0057] Step S4: Select the original in-context example data from the data matching dataset, input the original in-context example data into the large language model, and obtain the compressed in-context example data.

[0058] Specifically, the step S4 includes the following sub-steps:

[0059] Step S401, select the original in-context example data from the data matching data set ;

[0060] Step S402, use the original in-context example data to request a large language model to obtain the compressed and restated in-context example data;

[0061] Step S5, construct a prompt based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context example data.

[0062] Specifically, the step S5 includes the following sub-steps:

[0063] Step S501, initialize the enhanced example data list.

[0064] Step S502, traverse all the compressed in-context example data; set the initially traversed compressed in-context example data i = 0; for the i-th compressed in-context example data, refer to the rule set to produce the reasoning and answer based on the rule set ; concatenate the reasoning and answer to the end of the i-th compressed in-context example data to obtain the enhanced example data corresponding to the i-th compressed in-context example data, and add it to the enhanced example data list.

[0065] Step S503, i = i + 1; if i < N', where N' is the number of compressed in-context example data, then repeat step S502, otherwise enter step S504.

[0066] Step S504, initialize the prompt list, set the current traversal subscript i = 0;

[0067] Step S505, obtain the current sample with subscript i in the data matching data set ; the number of knowledge sets is n, obtain a total of n knowledge lists with subscript i in the n knowledge sets. Traverse the n knowledge lists, and for each knowledge list, concatenate the task instruction, the rule set, the enhanced example data list, the i-th sample, and the current knowledge list. Traverse each knowledge list to obtain n prompts corresponding to the i-th sample.

[0068] Further, the knowledge set includes several first knowledge sets obtained by repeating step S2 and several second knowledge sets obtained by repeating step S3.

[0069] Step S6, input the prompt into the large language model to obtain a data matching result.

[0070] Specifically, step S6 includes the following sub-steps:

[0071] Step S601, inputting n prompts corresponding to the i-th sample in the data matching data set into the large language model to obtain n request results;

[0072] Step S602, performing data cleaning on the n request results to obtain n binary classification results;

[0073] Step S603, performing majority voting on the n binary classification results, and taking the classification result with the most votes as the data matching result of the i-th sample.

[0074] Step S604, traverse each sample in the data matching data set to obtain a data matching result corresponding to each sample.

[0075] Furthermore, in this example, the large language model may adopt Llama-3, GLM-4, etc.

[0076] Example 1

[0077] In this example, taking the pattern matching task on the pattern matching dataset Synthea in the field of public health as an example, the method proposed in the present invention of combining external knowledge retrieval to enhance the large language model for data matching tasks is further explained.

[0078] Step S1, decomposing the pattern matching task in the field of public health into a set of rules in natural language form;

[0079] Furthermore, in this example, a total of 4 rules are designed for the pattern matching task in the field of public health, which respectively constrain the column data type, table semantics, and pattern pointing concept of the target data pattern. They include:

[0080] Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached;

[0081] Rule 2: If the columns of the two schemas have different data types in the database, then the answer is no, otherwise check rule 3;

[0082] Rule 3: If the tables of the two schemas are not semantically identical, then the answer is no, otherwise check rule 4;

[0083] Fourth rule: If the columns of the two patterns do not refer to the same concept, the answer is no, otherwise the answer is yes.

[0084] Step S2, discover the entities corresponding to the table objects in the Synthea database from the Synthea sample of the pattern matching dataset in the public health field, and traverse the dataset to retrieve the table description metadata related to the table object entities, and use the discovered table object entities and table description metadata to build a knowledge set, whose single knowledge format is [table name, table description].

[0085] Furthermore, the length N of the test set of the public health pattern matching dataset Synthea used in this example is 2964.

[0086] Step S3, use the Synthea sample data set in the public health field to request the large language model to extract keywords, use the keywords to query the external medical field knowledge base and serialize the query results into a natural language format, and use the keywords and the serialized query results to build a knowledge set.

[0087] Furthermore, in this example, the large language model selected is the GPT-3.5 model, the external medical field knowledge base is Snomed-CT, the retrieved one-hop relationship is the parents and children relationship of the corresponding entity in Snomed-CT, and the retrieved parents and children relationship is serialized into a natural language format in the form of {children] is a kind of {parents} through the entity name in the corresponding relationship.

[0088] In step S4, the user selects 4 pieces of data from the training set of Synthea, a pattern matching dataset in the public health field, as original in-context example data; uses the selected 4 original in-context example data to request GPT-3.5 to generate a summary of the pattern tables and columns in the given example data as compressed and restated in-context example data.

[0089] Step S5, construct prompts based on the rule set, knowledge set, and in-context sample data obtained in the above steps; traverse the selected 4 in-context sample data, based on the 4 rules designed for this example, the user checks the above 4 rules in sequence and produces at most 4 reasoning processes and the final answer produced by the rules at the end of the reasoning, and splices them to the end of the current in-context sample data to obtain the current enhanced sample data. After the traversal is completed, a total of 4 enhanced sample data are obtained. On this basis, the test set of Synthea, a pattern matching dataset in the public health field, is traversed. For the current target data traversed, a total of 3 knowledge lists are obtained, which are derived from Wikidata and the knowledge set constructed in step S2, Wikipedia, and the knowledge set constructed in step S3. Based on the above 3 knowledge lists, 3 prompts for the current target data are spliced ​​in sequence.

[0090] Step S6, traverse the 3N prompts obtained in step S5, and for the current 3 prompts, use the 3 prompts to request GPT-3.5 three times respectively, and obtain 3 request results; use regular expression matching to obtain the binary classification results in the 3 request results, a total of 3 binary classification results; based on the above 3 binary classification results, a majority vote of 3 votes is performed, and the classification result with the most votes is finally output as the method corresponding to the current target data.

[0091] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned data matching method for enhancing a large language model in combination with external knowledge retrieval. Figure 4 As shown, a hardware structure diagram of any device with data processing capability for the data matching method for enhancing a large language model by combining external knowledge retrieval provided by an embodiment of the present invention is shown, except Figure 4 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0092] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the data matching method of the large language model enhanced by combining external knowledge retrieval as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or replace the technical solutions of the present invention with equivalents without departing from the principle and scope of the present invention. The protection scope of the present invention shall be based on the claims.

Claims

1. A data matching method for a large language model enhanced by combining external knowledge retrieval, characterized in that: The method comprises: Decompose the data matching task into a set of rules in natural language form; Traverse each sample in the data matching data set; for the i-th sample, obtain the database role entity related to the source database in the i-th sample; traverse the samples other than the i-th sample in the data matching data set to obtain the metadata of the database role entity corresponding to the i-th sample; use the database role entity as an index and its corresponding metadata as a value to construct a first knowledge set; Traverse each sample in the data matching data set; for the i-th sample, input the i-th sample into the large language model to extract keywords; search the external domain knowledge base based on the keywords, obtain the entity corresponding to the keyword in the external domain knowledge base, and search the one-hop relationship of the entity in the external domain knowledge base; use the keyword as the index and its corresponding one-hop relationship as the value to construct the second knowledge set; Select original in-context sample data from the data matching dataset, input the original in-context sample data into the large language model, and obtain compressed in-context sample data; Constructing prompts based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context example data; Input the prompt into the large language model to obtain data matching results.

2. According to claim 1, a data matching method combining external knowledge retrieval to enhance a large language model is characterized in that: When the data matching task is pattern matching, the rule set includes: Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached; Rule 2: If the columns of the two schemas have different data types in the database, then the answer is no, otherwise check rule 3; Rule 3: If the tables of the two schemas are not semantically identical, then the answer is no, otherwise check rule 4; Fourth rule: If the columns of the two patterns do not refer to the same concept, the answer is no, otherwise the answer is yes; When the data matching task is entity matching, the rule set includes: Rule 1: Rules 2, 3, and 4 must be checked in sequence until a conclusion is reached; Rule 2: If the first entity is a shortened version of the second entity or vice versa, then the answer is yes, otherwise check rule 3; Rule 3: If the first entity is an alias of the second entity or vice versa, then the answer is yes, otherwise check rule 4; Rule 4: If the first entity and the second entity refer to the same real-world concept, then the answer is yes, otherwise the answer is no.

3. According to claim 1, a data matching method combining external knowledge retrieval and enhancing a large language model is characterized in that: When the data matching task is pattern matching, the samples in the data matching data set include a first pattern and a second pattern to be matched; When the data matching task is entity matching, the samples in the data matching data set include a first entity and a second entity to be matched.

4. A data matching method for enhancing a large language model in combination with external knowledge retrieval according to claim 1 or 3, characterized in that: The process of building the first knowledge set includes: Initialize entity list and metadata list; Traverse each sample in the data matching data set; for the i-th sample, obtain the database role entity related to the source database in the i-th sample, and add the database role entity to the entity list; Traverse the samples except the i-th sample in the data matching dataset to obtain the metadata of the database role entity corresponding to the i-th sample, and add it to the metadata list corresponding to the entity; The entities in the entity list are used as indexes, and the metadata list corresponding to the entities is used as values ​​to construct the first knowledge set.

5. The data matching method of combining external knowledge retrieval and enhancing a large language model according to claim 1 or 3, characterized in that: The process of building the second knowledge set includes: Traverse each sample in the data matching dataset; for the i-th sample, input the i-th sample into the large language model to extract keywords; Search the external domain knowledge base based on keywords to obtain the entities corresponding to the keywords in the external domain knowledge base; Based on the entity, the one-hop relationship of the entity is retrieved again in the external domain knowledge base to obtain a one-hop relationship list; The one-hop relationship list is serialized into a natural language format, and the keywords corresponding to the samples are used as indexes and the corresponding one-hop relationships are used as values ​​to construct the second knowledge set.

6. The data matching method of combining external knowledge retrieval and enhancing a large language model according to claim 1, characterized in that: The process of constructing prompt words based on the rule set, the first knowledge set, the second knowledge set, and the compressed in-context sample data includes: Initialize the enhanced sample data list; Traverse all compressed in-context sample data; for the i-th compressed in-context sample data, generate inferences and answers based on the rule set; concatenate the inferences and answers to the end of the i-th compressed in-context sample data to obtain the enhanced sample data corresponding to the i-th compressed in-context sample data, and add them to the enhanced sample data list; Traverse each sample in the data matching data set; for the i-th sample, the total number of the first knowledge set and the second knowledge set is n, obtain the n knowledge lists corresponding to the i-th sample in the first knowledge set and the second knowledge set; for a certain knowledge list, concatenate the task instructions, rule set, enhanced example data list, i-th sample, and current knowledge list to obtain the prompt corresponding to the current knowledge list of the i-th sample; traverse each knowledge list to obtain the n prompts corresponding to the i-th sample.

7. A data matching method for enhancing a large language model in combination with external knowledge retrieval according to claim 1 or 6, characterized in that: The process of inputting the prompt word into the large language model and obtaining the data matching result includes: Traverse the data to match each sample in the data set; Match the data to the n prompts corresponding to the i-th sample in the dataset, input them into the large language model, and obtain n request results; Perform data cleaning on n request results to obtain n binary classification results; Perform majority voting on the n binary classification results, and take the classification result with the most votes as the data matching result of the i-th sample.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the data matching method of combining external knowledge retrieval to enhance the large language model as described in any one of claims 1 to 7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data matching method combining external knowledge retrieval and enhancing a large language model as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the data matching method of any one of claims 1-7 combined with external knowledge retrieval to enhance the large language model is implemented.

Images