Table data-oriented dual-path knowledge retrieval enhanced generation method and device

Abstract

The invention provides a table data-oriented dual-path knowledge retrieval enhancement generation method and device, and relates to the technical field of retrieval enhancement generation. The method comprises the steps of obtaining knowledge base table data; based on the description text template, performing text conversion on the knowledge base table data to obtain a descriptive text; constructing a knowledge base according to the descriptive text to obtain a direct path knowledge base and an indirect path knowledge base; constructing a structured query text according to the user query text; performing matching retrieval based on the direct path knowledge base to obtain a direct retrieval result; performing extended retrieval based on the indirect path knowledge base to obtain an indirect retrieval result; and based on the retrieval result prompt text, performing text reasoning by using a large language model according to the user query text, the direct retrieval result and the indirect retrieval result to obtain a user query result. The invention relates to an efficient and accurate retrieval enhancement generation method which aims at table data and adopts double-path knowledge retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of retrieval enhancement generation, and in particular to a dual-path knowledge retrieval enhancement generation method and device for table data. Background Art

[0002] Retrieval-augmented Generation (RAG) is an AI technology that combines information retrieval techniques with language generation models. It is a key enabling technology for large language models (LLMs). While traditional LLMs possess powerful text generation capabilities, their training data is fixed and can become outdated over time, lacking the ability to provide accurate, contextually relevant responses. RAG technology, by building a knowledge retrieval and question-answering augmented agent and leveraging external knowledge sources to enhance the performance of generative models, effectively addresses the limitations of generative models caused by insufficient or outdated training knowledge, significantly improving the service quality of large models. It can effectively expand the knowledge scope of LLMs and protect data privacy. It also displays citation sources, reduces the need for model fine-tuning, is highly adaptable, and effectively reduces LLM hallucinations. Its role is particularly prominent in LLM service systems that require high levels of specialized or domain knowledge or require high levels of information freshness.

[0003] RAG technology often provides relatively poor service quality when used with tabular data. Knowledge bases based on tabular data often have disorganized information, rather than coherent text, and header information is easily lost. In RAG scenarios, large models struggle to obtain high-quality knowledge retrieval results, often with disorganized results that are difficult for them to understand. Currently, the knowledge retrieval process primarily uses methods such as similarity matching to obtain text fragments. This approach has poor matching accuracy when retrieving and matching tabular data, and can contain a large amount of irrelevant information, further misleading the large model. Existing retrieval mechanisms can generally only match knowledge fragments related to user questions, while some indirect knowledge related to them cannot be obtained, limiting the scope of knowledge search and failing to support question-and-answer scenarios that require the retrieval of related knowledge.

[0004] In the prior art, there is a lack of an efficient and accurate retrieval enhancement generation method for tabular data using dual-path knowledge retrieval. Summary of the Invention

[0005] In order to solve the technical problems of the existing technology such as a cluttered knowledge base, low retrieval quality and narrow retrieval scope, the embodiment of the present invention provides a dual-path knowledge retrieval enhancement generation method and device for tabular data. The technical solution is as follows:

[0006] In one aspect, a dual-path knowledge retrieval enhancement generation method for tabular data is provided. The method is implemented by a dual-path knowledge retrieval enhancement generation device, and the method includes:

[0007] Acquire knowledge base table data; perform text conversion on the knowledge base table data based on the description text template to obtain descriptive text;

[0008] Based on the named entity recognition model and the pre-trained embedding model, a knowledge base is constructed according to the descriptive text to obtain a direct path knowledge base;

[0009] Based on the Neo4j graph database, the knowledge base is constructed according to the descriptive text to obtain the indirect path knowledge base;

[0010] Obtain user query text; construct structured query text based on the user query text based on the named entity recognition model and the pre-trained text classification model;

[0011] Based on the direct path knowledge base, according to the structured query text, the cosine similarity method is used to perform matching retrieval to obtain direct retrieval results;

[0012] Based on the indirect path knowledge base, according to the structured query text, the Cypher query statement is used to perform extended search to obtain indirect search results;

[0013] Based on the retrieval result prompt text, a large language model is used to perform text reasoning based on the user query text, direct retrieval results, and indirect retrieval results to obtain the user query results.

[0014] On the other hand, a dual-path knowledge retrieval enhancement generation device for tabular data is provided, which is applied to a dual-path knowledge retrieval enhancement generation method for tabular data. The device includes:

[0015] The table data acquisition module is used to obtain the knowledge base table data; based on the description text template, the knowledge base table data is converted into text to obtain descriptive text;

[0016] The direct path knowledge base construction module is used to construct a knowledge base based on the named entity recognition model and the pre-trained embedding model according to the descriptive text to obtain a direct path knowledge base;

[0017] The indirect path knowledge base construction module is used to construct a knowledge base based on the Neo4j graph database and descriptive text to obtain an indirect path knowledge base;

[0018] The user query acquisition module is used to obtain user query text; based on the named entity recognition model and the pre-trained text classification model, it constructs a structured query text according to the user query text;

[0019] Direct retrieval module, which is used to perform matching retrieval based on the direct path knowledge base and the structured query text using the cosine similarity method to obtain direct retrieval results;

[0020] The indirect search module is used to perform extended search based on the indirect path knowledge base and the structured query text through Cypher query statements to obtain indirect search results;

[0021] The query result generation module is used to obtain user query results based on the retrieval result prompt text, the user query text, direct retrieval results and indirect retrieval results, and the text reasoning using a large language model.

[0022] On the other hand, a dual-path knowledge retrieval enhancement generation device is provided, which includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned dual-path knowledge retrieval enhancement generation methods for tabular data is implemented.

[0023] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned dual-path knowledge retrieval enhancement generation methods for tabular data.

[0024] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0025] The present invention proposes a dual-path knowledge retrieval enhancement generation method for tabular data, which uses a graph database to construct a RAG knowledge base by performing correlation analysis on samples of tabular knowledge base data. On this basis, the user query text is parsed to automatically obtain the entity or associated entity information therein; the associated samples in the knowledge base are queried through the graph database support unit to obtain the retrieval results; the retrieved tabular data is converted into descriptive text suitable for understanding by a large language model; the user's query text and the obtained retrieval enhancement data are integrated and input into the large language model to obtain the corresponding results. The method proposed by the present invention has high knowledge retrieval efficiency and a large knowledge retrieval scope, and performs special knowledge base construction and data conversion for tabular data, so that the large model can easily understand and reason about the retrieved information, thereby improving the quality of the output results. The present invention is an efficient and accurate retrieval enhancement generation method for tabular data using dual-path knowledge retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 This is a flow chart of a dual-path knowledge retrieval enhancement generation method for tabular data provided by an embodiment of the present invention;

[0028] Figure 2 This is a block diagram of a dual-path knowledge retrieval enhancement generation device for tabular data provided by an embodiment of the present invention;

[0029] Figure 3 It is a structural diagram of a dual-path knowledge retrieval enhancement generation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0031] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0032] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0033] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0034] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0035] The embodiment of the present invention provides a dual-path knowledge retrieval enhancement generation method for table data, which can be implemented by a dual-path knowledge retrieval enhancement generation device, which can be a terminal or a server. Figure 1 The flowchart of the dual-path knowledge retrieval enhancement generation method for tabular data is shown. The processing flow of the method may include the following steps:

[0036] S1. Acquire knowledge base table data; perform text conversion on the knowledge base table data based on a description text template to obtain descriptive text.

[0037] In one feasible implementation, the present invention converts each row in a table (e.g., Excel or database table) into a text snippet describing a natural language, facilitating subsequent natural language processing and vectorization. This step is a core task in the initialization phase, providing the foundational data for subsequent queries.

[0038] For example, a table row {Company: "A Technology", Location: "Beijing", Industry: "Artificial Intelligence"} is converted to the text: "A Technology is an artificial intelligence company located in Beijing." A more direct approach can also be used, as shown in Table 1 below (example table data).

[0039] Table 1 Example table data

[0040]

[0041] Convert the data in Table 1 into the following two descriptive texts:

[0042] Row 1: Zhang San is male, 25 years old, 177 cm tall, and weighs 72 kg. Row 2: Li Si is male, 30 years old, 172 cm tall, and weighs 65 kg.

[0043] This step effectively preserves the semantic association between the table header and the data, ensuring that subsequent vectorization captures the meaning of the fields. The output of this step is a textualized dataset (such as a JSON list or plain text file), which serves as the input source for subsequent processing.

[0044] S2. Based on the named entity recognition model and the pre-trained embedding model, a knowledge base is constructed according to the descriptive text to obtain a direct path knowledge base.

[0045] Optionally, based on the named entity recognition model and the pre-trained embedding model, a knowledge base is constructed according to the descriptive text to obtain a direct path knowledge base, including:

[0046] Based on the named entity recognition model, key entities are extracted according to the descriptive text to obtain the first entity set;

[0047] According to the first entity set, high-dimensional vectors are generated by using a pre-trained embedding model to obtain a first entity vector set;

[0048] Based on a preset vectorized dictionary, a knowledge base is constructed according to the first entity set and the first entity vector set to obtain a direct path knowledge base.

[0049] In one feasible implementation, an embedding model is used to convert textual tabular data into a vectorized dictionary to obtain a direct path knowledge base.

[0050] Use named entity recognition (NER) models, such as the industrial-grade natural language processing tool spaCy or BERT-NER (Bidirectional Encoder Representations from Transformers Named Entity Recognition) to extract key entities (such as "A Technology," "Beijing," and "Artificial Intelligence") from text. You can also specify a column in table data as the entity for each row.

[0051] For each entity, a high-dimensional vector is generated through the pre-trained embedding model Sentence-BERT, which lays a foundation for the subsequent calculation of vector similarity; the high-dimensional vector generated by the present invention has a dimension of 768.

[0052] The structure of the vectorized dictionary is {entity: vector, ...}, and a vector index is established, a clustering and similarity search library (Facebook AI Similarity Search, FAISS) or an efficient approximate nearest neighbor search library (Approximate Nearest Neighbors Oh Yeah, Annoy) to accelerate similarity retrieval.

[0053] S3. Based on the Neo4j graph database, a knowledge base is constructed according to the descriptive text to obtain an indirect path knowledge base.

[0054] Optionally, a knowledge base is constructed based on the descriptive text based on the Neo4j graph database to obtain an indirect path knowledge base, including:

[0055] Based on the descriptive text, the dependency parsing method is used to extract relations and obtain an explicit relation set.

[0056] Based on the descriptive text, the relationship extraction model is used to mine potential connections and obtain implicit relationship sets;

[0057] Based on the preset relationship storage structure, a relationship dictionary is constructed according to the explicit relationship set and the implicit relationship set;

[0058] Based on the Neo4j graph database, the database is constructed according to the relational dictionary to obtain the indirect path knowledge base.

[0059] In a feasible implementation, the tabular data is subjected to relational parsing and further converted into an indirect path knowledge base including entity-attribute-relationship knowledge graph triples.

[0060] Identify the relationship types between entities based on the entities and their context in the textual data and the entity relationship categories preset by the user. The relationship types of entities can also be automatically identified by the relationship recognition model without being predefined.

[0061] Identify direct connections using rules or the dependency parsing method Stanford Parser. For example, extract the relationship (A Technology, located in, Beijing) from "Technology A is located in Beijing." Use a relation extraction model to discover potential connections. For example, infer (Technology A, partner, Company B) from "Technology A collaborates with Company B." Build a relation dictionary with the structure {entity: [{relationship type: target entity}, ...], ...}.

[0062] Use graph databases such as Neo4j to build a knowledge graph based on the entities and their relationships in the relational dictionary, laying the foundation for subsequent association queries and outputting a graph database instance containing entities and relationships.

[0063] Treat each entity as a node in a graph database, attaching attributes (such as industry and location). Treat relationships as directed edges, annotating them with types (such as "located in" and "partner"). Create indexes for frequently queried fields (such as location and industry) to speed up traversal. Use a graph database (such as Neo4j) to store data, forming an entity-relationship table that supports the Cypher query language.

[0064] S4. Obtain user query text; construct a structured query text based on the user query text based on the named entity recognition model and the pre-trained text classification model.

[0065] Optionally, based on the named entity recognition model and the pre-trained text classification model, a structured query text is constructed according to the user query text, including:

[0066] Perform data cleaning on user query text to obtain cleaned text;

[0067] Based on the named entity recognition model, key entities are extracted from the cleaned text to obtain the second entity set;

[0068] According to the text after cleaning, perform intent classification through a pre-trained text classification model to obtain the query intent and the constraints of the query intent;

[0069] Based on the query text template, construct a structured query text according to the second entity set, query intent, and constraints.

[0070] In a feasible implementation, the user inputs the text of the query Query, for example: the user's natural language Query (such as "What are the technology companies in Beijing?").

[0071] Perform data cleaning on the Query, remove stop words, and correct spelling mistakes (such as "technology company" → "technology company").

[0072] Use an entity recognition model, such as a NER model or a large language model (Large Language Models, LLM) to identify the core entities in the Query (such as "Beijing" "technology company"). When the Query entered by the user contains too many entity types, set parameters to limit the maximum number of entities to avoid affecting the retrieval efficiency.

[0073] Through a pre-trained text classification model, such as a text classification model based on Bidirectional Encoder Representations from Transformers (BERT) or text similarity analysis, to judge the user's intent (such as "entity list query" "association relationship query"). If it is an association relationship query, it is necessary to further clarify its relationship type for subsequent implementation of the association query. This process can also be completed using an LLM.

[0074] After the above process, output the structured query text. If it is a direct query for certain information, return the direct query result. The format example of the structured query text is as follows:

[0075]

[0076] When querying related entities, assume the user's Query is "Who are the customers of Company A?", and the format example of the structured query text is as follows:

[0077]

[0078] [[ID=so]]S5. Based on the direct path knowledge base, according to the structured query text, use the cosine similarity method for matching retrieval to obtain the direct retrieval result.

[0079] Optionally, based on the direct path knowledge base, according to the structured query text, use the cosine similarity method for matching retrieval to obtain the direct retrieval result, including:

[0080] According to the second entity set in the structured query text, a high-dimensional vector is generated through the BERT model to obtain the second entity vector set;

[0081] Based on the direct path knowledge base, cosine similarity calculation is performed on the second entity vector set to obtain a first similarity set;

[0082] Based on the direct path knowledge base, performing matching retrieval according to the first similarity set to obtain preliminary direct retrieval results;

[0083] According to the constraints in the structured query text, the preliminary direct search results are screened to obtain direct search results.

[0084] In one feasible implementation, the entity list in the structured query text is used to obtain the corresponding direct entity information (including entities and attributes). The entities in the query (such as "technology company") are converted into vectors. Cosine similarity is calculated in the vectorized dictionary, and the top K largest elements in the knowledge base data are returned to match entities (such as "A technology" and "C electronics"). For example, the BERT model is first used to vectorize the data of each sample, and cosine similarity is further used to measure the data association. For two data vectors, and , the cosine similarity is defined as follows (1);

[0085] (1);

[0086] in, Represents a vector and vector The inner product of and vector and vector Module (or length of the vector). Cosine similarity It is a value between 0 and 1. The closer it is to 1, the higher the similarity. Then, the direct search results are filtered according to the constraints in the structured query text (such as "industry=technology").

[0087] S6. Based on the indirect path knowledge base and according to the structured query text, an extended search is performed through a Cypher query statement to obtain indirect search results.

[0088] Optionally, based on the indirect path knowledge base and according to the structured query text, an extended search is performed using a Cypher query statement to obtain indirect search results, including:

[0089] According to the second entity set in the structured query text, a high-dimensional vector is generated through the BERT model to obtain the second entity vector set;

[0090] Based on the indirect path knowledge base, performing cosine similarity calculation on the second entity vector set to obtain a second similarity set;

[0091] Based on the indirect path knowledge base, query entity expansion is performed according to the second similarity set and a preset similarity threshold to obtain an expanded entity range;

[0092] Based on the indirect path knowledge base and the extended entity scope, a multi-hop query is performed through Cypher query statements to obtain preliminary indirect retrieval results;

[0093] Based on the preliminary indirect retrieval results, a large language model is used to remove incorrectly associated entities to obtain indirect retrieval results.

[0094] In a feasible implementation, according to the entity list and the corresponding relationship type in the structured query text, the corresponding indirectly related entity information is obtained to obtain the indirect retrieval result.

[0095] Expand the entity scope based on vector similarity (e.g., "Technology Company" → "Artificial Intelligence Company," "IT Service Provider"). Perform multi-hop queries (e.g., MATCH (a:Company)-[:Partner]->(b) WHERE a.name="A Technology" RETURN b), then return related entities (e.g., "B Company," "D Research Institute"). LLM performs semantic verification on indirect results to remove noise (e.g., the mistaken association of "E Catering").

[0096] S7. Based on the retrieval result prompt text, the large language model is used to perform text reasoning according to the user query text, direct retrieval results and indirect retrieval results to obtain the user query results.

[0097] In one feasible implementation, direct search results are combined with indirect search results to form entity query results (direct + indirect), ensuring that both explicitly matched and implicitly associated entities are covered. For example, for indirect query scenarios, the combined structured prompt is as follows:

[0098]

[0099] Based on the retrieval result prompt text, the user's query text and entity retrieval results are combined to construct the user query result, which is input into the LLM to complete the reasoning process and return the result to the user.

[0100] The present invention proposes a dual-path knowledge retrieval enhancement generation method for tabular data, which uses a graph database to construct a RAG knowledge base by performing correlation analysis on samples of tabular knowledge base data. On this basis, the user query text is parsed to automatically obtain the entity or associated entity information therein; the associated samples in the knowledge base are queried through the graph database support unit to obtain the retrieval results; the retrieved tabular data is converted into descriptive text suitable for understanding by a large language model; the user's query text and the obtained retrieval enhancement data are integrated and input into the large language model to obtain the corresponding results. The method proposed by the present invention has high knowledge retrieval efficiency and a large knowledge retrieval scope, and performs special knowledge base construction and data conversion for tabular data, so that the large model can easily understand and reason about the retrieved information, thereby improving the quality of the output results. The present invention is an efficient and accurate retrieval enhancement generation method for tabular data using dual-path knowledge retrieval.

[0101] Figure 2 This is a block diagram of a dual-path knowledge retrieval enhancement generation device for table data according to an exemplary embodiment. The device is used in a dual-path knowledge retrieval enhancement generation method for table data. Figure 2 The device includes a table data acquisition module 210, a direct path knowledge base construction module 220, an indirect path knowledge base construction module 230, a user query acquisition module 240, a direct retrieval module 250, an indirect retrieval module 260, and a query result generation module 270.

[0102] The table data acquisition module 210 is used to acquire the knowledge base table data; based on the description text template, the knowledge base table data is converted into text to obtain descriptive text;

[0103] A direct path knowledge base construction module 220 is configured to construct a knowledge base based on the named entity recognition model and the pre-trained embedding model according to the descriptive text to obtain a direct path knowledge base;

[0104] An indirect path knowledge base construction module 230 is used to construct a knowledge base based on a Neo4j graph database and descriptive text to obtain an indirect path knowledge base;

[0105] The user query acquisition module 240 is used to acquire user query text; based on the named entity recognition model and the pre-trained text classification model, construct a structured query text according to the user query text;

[0106] A direct search module 250 is configured to perform matching search based on a direct path knowledge base and a structured query text using a cosine similarity method to obtain direct search results;

[0107] The indirect search module 260 is used to perform an extended search based on the indirect path knowledge base and the structured query text using a Cypher query statement to obtain indirect search results;

[0108] The query result generation module 270 is used to perform text reasoning using a large language model based on the retrieval result prompt text according to the user query text, direct retrieval results and indirect retrieval results to obtain the user query result.

[0109] Optionally, the direct path knowledge base construction module 220 is further configured to:

[0110] Based on the named entity recognition model, key entities are extracted according to the descriptive text to obtain the first entity set;

[0111] According to the first entity set, high-dimensional vectors are generated by using a pre-trained embedding model to obtain a first entity vector set;

[0112] Based on a preset vectorized dictionary, a knowledge base is constructed according to the first entity set and the first entity vector set to obtain a direct path knowledge base.

[0113] Optionally, the indirect path knowledge base construction module 230 is further configured to:

[0114] Based on the descriptive text, the dependency parsing method is used to extract relations and obtain an explicit relation set.

[0115] Based on the descriptive text, the relationship extraction model is used to mine potential connections and obtain implicit relationship sets;

[0116] Based on the preset relationship storage structure, a relationship dictionary is constructed according to the explicit relationship set and the implicit relationship set;

[0117] Based on the Neo4j graph database, the database is constructed according to the relational dictionary to obtain the indirect path knowledge base.

[0118] Optionally, the user query acquisition module 240 is further configured to:

[0119] Perform data cleaning on user query text to obtain cleaned text;

[0120] Based on the named entity recognition model, key entities are extracted from the cleaned text to obtain the second entity set;

[0121] Based on the cleaned text, intent classification is performed using a pre-trained text classification model to obtain the query intent and query intent constraints.

[0122] Based on the query text template, a structured query text is constructed according to the second entity set, query intent and constraint conditions.

[0123] Optionally, the direct retrieval module 250 is further configured to:

[0124] According to the second entity set in the structured query text, a high-dimensional vector is generated through the BERT model to obtain the second entity vector set;

[0125] Based on the direct path knowledge base, cosine similarity calculation is performed on the second entity vector set to obtain a first similarity set;

[0126] Based on the direct path knowledge base, performing matching retrieval according to the first similarity set to obtain preliminary direct retrieval results;

[0127] According to the constraints in the structured query text, the preliminary direct search results are screened to obtain direct search results.

[0128] Optionally, the indirect retrieval module 260 is further configured to:

[0129] According to the second entity set in the structured query text, a high-dimensional vector is generated through the BERT model to obtain the second entity vector set;

[0130] Based on the indirect path knowledge base, performing cosine similarity calculation on the second entity vector set to obtain a second similarity set;

[0131] Based on the indirect path knowledge base, query entity expansion is performed according to the second similarity set and a preset similarity threshold to obtain an expanded entity range;

[0132] Based on the indirect path knowledge base and the extended entity scope, a multi-hop query is performed through Cypher query statements to obtain preliminary indirect retrieval results;

[0133] Based on the preliminary indirect retrieval results, a large language model is used to remove incorrectly associated entities to obtain indirect retrieval results.

[0134] The present invention proposes a dual-path knowledge retrieval enhancement generation method for tabular data, which uses a graph database to construct a RAG knowledge base by performing correlation analysis on samples of tabular knowledge base data. On this basis, the user query text is parsed to automatically obtain the entity or associated entity information therein; the associated samples in the knowledge base are queried through the graph database support unit to obtain the retrieval results; the retrieved tabular data is converted into descriptive text suitable for understanding by a large language model; the user's query text and the obtained retrieval enhancement data are integrated and input into the large language model to obtain the corresponding results. The method proposed by the present invention has high knowledge retrieval efficiency and a large knowledge retrieval scope, and performs special knowledge base construction and data conversion for tabular data, so that the large model can easily understand and reason about the retrieved information, thereby improving the quality of the output results. The present invention is an efficient and accurate retrieval enhancement generation method for tabular data using dual-path knowledge retrieval.

[0135] Figure 3 Schematic diagram of the structure of a dual-path knowledge retrieval enhancement generation device provided by an embodiment of the present invention. Figure 3 As shown, the dual-path knowledge retrieval enhancement generation device may include the above Figure 2 Optionally, the dual-path knowledge retrieval enhancement generation device 310 may include a first processor 2001 .

[0136] Optionally, the dual-path knowledge retrieval enhancement generation device 310 may further include a memory 2002 and a transceiver 2003 .

[0137] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0138] The following combination Figure 3 The components of the dual-path knowledge retrieval enhancement generation device 310 are described in detail:

[0139] The first processor 2001 is the control center of the dual-path knowledge retrieval enhancement generation device 310 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0140] Optionally, the first processor 2001 may execute various functions of the dual-path knowledge retrieval enhancement generation device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0141] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0142] In a specific implementation, as an embodiment, the dual-path knowledge retrieval enhancement generation device 310 may also include multiple processors, such as Figure 3 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0143] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0144] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and access the memory through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0145] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0146] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0147] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and enhance the interface circuit of the generation device 310 through the dual-path knowledge retrieval. Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0148] It should be noted that Figure 3 The structure of the dual-path knowledge retrieval enhancement generation device 310 shown in the figure does not constitute a limitation on the router. The actual knowledge structure identification device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0149] In addition, the technical effects of the dual-path knowledge retrieval enhancement generation device 310 can refer to the technical effects of the dual-path knowledge retrieval enhancement generation method for tabular data described in the above method embodiment, and will not be repeated here.

[0150] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0151] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0152] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0153] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0154] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0155] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0156] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0157] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0161] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0162] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A dual-path knowledge retrieval enhancement generation method for tabular data, characterized in that: The method comprises: Acquire knowledge base table data; perform text conversion on the knowledge base table data based on the description text template to obtain descriptive text; Based on the named entity recognition model and the pre-trained embedding model, a knowledge base is constructed according to the descriptive text to obtain a direct path knowledge base; Based on the Neo4j graph database, the knowledge base is constructed according to the descriptive text to obtain the indirect path knowledge base; Obtain user query text; construct structured query text based on the user query text based on the named entity recognition model and the pre-trained text classification model; Based on the direct path knowledge base, according to the structured query text, the cosine similarity method is used to perform matching retrieval to obtain direct retrieval results; Based on the indirect path knowledge base, according to the structured query text, the Cypher query statement is used to perform extended search to obtain indirect search results; Based on the retrieval result prompt text, a large language model is used to perform text reasoning based on the user query text, direct retrieval results, and indirect retrieval results to obtain the user query results.

2. The dual-path knowledge retrieval enhancement generation method for tabular data according to claim 1 is characterized in that: The method is based on the named entity recognition model and the pre-trained embedding model, and constructs a knowledge base according to the descriptive text to obtain a direct path knowledge base, including: Based on the named entity recognition model, key entities are extracted according to the descriptive text to obtain the first entity set; According to the first entity set, high-dimensional vectors are generated by using a pre-trained embedding model to obtain a first entity vector set; Based on a preset vectorized dictionary, a knowledge base is constructed according to the first entity set and the first entity vector set to obtain a direct path knowledge base.

3. The dual-path knowledge retrieval enhancement generation method for tabular data according to claim 1 is characterized in that: The indirect path knowledge base is obtained by constructing a knowledge base based on the Neo4j graph database and the descriptive text, including: Based on the descriptive text, the dependency parsing method is used to extract relations and obtain an explicit relation set. Based on the descriptive text, the relationship extraction model is used to mine potential connections and obtain implicit relationship sets; Based on the preset relationship storage structure, a relationship dictionary is constructed according to the explicit relationship set and the implicit relationship set; Based on the Neo4j graph database, the database is constructed according to the relational dictionary to obtain the indirect path knowledge base.

4. The dual-path knowledge retrieval enhancement generation method for tabular data according to claim 1 is characterized in that: The method of constructing a structured query text based on the user query text based on the named entity recognition model and the pre-trained text classification model includes: Perform data cleaning on user query text to obtain cleaned text; Based on the named entity recognition model, key entities are extracted from the cleaned text to obtain the second entity set; Based on the cleaned text, intent classification is performed using a pre-trained text classification model to obtain the query intent and query intent constraints. Based on the query text template, a structured query text is constructed according to the second entity set, query intent and constraint conditions.

5. The dual-path knowledge retrieval enhancement generation method for tabular data according to claim 1 is characterized in that: The direct path knowledge base is based on the structured query text and the cosine similarity method is used to perform matching retrieval to obtain direct retrieval results, including: According to the second entity set in the structured query text, a high-dimensional vector is generated through the BERT model to obtain the second entity vector set; Based on the direct path knowledge base, cosine similarity calculation is performed on the second entity vector set to obtain a first similarity set; Based on the direct path knowledge base, performing matching retrieval according to the first similarity set to obtain preliminary direct retrieval results; According to the constraints in the structured query text, the preliminary direct search results are screened to obtain direct search results.

6. The dual-path knowledge retrieval enhancement generation method for tabular data according to claim 1 is characterized in that: The indirect path knowledge base is based on the structured query text and an extended search is performed through a Cypher query statement to obtain indirect search results, including: According to the second entity set in the structured query text, a high-dimensional vector is generated through the BERT model to obtain the second entity vector set; Based on the indirect path knowledge base, performing cosine similarity calculation on the second entity vector set to obtain a second similarity set; Based on the indirect path knowledge base, query entity expansion is performed according to the second similarity set and a preset similarity threshold to obtain an expanded entity range; Based on the indirect path knowledge base and the extended entity scope, a multi-hop query is performed through Cypher query statements to obtain preliminary indirect retrieval results; Based on the preliminary indirect retrieval results, a large language model is used to remove incorrectly associated entities to obtain indirect retrieval results.

7. A dual-path knowledge retrieval enhancement generation device for tabular data, the dual-path knowledge retrieval enhancement generation device for tabular data being used to implement the dual-path knowledge retrieval enhancement generation method for tabular data as claimed in any one of claims 1 to 6, characterized in that: The device comprises: The table data acquisition module is used to obtain the knowledge base table data; based on the description text template, the knowledge base table data is converted into text to obtain descriptive text; The direct path knowledge base construction module is used to construct a knowledge base based on the named entity recognition model and the pre-trained embedding model according to the descriptive text to obtain a direct path knowledge base; The indirect path knowledge base construction module is used to construct a knowledge base based on the Neo4j graph database and descriptive text to obtain an indirect path knowledge base; The user query acquisition module is used to obtain user query text; based on the named entity recognition model and the pre-trained text classification model, it constructs a structured query text according to the user query text; Direct retrieval module, which is used to perform matching retrieval based on the direct path knowledge base and the structured query text using the cosine similarity method to obtain direct retrieval results; The indirect search module is used to perform extended search based on the indirect path knowledge base and the structured query text through Cypher query statements to obtain indirect search results; The query result generation module is used to obtain user query results based on the retrieval result prompt text, the user query text, direct retrieval results and indirect retrieval results, and the text reasoning using a large language model.

8. The dual-path knowledge retrieval enhancement generation method for tabular data according to claim 7 is characterized in that: The user query acquisition module is further used to: Perform data cleaning on user query text to obtain cleaned text; Based on the named entity recognition model, key entities are extracted from the cleaned text to obtain the second entity set; Based on the cleaned text, intent classification is performed using a pre-trained text classification model to obtain the query intent and query intent constraints. Based on the query text template, a structured query text is constructed according to the second entity set, query intent and constraint conditions.

9. A dual-path knowledge retrieval enhancement generation device, characterized in that: The dual-path knowledge retrieval enhancement generation device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images