Multi-hop table retrieval method and system for open domain Text-to-SQL (Structured Query Language), storage medium and equipment

Abstract

The invention discloses an open domain Text-to-SQL (Structured Query Language) multi-hop table retrieval method and system, a storage medium and equipment, and belongs to the technical field of Text-to-SQL. The method aims at solving the problems that according to an existing table retrieval method, an unrelated table is retrieved due to the fact that an entity is similar to an entity in a plurality of unrelated tables, and the retrieved table is farther away from a related field due to the fact that the entity is possibly not matched with the related field. According to the method, multi-hop retrieval is adopted, an original user question retrieval table is adopted in the first hop, and a tabular question retrieval table rewritten by LLM is adopted in the later hop steps; a plurality of search results are kept in each hop by adopting beam search, the probability of correlation between a table and a question is calculated after each hop of search stage is ended, and a plurality of groups of most possible search results are selected and input to a rewriting stage together with an original user question; all the tables are sorted, and the tables are selected according to the tables for retrieval.

Description

Technical Field

[0001] The present invention belongs to the field of Text-to-SQL technology, and in particular relates to a Text-to-SQL table retrieval method, system, storage medium and device. Background Art

[0002] Text-to-SQL systems convert natural language descriptions into corresponding SQL queries. Open-domain text-to-SQL retrieves tables related to a question from a large database and then converts the question into SQL based on the tables, a critical task that is close to real-world scenarios. However, existing single-hop retrieval methods fail to address the inherent challenge of text-to-SQL: schema linking, which involves aligning entities in the question with those in the table. This challenge manifests itself in two ways: similar but unrelated entities, and entities with mismatched domains.

[0003] Similar unrelated entities refer to instances where certain entities in a question are similar to entities in many unrelated tables, resulting in these unrelated tables being retrieved while related tables corresponding to other entities in the question are not. Domain-mismatched entities refer to instances where entities in a question may not match the relevant domain, resulting in retrieved tables that are further from the relevant domain. Therefore, a solution is needed to address these two limitations. Summary of the Invention

[0004] The present invention aims to solve the problem in existing Text-to-SQL table retrieval methods that irrelevant tables are retrieved due to the similarity between entities and entities in many irrelevant tables, and the problem that domain-mismatched entities refer to entities in the question that may not match the relevant domain, resulting in the retrieved tables being further away from the relevant domain.

[0005] The multi-hop table retrieval method for open-domain Text-to-SQL uses a multi-hop retrieval method. The first hop uses the original user question retrieval table, and in the hops after the first hop, the tabular question retrieval table is rewritten using the large-scale language model (LLM). A beam search is used to maintain multiple retrieval results in each hop. After each retrieval stage, the probability of the table being related to the question is calculated, and multiple groups of the most likely retrieval results are selected and input into the rewriting stage together with the original user question. The above process is repeated until the maximum number of hops is reached or the early stopping condition is met. Finally, all tables are sorted and selected accordingly to input into the text-to-SQL module.

[0006] Furthermore, after sorting all the tables and selecting the tables to input into the text-to-SQL module, the text-to-SQL module generates an SQL query based on the user question and the retrieved tables.

[0007] Furthermore, the large-scale language model is an LLM model.

[0008] Furthermore, each hop in the multi-hop search method includes a search phase and a rewriting phase;

[0009] The processing process of the retrieval stage includes:

[0010] First, an encoder is used to encode the user question and all candidate tables (a total of N) into vectors. The cosine similarity between the question vector and the table vector is calculated as the retrieval score of the table in this hop. Then, following the beam search, the tables retrieved in the current hop are added to the tables retrieved previously in this bundle, as the retrieval results of this bundle up to this hop. The product of the retrieval scores of all tables in the current retrieval results of this bundle is calculated as the overall score of the bundle. The tables with the top k overall scores are selected as the corresponding retrieval results. Each result is input into the Large Scale Language Model (LLM) along with the original user question for rewriting.

[0011] The processing process of the rewriting stage includes:

[0012] The original user question and the current search results are input to the LLM, prompting the LLM to guess which tables are needed to answer the user question based on the original user question and the retrieved tables, that is, the related tables that have not been retrieved and output them as the next hop search query; at the same time, the LLM is prompted to generate a special tag if the retrieved tables can fully answer the user question; if the LLM outputs this special tag, it stops early at this hop.

[0013] Furthermore, in the rewriting stage, the LLM is prompted. If the retrieved table is able to fully answer the user's question, a special mark is generated. The specific prompt is "<question><retrieved table> Please guess the table that is needed to answer the question but has not been retrieved based on the question and the retrieved table; if the retrieved table is sufficient to answer the question, None is generated;".

[0014] Furthermore, the process of sorting all tables and selecting tables to input into text-to-SQL includes the following steps:

[0015] After the multi-hop search is completed, the overall scores of each bundle are compared, the bundle with the highest score is selected, and each table in this bundle of search results and the original user question are input into the text-to-SQL module.

[0016] The open domain Text-to-SQL multi-hop table retrieval system includes a retrieval module, a rewriting module, and a sorting module; among them,

[0017] The processing process of the retrieval module includes:

[0018] First, an encoder is used to encode the user question and all candidate tables (a total of N) into vectors. The cosine similarity between the question vector and the table vector is calculated as the retrieval score of the table in this hop. Then, following the beam search, the tables retrieved in the current hop are added to the tables retrieved previously in this bundle, as the retrieval results of this bundle up to this hop. The product of the retrieval scores of all tables in the current retrieval results of this bundle is calculated as the overall score of the bundle. The tables with the top k overall scores are selected as the corresponding retrieval results. Each result is input into the Large Scale Language Model (LLM) along with the original user question for rewriting.

[0019] The processing of the rewriting module includes:

[0020] The original user question and the current search results are input to the LLM. The LLM is prompted to guess which tables are needed to answer the user question based on the original user question and the retrieved tables. This means that the unretrieved related tables are output as the next hop search query. At the same time, the LLM is prompted to generate a special tag if the retrieved tables can fully answer the user question. If the LLM outputs this special tag, it stops early at this hop.

[0021] The processing process of the sorting module includes:

[0022] After the multi-hop search is completed, the overall scores of each bundle are compared, the bundle with the highest score is selected, and each table in this bundle of search results and the original user question are input into the text-to-SQL module.

[0023] Furthermore, the system also includes a text-to-SQL module, which generates SQL queries based on user questions and retrieved tables.

[0024] A computer storage medium stores at least one instruction, which is loaded by a processor and runs the open domain Text-to-SQL multi-hop table retrieval system.

[0025] An open domain Text-to-SQL multi-hop table retrieval device includes a processor and a memory. The memory stores at least one instruction, which is loaded by the processor and runs the open domain Text-to-SQL multi-hop table retrieval system.

[0026] Beneficial effects:

[0027] To address the two limitations of the prior art, this paper introduces multi-hop search and proposes a multi-hop table search method for open-domain text-to-SQL, alleviating the challenges of schema chaining and improving search performance. This method removes already retrieved information at each hop to guide the model in retrieving unretrieved tables. Furthermore, the beam search paradigm takes lower-ranked tables into account, enabling better retrieval of relevant tables that would be ranked lower in a single-hop search and alleviating the limitation of similar unrelated entities. Furthermore, our method rewrites the search query based on already retrieved tables, narrowing the gap between domains and alleviating the limitation of domain-mismatched entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The schematic diagrams retrieved for the multi-hop table.

[0029] Figure 2 This is a specific flowchart for multi-hop table retrieval. DETAILED DESCRIPTION

[0030] The following describes the specific implementation methods.

[0031] Specific implementation method 1: Combination Figure 1 and Figure 2 To explain this embodiment,

[0032] This embodiment is an open domain Text-to-SQL multi-hop table search method, which is applicable to open domain text-to-SQL table search.

[0033] The open domain Text-to-SQL multi-hop table search method described in this embodiment adopts a multi-hop search, wherein each hop includes a search phase and a rewriting phase.

[0034] The first step uses the original user question retrieval table. In subsequent steps, the table is rewritten using a large-scale language model (LLM). A beam search is used to maintain multiple retrieval results in each step. After each retrieval phase, the probability of the table being relevant to the question is calculated. The most likely retrieval results are selected and fed into the rewriting phase along with the original user question. This process is repeated until the maximum number of steps is reached or an early stopping condition is met. Finally, all tables are sorted and selected based on this sorting to be fed into the text-to-SQL module.

[0035] S100, retrieval stage:

[0036] The retrieval stage requires retrieving tables based on the user question in the current jump and the current bundle. This method first uses an encoder to encode the user question and all candidate tables, a total of N, into vectors. The cosine similarity between the question vector and the table vector is calculated as the retrieval score of the table in this jump. Then, following the beam search, the tables retrieved in the current jump are added to the tables retrieved previously in this bundle as the retrieval results for this bundle up to this jump. The product of the retrieval scores of all tables in the current retrieval results of this bundle is calculated as the overall score of this bundle. We select the top k tables with the highest overall scores and the corresponding retrieval results. Each result is input into the Large Scale Language Model (LLM) together with the original user question for rewriting.

[0037] For example, we set the bundle size to 3, maintaining a 3-bundle search with k = 3. In the first hop, we select the top three tables with the highest cosine similarity to the question encoding, and use them as the search results for the first, second, and third bundles of the first hop, respectively. Based on this, we feed the user question and these three tables into the LLM as three sets of inputs for rewriting, obtaining three rewritten questions, which are the questions corresponding to the second hop of the 3-bundle search. These three rewritten questions are then searched separately. For example, for the first bundle search, after the first rewritten question is encoded into a vector and its cosine similarity is calculated with all N table vectors, these N tables are combined with the first table retrieved in the first hop of the first bundle to form N tuples. The 3-bundle search then yields a total of 3*N tuples, which serve as the search results for the second hop. We continue to select the top three tuples from the search results for subsequent rewriting and retrieval.

[0038] S200, rewriting stage:

[0039] The rewriting stage needs to refer to the retrieved tables to rewrite the user question and determine whether the multi-hop search stops early at this hop. This method inputs the original user question and the current search results to the LLM, prompting the LLM to guess which tables are needed to answer the user question based on the original user question and the retrieved tables, that is, the related tables that have not been retrieved and output them as the next hop search query. At the same time, the LLM is prompted to generate a special tag if the retrieved tables can fully answer the user question. If the LLM outputs this special tag, the method stops early at this hop. The specific prompt is translated into Chinese as "<Question><Retrieved tables> Please guess the tables that are needed to answer the question but have not been retrieved based on the question and the retrieved tables. If the retrieved tables are sufficient to answer the question, None is generated."

[0040] for example Figure 1In the rewriting phase of the first hop, the LLM is prompted to guess the unretrieved related tables based on the user question "What countries that not playing cartoons written by Todd Casey?" and the already retrieved table "tvshow.cartoon(...)". The LLM output "tvshow.country(country id, countryname, language)" is used as the search query for the second hop. In the second hop, the LLM is prompted to guess the unretrieved tables based on the user question and the already retrieved tables "tvshow.cartoon(...)" and "tvshow.tv channel(...)". The LLM believes that the information in these two tables can fully answer the user question, that is, the tables "tvshow.cartoon(...)" and "tvshow.tv channel(...)" can answer the question "What countries that not playing cartoons written by Todd Casey?" Therefore, a special marker for early stopping is output, and the multi-hop search stops at this hop.

[0041] S300, sorting stage:

[0042] After the multi-hop search is completed, this solution compares the overall scores of each bundle and selects the bundle with the highest score. Each table in this bundle of search results and the original user question are input into the text-to-SQL module. The model generates an SQL query based on the user question and the retrieved tables.

[0043] The key of the present invention is to adopt multi-hop table retrieval for open domain text-to-SQL and to adopt beam search and rewriting method in multi-hop retrieval for similar unrelated entities and domain mismatch entity restrictions of pattern links. Specific implementation method 2:

[0045] This embodiment is an open domain Text-to-SQL multi-hop table retrieval system, including a retrieval module, a rewriting module, and a sorting module, which may also include a text-to-SQL module; wherein,

[0046] The processing process of the retrieval module includes:

[0047] First, an encoder is used to encode the user question and all candidate tables (a total of N) into vectors. The cosine similarity between the question vector and the table vector is calculated as the retrieval score of the table in this hop. Then, following the beam search, the tables retrieved in the current hop are added to the tables retrieved previously in this bundle, as the retrieval results of this bundle up to this hop. The product of the retrieval scores of all tables in the current retrieval results of this bundle is calculated as the overall score of the bundle. The tables with the top k overall scores are selected as the corresponding retrieval results. Each result is input into the Large Scale Language Model (LLM) along with the original user question for rewriting.

[0048] The processing of the rewriting module includes:

[0049] The original user question and the current search results are input to the LLM. The LLM is prompted to guess which tables are needed to answer the user question based on the original user question and the retrieved tables. This means that the unretrieved related tables are output as the next hop search query. At the same time, the LLM is prompted to generate a special tag if the retrieved tables can fully answer the user question. If the LLM outputs this special tag, it stops early at this hop.

[0050] The processing process of the sorting module includes:

[0051] After the multi-hop search is completed, the overall scores of each bundle are compared, the bundle with the highest score is selected, and each table in this bundle of search results and the original user question are input into the text-to-SQL module.

[0052] The text-to-SQL module generates SQL queries based on user questions and retrieved tables. Specific implementation method three:

[0054] This embodiment is a computer storage medium, wherein at least one instruction is stored in the storage medium. The at least one instruction is loaded by a processor and runs the open domain Text-to-SQL multi-hop table retrieval system.

[0055] It should be understood that the instructions include computer program products, software, or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system or other electronic device. Computer storage media may include readable media on which instructions are stored, and may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions. Specific implementation method four:

[0057] This embodiment is an open domain text-to-SQL multi-hop table search device, which includes a processor and a memory. It should be understood that the device includes any device including a processor and a memory described in the present invention, and may also include other units or modules that perform display, interaction, processing, control, and other functions through signals or instructions.

[0058] The memory stores at least one instruction, which is loaded and executed by the processor to execute the open domain Text-to-SQL multi-hop table retrieval system.

[0059] Those skilled in the art will appreciate that at least one instruction stored is a computer program product corresponding to the method or system. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0060] The present application is described with reference to the flowcharts and / or block diagrams of the methods, systems, and computer program products according to the embodiments of the present application, and can also be used for corresponding devices. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0063] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0064] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0065] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. An open domain Text-to-SQL multi-hop table retrieval method, characterized in that: A multi-hop search method is used for retrieval. The first hop uses the original user question retrieval table, and in the hops after the first hop, the tabular question retrieval table is rewritten using the large-scale language model LLM. Beam search is used to maintain multiple retrieval results in each hop. After each hop retrieval stage, the probability of the table being related to the question is calculated, and multiple groups of the most likely retrieval results are selected and input into the rewriting stage together with the original user question. The above process is repeated until the maximum number of hops is reached or the early stopping condition is met. Finally, all tables are sorted and the tables are selected accordingly to input into the text-to-SQL module.

2. The open domain Text-to-SQL multi-hop table retrieval method according to any one of claim 1, characterized in that: After sorting all the tables and selecting the tables to input into the text-to-SQL module, the text-to-SQL module generates SQL queries based on the user questions and the retrieved tables.

3. The open domain Text-to-SQL multi-hop table retrieval method according to claim 1 or 2, characterized in that: The large-scale language model is an LLM model.

4. The multi-hop table retrieval method of open domain Text-to-SQL according to claim 3, characterized in that: Each hop in the multi-hop retrieval method includes a retrieval phase and a rewriting phase; The processing process of the retrieval stage includes: First, the encoder is used to encode the user question and all candidate tables, a total of N, into vectors, and the cosine similarity between the question vector and the table vector is calculated as the retrieval score of the table in this jump; then, following the beam search, the tables retrieved in the current jump are added to the tables retrieved before this bundle, as the retrieval results of this bundle up to this jump; the product of the retrieval scores of all tables in the current retrieval results of this bundle is calculated as the overall score of this bundle; the tables with the top k overall scores are selected to correspond to the retrieval results, and each result is input into the large-scale language model LLM together with the original user question for rewriting; The processing process of the rewriting stage includes: The original user question and the current search results are input to LLM, and LLM is prompted to guess which tables are needed to answer the user question based on the original user question and the retrieved tables, that is, the related tables that have not been retrieved and output them as the next hop search query; at the same time, LLM is prompted to generate a special tag if the retrieved tables can fully answer the user question; if LLM outputs this special tag, it stops early at this hop.

5. The multi-hop table retrieval method of open domain Text-to-SQL according to claim 4, characterized in that: During the rewriting phase, LLM is prompted. If the retrieved table is able to fully answer the user's question, a special mark is generated. The specific prompt is "<question><retrieved table> Please guess the table that is needed to answer the question but has not been retrieved based on the question and the retrieved table; if the retrieved table is sufficient to answer the question, None is generated;".

6. The open domain Text-to-SQL multi-hop table retrieval method according to claim 5, characterized in that: The process of sorting all tables and selecting tables for text-to-SQL input includes the following steps: After the multi-hop search is completed, the overall scores of each bundle are compared, and the bundle with the highest score is selected. Each table in this bundle of search results and the original user question are input into the text-to-SQL module.

7. An open domain Text-to-SQL multi-hop table retrieval system, characterized by: It includes retrieval module, rewriting module and sorting module; among them, The processing process of the retrieval module includes: First, the encoder is used to encode the user question and all candidate tables, a total of N, into vectors, and the cosine similarity between the question vector and the table vector is calculated as the retrieval score of the table in this jump; then, following the beam search, the tables retrieved in the current jump are added to the tables retrieved before this bundle, as the retrieval results of this bundle up to this jump; the product of the retrieval scores of all tables in the current retrieval results of this bundle is calculated as the overall score of this bundle; the tables with the top k overall scores are selected to correspond to the retrieval results, and each result is input into the large-scale language model LLM together with the original user question for rewriting; The processing process of the rewriting module includes: Input the original user question and the current search results to LLM, prompt LLM to guess which tables are needed to answer the user question based on the original user question and the tables that have been retrieved, that is, the related tables that have not been retrieved and output them as the next hop search query; at the same time, prompt LLM to generate a special mark if the retrieved table can fully answer the user question; if LLM outputs this special mark, it stops early at this hop; The processing process of the sorting module includes: After the multi-hop search is completed, the overall scores of each bundle are compared, and the bundle with the highest score is selected. Each table in this bundle of search results and the original user question are input into the text-to-SQL module.

8. The open domain Text-to-SQL multi-hop table retrieval system according to claim 7, characterized in that: The system also includes a text-to-SQL module, which generates SQL queries based on user questions and retrieved tables.

9. A computer storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded by the processor and runs the open domain Text-to-SQL multi-hop table retrieval system according to any one of claims 7 to 8.

10. An open domain Text-to-SQL multi-hop table retrieval device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded by the processor and runs the open domain Text-to-SQL multi-hop table retrieval system according to any one of claims 7 to 8.

Images