A Table Retrieval Enhancement Method for Open-Domain Question Answering

Abstract

The present invention belongs to the field of natural language processing and information retrieval technology, and provides an execution-guided table retrieval enhancement method for open-domain question-answering. First, a retriever is used to preliminarily screen relevant tables from a table corpus to obtain a table pool. Then, for each table in the table pool, a deep learning Text-to-SQL model is used to convert the question into a standardized logical form such as SQL in combination with the question and table pattern information. Next, SQL is executed on the table and it is determined whether an error occurs in the execution result, which is used as a correlation basis to be incorporated into a new round of similarity calculation. The present invention makes full use of the table's pattern information in the process of table retrieval, incorporates the execution result into the retrieval similarity score, and effectively improves the accuracy of the table retrieval stage in the open-domain question-answering process.

Description

Technical Field

[0001] The present invention belongs to the technical fields of natural language processing and information retrieval, and relates to a method for enhancing table retrieval in open-domain question answering based on a Text-to-SQL deep learning model. Background Art

[0002] In open-domain question answering, compared with closed-domain question answering, the answer is not limited to a given reading material, but exists in a large corpus. As an important way to store information, tables exist in large numbers in web services and relational databases. Compared with free text, tables store a large amount of information and the content is more specific, which is an important information source for open-domain question answering. The table open-domain question answering task changes the retrieval object from text to table, and studies how to obtain the information that users are interested in from a large number of tables, which can play an important role in search engines and artificial customer service.

[0003] Currently, the mainstream method for open-domain question answering is a two-stage framework: the retrieval-reading model, which divides open-domain question answering into two stages: the purpose of the retrieval stage is to find text segments related to the question from thousands of texts, and the reading stage extracts answers from these relevant contents. For table open-domain question answering, the reading stage can be regarded as a closed-domain table question answering.

[0004] In terms of retrieval, traditional BM25 method and DPR method based on deep learning are both aimed at free text and do not make special optimizations for tables. Some scholars proposed DTR. Compared with DPR, it uses Tapas to replace Bert as the encoder and encodes the table structure, but the retrieval task is strongly related to the table content, and only encoding the table structure leads to unsatisfactory table retrieval results.

[0005] Great research progress has been made in closed-domain table question answering. There are two main methods. One is to encode the structure and content of the table and select the target cell as the answer, such as Tapas; the other is based on semantic parsing, which converts the question described in natural language into a logical form that can be executed on the table, such as SQL, and then executes the statement to obtain the corresponding answer to the question. Currently, the models for the closed-domain Text-to-SQL task have achieved an accuracy rate of over 90% on large datasets such as WikiSQL. Thus, table retrieval has become the bottleneck for improving the accuracy of the entire open-domain question answering. Summary of the Invention

[0006] In view of the above problems, the present invention provides an execution-guided table retrieval enhancement method for open-domain question answering. First, a retriever is used to preliminarily screen relevant tables from a table corpus to obtain a table pool. Then, for each table in the table pool, a deep learning Text-to-SQL model is used to convert the question into a standardized logical form such as SQL by combining the question and table pattern information. Next, SQL is executed on the table and it is determined whether an error occurs in the execution result, which is used as a correlation basis to be incorporated into a new round of similarity calculation.

[0007] The technical solution of the present invention is:

[0008] The process of open domain question answering is divided into: table retrieval stage and answer extraction stage; there are the following steps:

[0009] S1. Table preprocessing. Use tools such as SQLite to convert the table corpus into a db format so that SQL statements can be executed on any table. If the table itself exists in a relational database, skip the preprocessing stage.

[0010] S2. Perform a preliminary search. Expand the table by rows to form table content in text form; encode the content and load it into the index library; use the retriever to calculate the original similarity between the question and each table in the table corpus; sort the original similarity from large to small loss, and select the top N tables to form a table pool.

[0011] S3. Execute the statement on the table and obtain a new similarity score. For each table in the table pool, first input the question and table mode information into the Text-to-SQL deep learning model to obtain the corresponding SQL statement, and then execute the obtained SQL semantics on the table to determine whether an execution error occurs. If no execution error occurs, set the res EG =1, if an execution error occurs, then res EG =0; Calculate the new similarity score:

[0012] sim withEG =(1-α)·sim origin / maxsim origin +α·res EG

[0013] Among them, sim origin is the raw similarity score at the retrieval stage, maxsim origin is the maximum raw similarity score of all tables in the retrieval phase, sim withEG is the new similarity score;

[0014] S4, table reordering. The N tables in the table pool are sorted according to the new similarity score simwithEG Re - sort from largest to smallest, select the top - k tables with the highest scores, and enter the answer extraction stage;

[0015] S5. Answer extraction stage. Based on the input question sentence and the obtained top - k tables, use a deep - learning model for answer extraction such as cell classification or generation to obtain the answer.

[0016] The beneficial effects of the present invention are as follows: During the process of table retrieval, the pattern information of the table is fully utilized, and the execution result is incorporated into the retrieval similarity score, effectively improving the accuracy of the table retrieval stage in the open - domain question - answering process. Brief Description of the Drawings

[0017] Figure 1 It is a schematic diagram of a two - stage model for open - domain question - answering.

[0018] Figure 2 It is a flowchart of enhanced table retrieval based on execution guidance.

[0019] Figure 3 It is a comparison of experimental results between enhanced table retrieval based on execution guidance and non - enhanced table retrieval. Detailed Embodiments

[0020] The present invention will be described in detail below with reference to the accompanying drawings.

[0021] In the present invention, the two - stage model for table open - domain question - answering is as Figure 1 . The enhanced process of table retrieval based on execution guidance is as Figure 2 . Among them, the table retrieval stage may involve two deep - learning models: a retriever and a Text - to - SQL model. Before actual retrieval, the corresponding model training needs to be completed in combination with the labeled table corpus data.

[0022] Pre - processing of the table. Convert the table corpus into a db form using tools such as SQLite, so that SQL statements can be executed on any table. If the table already exists in a relational database, skip the pre - processing stage.

[0023] Flattening and expansion of the table. Before building the index library, process all the tables in the corpus into the form of continuous text. Concatenate the table title, column names, and row contents in the following form. If there is no table title, fill it with spaces.

[0024] Table title|Table column names|First row content|Second row content|…|nth row content

[0025] During the splicing process, delimiters are added to the spliced text after table conversion. Use ',' to represent the separation of each cell in a row, and '.' to represent the separation between rows. The delimiters enable the retriever to learn the table structure to a certain extent.

[0026] Perform a preliminary retrieval. The goal of the retriever is to retrieve N tables T1, T2,..., T from a large table corpus of tens of thousands, in descending order of scores according to the relevance to the question q. N , as the table pool related to the question sentence. Since this is a preliminary retrieval stage, N is generally a relatively large number. Considering the efficiency of subsequent screening, set N = 200.

[0027] The retriever can adopt traditional methods such as bm25. Load the preprocessed tables into the ElasticSearch index library, set the built-in similarity algorithm of ElasticSearch to bm25, set the algorithm parameters as k = 1.2, b = 0.75, and specify the number of query results to be N, then its implementation can be used for the preliminary retrieval of tables. The bm25 method does not require model training.

[0028] The retriever can also use the deep learning dense retriever DPR. DPR is a dual encoder based on Bert. The two encoders respectively encode the question q nl and the table T i into vectors v q and The vector length is the same as the encoding length of BERT, which is d = 728. After fine-tuning, DPR will convert the question and the table into vector forms and ensure that the inner product of semantically related question-table pairs is larger than that of other unrelated pairs, and can better capture semantic similarity, while traditional methods such as bm25 are more sensitive to keywords.

[0029] During the implementation process, train a DPR retriever based on the tiled table corpus. In the training process, select 1 non-gold table with the highest bm25 score as a hard example, set batch_size = 32 and use the In-Batch negative training method. After training, use one of the dual encoders of DPR: the table encoder to perform similar processing on all tables in the corpus to obtain the encoded table vectors, and load them into the dense vector retrieval library FASSI. During the preliminary retrieval, use one of the dual encoders of DPR: the question encoder to encode the question into a vector and search in the FASSI library.

[0030] For the target question sentence, the retriever returns the top N tables with relatively high original similarity as the table pool.

[0031] Table reordering. In this process, a Text-to-SQL deep learning model such as HydraNet is introduced as the semantic parser of the question. Before performing table reordering, a HydraNet model is trained based on the WikiSQL dataset with batch_size = 64 and learn_rate = 6*10 -6 , and the base model uses Roberta. During the table reordering process, for each table in the table pool, the question and table schema information are first input into the HydraNet model to parse the SQL statement corresponding to the question. Next, this SQL statement is executed on the table, and it is counted whether an error occurs during the execution.

[0032] The results obtained by executing on the candidate tables are converted into additional parameters according to different types: res EG , when no execution error occurs, res EG = 1, when an execution error occurs, res EG = 0. In reality, it is often expected that the result is not empty when asking a question, so an empty result can be regarded as a special execution error.

[0033] Calculate the new similarity score based on the following function,

[0034] sim withEG = (1 - α)·sim origin / maxsi origin +α·res EG

[0035] where sim origin is the original similarity score in the retrieval stage, maxsim origin is the maximum original similarity score of all tables in the retrieval stage, and sim withEG is the new similarity score; the coefficient α is used to measure the importance of the execution result, and the original retrieval relevance score and the execution result are linearly summed to obtain a new similarity estimation score. Different values of the coefficient α are taken according to the actual situation of the table corpus. In this implementation process, α = 0.9 is taken.

[0036] Reorder the N tables in the table pool according to the new similarity score sim withEG from largest to smallest, and select the top-k tables with the highest scores to enter the answer extraction stage;

[0037] Answer extraction stage. Based on the input question and the obtained top-k tables, a deep learning model for answer extraction such as cell classification or generation is used to obtain the answer.

Claims

1. A method for enhancing table retrieval in open-domain question answering, characterized in that, Including: A table retrieval stage and an answer extraction stage; In the table retrieval stage, based on the input question sentence and the given table corpus, the top k tables selected through retrieval and sorted by relevance are obtained. The specific method is as follows: the table is flattened row by row to form the table content in text form; the original similarity of each table in the table corpus is calculated by the retriever; the tables are sorted according to the loss from large to small of the original similarity, the first N tables are selected to form a table pool, and the top k tables are selected from the table pool; Among them, the specific method for selecting the top k tables from the table pool is as follows: for each table in the table pool, first input the question sentence and the table schema information into the Text-to-SQL deep learning model to obtain the corresponding SQL statement, and then execute the obtained SQL semantics on the table to determine whether an execution error occurs. If no execution error occurs, let res EG = 1. If an execution error occurs, let res EG = 0; calculate the new similarity score: sim withEG = (1 - α)·sim origin / maxsim origin + α·res EG where sim origin is the original similarity score in the retrieval stage, maxsim origin is the maximum original similarity score among all tables in the retrieval stage, and sim withEG is the new similarity score; Re - order the N tables in the table pool according to the new similarity score sim withEG in descending order, and select the top - k tables with the highest scores; In the answer extraction stage, an answer is obtained based on the input question sentence and the obtained top k tables.

Images