Table content query analysis method and system and electronic equipment

Abstract

The invention provides a table content query analysis method and system and electronic equipment, and belongs to the technical field of data processing. The method comprises the following steps: firstly, constructing a table data set, then converting each table in the table data set into a key value pair list form, and storing the key value pair list corresponding to each table as a knowledge slice into a knowledge base so as to construct and obtain a table knowledge base. The knowledge base is constructed by converting the table data structure into the form of the key value pair list, and the structure and content information of the table can be well reserved, so that the accuracy of the table retrieval stage is improved. Then, cells related to the query problem are screened through a pre-trained response cell prediction model, prompt content is obtained according to the query problem and cell construction and input into a large model for processing, and a tedious table does not need to be input, so that the analysis efficiency and accuracy of the large model can be improved; and the situation that the ultra-long table is cut off when being input into a large model can be avoided.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a table content query and analysis method, system and electronic device. Background Art

[0002] Tables are used to store and display structured / semi-structured data and are widely found in technical documents, industry reports, scientific literature, and other document materials. With the explosive growth in the number of document materials, the traditional manual method of looking up tables requires a lot of labor costs and time costs, which seriously limits the management and analysis of table data.

[0003] In the related art, the text content or part of the key text content in the table is stored in the knowledge base to realize the query and analysis of the table content based on the big model. However, this method stores the text content or part of the key text content in the table in the knowledge base, which cannot retain the structure of the table itself, resulting in low accuracy of table retrieval (screening). At the same time, in the table content analysis stage, directly using the big model to analyze the query questions and tables has low efficiency and accuracy, and there may be a situation where the overlong table input is truncated by the big model. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a table content query and analysis method, system and electronic device, aiming to improve the accuracy of table screening and improve the efficiency and accuracy of table analysis of large models.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a table content query analysis method, the method comprising:

[0006] Build a tabular dataset;

[0007] Convert each table in the table data set into a key-value pair list, and store the key-value pair list corresponding to each table as a knowledge slice in the knowledge base to construct a table knowledge base;

[0008] Obtaining a query question for table content analysis, and performing a similar vector search in the table knowledge base to obtain a candidate table related to the query question;

[0009] Invoking a pre-trained answer cell prediction model to process the query question and the candidate table, so as to filter out cells related to the query question from the candidate table;

[0010] The prompt content is obtained according to the query question and the cell structure, and the large model is called to obtain the prompt content and process it to obtain the table analysis summary content.

[0011] In one embodiment of the present application, converting each table in the table data set into a key-value pair list, and storing the key-value pair list corresponding to each table as a knowledge slice in the knowledge base to construct a table knowledge base includes:

[0012] Store the contents of each row of the table in the table data set as a list of key-value pairs, where the key is the header field corresponding to the cell and the value is the content corresponding to the cell;

[0013] The key-value pair list corresponding to each table is taken as a knowledge slice, and each knowledge slice is converted into a vector representation and stored in the vector database to construct a table knowledge base.

[0014] In one embodiment of the present application, the obtaining of a query question for table content analysis and performing a similarity vector search in the table knowledge base to obtain a candidate table related to the query question includes:

[0015] Obtaining a query question for table content analysis, and converting the query question into a query vector;

[0016] Calculating the similarity between all tables in the table knowledge base and the query vector;

[0017] All tables are sorted according to the similarity to select the top N tables with the highest similarity as candidate tables, where N is a positive integer.

[0018] In one embodiment of the present application, the calling of a pre-trained answer cell prediction model to process the query question and the candidate table to filter out cells related to the query question from the candidate table includes:

[0019] Convert the candidate table into a JS key-value pair data format;

[0020] Calling a pre-trained answer cell prediction model to process the query question and the candidate table in the JS key-value pair data format to output the answer probability of each row and the answer probability of each column in the candidate table;

[0021] According to the answer probability of each row and the answer probability of each column in the candidate table, cells related to the query question are screened out.

[0022] In one embodiment of the present application, the step of screening out cells related to the query question according to the answer probability of each row and the answer probability of each column in the candidate table includes:

[0023] Filter the top K target rows and target columns with the highest response probabilities in the candidate table, where K is a positive integer;

[0024] A cell in the candidate table that is located in both the target row and the target column is selected as a cell related to the query question.

[0025] In one embodiment of the present application, constructing a table data set includes:

[0026] Acquire a document data set, wherein each document data in the document data set contains a table;

[0027] A table in each document data in the document data set is extracted to construct a table data set.

[0028] In one embodiment of the present application, pre-training the response cell prediction model includes:

[0029] According to the table data set, synthesizing question-answer data through a large model to construct a question-answer data set, wherein the question-answer data includes a query question and an answer cell corresponding to the query question;

[0030] Construct an answer cell prediction model, and obtain the question and answer data set to train the answer cell prediction model, wherein a table in the JS key-value pair data format and the query question in the question and answer data are used as input, and the answer probability of the answer cell in the question and answer data is used as output, and the answer probability of the row where the answer cell is located and the answer probability of the column where the answer cell is located are both set to 1.

[0031] In one embodiment of the present application, synthesizing question-answer data by a large model based on the table data set to construct a question-answer data set includes:

[0032] Specifying some cells in the table of the table data set as response cells;

[0033] Generate a query question corresponding to the answer cell using the large model to construct question-answer data corresponding to the query question and the answer cell;

[0034] Collect all question and answer data to get the question and answer dataset.

[0035] To achieve the above purpose, a second aspect of an embodiment of the present application proposes a table content query and analysis system, including:

[0036] A first construction unit is used to construct a table data set;

[0037] A second construction unit is used to convert each table in the table data set into a key-value pair list, and store the key-value pair list corresponding to each table as a knowledge slice into the knowledge base to construct a table knowledge base;

[0038] A retrieval unit, used to obtain a query question for table content analysis, and perform a similar vector search in the table knowledge base to obtain a candidate table related to the query question;

[0039] A first processing unit, configured to call a pre-trained answer cell prediction model to process the query question and the candidate table, so as to filter out cells related to the query question from the candidate table;

[0040] The second processing unit is used to obtain prompt content according to the query question and the cell structure, and call the large model to obtain the prompt content and process it to obtain the table analysis summary content.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in any embodiment of the present application when executing the computer program.

[0042] In the technical solution provided in the embodiment of the present application, a table data set is first constructed, and then each table in the table data set is converted into a key-value pair list, and the key-value pair list corresponding to each table is stored in the knowledge base as a knowledge slice to construct a table knowledge base. By converting the table data structure into a key-value pair list to construct a knowledge base, the structure and content information of the table itself can be better retained, thereby improving the accuracy of the table retrieval stage. Next, based on the query question, a similar vector search is performed in the table knowledge base to obtain a candidate table related to the query question; then the pre-trained answer cell prediction model is called to process the query question and the candidate table, and the cells related to the query question can be screened out from the candidate table. Then, the prompt content is obtained based on the query question and the cell structure, and the large model is called to obtain the prompt content and process it to obtain the table analysis summary content. That is, first, the candidate tables related to the query question are retrieved through the table knowledge base to reduce the number of candidate tables, and then the cells related to the query question are screened through the pre-trained answer cell prediction model. Therefore, when using the large model to analyze the table content, the cells related to the query question can be directly entered without entering a lengthy table, thereby improving the analysis efficiency and accuracy of the large model, and avoiding the situation where an overly long table is entered into the large model and is truncated. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a table content query and analysis method provided by an embodiment of the present application;

[0044] Figure 2 is a flowchart of the steps of constructing a table data set provided by an embodiment of the present application;

[0045] Figure 3 It is a flowchart of the steps of converting each table in a table data set into a key-value pair list, and storing the key-value pair list corresponding to each table as a knowledge slice into a knowledge base to construct a table knowledge base, provided by an embodiment of the present application;

[0046] Figure 4 A flowchart of the steps of obtaining a query question for table content analysis and performing a similar vector search in a table knowledge base to obtain a candidate table related to the query question is provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of the steps of calling a pre-trained answer cell prediction model to process a query question and a candidate table to filter out cells related to the query question from the candidate table, provided in an embodiment of the present application;

[0048] Figure 6 is a flowchart of the steps of pre-training a response cell prediction model provided in an embodiment of the present application;

[0049] Figure 7 It is a flowchart of steps for synthesizing question and answer data through a large model based on a table data set to construct a question and answer data set according to an embodiment of the present application;

[0050] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] At present, large language models (LLMs) technology has been relatively mature in processing pure text information. With the help of natural language processing technology, large models can directly use text data for training and reasoning. However, in the industrial field, knowledge files contain not only text information, but also a large number of complex data forms such as tables. The existing large model domain knowledge enhancement technology still has many challenges and shortcomings in processing complex industrial tables.

[0055] Unlike text information that can be directly used by large models, table information contains not only data content but also complex table structures. Therefore, how to retrieve table information in large model domain knowledge enhancement is a major difficulty in the current industry. Tables usually have multi-level headers and a large number of merged cells, and there are complex associations between cells. When querying table content, it is usually necessary to query different levels of headers multiple times to locate the table value to be queried, which is not conducive to the extraction and use of table information.

[0056] In the related art, the text content or part of the key text content in the table is stored in the knowledge base to realize the query and analysis of the table content based on the big model. However, this method stores the text content or part of the key text content in the table in the knowledge base, which cannot retain the structure of the table itself, resulting in low accuracy of table retrieval (screening). At the same time, in the table content analysis stage, directly using the big model to analyze the query questions and tables has low efficiency and accuracy, and there may be a situation where the overlong table input is truncated by the big model.

[0057] Based on this, an embodiment of the present application proposes a table content query and analysis method, which aims to improve the accuracy of table screening and improve the efficiency and accuracy of table analysis of large models.

[0058] Reference Figure 1 , Figure 1 It is a flowchart of a table content query and analysis method provided in an embodiment of the present application, including but not limited to steps S110 to S150.

[0059] Step S110, constructing a table data set.

[0060] In the embodiment of the present application, a large number of table data sets are constructed to obtain a table data set, wherein the table data can be table data directly crawled by a crawling tool, or can be table data extracted from a large number of document data containing tables.

[0061] Reference Figure 2 , Figure 2 It is a flowchart of the steps of constructing a table data set provided by an embodiment of the present application, including but not limited to steps S210 to S220.

[0062] Step S210, obtaining a document data set, wherein each document data in the document data set contains a table;

[0063] Step S220 , extracting a table from each document data in the document data set to construct a table data set.

[0064] In the embodiment of the present application, a large-scale document data set can be formed by a document data source, and then a document table extraction method is used to extract the table from the document data in the document data set to obtain the table content information. Then the extracted tables are collected to construct a table data set. Specifically, if the document data is a word type document, the python-docx toolkit can be used to read the table content; if the document data is a pdf type document, the PDFP l umber tool can be used to extract the table content.

[0065] Step S120, converting each table in the table data set into a key-value pair list, and storing the key-value pair list corresponding to each table as a knowledge slice into the knowledge base to construct a table knowledge base.

[0066] In the embodiment of the present application, after constructing the table data set, each table in the table data set can be converted into a key-value pair list, and the key-value pair list corresponding to each table is stored in the knowledge base as a knowledge slice to construct a table knowledge base. Thus, the constructed table knowledge base can better retain the structure and content information of the table itself, thereby improving the accuracy of subsequent table retrieval and screening.

[0067] Reference Figure 3 , Figure 3 An embodiment of the present application provides a step flow chart for converting each table in a table data set into a key-value pair list, and storing the key-value pair list corresponding to each table as a knowledge slice into a knowledge base to construct a table knowledge base, including but not limited to steps S310 to S320.

[0068] Step S310, storing the content of each row of the table in the table data set as a key-value pair list, wherein the key is the header field corresponding to the cell, and the value is the content corresponding to the cell;

[0069] Step S320, taking the key-value pair list corresponding to each table as a knowledge slice, and converting each knowledge slice into a vector representation and storing it in a vector database to construct a table knowledge base.

[0070] In an embodiment of the present application, each row of the table in the table data set is first stored as a key-value pair list, wherein the key is the header field corresponding to the cell, and the value is the content corresponding to the cell. Exemplarily, the table data set includes the table shown in Table 1 below, and each row of the table shown in Table 1 is converted into a key-value pair list, which is: [{"field 1": n11, "field 2": n12, "field 3": n13}, {"field 1": n21, "field 2": n22, "field 3": n23}, ...]. The key-value pair list corresponding to each table is then used as a knowledge slice, and each knowledge slice can be converted into a vector representation using a Chinese pre-trained model (bert-base-ch i nese) and stored in the Faiss vector database, thereby constructing a table knowledge base.

[0071] Table 1

[0072] Field 1 Field 2 Field 3 n11 n12 n13 n21 n22 n23 … … …

[0073] In an embodiment of the present application, a table knowledge base is constructed by converting a table data structure into a key-value pair list and storing a table as a knowledge slice into a vector knowledge base. This can better preserve the structure and content information of the table itself and improve the accuracy of the table retrieval stage.

[0074] Step S130 , obtaining a query question for table content analysis, and performing a similar vector search in a table knowledge base to obtain candidate tables related to the query question.

[0075] In an embodiment of the present application, after constructing the table knowledge base, query questions for the table content (such as change analysis, trend summary, etc.) are first obtained, and then a similarity vector search is performed in the table knowledge base to obtain at least one candidate table most relevant to the query question, such as the top 5 candidate tables most relevant to the query question.

[0076] Reference Figure 4 , Figure 4 It is a flowchart of the steps of obtaining a query question for table content analysis and performing similarity vector retrieval in a table knowledge base to obtain candidate tables related to the query question, including but not limited to steps S410 to S430, provided by an embodiment of the present application.

[0077] Step S410, obtaining a query question for table content analysis, and converting the query question into a query vector;

[0078] Step S420, calculating the similarity between all tables in the table knowledge base and the query vector;

[0079] Step S430 , sorting all tables according to the similarity, so as to select the first N tables with the highest similarity as candidate tables, where N is a positive integer.

[0080] In the embodiment of the present application, a query question for the table content (such as change analysis, trend summary, etc.) can be obtained first, and then the query question can be converted into a query vector using a Chinese pre-trained model (bert-base-chinese), and then the similarity between all tables in the table knowledge base and the query vector is calculated. Then all tables are sorted according to the similarity, so that the top N tables with the highest similarity can be selected as candidate tables, where N is a positive integer.

[0081] In the embodiment of the present application, by calculating the similarity between all tables in the table knowledge base and the query vector, multiple candidate tables that are most relevant to the query question can be screened out based on the similarity. That is, the user's query intent and the table are semantically matched to determine the corresponding relationship between the query intent and the table content. Then, based on the semantic matching results, multiple tables that are most relevant to the query intent are selected as candidate tables, which can effectively reduce the number of tables to be analyzed, thereby improving the efficiency and accuracy of the subsequent large model for table content analysis.

[0082] Step S140, calling a pre-trained answer cell prediction model to process the query question and the candidate table to filter out cells related to the query question from the candidate table.

[0083] In an embodiment of the present application, after selecting multiple candidate tables that are most relevant to the query question from the table knowledge base, a pre-trained answer cell prediction model can be further called to process the query question and the candidate tables to select cells related to the query question from the candidate tables. Specifically, for each candidate table, the query question and the candidate table are used as input, and then the pre-trained answer cell prediction model is used for processing, and the answer probability corresponding to each row and each column in the candidate table can be output, so that at least one cell in the candidate table that is most relevant to the query question can be selected.

[0084] Exemplarily, three candidate tables most relevant to the query question are screened out from the table knowledge base, which are the first candidate table, the second candidate table and the third candidate table. First, the first candidate table and the query question are input into the pre-trained answer cell prediction model for processing. The answer cell prediction model can output the answer probability corresponding to each row in the first candidate table and the answer probability corresponding to each column, so that the cells (such as 3) most relevant to the query question in the first candidate table can be determined. Similarly, the second candidate table and the query question are input into the pre-trained answer cell prediction model for processing. The answer cell prediction model can output the answer probability corresponding to each row in the second candidate table and the answer probability corresponding to each column, so that the cells (such as 4) most relevant to the query question in the second candidate table can be determined. Then the third candidate table and the query question are input into the pre-trained answer cell prediction model for processing. The answer cell prediction model can output the answer probability corresponding to each row in the third candidate table and the answer probability corresponding to each column, so that the cells (such as 3) most relevant to the query question in the third candidate table can be determined. Thus, 3+4+3=10 cells related to the query question can be screened out from all candidate tables.

[0085] Reference Figure 5 , Figure 5 It is a flowchart of the steps of calling a pre-trained answer cell prediction model to process query questions and candidate tables to filter out cells related to the query questions from the candidate tables, including but not limited to steps S510 to S530, provided in an embodiment of the present application.

[0086] Step S510, converting the candidate table into a JS key-value pair data format;

[0087] Step S520, calling a pre-trained answer cell prediction model to process the query question and the candidate table in the JS key-value pair data format to output the answer probability of each row and each column in the candidate table;

[0088] Step S530 , based on the answer probability of each row and the answer probability of each column in the candidate table, cells related to the query question are screened out.

[0089] In an embodiment of the present application, each candidate table is first converted into a JS key-value pair data format (i.e., json format). Taking the table shown in Table 1 as an example, Table 1 is converted into a JS key-value pair data format as follows: {"header": ["field 1", "field 2", "field 3"], "table row": [[n11, n12, n13], [n21, n22, n23], ...]}. Then the query question and the candidate table in the JS key-value pair data format are input into a pre-trained response cell prediction model, so that the pre-trained response cell prediction model can output the response probability of each row and the response probability of each column in the candidate table. Then, according to the response probability of each row and the response probability of each column in the candidate table, the cells related to the query question can be screened out. Specifically, the top K target rows and target columns with the highest response probability in the candidate table can be screened, where K is a positive integer; then the cells in the candidate table that are simultaneously located in the target row and the target column are selected as cells related to the query question.

[0090] Exemplarily, the top three target rows and target columns with the highest probability of answering in the candidate table are screened out through the pre-trained answer cell prediction model, wherein the top three target rows with the highest probability of answering are row 4, row 1 and row 5, respectively, and the top three target columns with the highest probability of answering are column 2, column 1 and column 3, respectively. Thus, the cells related to the query question include: row 1, column 1, row 1, column 2, row 1, column 3, row 4, column 1, row 4, column 2, row 4, column 3, row 5, column 1, row 5, column 2, and row 5, column 3. After sorting, it is shown in Table 2 below:

[0091] Table 2

[0092] Field 1 Field 2 Field 3 n11 n12 n13 n41 n42 n43 n51 n52 n53

[0093] In an embodiment of the present application, a pre-trained response cell prediction model can be used to screen out multiple cells most relevant to the query question from the candidate table, which can effectively reduce the size of the table to be analyzed and avoid truncation of the large model when inputting an overly long table.

[0094] Reference Figure 6 , Figure 6 It is a flowchart of the steps of pre-training a response cell prediction model provided in an embodiment of the present application, including but not limited to steps S610 to S620.

[0095] Step S610, based on the table data set, question and answer data is synthesized through a large model to construct a question and answer data set, where the question and answer data includes query questions and answer cells corresponding to the query questions.

[0096] In an embodiment of the present application, in order to obtain cells related to the query question in the candidate table, it is necessary to pre-train the answer cell prediction model, and then use the trained answer cell prediction model to screen out the cells related to the query question in the candidate table. Among them, the process of pre-training the answer cell prediction model is: first, each table in the table data set is standardized, that is, each table is uniformly converted into a JS key-value pair data format. Taking the table shown in Table 1 as an example, Table 1 is converted into a JS key-value pair data format: {"header": ["field 1", "field 2", "field 3"], "table row": [[n11,n12,n13], [n21,n22,n23], ...]}. Then, the question and answer data is synthesized based on the table data set through a large model, wherein the question and answer data includes query questions and answer cells corresponding to the query questions. Then all the synthesized question and answer data are collected to obtain a question and answer data set.

[0097] Reference Figure 7 , Figure 7 It is a flowchart of the steps provided by an embodiment of the present application for synthesizing question and answer data through a large model based on a table data set to construct a question and answer data set, including but not limited to steps S710 to S730.

[0098] Step S710, specifying some cells in a table of a table data set as response cells;

[0099] Step S720, using the large model to generate query questions corresponding to the answer cells, so as to construct question-answer data corresponding to the query questions and the answer cells;

[0100] Step S730, collect all the question and answer data to obtain a question and answer data set.

[0101] In the embodiment of the present application, a portion of the table can be randomly extracted from the table data set, and a cell in the table can be randomly designated as the answer to the table query (i.e., as the answer cell). Then, the prompt engineering method is used to use the large model to generate corresponding query questions for each answer cell, and then the query question and the corresponding answer cell form a question and answer data. All the question and answer data are collected to construct a question and answer data set.

[0102] Step S620, construct an answer cell prediction model, and obtain a question and answer data set to train the answer cell prediction model, wherein a table in the JS key-value pair data format and the query question in the question and answer data are used as input, and the answer probability of the answer cell in the question and answer data is used as output, and the answer probability of the row where the answer cell is located and the answer probability of the column where the answer cell is located are both set to 1.

[0103] In an embodiment of the present application, a response cell prediction model is constructed, the input of which is a query question and a table content in json format, and the output is the response probability of each row and the response probability of each column in the table. Among them, the cell formed by the row and column with the highest response probability is the predicted response cell. Taking the table shown in Table 1 as an example, the input is a query question and {"header": ["field 1", "field 2", "field 3"], "table row": [[n11, n12, n13], [n21, n22, n23], ...]}, and the output is the response probability Prow = [p1, p2, ...] of each row in Table 1 and the response probability Pco l umn = [p1', p2', p3']. The question and answer data set is used as training data to train the response cell prediction model. Among them, for each question-and-answer data, the query question in the question-and-answer data and the table content in json format are used as the input of the training sample, and the answer probability of the row and column where the answer cell is located is specified as 1, and the other rows and columns are specified as 0 as the output of the training sample. A set of inputs and outputs constitutes a training sample, and a response cell prediction model is obtained after training. Taking the table shown in Table 1 as an example, assuming that the response cell is n13, the output of the constructed training sample is Prow = [1,0,…] and Pcolumn = [0,0,1]. In this way, the response cell prediction model can be trained.

[0104] In an embodiment of the present application, a response data set is constructed to train a response cell prediction model, so that cells related to the query question in the candidate table can be screened out through the trained response cell prediction model, which can effectively reduce the size of the table to be analyzed, thereby improving the efficiency and accuracy of subsequent large models in performing table content analysis.

[0105] Step S150, obtaining prompt content according to the query question and the cell structure, and calling the large model to obtain the prompt content and process it to obtain the table analysis summary content.

[0106] In the embodiment of the present application, after the cells related to the query question in the candidate table are screened out, the query question and the screened cells related to the query question can be constructed into prompt content, and input into the big model, so that the big model can generate the table analysis summary content. Specifically, based on the cells related to the query question and the user's query question, the prompt content that guides the big model to analyze the table can be constructed, and then the prompt content can be input into the big model (such as ChatGLM, China-Vicuna, Qwen, etc.), so that the big model can generate the corresponding table analysis summary content.

[0107] In an embodiment of the present application, a table knowledge base is constructed by converting a table data structure into a list of key-value pairs and storing a table as a knowledge slice in the knowledge base. This can better preserve the structure and content information of the table itself and improve the accuracy of the table retrieval stage. In the table content query stage, a pre-trained response cell prediction model is used to filter cells related to the query question, which can effectively reduce the size of the table to be analyzed, improve the efficiency and accuracy of large model analysis, and avoid the truncation of the large model when inputting an over-long table.

[0108] The present application also provides a table content query and analysis system, including:

[0109] A first construction unit is used to construct a table data set;

[0110] The second construction unit is used to convert each table in the table data set into a key-value pair list, and store the key-value pair list corresponding to each table as a knowledge slice into the knowledge base to construct a table knowledge base;

[0111] A retrieval unit is used to obtain a query question for table content analysis and perform similarity vector retrieval in a table knowledge base to obtain candidate tables related to the query question;

[0112] A first processing unit is used to call a pre-trained answer cell prediction model to process the query question and the candidate table to filter out cells related to the query question from the candidate table;

[0113] The second processing unit is used to obtain prompt content according to the query question and the cell structure, and call the large model to obtain the prompt content and process it to obtain the table analysis summary content.

[0114] In an embodiment of the present application, a table knowledge base is constructed by converting a table data structure into a list of key-value pairs and storing a table as a knowledge slice in the knowledge base. This can better preserve the structure and content information of the table itself and improve the accuracy of the table retrieval stage. In the table content query stage, a pre-trained response cell prediction model is used to filter cells related to the query question, which can effectively reduce the size of the table to be analyzed, improve the efficiency and accuracy of large model analysis, and avoid the truncation of the large model when inputting an over-long table.

[0115] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above table content query and analysis method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0116] See also Figure 8 , Figure 8 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0117] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0118] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the table content query and analysis method of the embodiment of this application;

[0119] Input / output interface 803, used to implement information input and output;

[0120] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0121] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0122] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0123] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0124] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0125] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0127] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0128] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0133] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and substance of the present invention should be within the scope of the rights of the present invention.

Claims

1. A table content query and analysis method, characterized in that: The method comprises: Build a tabular dataset; Convert each table in the table data set into a key-value pair list, and store the key-value pair list corresponding to each table as a knowledge slice in the knowledge base to construct a table knowledge base; Obtaining a query question for table content analysis, and performing a similar vector search in the table knowledge base to obtain a candidate table related to the query question; Calling a pre-trained answer cell prediction model to process the query question and the candidate table to filter out cells related to the query question from the candidate table; The prompt content is obtained according to the query question and the cell structure, and the large model is called to obtain the prompt content and process it to obtain the table analysis summary content.

2. The method according to claim 1, characterized in that The step of converting each table in the table data set into a key-value pair list, and storing the key-value pair list corresponding to each table as a knowledge slice into the knowledge base to construct a table knowledge base includes: Store the contents of each row of the table in the table data set as a key-value pair list, wherein the key is the header field corresponding to the cell, and the value is the content corresponding to the cell; The key-value pair list corresponding to each table is taken as a knowledge slice, and each knowledge slice is converted into a vector representation and stored in the vector database to construct a table knowledge base.

3. The method according to claim 1, characterized in that The obtaining of a query question for table content analysis and performing a similar vector search in the table knowledge base to obtain a candidate table related to the query question includes: Obtaining a query question for table content analysis, and converting the query question into a query vector; Calculating the similarity between all tables in the table knowledge base and the query vector; All tables are sorted according to the similarity to select the top N tables with the highest similarity as candidate tables, where N is a positive integer.

4. The method according to claim 1, characterized in that: The calling of a pre-trained answer cell prediction model to process the query question and the candidate table to filter out cells related to the query question from the candidate table includes: Convert the candidate table into a JS key-value pair data format; Calling a pre-trained answer cell prediction model to process the query question and the candidate table in the JS key-value pair data format to output the answer probability of each row and the answer probability of each column in the candidate table; According to the answer probability of each row and the answer probability of each column in the candidate table, cells related to the query question are screened out.

5. The method according to claim 4, characterized in that The step of screening out cells related to the query question according to the answer probability of each row and the answer probability of each column in the candidate table includes: Filter the top K target rows and target columns with the highest response probabilities in the candidate table, where K is a positive integer; A cell in the candidate table that is located in both the target row and the target column is selected as a cell related to the query question.

6. The method according to claim 1, characterized in that The constructing of the table data set comprises: Acquire a document data set, wherein each document data in the document data set contains a table; A table in each document data in the document data set is extracted to construct a table data set.

7. The method according to claim 1, characterized in that Pre-trained answer cell prediction models include: According to the table data set, synthesizing question-answer data through a large model to construct a question-answer data set, wherein the question-answer data includes a query question and an answer cell corresponding to the query question; Construct an answer cell prediction model, and obtain the question and answer data set to train the answer cell prediction model, wherein a table in JS key-value pair data format and the query question in the question and answer data are used as input, and the answer probability of the answer cell in the question and answer data is used as output, and the answer probability of the row where the answer cell is located and the answer probability of the column where the answer cell is located are both set to 1.

8. The method according to claim 1, characterized in that The step of synthesizing question-answer data by using a large model according to the table data set to construct a question-answer data set includes: Specifying some cells in the table of the table data set as response cells; Generate a query question corresponding to the answer cell using the large model to construct question-answer data corresponding to the query question and the answer cell; Collect all question and answer data to get the question and answer dataset.

9. A table content query and analysis system, characterized in that: include: A first construction unit is used to construct a table data set; A second construction unit is used to convert each table in the table data set into a key-value pair list, and store the key-value pair list corresponding to each table as a knowledge slice into the knowledge base to construct a table knowledge base; A retrieval unit, used to obtain a query question for table content analysis, and perform a similar vector search in the table knowledge base to obtain a candidate table related to the query question; A first processing unit, configured to call a pre-trained answer cell prediction model to process the query question and the candidate table, so as to filter out cells related to the query question from the candidate table; The second processing unit is used to obtain prompt content according to the query question and the cell structure, and call the large model to obtain the prompt content and process it to obtain the table analysis summary content.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

Images