Question and answer method and system for table knowledge extraction in water taking permission field
By reconstructing the structure of the table in the water withdrawal license field and extracting knowledge of large language models, combining with prompt engineering, a knowledge graph database was built, and the problems of difficulty in retrieving table content and low flexibility were solved, and efficient use of table content and quick question-and-answer questions and answers were achieved. .
Patent Information
- Application Number
- CN202411140592.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-06
AI Technical Summary
In the field of water withdrawal licensing, the prior art is difficult to effectively process multiple types of tables, which leads to difficulty in retrieving table contents, consumes a lot of manpower and material resources, and has low flexibility in automation solutions.
By obtaining the water withdrawal permission field tables, it is transformed into text format, and using a large language model and prompt engineering to extract structured fact triple data to build a knowledge graph database to achieve efficient retrieval of user questions.
The utilization rate and analysis efficiency of table content in the field of water withdrawal permits have been improved, the problems of difficulty in retrieving table content and low flexibility have been solved, and the rapid question and answer have been achieved.
Smart Images

Figure CN119088921A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of knowledge base question answering, and in particular to a question answering method and system for extracting table knowledge in the field of water abstraction permit. Background Art
[0002] In the field of water abstraction permit documents, tables are an important information carrier that carries key data and analysis results. The tables contain key information such as precipitation and water abstraction, and are an important basis for water abstraction permit decision-making.
[0003] When professionals in the field of water permits manage water permits, forms account for a large proportion of the entire text, and it usually takes a lot of manpower and material resources to obtain useful information from the water permit document data forms. Therefore, it is necessary to realize intelligent question and answer of water permit data forms to provide convenience and support for water permit decision-making.
[0004] In the field of water abstraction permits, there are many types of tables in documents due to different regulatory requirements of different regions and organizations and different personal language habits. These tables can be classified into vertical tables, horizontal tables, hierarchical tables and complex tables according to the different arrangements of the table headers. A vertical table specifically refers to the first row as the table header, and the subsequent rows arrange the data vertically; the characteristic of a horizontal table is that its first column acts as a row header, and the data is spread out horizontally; a hierarchical table presents a more complex structure, its header is hierarchical, and the table may include merged cells, which requires considering both horizontal and vertical header information when interpreting the data; a complex table goes beyond the above three types, and its header may be located anywhere in the table.
[0005] In addition, these tables often contain a large number of professional terms and specific geographical terms, such as "sediment transport volume", "functional area water quality compliance rate" and "Beifushantang Changshu buffer zone", which greatly increases the difficulty of retrieving and using information directly from the tables. Taking the "China Water Resources Bulletin" as an example, the tables in this report occupy a large amount of space and reflect various aspects of water resources. In the entire document, there are 13 tables in total. These tables occupy a large amount of space in the document and contain a lot of content. All four types of tables appear multiple times, among which the more complex hierarchical tables and complex tables appear a total of 5 times. At the same time, because there are many professional terms in the tables, such as water storage dynamics and groundwater dynamics, this brings additional challenges to understanding the content of the tables.
[0006] In view of the above difficulties, the existing methods for realizing intelligent question and answer of water permit data forms have many defects. For example, the traditional manual input is inefficient, the template matching has a small scope of application, and it is difficult to cope with the challenges of diverse table types. Some automated solutions such as NL2SQL technology attempt to provide table question and answer services, but they usually lack flexibility. Summary of the invention
[0007] The present invention provides a question-answering method and system for extracting knowledge from tables in the field of water abstraction permits, which are used to solve the problems of difficulty in retrieving table contents of water abstraction permit information tables in the prior art, large consumption of manpower and material resources, and low flexibility.
[0008] In a first aspect, the present invention provides a question-answering method for extracting table knowledge in the field of water abstraction permit, comprising:
[0009] Obtain the water abstraction permit field table to reconstruct the table structure and obtain a text format table;
[0010] According to the text format table, the prompt engineering method is used to extract knowledge based on the large language model to obtain structured fact triple data;
[0011] Constructing a knowledge graph database based on the fact triple data;
[0012] Obtaining a user search question, and obtaining a target answer set based on the user search question and the knowledge graph database;
[0013] The target answer set is input into a large language model to output a retrieval result.
[0014] According to the question-answering method for extracting knowledge from water abstraction permit field tables, the step of obtaining a water abstraction permit field table and reconstructing the table structure to obtain a text format table includes:
[0015] Extract the water abstraction permit field table from the document;
[0016] Reconstructing the merged cells in the row header and column header of the water withdrawal permit field table to obtain the cell text content; the reconstruction includes splitting and / or merging cells;
[0017] The text content of the cell is converted to obtain a text format table.
[0018] According to the question-answering method for extracting knowledge from water abstraction permit domain tables, the step of extracting water abstraction permit domain tables from documents includes:
[0019] Locate the data body of the water abstraction permit field table in the document and obtain the table index;
[0020] According to the table index, obtain the table title;
[0021] The data body and the corresponding table title are combined to obtain a water withdrawal permit field table.
[0022] According to the question-answering method for extracting table knowledge in the field of water abstraction permit, the step of obtaining the table title according to the table index includes:
[0023] Retrieving adjacent paragraphs of the data body according to the table index;
[0024] According to the adjacent paragraphs and the preset table title set, the adjacent paragraphs are matched using regular expressions to generate matching results;
[0025] According to the matching results, get the table title.
[0026] According to the question-answering method for extracting knowledge from tables in the field of water abstraction permits, the step of extracting knowledge based on a large language model using a prompt engineering method according to a text format table to obtain structured fact triple data includes:
[0027] Build a prompt engineering template based on the text format table;
[0028] Based on the large language model, knowledge is extracted from the text format table according to the prompt engineering template to obtain structured fact triple data.
[0029] According to the question-answering method for extracting table knowledge in the field of water abstraction permit, the step of obtaining a target answer set based on the user search question and the knowledge graph database includes:
[0030] According to the user search question, obtain question entities and question attributes;
[0031] Generate candidate triples by using fuzzy matching based on the problem entity and the knowledge graph database, and the candidate triples are used to determine the fact triple data related to the problem entity;
[0032] Vector matching is adopted according to question attributes and candidate triples to generate a target answer set; the target answer set is used to determine fact triple data related to the question attributes in the candidate triples.
[0033] In a second aspect, the present invention further provides a question-answering system for extracting table knowledge in the field of water abstraction permits, comprising:
[0034] Table extraction module, used to obtain water abstraction permit field tables and reconstruct table structures to obtain text format tables;
[0035] The knowledge extraction module is used to extract knowledge based on the text format table using the prompt engineering method based on the large language model to obtain structured fact triple data;
[0036] A knowledge base construction module, used to construct a knowledge graph database based on the fact triple data;
[0037] A knowledge base retrieval module is used to obtain a user search query and obtain a target answer set based on the user search query and the knowledge graph database;
[0038] The table question answering module is used to input the target answer set into the large language model to output the retrieval result.
[0039] According to the question-answering system for extracting table knowledge in the field of water abstraction permit, the table extraction module includes:
[0040] Table extraction unit: used to extract the water abstraction permit field table in the document;
[0041] Reconstruction unit: reconstruct the merged cells in the row header and column header of the water withdrawal permit field table to obtain the cell text content; the reconstruction includes splitting and / or merging cells;
[0042] The conversion unit is used to convert the text content of the cell to obtain a text format table.
[0043] According to the question-answering system for extracting knowledge from tables in the field of water abstraction permit, the knowledge extraction module includes:
[0044] A prompt engineering template building unit is used to build a prompt engineering template according to a text format table;
[0045] The extraction unit is used to extract knowledge from the text format table based on the large language model according to the prompt engineering template to obtain structured fact triple data.
[0046] According to the question-answering system for extracting table knowledge in the field of water abstraction permit, the knowledge base retrieval module includes:
[0047] A processing unit, configured to obtain question entities and question attributes according to the user search question;
[0048] A first matching unit, configured to generate candidate triples by fuzzy matching based on the question entity and the knowledge graph database, wherein the candidate triples are used to determine fact triple data related to the question entity;
[0049] The second matching unit is used to generate a target answer set by using vector matching according to the question attributes and the candidate triples; the target answer set is used to determine the fact triple data related to the question attributes in the candidate triples.
[0050] The question-answering method and system for extracting table knowledge in the field of water abstraction permit provided by the present invention have the following advantages over the prior art:
[0051] By obtaining the table in the field of water abstraction permit and reconstructing the table structure, a text format table is obtained. Based on the text format table, knowledge is extracted based on the large language model using the prompt engineering method to obtain structured fact triple data. The present invention combines the large language model with table knowledge extraction by using the prompt engineering method, and can accurately extract various types of table knowledge. Then, based on the fact triple data, a knowledge graph database is constructed; user search questions are obtained, and the target answer set is obtained based on the user search questions and the knowledge graph database; the target answer set is input into the large language model to output the search results. It can realize efficient analysis of the table in the field of water abstraction permit, effectively complete knowledge extraction, and then realize fast question and answer. It greatly improves the utilization rate of the content of the table in the field of water abstraction permit, as well as the analysis efficiency, and solves the problems of difficulty in retrieving the table content of the water abstraction permit information table in the prior art, consumption of a large amount of manpower and material resources, and low flexibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0053] Figure 1 It is a flowchart of a question-answering method for extracting table knowledge in the field of water abstraction permits provided by an embodiment of the present invention;
[0054] Figure 2 is a schematic diagram of a process of reconstructing merged cells in a row header provided by an embodiment of the present invention;
[0055] Figure 3 It is a schematic diagram of a process of reconstructing merged cells in a column header provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.
[0058] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0059] Combine the following Figure 1-Figure 3 The present invention describes a question-answering method and system for extracting table knowledge in the field of water abstraction permits provided in an embodiment of the present invention.
[0060] Figure 1 is a flow chart of a question-answering method for extracting table knowledge in the field of water abstraction permits provided by an embodiment of the present invention, such as Figure 1 As shown, including but not limited to the following steps:
[0061] S101: Obtain a water abstraction permit field table and reconstruct the table structure to obtain a text format table.
[0062] Among them, the forms in the field of water abstraction permit are text materials in the form of tables in the document materials in the field of water abstraction permit, such as water abstraction permit application form, water abstraction permit change application form, water abstraction permit cancellation application form, water abstraction permit application form, etc.
[0063] Optionally, the step of obtaining the water withdrawal permit field table and reconstructing the table structure to obtain a text format table includes:
[0064] Extract the water abstraction permit field table from the document;
[0065] Reconstructing the merged cells in the row header and column header of the water withdrawal permit field table to obtain the cell text content; the reconstruction includes splitting and / or merging cells;
[0066] The text content of the cell is converted to obtain a text format table.
[0067] Optionally, the step of extracting the water abstraction permit field table in the document includes:
[0068] Locate the data body of the water abstraction permit field table in the document and obtain the table index;
[0069] According to the table index, obtain the table title;
[0070] The data body and the corresponding table title are combined to obtain a water withdrawal permit field table.
[0071] Optionally, the step of obtaining a table title according to the table index includes:
[0072] Retrieving adjacent paragraphs of the data body according to the table index;
[0073] According to the adjacent paragraphs and the preset table title set, the adjacent paragraphs are matched using regular expressions to generate matching results;
[0074] According to the matching results, get the table title.
[0075] Specifically, before using a large language model to process the table in the document, the table needs to be extracted from the document. Extracting the water withdrawal permit field table in the document includes: the first step is to input the document. For example, the custom blocks function in the python-docx library can be used. Through the blocks function, the paragraphs and tables in the document are directly identified in sequence, and then all the tables in the document are located. The second step is that for each found table, there will be a corresponding index. This index represents the position of the table in the document. According to this index, look up one or two adjacent paragraphs. The third step is to analyze the text of these paragraphs, and use regular expressions such as "Table\d+-\d+\s+.*" and "Table\d+\s+.*" to match the two types of table titles such as "Table 1-1A Plant Water Withdrawal" and "Table 3B Ground Precipitation", and then identify the table title based on the matching results. The fourth step is to form a complete table with the identified table title and the corresponding located table to obtain a text format table.
[0076] Further, the merged cells in the row header and column header of the water withdrawal permit field table are reconstructed to obtain the cell text content, and the cell text content is converted to obtain a text format table including:
[0077] After extracting the table title, the cells in the complex table are reconstructed. After analyzing a large number of water permit data tables, it is found that merged cells mostly appear in row headers and column headers, which play an important role in the table. These merged cells make the structure of the table more complex. In order to enable the large language model to extract table knowledge more accurately, it is first necessary to reconstruct the merged cells in the table row headers and column headers.
[0078] The first step is to find the merged cells in the row header and column header, traverse the cells in the row header and column header, check whether the cell is a merged cell, and display the row number and column number of the merged cell. The second step is to reconstruct these merged cells. Reconstruction includes splitting and / or merging cells; for example, first reconstruct the merged cells in the row header, split each merged cell in the row header into the same number of cells as the corresponding sub-cells on the right, and the contents of these cells are the same as the contents of the merged cells, and then each cell and its corresponding right cell are merged into a large cell.
[0079] like Figure 2 As shown, "Rice", "Wheat", "Corn" and "Rapeseed" belong to the merged cells in the row header. Taking the merged cell "Rice" as an example, first split "Rice" into two cells with the same content "Rice" on the upper and lower sides. These two cells and the two corresponding sub-cells "Mean" and "Variation" on the right are merged respectively, and finally the upper and lower cells of "Rice Mean" and "Rice Variation" of the same size as "Crop" are obtained.
[0080] Then reconstruct the merged cells in the column header. Split each merged cell in the column header into the same number of cells as the corresponding sub-cells below. The contents of these cells are the same as the contents of their merged cells. Then merge each cell with its corresponding sub-cell below to form a large cell.
[0081] like Figure 3 As shown, "Sluice A" and "Sluice B" belong to the merged cells in the column header. Taking the merged cell "Sluice A" as an example, first split "Sluice A" into two cells with the same content "Sluice A" on the left and right. These two cells and the two corresponding sub-cells "Water Diversion" and "Drainage" below them are merged respectively, and finally two cells of "Sluice A Diversion" and "Sluice A Drainage" with the same size as "Year" are obtained.
[0082] The splitting of merged cells in a table includes: after reconstructing the merged cells in the table row header and column header, traversing each row in the table row by row. At this time, the merged cells in the table will be split into small cells containing the same content, and then converted into plain text format for subsequent knowledge extraction using a large language model.
[0083] S102: extracting knowledge based on the text format table using a prompt engineering approach based on a large language model to obtain structured fact triple data.
[0084] The fact triple data includes: subject, predicate and object. Subject corresponds to entity, which refers to the first element in the triple, usually a specific entity or object. Predicate corresponds to property, which refers to the second element in the triple, indicating the relationship or property between the subject and the object. Object corresponds to value, which refers to the third element in the triple, usually the attribute value of the subject or another entity related to the subject.
[0085] The present invention unifies all tables into a (entity, attribute, value) structure through triples, and the structured feature of triple data is an ordered set consisting of subject (entity), predicate (relationship) and object (entity).
[0086] Tips for building project templates, for example:
[0087] Build a prompt engineering template based on the text format table;
[0088] Specifically, the instructions for building a large language model are: The user will enter a form. Your job is to organize the information in the form and list them clearly. Pay attention to every row and column in the form and don't miss any.
[0089] Example:
[0090] Input: The following table is from the chapter '2.2.3 Water Intake Rationality Analysis' of the Water Resources Argumentation for the Expansion Project of Plant A at Site A, 'Table 2-1 Circulating Water Requirement (Unit: t / h)'
[0091] |||||
[0092] |:-:|:-:|:-:|:-:|
[0093] |Equipment type|Power(kw / h)|Circulating water volume|Circulating water volume|
[0094] |Equipment type|Power(kw / h)|Hot season|Cold season|
[0095] |1×Pasteurized|576.95|34617|22501|
[0096] |2×High Temperature Sterilization|1153.9|69234|45002|"
[0097] Output:
[0098] |Entity|Attribute|Value|
[0099] |---|---|---|
[0100] | Plant A expansion project at site A | Power of 1x pasteurization equipment | 576.95kw / h |
[0101] | Plant A expansion project at site A | 1x circulating water volume of pasteurization equipment in hot season | 34617t / h |
[0102] | Plant A expansion project in site A | 1x cold season circulating water volume of pasteurization equipment | 22501t / h |
[0103] | Plant A expansion project in site A | 2x circulating water volume of high temperature sterilization equipment in hot season | 69234t / h |
[0104] | Plant A expansion project in site A | 2x circulating water volume of high temperature sterilization equipment in cold season | 45002t / h |
[0105] Actual input: The following table is from the chapter "{title}" of {file Name}. The table is {table}.
[0106] The specific steps for the large language model to extract table knowledge according to the prompt engineering template are as follows:
[0107] The name of the document, the chapter name and the table corresponding to the table will be filled into {fileName}, {title} and {table} in the prompt project template, forming a large paragraph of text to be sent to the large language model for processing. The large language model extracts knowledge from the table according to the requirements and examples in the prompt project template. The extraction results are in the form of triples, just like the example output. The large language model extracts one table at a time until all tables are processed before the table knowledge extraction ends.
[0108] The following is an example of a large language model extracting knowledge from a water permit table:
[0109] Table 1 Results of extracting tabular knowledge from large language models
[0110]
[0111]
[0112] It should be noted that, referring to the content introduced in the above background technology, the types of large tables in water abstraction permit materials are diverse. Although the large language model can handle large tables and understand the contents of the tables well due to its powerful contextual reasoning ability and long text processing ability, this method has encountered limitations in practical applications.
[0113] Extracting table contents into triples has two main advantages for table question answering systems.
[0114] From a functional perspective, although the tables in the water permit data belong to structured data, the structures of the tables vary greatly. The triples of the present invention unify all tables into a structure of (entity, attribute, value), so that the table contents can be stored in the knowledge base in a more organized form, which greatly facilitates the subsequent knowledge retrieval work.
[0115] From the perspective of content, a triple is composed of (entity, attribute, value). In water abstraction permit data, most tables record numerical information of a certain place, such as (Northern Zone 6, precipitation, 340.6 mm). The use of this triple can well display the content of the water abstraction permit data table.
[0116] Large language models can understand the information in tables, but they do not support directly converting this information into triples, which makes it difficult to effectively integrate the understood table content into the knowledge base. After the introduction of prompt engineering, the situation has changed.
[0117] By designing a sophisticated prompt engineering template, the large language model can not only understand the content of the table, but also generate structured triple information based on the specific requirements of the prompt engineering.
[0118] S103: Construct a knowledge graph database based on the fact triple data.
[0119] Specifically, the fact triple data extracted by the large language model is put into the knowledge base.
[0120] S104: Obtain a user search question, and obtain a target answer set based on the user search question and the knowledge graph database.
[0121] Optionally, the step of obtaining a target answer set according to the user search question and the knowledge graph database includes:
[0122] According to the user search question, obtain question entities and question attributes;
[0123] Generate candidate triples by using fuzzy matching based on the problem entity and the knowledge graph database, and the candidate triples are used to determine the fact triple data related to the problem entity;
[0124] Vector matching is adopted according to question attributes and candidate triples to generate a target answer set; the target answer set is used to determine fact triple data related to the question attributes in the candidate triples.
[0125] Specifically, the questions input by users are extracted through the large language model to extract the entities and attributes of the questions. For example, in the question “What is the area occupied by the construction project of the instant noodle factory in Village A?”, the entity extracted from the question is “Construction project of the instant noodle factory in Village A” and the attribute is “area occupied”.
[0126] After the question extraction is completed, it is necessary to compare the knowledge in the knowledge base and select the correct answer.
[0127] The method used in knowledge base retrieval is to first roughly match the question entity with the entity in the knowledge base to find candidate triples, and then match the attributes of the question with the attributes of the candidate triples to find the most relevant N triples.
[0128] Specific steps for knowledge base retrieval: There are a large number of place names and water conservancy professional terms in the table in the field of water abstraction permits. These place names and water conservancy professional terms are usually the entities of the problem. The word vector embedding model cannot effectively vectorize these professional terms, which will lead to a decrease in the accuracy of knowledge base retrieval.
[0129] Therefore, we first use fuzzy matching between the entities in the question and the entities in the knowledge base to find candidate triples related to the question entities.
[0130] For attribute matching, direct text comparison is performed, which cannot identify the semantic information of the text.
[0131] Therefore, it is necessary to first embed all the attributes in the triples in the knowledge base and the extracted question attributes into vector form through the Bert model. By comparing in the form of vectors, the semantic information of the triple words can be more accurately reflected, and the retrieval accuracy is higher.
[0132] Compare the attribute vector in the question with all the attribute vectors in the candidate triples for vector similarity. Here, the cosine similarity algorithm is used to obtain the N pieces of knowledge with the highest scores.
[0133] S105: Input the target answer set into a large language model to output retrieval results.
[0134] Combined with the powerful natural language processing capabilities of the large language model, the N pieces of knowledge retrieved from the knowledge base are input into the large language model, and the large language model generates accurate, fluent and natural language answers to meet the user's query needs.
[0135] It should be noted that the present invention focuses on the combination of table extraction and structure reconstruction with a large language model, using prompt engineering and a large language model to extract table knowledge. Inputting the target answer set into a large language model for language integration to output accurate, fluent and natural language retrieval results belongs to the prior art and will not be described in detail in this embodiment.
[0136] In a second aspect, the present invention further provides a question-answering system for extracting table knowledge in the field of water abstraction permits, the system comprising:
[0137] Table extraction module, used to obtain water abstraction permit field tables and reconstruct table structures to obtain text format tables;
[0138] The knowledge extraction module is used to extract knowledge based on the text format table using the prompt engineering method based on the large language model to obtain structured fact triple data;
[0139] A knowledge base construction module, used to construct a knowledge graph database based on the fact triple data;
[0140] A knowledge base retrieval module is used to obtain a user search query and obtain a target answer set based on the user search query and the knowledge graph database;
[0141] The table question answering module is used to input the target answer set into the large language model to output the retrieval result.
[0142] Furthermore, the table extraction module includes:
[0143] Table extraction unit: used to extract the water abstraction permit field table in the document;
[0144] Reconstruction unit: reconstruct the merged cells in the row header and column header of the water withdrawal permit field table to obtain the cell text content; the reconstruction includes splitting and / or merging cells;
[0145] The conversion unit is used to convert the text content of the cell to obtain a text format table.
[0146] Furthermore, the knowledge extraction module includes:
[0147] A prompt engineering template building unit is used to build a prompt engineering template according to a text format table;
[0148] The extraction unit is used to extract knowledge from the text format table based on the large language model according to the prompt engineering template to obtain structured fact triple data.
[0149] Furthermore, the knowledge base retrieval module includes:
[0150] A processing unit, configured to obtain question entities and question attributes according to the user search question;
[0151] A first matching unit, configured to generate candidate triples by fuzzy matching based on the question entity and the knowledge graph database, wherein the candidate triples are used to determine fact triple data related to the question entity;
[0152] The second matching unit is used to generate a target answer set by using vector matching according to the question attributes and the candidate triples; the target answer set is used to determine the fact triple data related to the question attributes in the candidate triples.
[0153] In summary, the question-answering method and system for extracting table knowledge in the field of water abstraction permit provided by the present invention has the following advantages over the prior art:
[0154] (1) Combination of table extraction and structure reconstruction with a large language model
[0155] In documents where tables and text are interspersed, the large language model processes text and tables differently and cannot directly process the tables in the document. Therefore, it is necessary to extract the tables in the document, reconstruct the table structure, simplify the table structure, convert it into a plain text format table, and then input it into the large language model. With its powerful context understanding ability and long text processing ability, the large language model can effectively extract the table content.
[0156] (2) Using hint engineering and large language models to extract tabular knowledge
[0157] Conventional table question-answering systems have difficulty effectively handling complex table contents and the problem of how to put table contents into the knowledge base. The large language model can accurately extract complex table contents through prompt engineering, convert them into triples and put them into the knowledge base, and it can handle more table styles.
[0158] (3) Using triple retrieval in knowledge base retrieval
[0159] There are a large number of place names and water conservancy professional terms in the tables in the field of water abstraction permits. The word vector embedding model cannot effectively vectorize these professional terms, and the entities in user questions are usually related to these place names and water conservancy professional terms. If vector matching is used directly, this will lead to a decrease in the accuracy of knowledge base retrieval.
[0160] Performing fuzzy matching on the question entity first and then vector matching on the question attributes can avoid such problems and improve retrieval accuracy.
[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A question-answering method for extracting table knowledge in the field of water abstraction permits, characterized in that: include: Obtain the water abstraction permit field table to reconstruct the table structure and obtain a text format table; According to the text format table, the prompt engineering method is used to extract knowledge based on the large language model to obtain structured fact triple data; Constructing a knowledge graph database based on the fact triple data; Obtaining a user search question, and obtaining a target answer set based on the user search question and the knowledge graph database; The target answer set is input into a large language model to output a retrieval result.
2. The question-answering method for extracting table knowledge in the field of water abstraction permit according to claim 1 is characterized in that: The step of obtaining the water withdrawal permit field table and reconstructing the table structure to obtain a text format table includes: Extract the water abstraction permit field table from the document; Reconstructing the merged cells in the row header and column header of the water withdrawal permit field table to obtain the cell text content; the reconstruction includes splitting and / or merging cells; The text content of the cell is converted to obtain a text format table.
3. The question-answering method for extracting table knowledge in the field of water abstraction permit according to claim 2 is characterized in that: The step of extracting the water abstraction permit field table in the document comprises: Locate the data body of the water abstraction permit field table in the document and obtain the table index; According to the table index, obtain the table title; The data body and the corresponding table title are combined to obtain a water withdrawal permit field table.
4. The question-answering method for extracting table knowledge in the field of water abstraction permit according to claim 3 is characterized in that: The step of obtaining a table title according to the table index comprises: Retrieving adjacent paragraphs of the data body according to the table index; According to the adjacent paragraphs and the preset table title set, the adjacent paragraphs are matched using regular expressions to generate matching results; According to the matching results, get the table title.
5. The question-answering method for extracting table knowledge in the field of water abstraction permit according to claim 1 or 2, characterized in that: The step of extracting knowledge based on the text format table using a prompt engineering method based on a large language model to obtain structured fact triple data includes: Build a prompt engineering template based on the text format table; Based on the large language model, knowledge is extracted from the text format table according to the prompt engineering template to obtain structured fact triple data.
6. The question-answering method for extracting table knowledge in the field of water abstraction permit according to claim 1 is characterized in that: The step of obtaining a target answer set according to the user search question and the knowledge graph database comprises: According to the user search question, obtain question entities and question attributes; Generate candidate triples by using fuzzy matching based on the problem entity and the knowledge graph database, and the candidate triples are used to determine the fact triple data related to the problem entity; Vector matching is adopted according to question attributes and candidate triples to generate a target answer set; the target answer set is used to determine fact triple data related to the question attributes in the candidate triples.
7. A question-answering system for extracting table knowledge in the field of water abstraction permits, characterized in that: include: Table extraction module, used to obtain water abstraction permit field tables and reconstruct table structures to obtain text format tables; The knowledge extraction module is used to extract knowledge based on the text format table using the prompt engineering method based on the large language model to obtain structured fact triple data; A knowledge base construction module, used to construct a knowledge graph database based on the fact triple data; A knowledge base retrieval module is used to obtain a user search query and obtain a target answer set based on the user search query and the knowledge graph database; The table question answering module is used to input the target answer set into the large language model to output the retrieval result.
8. The question-answering system for extracting knowledge from tables in the field of water abstraction permit according to claim 7, characterized in that: The table extraction module comprises: Table extraction unit: used to extract the water abstraction permit field table in the document; Reconstruction unit: reconstruct the merged cells in the row header and column header of the water withdrawal permit field table to obtain the cell text content; the reconstruction includes splitting and / or merging cells; The conversion unit is used to convert the text content of the cell to obtain a text format table.
9. The question-answering system for extracting knowledge from tables in the field of water abstraction permit according to claim 6, characterized in that: The knowledge extraction module comprises: A prompt engineering template building unit is used to build a prompt engineering template according to a text format table; The extraction unit is used to extract knowledge from the text format table based on the large language model according to the prompt engineering template to obtain structured fact triple data.
10. The question-answering system for extracting knowledge from tables in the field of water abstraction permit according to claim 6, characterized in that: The knowledge base retrieval module includes: A processing unit, configured to obtain question entities and question attributes according to the user search question; A first matching unit, configured to generate candidate triples by fuzzy matching based on the question entity and the knowledge graph database, wherein the candidate triples are used to determine fact triple data related to the question entity; The second matching unit is used to generate a target answer set by using vector matching according to the question attributes and the candidate triples; the target answer set is used to determine the fact triple data related to the question attributes in the candidate triples.
Citation Information
Cited By
Application level retrieval enhancement generation strategy method and device
CN119336873A
Table data extraction method and device, computer equipment and storage medium
CN120030162A
