Question answering method and system based on table semantic fasttext question parsing

Abstract

The application provides a kind of based on table semantic fasttext question analysis question and answer method and system, wherein, method includes: data preparation stage: table is rewritten as the text of hierarchical structure of chapter semantics, obtain the analysis dictionary for question analysis, establish question and answer annotation corpus and establish the data required for the establishment of table semantic fasttext question analysis question and answer method of field related fasttext similarity calculation model;Question analysis stage: after question is parsed through analysis dictionary, it is parsed with the question and answer annotation corpus according to fasttext similarity calculation model;Obtain answer stage: according to the question after semantic analysis, operation is carried out to construct complete answer.The application rewrites complex form into natural language text with chapter hierarchical structure, overcomes the difficulty that complex table structure is difficult to be standardized into database task;Through fasttext question analysis model, a general question multi-attribute solution is obtained by comparing an approximate solution, which greatly meets the engineering needs.

Description

Technical Field

[0001] This document relates to the field of question parsing technology, and in particular to a question-answering method and system based on tabular semantic FastText question parsing. Background Technology

[0002] Question answering technology is a typical application of natural language processing technology in artificial intelligence. A major problem in reality is the contradiction between the arbitrariness of question answers and the uniqueness required in engineering. The shortcoming of existing technologies lies in their inability to guarantee the accuracy of answers, which is unacceptable in engineering. Existing question answering technologies have the following drawbacks:

[0003] (1) Ambiguity in Question Semantic Understanding. The question-and-answer framework defined in frame semantics methods is arbitrary and ambiguous because it is artificially defined. For example, "What are the wells in the Puguang gas field with a depth greater than 6000 meters?" can be defined as "Puguang gas field + attribute", "Puguang gas field + well + attribute", "Puguang gas field + well + attribute constraint", etc., with each person defining different semantics. Ultimately, there is no authoritative semantic definition, leading to difficulties in corpus annotation and failing to meet the requirements of uniqueness and accuracy in engineering. If question parsing is performed according to the everyday 5W1H method, many non-question sentences and multi-conditional questions in engineering become difficult to parse. For example, "yesterday's daily output" is not a question but a numerical query; "What are the wells in the Puguang gas field with a depth greater than 6000 meters?" includes two "What" questions: "What are the wells?" and "What are the wells with a depth greater than 6000 meters?". This makes it difficult to use a single-layer sequence of 5W1H for question parsing.

[0004] (2) Difficulty in handling complex hierarchical structures in questions. Question-and-answer systems in engineering are built within the spatial and temporal scope of certain businesses of a company. Engineering forms used in engineering, based on the principles of readability and unambiguity, generally have complex spatial and temporal structures. These structures are difficult to break down into simpler diagrams or smaller tables. Generally, a single table contains complex business, structural, and temporal relationships, which can be understood as a summary table organically pieced together from multiple smaller tables or diagrams. For example, an oilfield production daily report includes all businesses such as production wells, drilling, purification plants, and production service centers, as well as the spatial relationships of facilities such as the main body, lines, wells, and various levels of joint facilities of purification units, and the temporal hierarchical relationships of year, month, and day. Moreover, the objects in the structure exist both horizontally and vertically in the form, rather than being arranged in a single table format of objects + attributes vertically. If we follow the traditional approach of parsing simple questions, we would need to break down the complex table in this project into many smaller tables or reconstruct a complex knowledge graph from this form. These anti-engineering approaches complicate simple problems, adding a lot of unnecessary work, making it difficult to ensure the rationality of the project logic and the implementation schedule.

[0005] (3) Too many dictionaries are required, which is not engineering-friendly. For example, an oilfield question-and-answer system requires 39 dictionaries to realize the transformation of words in the question → entities (time, equipment at all levels) → concepts (attributes) → sentence semantics (question form) and complete the semantic parsing task of the question. This task is huge. The core problem is that after decomposing a form with a complex structure into smaller granularities, the content described by these extra dictionaries is the content of the existing structure at each level in the engineering form. It is just a re-explicit description using multiple small tables or small diagrams, without most effectively directly using the implicit knowledge in the engineering.

[0006] (4) Excessive computation time due to multiple implicit nodes. In knowledge graph-based methods, cross-node question answering can lead to excessively long extension times. Crashes also occur when performing cross-node calculations at a large time granularity. For example, in a daily production report, calculating the annual value can take too long, preventing timely responses. For instance, the question "What is the cumulative production of line XX?" has an implicit inclusion relationship between "line → well" and a time hierarchy of "year → month → day." With one daily production report, frequent database table calls and cumulative read operations directly cause computer crashes.

[0007] Therefore, a question-and-answer method is urgently needed to address the aforementioned shortcomings. Summary of the Invention

[0008] This invention provides a question-answering method and system based on table semantic fasttext question parsing, aiming to solve the above-mentioned problems.

[0009] This invention provides a question-answering method based on table semantic fasttext question parsing, including:

[0010] S1. Data preparation stage: Rewrite the table into a hierarchical semantic text, obtain the parsing dictionary for question parsing, establish a question-answering annotation corpus and establish a domain-related fasttext similarity calculation model to establish the data required for the question-answering method of table semantic fasttext question parsing;

[0011] S2, Question Parsing Stage: The questions are parsed using a parsing dictionary, and the parsed questions are semantically analyzed against the question-and-answer labeled corpus based on the FastText similarity calculation model;

[0012] S3. Answer Acquisition Stage: Perform calculations based on the semantically parsed question to construct a complete answer.

[0013] This invention provides a question-answering system based on table semantic fasttext question parsing, comprising:

[0014] The data module is used to rewrite tables into hierarchical semantic text, obtain parsing dictionaries for question parsing, build question-answering labeled corpora, and establish domain-related fasttext similarity calculation models to build the data required for question-answering methods of table semantic fasttext question parsing;

[0015] The processing module is used to parse the question using a parsing dictionary and then perform semantic parsing of the question based on the fasttext similarity calculation model;

[0016] The application module is used to obtain the answer from the semantically parsed question using the question-answer labeled corpus, and to assemble the answer to obtain the complete answer.

[0017] This invention overcomes the problems of ambiguity in question interpretation and multi-level cross-boundary node calculation by directly constructing question-and-answer structures from complex engineering forms, leveraging the inherent complete meaning and complex structure of the forms themselves. It rewrites complex forms into natural language text with a hierarchical structure, overcoming the difficulty of standardizing complex table structures for database tasks. Furthermore, through the FastText question parsing model, it obtains multi-attribute solutions for general questions by comparing an approximate solution, greatly satisfying the requirements for data accuracy in engineering. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a question-answering method based on tabular semantic fasttext question parsing according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of a question-answering system based on table semantic fasttext question parsing according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a specific question-and-answer system based on a table query mode, as an embodiment of this system.

[0022] Figure 4 This is a schematic diagram of the daily report structure according to an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the chapter structure of the daily report according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the daily report knowledge base according to an embodiment of the present invention;

[0025] Figure 7 This is a schematic diagram of the table name dictionary according to an embodiment of the present invention;

[0026] Figure 8 This is a schematic diagram of a spoken language dictionary according to an embodiment of the present invention;

[0027] Figure 9 This is a schematic diagram of the object dictionary according to an embodiment of the present invention;

[0028] Figure 10 This is a schematic diagram of the object attribute dictionary according to an embodiment of the present invention;

[0029] Figure 11 This is a schematic diagram of the attribute value dictionary according to an embodiment of the present invention;

[0030] Figure 12 This is a schematic diagram of a time dictionary according to an embodiment of the present invention;

[0031] Figure 13 This is a schematic diagram of the interrogative word dictionary according to an embodiment of the present invention;

[0032] Figure 14 This is a schematic diagram of the question annotation corpus according to an embodiment of the present invention;

[0033] Figure 15 This is a flowchart of the question-answering technology based on table semantic fasttext question parsing according to an embodiment of the present invention;

[0034] Figure 16 This is a question posed in an embodiment of the present invention;

[0035] Figure 17 This is a schematic diagram of the question parsing results in an embodiment of the present invention. Detailed Implementation

[0036] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0037] Method Implementation Examples

[0038] This invention provides a question-answering method based on tabular semantic fasttext question parsing. Figure 1 This is a flowchart of a question-answering method based on tabular semantic fasttext question parsing according to an embodiment of the present invention. Figure 1 As shown, the question-answering method based on table semantic fasttext question parsing in this embodiment of the invention includes:

[0039] Step S101, Data Preparation Stage: The table is rewritten into hierarchical semantic text; a parsing dictionary for question parsing is obtained; a question-answering labeled corpus is established; and a domain-related fasttext similarity calculation model is built. This establishes the data required for the question-answering method of table semantic fasttext question parsing. Step S101 specifically includes:

[0040] Establish a query knowledge base and prepare the data involved in question answering. Rewrite the tables from whole to part into hierarchical semantic text, serving as the knowledge base (inverted index) for question search. The large table is converted into a document with a text structure, which is the template for the same large table; smaller tables represent chapters, with the chapter name as the table header and the paragraph text below the chapter as the records; when transforming the structured smaller tables into paragraph text sentences, semantic information needs to be added. This process of converting the project record (large table → smaller table → table row → table record) into the required text → chapter → sentence → word for search builds a textual description of the entire table from whole to part—a 1-1 process. However, querying is a 1-to-n process from bottom to top, from part to whole; that is, from bottom to top, multiple search combinations may be generated. This requires more exclusion conditions in dictionary form at different levels to eliminate ambiguity and obtain a unique answer. To implement the search, all words in the text need to be placed in a word segmentation dictionary, serving as the inverted index for search.

[0041] Step S101, which involves rewriting the table into hierarchical semantic text, specifically includes:

[0042] The daily report tables are converted into a hierarchical text knowledge base based on the daily report template. The daily report template breaks down the table structure, and the text knowledge base contains four levels: words, constructed sentences, sub-table names, and table names, which correspond to the four levels of words, sentences, chapters, and articles in natural language, respectively.

[0043] Step S101, obtaining the parsing dictionary for question parsing, specifically includes:

[0044] Obtain the table name dictionary for passage and chapter-level parsing used for question parsing. The table name dictionary is manually configured according to business needs, and the table name is determined by the specific question. The table name dictionary is updated according to the specific question.

[0045] This system retrieves a colloquial dictionary, an object dictionary, an object attribute dictionary, an attribute value dictionary, a time dictionary, and an interrogative word dictionary for word-level parsing of questions. The colloquial dictionary converts colloquial words in questions into standard names; the object dictionary corresponds to the indexes in the table and represents semantic objects or hierarchical objects; the object attribute dictionary collects the contents of all horizontal attributes in the table, and the attribute value dictionary collects the specific values ​​of all horizontal attributes in the table; the time dictionary describes the general time structure; and the interrogative word dictionary describes the interrogative words appearing in the question and the methods used to obtain the answer.

[0046] Step S101, which involves establishing a question-and-answer labeled corpus, specifically includes:

[0047] A question-and-answer tagged corpus is formed by collecting all known questions and the correct parsing results of the corresponding questions after manual proofreading. The question-and-answer tagged corpus is continuously formed in an incremental manner. The result of automatic parsing of each new question is used as a reference, and then manually proofread to obtain the final correct parsing result. At the same time, all related parsing dictionaries are added.

[0048] Step S101, establishing a domain-related fasttext similarity calculation model, specifically includes:

[0049] The input question and output table name, object, and attribute are digitized and tensed using a v2w approach. Then, a multi-level neural network mapping relationship is established between the input tensor and the output attribute tensor to obtain the parsing parameters of the question.

[0050] Step S102, Question Parsing Stage: The question is parsed using a parsing dictionary. The parsed question is then semantically analyzed against the question-and-answer labeled corpus using the FastText similarity calculation model. Step S102 specifically includes:

[0051] The FastText model is used to parse the question y = f(x). Following the principle of best character matching, the closest reference parsing result y0 = f0(x0) is found by calculating the similarity between the question and known questions in the question-answer corpus. Question parsing is performed at two levels: character-level and semantic-level. The question and reference question are compared, and character-level decomposition is performed using a non-overlapping linear sequence annotation method, without considering semantic phenomena such as word fusion or borrowing (e.g., fusion such as "production amount of gas = daily gas production, amount", borrowing such as "depth of how much = well depth, amount"). The decomposed characters are mapped to semantic elements through dictionary comparison, achieving character-to-semantic parsing, that is, mapping characters to attributes and attribute values ​​in a table. At this point, semantic phenomena such as fusion and borrowing are directly parsed using dictionary comparison. If y0, f0, and x0 appear in the question, then y = y0, f = f0, and x = x0; otherwise, for y0, f0, and x0 not appearing in the question, dictionary matching is required to finally complete the parsing of y = f(x).

[0052] Step S103, Answer Acquisition Stage: A complete answer is constructed based on the semantically parsed question. Step S103 specifically includes:

[0053] Based on the parsed result, the question y = f(x) is used to perform calculations and construct the answer. y and x are used to retrieve valid knowledge entries containing complete semantics and data from the question → knowledge base, equivalent to loading the table header and data. f is used for calculations in the knowledge base → answer, equivalent to processing the data to obtain the final result. f typically performs query operations on the table data; complex operations include limited queries and statistical analysis. Based on people's question-answering habits, an answer template is constructed, and the retrieved data is used to build the answer according to the template requirements, outputting the complete answer.

[0054] Reference Figure 15 A specific embodiment of this method includes three main processes: knowledge base construction, question parsing, and answer generation, detailed as follows:

[0055] Step 151: Knowledge Base Construction

[0056] The more than 1,000 related tables used in the project were rewritten into sentences for question answering, transforming structured data queries into natural language search.

[0057] Step 151-1: Read the report corpus

[0058] The report is a complete project schedule, such as Figure 4 As shown, the table format has been merged according to the needs of the project understanding, and multiple tables of different specifications have been merged together. In particular, there are some texts that are not tables. Therefore, when reading the table using pandas, do not recognize the table header.

[0059] Step 151-2: Read the report template

[0060] Reporting modes such as Figure 5 As shown, the report template enables the positioning of the text structure of the report. The report mode is universal, and the template is the same for reports of the same type. Therefore, the report template is the foundation for automated report recognition and conversion.

[0061] Step 151-3: Processing multiple rows in the report

[0062] In an Excel spreadsheet, multiple rows containing business meaning are typically enclosed in quotation marks, such as "oil temperature\r\n.C" indicating the relationship between oil temperature and its units. Replacing \r\n ensures that each table contains only one string without \r\n; otherwise, \r\n can cause numerous visualization problems during text processing.

[0063] Step 151-4: Report Row Concatenation

[0064] The report is separated by columns, which does not conform to the natural language processing habit of sentences. Therefore, all numeric cells in each row are first removed, and only non-array cells are kept. Then, all cells are sorted using the sort() method. This ensures that tables with the same header but different column positions have the same semantic meaning. Finally, all cells are concatenated together with @@@ to form the sentence text of each record, such as "Station Name@@@Number of Wells@@@Production Time@@@Production Time h@@@Wellhead Temperature@@@Wellhead Pressure (MPa)@@@Production Allocation@@@Daily Production@@@Monthly Production@@@Yearly Cumulative@@@Cumulative" to represent a header.

[0065] Step 151-5: Report Text Structure Recognition

[0066] By comparing the line text of the report with the template, the position of the table header in the report is marked. If the table is regarded as a document, identifying the table header is to identify the chapter structure of the table.

[0067] Step 151-6: Multi-row header processing

[0068] Multi-row headers represent the hierarchical structure of attributes. Each attribute appears only once at the upper level and is filled into the first table. Therefore, for rows marked as sub-tables, all rows should be filled using the `ffill()` method. Multi-level headers are merged using `@@@` to connect different levels. Then, redundant headers are removed, leaving only a single merged header. Examples of headers like "Wellhead Pressure (MPa)@@@ Oil Pressure", "Wellhead Pressure (MPa)@@@ Casing Pressure", "Production Allocation@@@ 104 m3", "Daily Production@@@ Wellhead Gas 104 m3", "Daily Production@@@ Liquid m3", "Monthly Production@@@ Wellhead Gas 104 m3", "Monthly Production@@@ Liquid m3", "Annual Cumulative@@@ Wellhead Gas 108 m3", "Annual Cumulative@@@ Liquid 104 m3", "Cumulative@@@ Wellhead Gas 108 m3", and "Cumulative@@@ Liquid 104 m3" all represent a set of two-level indicators.

[0069] Step 151-7: Single-line text and dictionary generation

[0070] Single-line text refers to attribute-value pairs arranged in a single line. First, identify the single-line text, then use the `pd.dropna(how='all', axis=1, inplace=True)` method to remove all empty columns, constructing a multi-line structure with alternating attribute-value pairs. Then, extract the attribute-value pairs sequentially and write them as a key:value dictionary. This dictionary can be used directly as a sentence, such as `s={'Measurement output (104m3):0.0','New well output (104m3):0.0','Old well output (104m3):2433.0'}`, which is the text of a single-line record.

[0071] At the same time, all table records without numbers are output as word segmentation dictionaries. For example, c = {'Measure output (104m3)', 'Old well output (104m3)', 'New well output (104m3)'} is a word segmentation dictionary. If the input question has an intersection with this dictionary c after word segmentation, the corresponding sentence is s, and s is a table with complete semantic records.

[0072] Steps 151-8: Generating small table text and dictionary

[0073] Because the smaller tables have a defined structure, the header information needs to be included when transforming them into sentences. First, extract all entries from the smaller tables and use `pd.dropna(how='all', axis=1, inplace=True)` to remove all empty columns. Then, extract the row numbers of the smaller tables. Next, construct two tables with headers and without headers between the two smaller tables' row numbers. Then, remove empty columns from the table without row numbers and use its `columns` property to select the table with headers, where the headers are stored in the first row. Finally, use the `pd.to_dict('records')` method to transform each record into a sentence with header fields, structured as a dictionary; the first row is the header row and needs to be removed.

[0074] Simultaneously output as Figure 10 The attribute dictionary shown and such Figure 11 The dictionary of attribute values ​​shown also indicates the position of the attribute and its value within the text structure.

[0075] Step 152: Question Analysis

[0076] By performing similarity calculation on the questions based on the fasttext model, the reference parsing result y0=f0(x0) is obtained from the known question-answering corpus. By comparing the reference result, the final parsing result y=f(x) is obtained.

[0077] Step 152-1: Read the question-and-answer tagged corpus and various dictionaries

[0078] Read Figure 9-13 The dictionary, and Figure 4 The corpus shown.

[0079] Step 152-2: Read the question

[0080] Read Figure 16 The questions shown include some that are the same as those in the corpus, representing repeated questions, and some that are new questions, which are the focus of question analysis.

[0081] Step 152-3: Calculate the similarity between the question and the corpus.

[0082] The similarity between the question and the sequence of question sentences in the corpus is calculated using the fasttext model, employing the cosine method from scipy.spatial as follows:

[0083] diss=1-scipy.spatial.distance.cosine(model.get_word_vector(word),model.get_word_vector(word1))

[0084] Then, the diss entries are sorted from largest to smallest, and the first one is taken as the reference question that is most similar to the question.

[0085] Step 152-4: Obtain the reference y0 = f0(x0)

[0086] Extract the corresponding character parsing results and semantic parsing results from the question annotation, and use them as y0 = f0(x0), i.e., the reference parsing results.

[0087] Step 152-5: Calculate the part y0\f0\x0 in the question.

[0088] If the character y0\f0\x0 is in the question, then this part of the parsing result is taken directly as part of the final parsing result y\f\x.

[0089] Replace the original question with known data: q1 = q.replace(j,'#',1), continuously narrowing down the original question to reduce interference from finding new elements.

[0090] Step 152-6: Calculate the remaining y0\f0\x0

[0091] For y0, f0, and x0 that are not in the question, first parse the semantic element dictionary type from the reference semantics, then use the corresponding dictionary lookup in q1, and take the longest result as the parsed semantic element. There is no need to use a spoken dictionary lookup in q1.

[0092] Once the semantic attributes of the newly added string are confirmed, simply replace y0 with y as the semantic parsing result:

[0093] Step 152-7: Obtain the complete y = f(x)

[0094] After steps 152-5 and 152-6, the complete analytical result of the question y = f(x) is obtained as follows: Figure 17 As shown, questions with a similarity of 1 are repeated questions, and their parsing results can be directly copied from the corpus annotation results.

[0095] Regarding the question "How many wells were put into production in Puguang Main Body in 2011?", referring to the known question "How many wells were put into production in Puguang Main Body in 2012?", the analysis result is as follows:

[0096] Character Analysis: Puguang Main Body @@@2011@@@Annual@@@Number of Wells Put into Production (wells)

[0097] Semantic parsing: `Puguang` main body: block / object `Obj@@@2011: number / attribute value @@@year / attribute t @@@number of wells in production (wells) / attribute y`

[0098] This shows that "2011" is a time object converted from a compared string.

[0099] Step 153: Answer Generation

[0100] The data in the knowledge base is found based on the character parsing results of the question, the value of y is calculated based on the semantic parsing results, and finally the answer is concatenated and output.

[0101] Step 153-1: Search data based on x and y

[0102] The parsed semantic elements are compiled into a set, which is then matched against words in the knowledge base. The knowledge item with the most matches is the required data. (See...) Figure 17 For example, the sentence that can be matched by this set {'number of wells put into production', 'block', 'Puguang main body', '2011', 'year'} obviously contains all the information needed for "how many wells were put into production in Puguang main body in 2011".

[0103] Step 153-2: Data Recovery

[0104] Since the retrieved data is a complete record, and questions and answers generally do not require much data, when there are statistical calculations, the data needs to be truncated according to the semantically given table header to obtain a smaller table for calculation.

[0105] Step 153-3: Perform data calculations based on f

[0106] Method f is the calculation method for obtaining the data of y, such as querying, statistical operation, etc. Each f corresponds to a semantic frame or a template.

[0107] Step 153-4: Obtain the answer template

[0108] Extract the answer template directly from the reference annotations.

[0109] Step 153-5: Assemble the answer

[0110] Replace the corresponding words in the answer template with the parsed semantic words and calculation results to obtain the final answer.

[0111] By employing the embodiments of the present invention, the following beneficial effects are reported:

[0112] This patent overcomes the challenges of interpreting ambiguities in questions and calculating multi-level cross-boundary nodes by directly constructing question-and-answer responses for complex engineering forms, leveraging the inherent meaning and complex structure of these forms. It rewrites complex forms into natural language text with a hierarchical structure, overcoming the difficulty of standardizing complex table structures for database tasks. Furthermore, through the FastText question parsing model, it obtains multi-attribute solutions for general questions by comparing approximate solutions, greatly satisfying engineering needs. In practice, question-and-answer responses were applied to 13 engineering forms at an oilfield production site, achieving an accuracy rate of 98%, meeting the engineering requirements for data accuracy.

[0113] System Implementation Examples

[0114] This invention provides a question-answering system based on table semantic FastText question parsing. Figure 2 This is a schematic diagram of a question-answering system based on tabular semantic fasttext question parsing according to an embodiment of the present invention. Figure 2 As shown, the question-answering system based on table semantic fasttext question parsing in this embodiment of the invention specifically includes:

[0115] Data module 20 is used to rewrite tables into hierarchical semantic text, obtain parsing dictionaries for question parsing, build question-answering labeled corpora, and build domain-related fasttext similarity calculation models to establish the data required for question-answering methods of table semantic fasttext question parsing;

[0116] Processing module 22 is used to parse the question using a parsing dictionary and then perform semantic parsing on the question based on the fasttext similarity calculation model;

[0117] Application module 24 is used to obtain the answer from the semantically parsed question through the question-answer labeled corpus, and to assemble the answer to obtain the complete answer.

[0118] Figure 3This diagram illustrates a specific question-and-answer system based on a table query model, as described in this embodiment. It comprises a data layer 1, a processing layer 2, and an application layer 3. Data layer 1 corresponds to the data module 20 of the question-and-answer system based on table semantic fasttext question parsing in this embodiment. Processing layer 2 corresponds to the processing module 22 of the same system. Application layer 3 corresponds to the application module. In a specific embodiment, the data layer stores, reads, writes, and modifies files, containing engineering materials needed for question-and-answer processing. These materials include pre-prepared reports, dictionaries, models, and annotated question-and-answer corpora used for natural language parsing of questions. The processing layer parses questions, performs table lookups, and generates answers, effectively transforming the entire question-and-answer system into a query system. The application layer handles user interaction, including question input and answer output.

[0119] Data Layer 1 consists of four parts: Report Knowledge Base 1-1, Dictionary 1-2, Petrochemical V2W Vector Model 1-3, and Question-Answer Annotated Corpus 1-4. The entire data layer provides various data required for data output. Report Knowledge Base 1-1 comprises Daily Report 1-1-1, Daily Report Template 1-1-2, and Daily Report Knowledge Base 1-1-3, converting daily reports from the project into a natural language knowledge base for querying. The typical structure of Daily Report 1-1-1 is as follows: Figure 4 As shown (but not limited to this structure), Figure 4 It is composed of seven different business summaries. In addition to the traditional vertical table of objects and horizontal tables of attributes, each structure includes a sixth type where only object attributes are arranged horizontally, used for handling total metrics or cases with limited data. These forms are views generated from many databases, and their composition conforms to engineering reading habits. Therefore, directly handling the questions and answers for these forms is also a practice consistent with engineering practices. The daily report template 1-1-2 is as follows... Figure 5 As shown, the table can be viewed as an article, with each sub-table as a chapter. The chapter name is formed by concatenating the header fields of the sub-tables. (Daily Knowledge Base 1-1-3) Figure 6 As shown, a daily engineering report is rewritten into a hierarchical text knowledge base. This knowledge base contains four levels: words → constructed sentences → sub-table names → table names, corresponding to the four levels of text: words → sentences → chapters → articles. Words are used for searching, selecting candidate materials through questions. A sentence is a complete form record containing object attributes and their values ​​for each field, representing an instantiation of the sub-table's semantics. A sub-table name represents a partial semantic framework; sentences with the same sub-table name share the same table structure or framework semantics. The table name is the name of the entire table, representing a complete engineering scenario.

[0120] Dictionary 1-2 further includes a discourse-level parsing dictionary and a table name dictionary 1-2-1 for question parsing, and colloquial dictionaries 1-2-2, 1-2-3, 1-2-4, 1-2-5, 1-2-6, and 1-2-7 for sentence-level parsing (including object dictionary 1-2-2, object attribute dictionary 1-2-4, attribute value dictionary 1-2-5, time dictionary 1-2-6, and interrogative word dictionary 1-2-7). These dictionaries facilitate the conversion between meaningless characters, standard terms in the table, table attributes, and operational methods. By constructing a search text from all possible questions for the table, these word-level dictionaries constitute the word segmentation library capable of retrieving this text. Table name dictionary 1-2-1 consists of table names, such as... Figure 7 As shown, this typically includes all data tables and views involved in the project, exceeding 1000 in number, but generally no more than 10 are project-specific to keep the project scope focused. The table name dictionary is primarily defined based on business needs, as the same question may correspond to many business processes, i.e., many tables. However, for a specific scenario, only one table needs to be selected, requiring manual intervention to specify the table name. The table name is determined by the specific question, mainly based on the objects and attributes appearing in the question. For example, "sulfur" only belongs to the daily report, so the table name dictionary must be updated according to the specific question. Automatic table name recognition is achieved through a question classification algorithm. Regarding sub-tables, the sub-table is defined as the basic query structure. The basic semantic structure of the sub-table is defined as a 3-tuple of object + attribute + value, where objects are arranged vertically and attributes horizontally. The intersection of the vertical objects and horizontal attributes determines the position of the value, but the value's position is not necessarily at the intersection; there are exceptions. 1-2-2 is a colloquial dictionary, collecting everyday expressions for objects and attributes appearing in spoken language. Because spoken language is highly regional and not a widely accepted consensus, it is treated as a separate dictionary, allowing for additions, deletions, and adjustments as needed to adapt to different application scenarios. Figure 8 As shown. Object dictionary 1-2-3 corresponds to the index in the table, which is a semantic object or an object with a hierarchy, such as... Figure 9 As shown; the object attribute dictionary 1-2-4 contains the contents of all horizontal attributes in the table, such as... Figure 10 As shown, it also lists multi-level attributes appearing in many tables, using @@@ to represent hierarchical relationships. The attribute value dictionary is as follows: Figure 11 As shown, 1-2-3, 1-2-4, and 1-1-5 together describe the elements of the table. The time dictionary 1-2-6 is as follows... Figure 12 As shown, it contains many regular expressions to describe a general time structure. (Question word dictionary 1-2-7 is shown below.) Figure 13 As shown, the question words appearing in the question and the methods used to obtain the answer are described, and these methods directly correspond to the methods in the program.

[0121] Question-answering tagged corpora 1-3 consist of all known questions and correctly parsed results that have been manually proofread. This is the corpus upon which all models rely, and its format is as follows: Figure 14 As shown, the annotations express the relationships between meaningless characters, standard terms, table attributes, and question semantics. For example, the annotation "Number of wells in production: Number of wells in production (mouth) / attribute / y" indicates that the annotation term for the string "Number of wells in production" is "Number of wells in production (mouth)," which is a table attribute, i.e., the table header, representing the required result y in the question. As the questions become more complex, the corpus will tend to converge to a certain data point within a limited scenario. The corpus is continuously formed incrementally, using the automatically parsed results of each new sentence as a reference, and then manually proofreading to obtain the final correct parsing result. These annotations will also add all related dictionaries 1-2. The corpus annotation is a sentence-level parsing corpus, and its addition directly affects the content of its lower-level word-level parsing dictionaries, with an automatic linkage between the upper and lower levels.

[0122] 1-4 is a FastText W2V vector data model in the petrochemical field, using 2.66 million articles (titles, abstracts, and content) from professional literature in the petrochemical field, totaling 22 million sentences, as the corpus. The vector dimension is 120, and the training time is 5.5 hours. Because the question-answering corpus contains many table names, objects, and attributes, and these are constantly changing, a classification model with a fixed number of categories cannot be used. Instead, a V2W approach is used to digitize and tensorize the input questions and output table names, objects, and attributes. Then, a multi-layered neural network mapping relationship is established between the input tensor and the output attribute tensor, enabling the calculation of various question parsing parameters from the input questions.

[0123] Processing layer 2 includes three modules: knowledge base construction 2-1, question parsing 2-2, and answer generation 2-3. These modules perform question parsing, disambiguation, calculation, and finally, question generation. Knowledge base construction 2-1 further includes two modules: daily knowledge base construction 2-1-1 and dictionary construction 2-1-2. These modules respectively construct the knowledge base from tables in the database from top to bottom and output various corresponding dictionaries. Question parsing 2-2 includes question fasttext similarity calculation 2-2-1, question character comparison 2-2-2, and question semantic element query 2-2-3 to parse the question from characters to semantics. Answer generation 2-3 includes two modules: question result calculation 2-3-1 and answer assembly 2-3-2. These modules process the data and assemble the answer according to a template, ultimately producing a complete answer.

[0124] Application layer 3 includes two modules: question input 3-1 and answer output 3-2, which are used to implement question input and final answer presentation on the interface.

[0125] By employing the embodiments of the present invention, the following beneficial effects are reported:

[0126] This patent overcomes the challenges of interpreting ambiguities in questions and calculating multi-level cross-boundary nodes by directly constructing question-and-answer responses for complex engineering forms, leveraging the inherent meaning and complex structure of these forms. It rewrites complex forms into natural language text with a hierarchical structure, overcoming the difficulty of standardizing complex table structures for database tasks. Furthermore, through the FastText question parsing model, it obtains multi-attribute solutions for general questions by comparing approximate solutions, greatly satisfying engineering needs. In practice, question-and-answer responses were applied to 13 engineering forms at an oilfield production site, achieving an accuracy rate of 98%, meeting the engineering requirements for data accuracy.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A question-answering method based on tabular semantic fasttext question parsing, characterized in that, include: S1. Data preparation stage: Rewrite the table into a hierarchical semantic text, obtain the parsing dictionary for question parsing, establish a question-answering annotation corpus and establish a domain-related fasttext similarity calculation model to establish the data required for the question-answering method of table semantic fasttext question parsing; The step S1 of rewriting the table into hierarchical semantic text specifically includes: The daily report table is converted into a hierarchical text knowledge base based on the daily report template. The daily report template realizes the splitting of the table structure. The text knowledge base contains four levels: words, constructed sentences, sub-table names, and table names, which correspond to the four levels of words, sentences, chapters, and articles in natural language, respectively. The establishment of the question-and-answer annotation corpus in step S1 specifically includes: A question-and-answer annotation corpus is formed by collecting all known questions and the correct parsing results of the corresponding questions after manual proofreading. The question-and-answer annotation corpus is continuously formed in an incremental manner. The result of automatic parsing of each new question is used as a reference, and then manually proofread to obtain the final correct parsing result. At the same time, all related parsing dictionaries are added. S2, Question Parsing Stage: The question is parsed using the parsing dictionary, and the parsed question is semantically parsed with the question-and-answer annotation corpus based on the fasttext similarity calculation model; S3. Answer Acquisition Stage: Perform calculations based on the semantically parsed question to construct a complete answer.

2. The method according to claim 1, characterized in that, The step S1 of obtaining the parsing dictionary for question parsing specifically includes: Obtain a dictionary of table names for passage and chapter-level parsing used for question parsing. The table name dictionary is manually configured with table names based on business needs. The table names are determined by the specific questions. The table name dictionary is updated according to the specific questions. The system acquires a colloquial dictionary, an object dictionary, an object attribute dictionary, an attribute value dictionary, a time dictionary, and an interrogative word dictionary for word-level parsing of questions. The colloquial dictionary converts colloquial words in questions into standard names. The object dictionary corresponds to the indexes in the table and represents semantic objects or hierarchical objects. The object attribute dictionary collects the content of all horizontal attributes in the table, and the attribute value dictionary collects the specific values ​​of all horizontal attributes in the table. The time dictionary describes the general time structure, and the interrogative word dictionary describes the interrogative words appearing in the questions and the methods used to obtain the answers.

3. The method according to claim 1, characterized in that, The establishment of the domain-related fasttext similarity calculation model in step S1 specifically includes: The input question and output table name, object, and attribute are digitized and tensed using a v2w approach. Then, a multi-level neural network mapping relationship is established between the input tensor and the output attribute tensor to obtain the parsing parameters of the question.

4. The method according to claim 1, characterized in that, Step S2 specifically includes: Based on the principle of best matching of question characters, the reference parsing result is obtained by calculating the similarity between the question and the known questions in the question-answer annotation corpus.

5. A question-answering system based on tabular semantic fasttext question parsing, characterized in that, include: The data module is used to rewrite tables into hierarchical semantic text, obtain parsing dictionaries for question parsing, build question-answering labeled corpora, and establish domain-related fasttext similarity calculation models to build the data required for question-answering methods of table semantic fasttext question parsing; The data module specifically includes: The report knowledge base module is used to convert the daily report tables into a hierarchical text knowledge base based on the daily report template. The daily report template realizes the splitting of the table structure. The text knowledge base contains four levels: words, constructed sentences, sub-table names, and table names, which correspond to the four levels of words, sentences, chapters, and articles in natural language, respectively. The dictionary module is used to obtain dictionaries for word, text, and chapter-level parsing of questions; The corpus module is used to collect all known questions and the correct parsing results of the corresponding questions after manual proofreading to form a question-answer labeled corpus. The question-answer labeled corpus is continuously formed in an incremental manner. The result of automatic parsing of each new question is used as a reference, and then manually proofread to obtain the final correct parsing result; at the same time, all related parsing dictionaries will be added. The fasttext module is used to digitize and tensor the input question and the output table name, object, and attribute in a v2w manner. Then, a multi-level neural network mapping relationship is established between the input tensor and the output attribute tensor to obtain the parsing parameters of the question. The processing module is used to parse the question using the parsing dictionary and then perform semantic parsing on the question based on the fasttext similarity calculation model; The application module is used to obtain the answer sentence from the question and answer annotation corpus based on the semantically parsed question sentence, and to assemble the answer sentence to obtain the complete answer sentence.

6. The system according to claim 5, characterized in that, The dictionary module specifically includes: The table name dictionary is used for chapter and section-level parsing of questions. The table name dictionary is manually assigned table names according to business needs. The table names are determined by the specific questions. The table name dictionary is updated according to the specific questions. Obtain the spoken language dictionary, object dictionary, object attribute dictionary, attribute value dictionary, time dictionary, and interrogative word dictionary for word-level parsing of question sentences. A colloquial dictionary is used to convert colloquial words in questions to their standard names. The object dictionary corresponds to the index in the table, and is a semantic object or an object with a hierarchy; The object attribute dictionary collects the contents of all horizontal attributes within the table. The attribute value dictionary collects the specific values ​​of all horizontal attributes in the table; The time dictionary is used to describe a general time structure; The interrogative word dictionary is used to describe the interrogative words that appear in a question and the methods used to obtain the answer.

7. The system according to claim 6, characterized in that, The processing module is specifically used for: Based on the principle of best matching of question characters, the reference parsing result is obtained by calculating the similarity between the question and the known questions in the question-answer annotation corpus.

Images