Table term-oriented multi-modal structured extraction and answer aggregation method and system

Abstract

The invention relates to the technical field of document image understanding and information extraction, and provides a multi-modal structured extraction and answer aggregation method and system oriented to table terms. According to the method, through the steps of table detection and structure reconstruction, semantic role labeling, semantic alignment and cross-table aggregation, condition cutting, consistency checking and answer generation, accurate extraction of parameters-unit-threshold-conditions in a regulation table, cross-table alignment and checkable answer generation are realized; and the analysis accuracy and the result credibility of the complex table are effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of document image understanding and information extraction technology, specifically relating to a multimodal structured extraction and answer aggregation method and system for table clauses. Background Technology

[0002] In the field of intelligent processing of regulations, standards, and appendices, tables are crucial carriers of key parameters, units, thresholds, and applicable conditions. Existing technologies typically rely on general table parsing methods, combining optical character recognition (OCR) and layout analysis to recover the table structure, and utilizing rule-based or statistical learning models for information extraction. However, these methods suffer a significant drop in parsing accuracy when dealing with complex structures such as merged cells, hierarchical headers, and continuous tables spanning multiple pages, making it difficult to accurately establish semantic relationships between parameters and their corresponding units and thresholds.

[0003] Furthermore, existing methods lack indicator ontology and unit dimension systems specific to professional fields (such as power and engineering), failing to effectively address semantic alignment and unit normalization issues across tables and appendices. This leads to semantic fragmentation and numerical conflicts when integrating multi-source information. In the question-and-answer generation stage, existing technologies generally lack a deep understanding of the query context and the ability to tailor conditions, resulting in redundant and insufficiently targeted results. Additionally, the lack of standardized evidence citation mechanisms leads to poor traceability and verifiability of the generated answers, making it difficult to meet the needs of high-reliability application scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of inaccurate structure restoration, difficulty in semantic alignment, weak cross-table integration capability, and unverifiable results in existing technologies when parsing complex regulatory tables. It provides a multimodal structured extraction and answer aggregation method and system for table clauses. This method aims to achieve accurate extraction of "parameter-unit-threshold-applicable conditions" from tables, semantic and unit alignment across tables / appendices, intelligent condition pruning based on query context, and ultimately generate verifiable answers with precise citations and consistent numerical values.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows: This invention provides a multimodal structured extraction and answer aggregation method for table-based clauses, comprising the following steps: Step S101, Table Detection and Structure Reconstruction: Detect table regions and cells in the input document page image, construct a cell adjacency graph, and restore the standard grid structure of the table based on integer linear programming optimization, and parse the hierarchical path of the table header tree and the list header tree. Step S102, Semantic Role Labeling: Normalize the text in the cell and predict its semantic role label based on the sequence labeling model. Use graph matching to associate parameters, units, thresholds and conditions belonging to the same record to generate a structured record containing source metadata. Step S103, Semantic Alignment and Cross-Table Aggregation: Map the parameter text and table header path to vectors, perform concept alignment with the pre-built indicator ontology to obtain a semantic alignment score; use the pre-built unit dimension graph to perform unit consistency verification, and unify the values ​​to the target unit through the unit conversion chain; aggregate records that are semantically equivalent and have consistent units, and fuse the semantic alignment score, detection confidence and cross-source consistency to calculate the aggregation confidence. Step S104, Condition trimming: Parse the context of the user query, convert it into a set of predicate conditions and calculate the condition satisfaction; use a lightweight machine learning model to predict the probability of a record being selected; based on the condition satisfaction and selection probability, filter by a dual threshold to retain candidate records that meet the conditions. Step S105, Consistency Verification and Answer Generation: Calculate the numerical consistency index of candidate records from different sources. If the consistency is lower than a preset threshold, a conflict alarm is triggered. Construct a set of candidate answers and score them using a comprehensive scoring function. The scoring function integrates query element coverage, condition satisfaction, aggregation confidence, consistency score, and logical penalty items. Select the candidate answer with the highest score and output a verifiable answer with table number, row and column path, appendix, or clause location information according to a standardized citation template.

[0006] Furthermore, step S101 specifically includes: Use a deep learning-based detection model to obtain the bounding boxes of tables and cells; The Hough transform is applied to extract a set of lines from the document page image, which serves as the geometric constraint for reconstructing the table grid. With the goal of minimizing the cost of selecting adjacent edges of cells, under the constraints of consistent row and column alignment, no cell overlap, and closed grid boundaries, the row span and column span of merged cells are restored by solving the integer linear programming problem, and the row header tree and column header tree are generated by parsing.

[0007] Furthermore, step S102 specifically includes: The sequence labeling model is a conditional random field or a softmax classifier, and its input features include the text content of the cell and its row and column path encoding in the row header tree and the list header tree; The structured record The format is defined as follows: ; in, For indicator items; Unit; This represents a threshold operator paired with a numeric pair; For applicable conditions; Metadata includes at least the table number, the row and column paths of the cells, and the clause number to which they belong.

[0008] Furthermore, in step S103: The semantic alignment score is calculated by combining the cosine similarity between the parameter embedding vector and the indicator ontology concept embedding vector, as well as the matching result based on the alias mapping rule. The unit consistency check determines whether the dimension vector of the unit in the record is consistent with the dimension vector of the target unit, and allows numerical conversion and aggregation only when they are consistent.

[0009] Furthermore, in step S103, the aggregation operators used for cross-table aggregation include minimum value, maximum value, average value, or voting; the aggregation confidence is obtained by weighted summation of the semantic alignment score, detection confidence, and cross-source consistency, followed by compression using the Sigmoid function.

[0010] Furthermore, in step S104: Condition satisfaction The calculation formula is: ; in, For the first Predicate weights; Indicates in scene metadata Does the following predicate hold true? For indicator functions; The dual-threshold filtering refers to retaining only those that meet the criteria. and The records, among which and This is a preset threshold.

[0011] Furthermore, in step S105, the numerical consistency index The calculation formula is: ; in, For the same concept and under the same conditions, the first The value of each record; To prevent the stability constant from having a denominator of zero; when The system triggers conflict alarms and verifies the consistency between the upper and lower limit logical relationships and the dimensionality.

[0012] Furthermore, in step S105, the answer... In the query The comprehensive scoring function below for: ; in, To query a set of elements; For the set of answer elements; Measured for feature coverage; Aggregate confidence levels; This is a logical constraint penalty term; These are the weighting coefficients.

[0013] Furthermore, in step S105, the format of the referenced template is standardized. for: ; in, Number the tables; These are the row and list header paths, respectively. Appendix number; D represents the clause number; The generated answers will include unit conversion instructions and conflict warnings.

[0014] The present invention also provides a multimodal structured extraction and answer aggregation system for implementing the method, comprising: The document input module is used to receive document images and layout data; The table detection and reconstruction module is used to perform table detection, grid reconstruction and table header parsing, and its input is connected to the output of the document input module. The semantic extraction module is used for text normalization, role labeling and record generation, and its input is connected to the output of the table detection and reconstruction module. The cross-table alignment module is used for semantic alignment, unit conversion and record aggregation, and its input is connected to the output of the semantic extraction module. The conditional pruning module is used for condition parsing and record filtering, and its input is connected to the output of the cross-table alignment module. A consistency verification module is used for consistency calculation and anomaly detection, and its input is connected to the output of the conditional pruning module. The answer generation module is used for answer scoring, citation generation, and final output, and its input is connected to the output of the consistency verification module.

[0015] Compared with the prior art, the advantages of this invention are as follows: (1) High parsing accuracy: Through the multimodal fusion method of "visual inspection + grid reconstruction + graph optimization", the complex table structure is explicitly restored, which effectively solves the parsing problem of merged cells and hierarchical headers and ensures the accuracy of key information association.

[0016] (2) Strong semantic integration capability: The introduction of the “indicator ontology + unit dimension diagram” mechanism realizes semantic unification and unit consistency processing across tables and appendices, fundamentally eliminating information silos and ensuring the reliability and consistency of multi-source data fusion.

[0017] (3) Highly targeted answers: Conditions are tailored based on the query context, so that the generated answers closely match the user's actual scenario, avoiding information redundancy and improving the practical value of the results.

[0018] (4) Results are verifiable and traceable: Standardized evidence citation templates are used to accurately locate the original source of each numerical assertion. Combined with consistency verification and alarm mechanisms, the credibility of the results and the efficiency of audit review are greatly enhanced.

[0019] (5) High degree of automation: The end-to-end processing flow significantly reduces human intervention, thereby reducing the workload of compliance review and the risk of subjective error. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A flowchart of the multimodal structured extraction and answer aggregation method for table-oriented clauses provided by the present invention; Figure 2 The structural block diagram of the multimodal structured extraction and answer aggregation system for table clauses provided by the present invention; Figure 3 This is one of the hardware structure diagrams of the electronic device provided in the embodiments of the present invention; Figure 4 This is the second schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The terms "first," "second," etc., used in this specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] This invention provides a multimodal structured extraction and answer aggregation method for table-based clauses, characterized by the following steps: Step S101, Table Detection and Structure Reconstruction: Detect table regions and cells in the input document page image, construct a cell adjacency graph, and restore the standard grid structure of the table based on integer linear programming optimization, and parse the hierarchical path of the table header tree and the list header tree. Step S102, Semantic Role Labeling: Normalize the text in the cell and predict its semantic role label based on the sequence labeling model. Use graph matching to associate parameters, units, thresholds and conditions belonging to the same record to generate a structured record containing source metadata. Step S103, Semantic Alignment and Cross-Table Aggregation: Map the parameter text and table header path to vectors, perform concept alignment with the pre-built indicator ontology to obtain a semantic alignment score; use the pre-built unit dimension graph to perform unit consistency verification, and unify the values ​​to the target unit through the unit conversion chain; aggregate records that are semantically equivalent and have consistent units, and fuse the semantic alignment score, detection confidence and cross-source consistency to calculate the aggregation confidence. Step S104, Condition trimming: Parse the context of the user query, convert it into a set of predicate conditions and calculate the condition satisfaction; use a lightweight machine learning model to predict the probability of a record being selected; based on the condition satisfaction and selection probability, filter by a dual threshold to retain candidate records that meet the conditions. Step S105, Consistency Verification and Answer Generation: Calculate the numerical consistency index of candidate records from different sources. If the consistency is lower than a preset threshold, a conflict alarm is triggered. Construct a set of candidate answers and score them using a comprehensive scoring function. The scoring function integrates query element coverage, condition satisfaction, aggregation confidence, consistency score, and logical penalty items. Select the candidate answer with the highest score and output a verifiable answer with table number, row and column path, appendix, or clause location information according to a standardized citation template.

[0024] Step S101 specifically includes: Use a deep learning-based detection model to obtain the bounding boxes of tables and cells; The Hough transform is applied to extract a set of lines from the document page image, which serves as the geometric constraint for reconstructing the table grid. With the goal of minimizing the cost of selecting adjacent edges of cells, under the constraints of consistent row and column alignment, no cell overlap, and closed grid boundaries, the row span and column span of merged cells are restored by solving the integer linear programming problem, and the row header tree and column header tree are generated by parsing.

[0025] Step S102 specifically includes: The sequence labeling model is a conditional random field or a softmax classifier, and its input features include the text content of the cell and its row and column path encoding in the row header tree and the list header tree; The structured record The format is defined as follows: ; in, For indicator items; Unit; This represents a threshold operator paired with a numeric pair; For applicable conditions; Metadata includes at least the table number, the row and column paths of the cells, and the clause number to which they belong.

[0026] In step S103: The semantic alignment score is calculated by combining the cosine similarity between the parameter embedding vector and the indicator ontology concept embedding vector, as well as the matching result based on the alias mapping rule. It should be noted that the indicator ontology is a conceptual system built on domain knowledge (such as power and engineering standards), which includes standardized indicator names, synonyms, abbreviations and their semantic relationships.

[0027] The unit consistency check determines whether the dimension vector of the unit in the record is consistent with the dimension vector of the target unit, and allows numerical conversion and aggregation only when they are consistent.

[0028] It should be noted that the unit dimension diagram defines the basic dimensions under the International System of Units (SI) (such as length L, mass M, time T, etc.) and their derived relationships, and stores conversion factors between common units, for example, 1kW = 1000W. The aggregation operators used in the cross-table aggregation include minimum value, maximum value, average value or vote; the aggregation confidence is obtained by weighted summation of the semantic alignment score, detection confidence and cross-source consistency, and then compressed by the Sigmoid function.

[0029] In step S104: Condition satisfaction The calculation formula is: ; in, For the first Predicate weights; Indicates in scene metadata Does the following predicate hold true? For indicator functions; The dual-threshold filtering refers to retaining only those that meet the criteria. and The records, among which and This is a preset threshold.

[0030] In step S105, the numerical consistency index The calculation formula is: ; in, For the same concept and under the same conditions, the first The value of each record; To prevent the stability constant from having a denominator of zero; when The system triggers conflict alarms and verifies the consistency between the upper and lower limit logical relationships and the dimensionality.

[0031] Answer In the query The comprehensive scoring function below for: ; in, To query a set of elements; For the set of answer elements; Measured for feature coverage; Aggregate confidence levels; This is a logical constraint penalty term; These are the weighting coefficients.

[0032] Standardize the format of the template reference for: ; in, Number the tables; These are the row and list header paths, respectively. Appendix number; D represents the clause number; The generated answers will include unit conversion instructions and conflict warnings.

[0033] Combined Figure 2 As shown, the present invention also provides a multimodal structured extraction and answer aggregation system for implementing the method, which includes a document input module 1, a table detection and reconstruction module 2, a semantic extraction module 3, a cross-table alignment module 4, a conditional pruning module 5, a consistency verification module 6, and an answer generation module 7.

[0034] The document input module 1 is used to receive document images and layout data.

[0035] The table detection and reconstruction module 2 is used to perform table detection, grid reconstruction and table header parsing, and its input is connected to the output of the document input module.

[0036] The semantic extraction module 3 is used for text normalization, role labeling and record generation, and its input is connected to the output of the table detection and reconstruction module.

[0037] The cross-table alignment module 4 is used for semantic alignment, unit conversion and record aggregation, and its input is connected to the output of the semantic extraction module.

[0038] The conditional clipping module 5 is used for conditional parsing and record filtering, and its input is connected to the output of the cross-table alignment module.

[0039] The consistency verification module 6 is used for consistency calculation and anomaly detection, and its input is connected to the output of the conditional pruning module.

[0040] The answer generation module 7 is used for answer scoring, citation generation and final output, and its input is connected to the output of the consistency verification module.

[0041] The preferred embodiments of the present invention will now be described in detail. The specific details described herein are for ease of understanding and not intended to limit the scope of protection of the invention.

[0042] See Figure 1This invention provides a multimodal structured extraction and answer aggregation method for table clauses. The overall process includes the following steps: table detection and structure reconstruction (step S101), semantic role labeling (step S102), semantic alignment and cross-table aggregation (step S103), condition trimming (step S104), consistency verification and answer generation (step S105). Each step is described in detail below.

[0043] 1. Document Input and Preprocessing The system receives document page images input from document input module 1. Optionally, layout analysis results and text content can be received simultaneously or obtained through an OCR engine as supplementary multimodal information. The document input module is responsible for receiving this data uniformly and passing it to subsequent processing modules.

[0044] 2. Table Detection and Structure Reconstruction This step is performed by the table detection and reconstruction module 2, whose core task is to accurately locate the table from the document image and restore its logical structure.

[0045] Table and cell detection: Utilizing deep learning detectors based on DETR or Mask R-CNN architectures for document image detection. Perform inference and output the outer rectangle (table frame) of the table and the bounding boxes (cell frames) of all logical cells inside. Each detected target (table and cell) is accompanied by a detection confidence score.

[0046] Geometric prior extraction: To enhance robustness under complex layouts, the image... Apply the Hough transform to extract the candidate line set. , serving as a strong geometric constraint for the table grid lines.

[0047] Structure Reconstruction and Mesh Recovery: Constructing an Adjacency Graph from Detected Cells , where the node set V For each cell, edge set E This indicates the adjacency or row / column alignment relationships between cells. The grid structure is optimized by constructing and solving an integer linear programming (ILP) problem: ; The constraints include: Consistent row and column alignment: Cells belonging to the same row or column should be aligned.

[0048] No cell overlap: The bounding boxes of any two cells must not overlap.

[0049] Closed grid boundaries: The table should form a complete rectangular grid.

[0050] in, The 0 / 1 decision variable vector for edge selection; For the edge The cost (usually related to cell spacing and alignment errors); To select an edge The indicator variable is [the variable name]. By solving this ILP problem, the system can accurately recover the row span and column span of each cell.

[0051] Header hierarchy parsing: Based on the geometric containment relationship between cells (i.e., the bounding box of one cell completely contains the bounding box of another cell in the image coordinate system, and there is row and column alignment), a row header tree is generated. and list header tree Each data cell Each can obtain its unique line path. and column paths ,in, It is the first Layered table header node, It is the first Layer list header node.

[0052] 3. Semantic role labeling This step is performed by the semantic extraction module 3, which aims to extract structured "parameter-unit-threshold-condition" tuples from the reconstructed table.

[0053] Text normalization and segmentation: The text within each cell is processed according to rules, including unifying full-width / half-width characters, standardizing numbers and special symbols (such as "≥", "≤"), and converting subscripts and superscripts (such as "m²" to "m2"). Subsequently, the text is initially segmented into three segments: "Value", "Unit", and "Comment".

[0054] Role label prediction: Combining the cell's path encoding (i.e., row and column path information) with the normalized text content, a feature representation of the cell is formed. This feature is then fed into a Conditional Random Field (CRF) or Softmax classifier to predict its semantic role label. : ; in, For the first Predicted role labels for each cell. Iterate through the candidate role set {PARAM, UNIT, THRESH, COND, OTHER}; and These are the classifier weights and biases, respectively. Indicates the character The normalized probability score. Structured record generation: Based on the predicted roles, a graph matching algorithm is used to associate the "parameters (PARAM)," "units (UNIT)," "thresholds (THRESH)," "conditions (COND)," and "others (OTHER)" belonging to the same record, assembling them into a structured record. : ; in, For indicator items; Unit; This represents the threshold operator and numeric pairs (e.g.) ); For applicable conditions; The metadata field records the table ID (TableID) from which the record originated, and the row and column paths of the cell (…). , (and the relevant clause number, ClauseID, etc.)

[0055] 4. Semantic alignment and cross-table aggregation This step is performed by the cross-table alignment module 4, which aims to unify and integrate information across different tables or appendices.

[0056] Semantic alignment: For the parameter item PARAM in a record, its text content is aligned with its complete header path ( and ) splicing, through embedded functions (Pre-trained models such as BERT) are mapped to dense vectors. This vector is then mapped to all concept embeddings in the pre-built metric ontology. cosine similarity : .

[0057] The indicator ontology is a conceptual system built upon domain knowledge (such as power and engineering standards), containing standard indicator names, aliases, abbreviations, etc. Combined with predefined alias mapping rules, the final aligned standard concepts are determined. And calculate the semantic alignment score. : .

[0058] Unit normalization: Query a pre-built unit dimension map to retrieve the units in the records. dimensional vector and target unit dimensional vector The unit dimension diagram defines the dimensional system under the International System of Units (SI) and the conversion relationships between units. If Then refuse to aggregate; only if Numerical conversions are only allowed under certain conditions. This is achieved through a predefined conversion chain. Value Values ​​converted to the target unit The overall conversion factor The conversion chain is the product of the conversion factors for each intermediate unit along the link, as shown below: ; in, This is to convert the intermediate unit sequence on the link; Indicates multiplication.

[0059] Cross-table aggregation: Aggregating records from different tables or appendices that are aligned to the same ontology concept, have consistent units, and are comparable under comparable conditions. ; in, The representative value after aggregation; aggregation operator The selection can be tailored to the specific scenario, such as taking the minimum (min), maximum (max), average (mean), or voting. Simultaneously, semantic alignment scores are integrated. Confidence level during the detection phase and cross-source consistency metrics The overall confidence level of this aggregation is calculated using compression functions such as Sigmoid. : ; in, , , These are the weighting coefficients.

[0060] 5. Conditional clipping This step is performed by the conditional cropping module 5, which is used to filter the most relevant records based on the context of the user's query.

[0061] Query parsing: Using a fine-tuned sequence-to-sequence (Seq2Seq) model or semantic parser, the user's natural language query q is parsed to identify the implicit predicate conditions (such as "voltage level = 10kV", "ambient temperature > 25℃"), forming a predicate set. And assign weights to each predicate. .

[0062] Condition satisfaction calculation: combining the query scenario metadata (This can be parsed from the query or provided by the user), calculate the satisfaction level of the condition field `cond` in each record. : ; in, For the first Predicate weights; Indicates in scene metadata Does the following predicate hold true? This is an indicator function (1 if true, 0 otherwise).

[0063] Learning-based selection probabilities: Using a lightweight machine learning model (such as a multilayer perceptron (MLP) or gradient boosting decision tree (GBDT) based on records. Features (such as param, unit, cond, etc.) predict the probability that the record will be selected by the user. .

[0064] Dual threshold filtering: setting a satisfaction threshold (e.g., 0.7) and selection probability threshold (e.g., 0.5), only retain those that simultaneously satisfy the condition. and The records are then entered into the subsequent answer candidate pool.

[0065] 6. Consistency Check and Alarm This step is performed by consistency verification module 6 to ensure the reliability of cross-source data.

[0066] Consistency calculation: For multiple records that point to the same indicator concept 'o' and are comparable under the same conditions, calculate the consistency index of their values. : ; in, The consistency score (the higher the score, the more consistent the score). For the same concept and under the same conditions, the first The value of each record; It is a small positive number, used to prevent the denominator from being zero; This is the median normalization term.

[0067] Conflict alarm and logic verification: When Cons is lower than the preset threshold When the threshold is 0.8 (e.g., 0.8), the system triggers a conflict alarm. Furthermore, the logical relationship between thresholds is validated (e.g., ensuring the upper limit of the same metric is not less than the lower limit). If a logical contradiction is found, aggregation is rejected and an anomaly is flagged.

[0068] 7. Answer aggregation, scoring, and generation This step is executed by the answer generation module 7 and is the final output stage of the process.

[0069] Answer scoring: Within the candidate record pool, based on the user's target concept. Construct a set of candidate answers Calculate a comprehensive score for each candidate answer. : ; in, For the answer In the query The overall score below; This is a set of answer / query elements (including equipment, voltage level, environmental constraints, desired units, etc.). For feature coverage and matching metrics; For condition satisfaction; Aggregate confidence levels; Consistency score; This is a logical constraint penalty term; The weighting coefficients can be set manually or calibrated using labeled data and optimization algorithms.

[0070] Answer selection and output: Select the candidate with the highest score as the final answer. : When outputting answers, strictly follow the standardized citation template and attach citation tags. : ;in, For table number; These are the row and list header paths, respectively. Appendix number.

[0071] Additional information: If the numerical values ​​in the answer have undergone unit conversion, please indicate the original unit and conversion factor in parentheses (e.g., "340 GW (original unit: kW, conversion factor: 1 GW = 10^6 kW)"). If a conflict alarm is triggered, please append "(Data Inconsistency Warning)" to the end of the answer.

[0072] System Implementation and Training See Figure 2 The system used to implement the above methods includes a document input module 1, a table detection and reconstruction module 2, a semantic extraction module 3, a cross-table alignment module 4, a conditional pruning module 5, a consistency check module 6, and an answer generation module 7. These modules are connected sequentially to form an end-to-end processing pipeline.

[0073] The deep learning models involved in the system (detection, reconstruction, character annotation, alignment, cropping, etc.) are trained using weakly supervised data, synthetic data, and manually annotated data. The total loss function L is the weighted sum of the losses for each task: ; in, To detect the relevant losses; For grid reconstruction loss; Label the loss for the role; This is the loss due to the body alignment. Conditional trimming loss; represents the weighting coefficient for each loss.

[0074] Example Effects Through the above steps, this invention achieves accurate extraction and reliable generation of "parameters, units, thresholds, and applicable conditions" in regulatory tables. Compared with existing technologies, this invention can guarantee cross-table consistency, scenario relevance, and verifiability of results, providing reliable support for engineering, compliance, and audit retrieval.

[0075] like Figure 3 As shown, this embodiment of the invention also provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described multimodal structured extraction and answer aggregation method embodiment for table clauses and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0076] It should be noted that the first electronic device in the embodiments of the present invention includes the mobile electronic device and the non-mobile electronic device described above.

[0077] Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present invention.

[0078] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0079] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0080] It should be understood that, in this embodiment of the invention, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understandable that the aforementioned modem processor may not be integrated into the processor 710.

[0081] This invention also provides a readable storage medium storing a program or instructions. When executed by a processor, the program or instructions implement the various processes of the above-described multimodal structured extraction and answer aggregation method embodiments oriented towards table clauses, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0082] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0083] This invention also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described multimodal structured extraction and answer aggregation method embodiments oriented towards table clauses, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0084] It should be understood that the chip mentioned in the embodiments of the present invention may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0085] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0086] Furthermore, it should be noted that the scope of the methods and systems in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0087] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the scope of protection of the present invention, and all of these forms are within the protection scope of the present invention.

Claims

1. A multimodal structured extraction and answer aggregation method for table-based clauses, characterized in that, Includes the following steps: Step S101, Table Detection and Structure Reconstruction: Detect table regions and cells in the input document page image, construct a cell adjacency graph, and restore the standard grid structure of the table based on integer linear programming optimization, and parse the hierarchical path of the table header tree and the list header tree. Step S102, Semantic Role Labeling: Normalize the text in the cell and predict its semantic role label based on the sequence labeling model. Use graph matching to associate parameters, units, thresholds and conditions belonging to the same record to generate a structured record containing source metadata. Step S103, Semantic Alignment and Cross-Table Aggregation: Map the parameter text and table header path to a vector, perform concept alignment with the pre-built indicator ontology to obtain a semantic alignment score; use the pre-built unit dimension diagram to perform unit consistency verification, and unify the values ​​to the target unit through the unit conversion chain; Records that are semantically equivalent and have consistent units are aggregated, and the semantic alignment score, detection confidence and cross-source consistency are fused to calculate the aggregation confidence. Step S104, Condition trimming: Parse the context of the user query, convert it into a set of predicate conditions and calculate the condition satisfaction; use a lightweight machine learning model to predict the probability of a record being selected; based on the condition satisfaction and selection probability, filter by a dual threshold to retain candidate records that meet the conditions. Step S105, Consistency Verification and Answer Generation: Calculate the numerical consistency index of candidate records from different sources. If the consistency is lower than a preset threshold, a conflict alarm is triggered. Construct a set of candidate answers and score them using a comprehensive scoring function. The scoring function integrates query element coverage, condition satisfaction, aggregation confidence, consistency score, and logical penalty items. Select the candidate answer with the highest score and output a verifiable answer with table number, row and column path, appendix, or clause location information according to a standardized citation template.

2. The method according to claim 1, characterized in that, Step S101 specifically includes: Use a deep learning-based detection model to obtain the bounding boxes of tables and cells; The Hough transform is applied to extract a set of lines from the document page image, which serves as the geometric constraint for reconstructing the table grid. With the goal of minimizing the cost of selecting adjacent edges of cells, under the constraints of consistent row and column alignment, no cell overlap, and closed grid boundaries, the row span and column span of merged cells are restored by solving the integer linear programming problem, and the row header tree and column header tree are generated by parsing.

3. The method according to claim 1, characterized in that, Step S102 specifically includes: The sequence labeling model is a conditional random field or a softmax classifier, and its input features include the text content of the cell and its row and column path encoding in the row header tree and the list header tree; The structured record The format is defined as follows: ； in, For indicator items; As a unit; This represents a threshold operator paired with a numeric pair; For applicable conditions; Metadata includes at least the table number, the row and column paths of the cells, and the clause number to which they belong.

4. The method according to claim 1, characterized in that, In step S103: The semantic alignment score is calculated by combining the cosine similarity between the parameter embedding vector and the indicator ontology concept embedding vector, as well as the matching result based on the alias mapping rule. The unit consistency check determines whether the dimension vector of the unit in the record is consistent with the dimension vector of the target unit, and allows numerical conversion and aggregation only when they are consistent.

5. The method according to claim 1, characterized in that, In step S103, the aggregation operators used for cross-table aggregation include minimum value, maximum value, average value or voting; the aggregation confidence is obtained by weighted summation of the semantic alignment score, detection confidence and cross-source consistency, and then compressed by the Sigmoid function.

6. The method according to claim 1, characterized in that, In step S104: Condition satisfaction The calculation formula is: ； in, For the first Predicate weights; Indicates in scene metadata Does the following predicate hold true? For indicator functions; The dual-threshold filtering refers to retaining only those that meet the criteria. and The records, among which and This is a preset threshold.

7. The method according to claim 1, characterized in that, In step S105, the numerical consistency index The calculation formula is: ； in, For the same concept and under the same conditions, the first The value of each record; To prevent the stability constant from having a denominator of zero; when The system triggers conflict alarms and verifies the consistency between the upper and lower limit logical relationships and the dimensionality.

8. The method according to claim 1, characterized in that, In step S105, the answer is... In the query The comprehensive scoring function below for: ； in, To query a set of elements; For the set of answer elements; Measured for feature coverage; Aggregate confidence levels; This is a logical constraint penalty term; These are the weighting coefficients.

9. The method according to claim 1, characterized in that, In step S105, the format of the referenced template is standardized. for: ； in, Number the tables; These are the row and list header paths, respectively; Appendix number; D represents the clause number; The generated answers will include unit conversion instructions and conflict warnings.

10. A multimodal structured extraction and answer aggregation system for implementing the method of any one of claims 1 to 9, characterized in that, include: The document input module is used to receive document images and layout data; The table detection and reconstruction module is used to perform table detection, grid reconstruction and table header parsing, and its input is connected to the output of the document input module. The semantic extraction module is used for text normalization, role labeling and record generation, and its input is connected to the output of the table detection and reconstruction module. The cross-table alignment module is used for semantic alignment, unit conversion and record aggregation, and its input is connected to the output of the semantic extraction module. The conditional pruning module is used for condition parsing and record filtering, and its input is connected to the output of the cross-table alignment module. A consistency verification module is used for consistency calculation and anomaly detection, and its input is connected to the output of the conditional pruning module. The answer generation module is used for answer scoring, citation generation, and final output, and its input is connected to the output of the consistency verification module.

Images