A document structuring method based on multi-modal data fusion and dynamic rule driving and a storage medium

Abstract

The present application relates to the technical field of document processing, and more particularly to a document structuring method based on multi-modal data fusion and dynamic rule driving and a storage medium, which converts multi-modal document layout understanding results into unified structured page information, automatically generates extraction strategies based on the structural patterns in the page information, drives differentiated executable workflows based on the extraction strategies, and adaptively generates hierarchical output control instructions based on the comparison results of quality evaluation results and preset thresholds, thereby reducing rule maintenance costs, improving the dynamic adaptation capability of complex document layouts, and ensuring data lake quality.

Description

Technical Field

[0001] This invention relates to the field of intelligent document processing technology, specifically to a document structuring method based on multimodal data fusion and dynamic rule-driven approach. Background Technology

[0002] As enterprises deepen their digital transformation, a large number of unstructured documents need to be automatically parsed into structured data and written into data warehouses or data lakes to support downstream business analysis, audit traceability, and intelligent decision-making. Currently, the mainstream technical approaches for document structuring include the following: Schemes based on OCR and fixed templates extract fields from pre-labeled areas after recognizing text and coordinates using OCR, and then employ regular expressions, XPath, or coordinate cropping. This approach has been applied in scenarios with relatively fixed layouts, such as bank receipts and invoices. Schemes based on multimodal fusion models, such as CN119227794B, construct knowledge graphs by fusing text and image features and using BERT for semantic analysis; CN114419304B and WO2023138023A1 obtain text blocks and coordinates through OCR and extract text, visual, and layout features, fusing them into graph nodes for node classification and link prediction; CN118260390A combines OCR, layout analysis, and prompt word engineering to extract key information and return the original text location. Schemes based on large language model prompt words extract key information through natural language question answering using prompt word engineering; some schemes further combine vector retrieval enhancement generation techniques.

[0003] The existing technical solutions mentioned above have the following shortcomings: First, existing layout analysis mostly stays at the level of region identification or feature fusion, and does not uniformly model the spatial association, semantic association, hierarchical structure and cross-page association of each element in the layout into a structured representation that can be directly called by downstream tasks. When the document layout changes, it needs to be manually re-annotated, and there is an engineering gap between the output form of the multimodal fusion model and rule extraction and process arrangement. Second, rule templates, regular expressions, prompt words and schemas need to be manually written and maintained for each layout. When the document type or field naming changes, it needs to be reconfigured, and the maintenance cost increases with the increase of document types. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a document structuring method and storage medium based on multimodal data fusion and dynamic rule-driven approach.

[0005] In a first aspect, embodiments of the present invention provide a document structuring method based on multimodal data fusion and dynamic rule-driven approach, the method comprising: The multimodal document is parsed to extract the attribute information of each page element and the relationship between page elements, and to construct a unified structured page information. The structural patterns in the page information are identified and summarized, and extraction strategies are dynamically generated based on the summarization and analysis results. The extraction strategies are used to locate the target business fields from the page information and extract the field values. The extraction strategy is subjected to dependency resolution and task orchestration to generate an executable workflow containing multiple processing nodes and the dependencies between them. Execute the executable workflow, drive each processing node to run according to the dependency relationship, and obtain the field extraction results and execution status information output by each processing node; Based on the field extraction results, execution status information, and unified structured page information, the field-level confidence level and process-level execution quality are determined. The field-level confidence level is determined based on the reliability of the field value itself and the integrity of the evidence, while the process-level execution quality is determined based on the stability and compliance of the workflow execution process. Based on the comparison results of field-level confidence and process-level execution quality with preset thresholds, hierarchical output control instructions are generated. These instructions are used to write data that meets the threshold conditions into the target storage system and to divert data that does not meet the threshold conditions to manual processing.

[0006] In conjunction with the first aspect, the steps of parsing the multimodal document, extracting the attribute information of each page element and the relationships between page elements, and constructing a unified structured page information include: Retrieve the text content, page coordinates, image features, and type labels of each page element in a multimodal document; the type labels include at least one of the following: title, paragraph, field name, field value, table, cell, image, and page number; Multimodal fusion is performed based on the page coordinates, text content, and image features of each page element to determine the spatial adjacency relationship between each page element; Determine the spatial containment relationship between page elements based on their page coordinates; Determine the hierarchical relationship between page elements based on their type tags and spatial inclusion relationships; Based on the page coordinates, type labels, and text content of each page element, determine the table-side relationships between cells belonging to the same table; Based on the page coordinates and text content of each page element, determine the cross-page succession relationship between elements belonging to different pages but belonging to the same table or paragraph; Based on spatial adjacency, hierarchical affiliation, same-table edge relationships, and cross-page continuity, a unified structured page information is constructed.

[0007] In conjunction with the first aspect, the steps for constructing a unified, structured page information based on spatial adjacency, hierarchical affiliation, relationships within the same table, and cross-page continuity include: A graph structure is constructed using each page element as a node and spatial adjacency, hierarchical affiliation, same-table edge relationship, and cross-page continuity relationship as edges. Use graph structure as a unified structure for page information.

[0008] Combining the first aspect, the steps of identifying and summarizing structural patterns in page information, and dynamically generating extraction strategies based on the summarization and analysis results, include: Based on the type tags of each page element in the unified structured page information, a subgraph structure consisting of at least two nodes and the relationships between nodes is matched in the unified structured page information, and the matched subgraph structure is used as the structural pattern; wherein, the relationships include at least one of spatial adjacency, hierarchical affiliation, same table edge relationship and cross-page continuation relationship; Count the frequency of the same structural pattern appearing in uniformly structured page information; Structural patterns that appear more frequently than a preset threshold are identified as recurring structural patterns. The spatial positional relationship and hierarchical affiliation relationship between nodes in the recurring structural pattern are used as rule conditions, and the extraction method of field value nodes in the recurring structural pattern is used as rule actions. Extraction rules are generated based on the rule conditions and rule actions. The extraction rules are treated as rule execution class processing units to generate a rule extraction strategy consisting of at least one rule execution class processing unit. Based on the type tags and text content of each page element in the unified structured page information, input and output data format specifications and prompt word templates are generated for the target extraction fields indicated by each type tag; the input and output data format specifications and prompt word templates are encapsulated into model inference processing units; and model inference strategies are generated by combining at least one model inference processing unit. The rule extraction strategy and the model inference strategy are used as the extraction strategy.

[0009] In conjunction with the first aspect, the steps for dependency resolution and task orchestration of the extraction strategy include: The processing logic categories of each processing unit in the extraction strategy and the dependencies between each processing unit are analyzed. The processing logic categories include rule execution and model reasoning. For each processing unit, the execution mode is determined based on the processing logic category of the processing unit and the preset field risk level; the execution mode includes at least one of the following: single-path execution mode, dual-path parallel execution mode, and hybrid execution mode. Configure the corresponding processing node for the processing unit according to the execution mode; Configure the dependencies between processing nodes based on the dependencies between processing units; An executable workflow is generated based on the processing nodes and the dependencies between them.

[0010] In conjunction with the first aspect, the steps of executing an executable workflow, driving each processing node to run according to its dependencies, and obtaining the field extraction results and execution status information output by each processing node include: Analyze the executable workflow to determine the processing nodes contained in the executable workflow and the dependencies between the processing nodes; Based on dependencies, schedule each processing node to execute sequentially or in parallel; When executing each processing node, the corresponding processing logic is executed according to the processing logic category corresponding to the processing node to obtain the field extraction results output by the processing node; Record the execution status information after the processing node has finished executing. The execution status information includes at least one of the following: whether the processing node has been successfully executed, execution time, input data integrity, and output data integrity. The field extraction results output by each processing node are summarized into field extraction results, and the execution status information of each processing node is summarized into execution status information.

[0011] In conjunction with the first aspect, based on the field extraction results, execution status information, and unified structured page information, the steps for determining field-level confidence and process-level execution quality include: Based on the field extraction results, execution status information, and unified structured page information, determine at least two dimensions among layout positioning confidence, rule matching confidence, model extraction confidence, cross-validation consistency, field value validity, and historical performance confidence, and perform weighted calculations on at least two dimensions to obtain field-level confidence. Based on the execution status information and field extraction results, at least one dimension of each processing node is determined, including execution status score, input integrity score, output compliance score, and evidence coverage score. The at least one dimension is then weighted to obtain the process-level execution quality.

[0012] In conjunction with the first aspect, the steps for generating hierarchical output control instructions based on the comparison results of field-level confidence and process-level execution quality with preset thresholds include: When the field-level confidence level is greater than or equal to the preset confidence level threshold, and the process-level execution quality is greater than or equal to the preset execution quality threshold, a first control instruction is generated. The first control instruction is used to instruct the field extraction results to be written to the target storage system. When the field-level confidence level is less than the confidence threshold, or the process-level execution quality is less than the execution quality threshold, a second control instruction is generated. The second control instruction is used to instruct the field extraction results to be diverted to manual processing.

[0013] In conjunction with the first aspect, the method also includes: Obtain the results of manual processing; Based on the results of manual processing, update the configuration parameters of the corresponding rules or models in the extraction strategy.

[0014] Secondly, this application also provides a storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0015] The embodiments of the present invention bring the following beneficial effects: The document structuring method and storage medium based on multimodal data fusion and dynamic rule driving provided in this application convert the multimodal document layout understanding results into unified structured page information, and automatically generate extraction strategies based on the structural patterns in the page information. The extraction strategies are used as the driving basis to dynamically arrange differentiated executable workflows, and then hierarchical output control instructions are adaptively generated by comparing the quality evaluation results with preset thresholds. This reduces the rule maintenance cost, improves the dynamic adaptation capability of complex document layouts, and ensures the quality of data entering the lake.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the document structuring method based on multimodal data fusion and dynamic rule-driven approach provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the principle of the method provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] To facilitate understanding of this embodiment, the technical terms used in this application will be briefly introduced below.

[0022] LayoutGraph is a data model that represents the page elements and their relationships in a multimodal document as a graph structure.

[0023] In document layout understanding, if the page elements are not modeled in a structured way, then the relationship between each element needs to be determined. When the number of page elements is N, the number of relationship pairs that need to be processed is N×(N-1) / 2, which means that the number of elements increases quadratically, resulting in the combinatorial explosion problem.

[0024] Skill, or model reasoning ability, is a description of an executable extraction capability for a specific document type, field, or processing scenario.

[0025] Rule Spec is an extraction rule description automatically generated by LayoutGraph.

[0026] A Workflow Spec is a workflow description generated from rules, skills, field risks, and business configurations.

[0027] A Directed Acyclic Graph (DAG) is used to represent dependencies between tasks and is executed by a scheduling system.

[0028] TaskDAG, or Directed Acyclic Graph, is used to describe the specific implementation of the executable workflow consisting of processing nodes and dependencies in this application.

[0029] Markdown (MD) is a lightweight markup format.

[0030] Iceberg, or Apache Iceberg, is an open data lake table format storage technology.

[0031] A schema, or data format specification, is used to define the structure, type, and constraints of data. In this application, the input schema defines the data format received by the processing unit, and the output schema defines the data format output by the processing unit.

[0032] After introducing the technical terms used in this application, the application scenarios and design concepts of the embodiments of this application will be briefly described below.

[0033] Existing technologies suffer from drawbacks such as layout understanding results not being able to directly drive downstream tasks, rule maintenance relying on manual configuration, and fixed and rigid processing procedures.

[0034] Based on this, embodiments of this application provide a document structuring method and a computer storage medium based on multimodal data fusion and dynamic rule-driven approach.

[0035] Example 1 This application provides a document structuring method based on multimodal data fusion and dynamic rule-driven approach, combining... Figure 1 As shown, the method includes: S110: Parse the multimodal document, extract the attribute information of each page element in the multimodal document and the relationship between page elements, and construct a unified structured page information.

[0036] S120 identifies and summarizes the structural patterns in the page information, and dynamically generates an extraction strategy based on the summary analysis results; the extraction strategy is used to locate the target business field in the page information and extract the field value.

[0037] S130 performs dependency resolution and task orchestration on the extraction strategy to generate an executable workflow containing multiple processing nodes and the dependencies between them.

[0038] S140 executes the executable workflow, driving each processing node to run according to its dependencies, and obtaining the field extraction results and execution status information output by each processing node.

[0039] S150, based on the field extraction results, execution status information, and unified structured page information, determine the field-level confidence and process-level execution quality. The field-level confidence is determined based on the reliability of the field value itself and the integrity of the evidence, while the process-level execution quality is determined based on the stability and compliance of the workflow execution process.

[0040] S160 generates hierarchical output control instructions based on the comparison results of field-level confidence and process-level execution quality with preset thresholds. The hierarchical output control instructions are used to write data that meets the threshold conditions into the target storage system and to divert data that does not meet the threshold conditions to manual processing.

[0041] The method provided in this application first parses the multimodal document to construct a unified structured page information containing page elements and their spatial adjacency, hierarchical affiliation, and relationships within the same table and across pages. Then, based on this page information, it dynamically generates an extraction strategy containing rule execution and model reasoning processing units by automatically extracting and summarizing recurring structural patterns. Next, it automatically generates an executable workflow based on this extraction strategy as the arrangement basis. After executing the workflow, it determines the quality evaluation result based on the field extraction results and execution status information output by each processing node. Finally, it generates hierarchical output control instructions based on the matching of the quality evaluation result with preset conditions to achieve the separation control of automatic entry into the lake and manual processing.

[0042] like Figure 2 As shown, the multimodal document is first converted into a LayoutGraph through the layout understanding and graph construction stage. The LayoutGraph, as a unified structured page information, is input into the rule generation channel and the skill generation channel respectively: the rule generation channel outputs a Rule Spec by automatically extracting and summarizing repetitive structural patterns, and the skill generation channel outputs a Skill Spec by automatically configuring input and output schemas and prompt word templates for each type of field. Together, they constitute the extraction strategy. The extraction strategy then enters the workflow orchestration stage, where it is compiled into an executable TaskDAG after dependency resolution and task orchestration. The TaskDAG is executed by the scheduling system according to the node dependencies. During the execution process, each processing node outputs field extraction results and execution status information. This information is then aggregated and input into the quality evaluation stage. Finally, based on the dual scoring results of field-level confidence and process-level execution quality, high-confidence data (i.e., data that meets the threshold conditions) is written to the target storage system, while low-confidence data (i.e., data that does not meet the threshold conditions) is diverted to manual confirmation for further processing.

[0043] In conjunction with the first aspect, step S110 includes: S111, obtain the text content, page coordinates, image features and type labels of each page element in the multimodal document; the type labels include at least one of the following: title, paragraph, field name, field value, table, cell, image, page number.

[0044] S112, perform multimodal fusion based on the page coordinates, text content and image features of each page element to determine the spatial adjacency relationship between each page element.

[0045] S113, Determine the spatial containment relationship between page elements based on the page coordinates of each page element.

[0046] S114. Determine the hierarchical relationship between page elements based on their type tags and spatial inclusion relationships.

[0047] S115, determine the same-table edge relationships between cells belonging to the same table based on the page coordinates, type labels, and text content of each page element.

[0048] S116, Based on the page coordinates and text content of each page element, determine the cross-page continuation relationship between elements belonging to different pages but belonging to the same table or the same paragraph.

[0049] S117. Construct a unified, structured page information based on spatial adjacency, hierarchical affiliation, same-table edge relationships, and cross-page continuity relationships.

[0050] In step S110, the multimodal document can be PDF, image, scanned copy, Word, Excel, form, contract, report, invoice and business document, etc. These document types cover a variety of unstructured or semi-structured documents commonly used in enterprise business.

[0051] Step S111 involves starting a multi-path feature extraction pipeline in parallel after receiving the multimodal document to obtain the basic attributes of each identifiable page element.

[0052] Regarding text content and page coordinates, the system uses an OCR engine to recognize each page image and outputs the character sequence of each text block and its bounding box coordinates (horizontal and vertical values ​​of the top left and bottom right corners). For digital documents containing native text layers, the system extracts the native text and its absolute coordinates, cross-validates them with the OCR results, takes the intersection and compares the highest value as the adoption value, thus balancing the text recall rate of tilted scans with the coordinate accuracy of digital documents.

[0053] Regarding image features, the system uses a visual backbone network to extract deep visual embedding vectors for each detected element region. These vectors capture microscopic page layout attributes that cannot be expressed by plain text, such as font thickness, color depth, underline, background color, seal texture, and handwriting.

[0054] For type label determination, the system assigns a most matching label to each element based on text semantics, visual features, and geometric attributes through a fusion classifier. The candidate set of type labels includes: titles (large font size, bold, centered, and containing subordinate paragraphs), paragraphs (large, continuous blocks of text), field names (short text ending with a colon or a space, serving as the key for data items), field values ​​(short text immediately following the field name, containing specific data), tables (regular matrix areas composed of cells), cells (the basic unit of a table container), images (non-text image areas), and page numbers (purely numerical identifiers at the page edges). The determination comprehensively utilizes keyword semantics (e.g., "total" and "amount" corresponding to field names), visual salience (table line detection response), and absolute position (footer numbers are determined as page numbers).

[0055] Each element is encapsulated as a temporary primitive node, containing the following attributes: pageId (global page ID, identifying the physical page or worksheet number where the element resides), nodeId (local element ID, uniquely identifying the element within the current document), bbox (bounding box coordinates, including four values: x1 (top-left corner), y1 (top-left corner), x2 (bottom-right corner), and y2 (bottom-right corner), text (text content, which can be empty for pure image elements), imageRef (image feature reference, pointing to the deep visual embedding vector), and type (type label). Among these, pageId and nodeId together ensure the element's global addressability; bbox and text provide spatial geometry and textual payloads, respectively; imageRef introduces visual modal information; and type provides semantic category priors. These six attributes are indispensable and together constitute the complete factual description of each primitive node. They will accompany the node as a fixed attribute slot throughout the entire lifecycle of the LayoutGraph. Subsequent relational inference nodes can access all its basic attributes in reverse through nodeId without repeatedly extracting the original data.

[0056] As an example, in the invoice scenario, the system records node 001 as {pageId:1,nodeId:"N001",bbox:(50,100,150,120),text:"Invoice Number",imageRef:[0.23,-0.56...0.78],type:"Field Name"}. This set of six tuples completely and unambiguously defines all the atomic features of the element.

[0057] Step S112 establishes spatial adjacency edges for the scattered elements from step S111 based on their physical proximity and semantic relevance on the page. Specifically: First, coordinate alignment and spatial measurement are performed to unify the bboxes of each element to the same coordinate system. The horizontal center distance, vertical center distance, and vertical projection overlap rate between each pair are calculated, with a focus on the right adjacency (field name - field value) and the bottom adjacency (heading - paragraph first line). The results are used as a candidate set.

[0058] Next, cross-attention image-text feature alignment is performed: the visual embedding vector and text semantic embedding vector of the candidate element pair are concatenated and sent to the cross-attention module to determine whether the visual appearance and text role of the two constitute a natural proximity pair. For example, even if there is a small gap between the short text ending with a bold colon and the short text with no bold data format on the right, if the visual consistency is strong, it is still given high affinity; if the visual styles are very different, the score will be reduced.

[0059] Finally, graph convolution is used to fuse spatial relationships: the graph convolutional network jointly models the local neighborhood structure of all elements on the page, comprehensively considers the relative position distribution and feature similarity between an element and multiple candidate elements around it, and through the propagation and aggregation of information in the graph structure, the initial adjacency value is weighted and corrected to resolve local ambiguity.

[0060] After fusion and determination, the system establishes a directed spatial adjacency edge for each pair of valid adjacent elements (the direction is determined by the relative orientation, such as right adjacency), and includes it in the LayoutGraph edge set. Each edge is accompanied by a weight (0~1) and an evidence field (recording key clues that support the determination, such as coordinate distance, cross attention alignment score, graph convolution aggregation correction value, etc.).

[0061] Step S113 iterates through all elements output in S111 and determines the containment relationship based on their respective bounding box coordinates: if the bounding box of element A completely contains the bounding box of element B (all four boundaries satisfy the condition that the boundary value of A is less than or equal to the corresponding boundary value of B, considering coordinate tolerance) If A is typically set to 2 pixels, then A is considered to contain B. To avoid fully connected element explosion, the system implements a nearest-parent pruning strategy: if A contains B and there exists a C such that A contains C and C contains B, then only the "AC" and "CB" edges are created, and the "AB" edge is not directly created, thus forming a precise tree-like hierarchical structure for the containment relationship. For non-rectangular elements such as images or seals, the system extracts the coordinates of their irregular polygonal outlines and uses ray casting to determine whether the text coordinates fall within their region.

[0062] All spaces, including edges, also record weight and evidence, and are incorporated into the LayoutGraph.

[0063] Step S114 elevates the spatial containment relationship to the document's logical structure, restoring the chapter hierarchy. For elements marked as headings in S111, the system first parses their numbering information and uses regular expression matching to identify the hierarchy depth. For headings without numbers, the system extracts the font size (font_size) and left indentation (margin-left) given by the OCR, and uses a clustering algorithm to group headings with similar font sizes and consistent indentation into the same level (parent headings usually have larger font sizes than child headings, and their left indentation is more left-aligned or centered).

[0064] The attribution determination adopts the nearest ancestor principle: trace the coordinate space upward from the current paragraph node. If the paragraph is included in the spatial range of a certain title and is not included in other titles at the same level, then the paragraph belongs to that title; at the same time, it checks whether the starting text of the paragraph corresponds semantically with the title text to increase credibility.

[0065] In addition to recording parent-child relationships, the hierarchical belonging edges also mark the hierarchical depth difference, providing a structured context for subsequent rule generation.

[0066] In step S115, for table structure parsing, a projection analysis method is adopted to process wired tables and wireless tables. The system first extracts the center point coordinates from all cell nodes under the table container, performs one-dimensional clustering on the y-coordinates (with the bandwidth set to 0.8 times the average character height), and the clustering result is a "row group"; one-dimensional clustering is performed on the x-coordinates, and the result is a "column group".

[0067] For cells spanning multiple rows and columns (merged cells), the system checks the bbox width and height of each cell one by one: if the left boundary is not on the column clustering center line or the right boundary spans multiple column center lines, it is marked as colspan>1; if the cell vertically spans multiple row center lines, it is marked as rowspan>1.

[0068] Subsequently, the system establishes horizontal same-table edges between adjacent column cells in each row, and establishes vertical same-table edges between adjacent row cells in each column; for merged cells, the system establishes a mapping relationship between the cell and all virtual grid units that it actually spans and occupies.

[0069] At the same time, combining the text content and type tags, if the text of the cells in the first row contains nouns such as "date", "amount" and the font is bold, the row is marked as a header row, and header mapping edges are established pointing to all data cells below the corresponding column.

[0070] In step S116, elements that belong to different pages but logically belong to the same integral unit are re-"stitched" through a tail-head matching algorithm. For a cross-page table, the system compares the table containers of Page N and Page N+1, and checks item by item whether the number of columns is consistent, whether the similarity of the column width ratio (calculated by normalizing the bbox width) is greater than 90%, and whether the Levenshtein edit distance of the header text is highly approximate. If the conditions are met, it is determined as the continuation of the same table, a cross-page continuous edge is established between the last data row of Page N and the first row of Page N+1, a "(continued on the next page)" mark is added to the tail of the table on Page N, and a "(continued from the previous page)" mark is added to the header of Page N+1.

[0071] For cross-page paragraphs, the system checks whether the end of the paragraph on Page N ends with a hyphen "-" or a dash, or whether the sentence grammar is incomplete (ends with "," or "de"); at the same time, it checks whether the beginning of the paragraph on Page N+1 is a lowercase letter or a non-sentence-initial capitalization (in Chinese, it starts with a preposition or conjunction). If they match, a cross-page text flow edge is established, and the text fields of the two paragraphs are logically spliced into a complete string for use in subsequent LLM extraction.

[0072] In step S117, uniformly structured page information is constructed based on all nodes and edges from S111 to S116.

[0073] In conjunction with the first aspect, S117 includes: S1171 uses each page element as a node and spatial adjacency, hierarchical affiliation, same-table edge relationship, and cross-page continuity relationship as edges to construct a graph structure.

[0074] S1172, using graph structure as unified structured page information.

[0075] In step S117, all page elements generated in stage S111 that carry complete attribute fields such as pageId, nodeId, bbox, text, imageRef, and type are used as the node set in the graph. At the same time, the spatial adjacency relationship generated in S112, the hierarchical relationship generated in S114, the same table edge relationship generated in S115, and the cross-page continuity relationship generated in S116 are used as the edge set in the graph to begin constructing a unified graph structure.

[0076] First, all original nodes output from S111 are traversed, each element is assigned a unique in-graph identifier (i.e., using nodeId) and added to the node set. Then, all edge relationships confirmed from S112 to S116 are traversed, and directed or undirected connections between nodes are established using the nodeId of the source and target nodes as indices. During the construction process, the system does not simply perform mechanical splicing, but simultaneously executes two key operations: First, it attaches complete relationship attributes to each edge, including relationship type labels (such as "right adjacency", "parent-child hierarchy", "horizontal same table", "spanning page continuation"), confidence weight (reflecting the reliability of the relationship determination, with a value of 0 to 1), and evidence field (recording key clues supporting the relationship determination, such as coordinate distance, cross-attention score, table header text similarity, etc.); Second, based on the edge connection density and type distribution, it automatically identifies table subgraphs with complete row and column structures, hierarchical subtrees with clear chapter titles, and continuous paragraphs or tables stitched together by spanning page edges, forming the initial indexes of tableLinks and sectionTree, respectively.

[0077] The original graph structure of S1171 is subjected to conflict resolution, redundancy compression, and index optimization to generate an engineered graph structure that can be directly traversed downstream. Specifically: The system first performs conflict resolution and redundancy removal: if a pair of nodes has multiple relationships (such as spatial adjacency and hierarchical affiliation), the system retains all multi-relationships, adds a "relationship type bitmask" to each edge, and stores independent weights and evidence respectively; if a circular dependency is detected (such as A contains B and B contains A), it decides based on the size of the spatial area, retaining the edge with the larger area containing the smaller area, and deleting the reverse edge.

[0078] Then, a global sorting of reading order is performed: a multi-key stable sort is performed based on the pageId, y-coordinate (from top to bottom), and x-coordinate (from left to right) of all nodes, generating a readingOrder array. During sorting, the edges of the table have the highest priority, ensuring that the table cells are arranged in order from left to right and from top to bottom, without being affected by header and footer interruptions.

[0079] After the above improvements, the system encapsulates the final output into a standardized data structure called LayoutGraph, as shown in Table 1. Its top-level structure consists of four core parts defined by the LayoutGraph table: LayoutNode (a list of nodes, each carrying all original features extracted in S111 and the graph attributes added in S1171), LayoutEdge (a list of edges, containing at least all relationships generated from S112 to S116 and their corresponding weights and evidence), readingOrder (an array of reading order), and sectionTree (a chapter tree index, organizing the document's logical outline in a tree structure). Furthermore, all stitching information for cross-page tables is summarized into tableLinks, recording the physical pagination mapping relationship for each cross-page table.

[0080]

[0081] In this embodiment, the graph structure is serialized into memory using Protocol Buffer or efficient JSON format. This completely transforms the original pixels and scattered OCR text into a semantically rich graph that is computationally comprehensible, traversable, and capable of subgraph cropping. The subsequent rule generation module can then accurately and dynamically locate the required fields using graph query statements such as "search type = field name and out-degree is 1, right adjacent," without needing to concern itself with changes in the absolute coordinates of the original layout.

[0082] It is worth emphasizing that the graph structure constructed in this step simultaneously accommodates multiple relationships: multiple edges may exist between the same pair of nodes (for example, a field name node and a field value node have both a "right adjacency" spatial edge and a "same table" table edge with the cell below because the field name is a column header). The system retains all of these and does not merge or delete relationships. Different types of edges will be consumed separately in different subsequent processing scenarios. Spatial adjacency edges are mainly used for field location rule generation, same table edges are mainly used for table structure parsing skills, and hierarchical belonging edges are mainly used for constructing LLM prompt words that are aware of chapter context.

[0083] Example 2 Compared with Embodiment 1, the difference in this embodiment is that step S120 includes: S121, based on the type tags of each page element in the unified structured page information, match a subgraph structure composed of at least two nodes and the relationship between the nodes in the unified structured page information, and use the matched subgraph structure as the structure pattern; wherein, the relationship includes at least one of spatial adjacency relationship, hierarchical belonging relationship, same table edge relationship and cross-page continuation relationship.

[0084] S122, Count the frequency of the same structural pattern appearing in the unified structured page information.

[0085] S123, determine the structure pattern that appears more frequently than a preset threshold as a recurring structure pattern.

[0086] S124, take the spatial position relationship and hierarchical relationship between nodes in the recurring structural pattern as the rule condition, take the extraction method of field value nodes in the recurring structural pattern as the rule action, and generate extraction rules based on the rule condition and rule action.

[0087] S125, The extraction rules are used as rule execution class processing units to generate a rule extraction strategy consisting of at least one rule execution class processing unit; S126, based on the type tags and text content of each page element in the unified structured page information, generate input / output data format specifications and prompt word templates for the target extraction fields indicated by each type tag; encapsulate the input / output data format specifications and prompt word templates into model inference processing units; and generate a model inference strategy by combining at least one model inference processing unit.

[0088] S127 uses rule extraction strategy and model inference strategy as extraction strategy.

[0089] In this embodiment, step S121 is the starting point for automatic rule generation, and its goal is to identify repetitive local structural patterns with business semantic value from the LayoutGraph. The system takes the LayoutGraph output in S117 as input, and matches subgraph structures composed of at least two nodes and their relationships in the graph structure according to the type tags of each page element (such as "field name", "field value", "title", "cell" etc.), and takes each matched subgraph as a candidate "structural pattern".

[0090] During the matching process, the system focuses on the types of relationships between nodes, including spatial adjacency (such as "right adjacency" connecting field names and field values), hierarchical relationship (such as parent-child connection between headings and subordinate paragraphs), same-table-edge relationship (such as horizontal row connection and vertical column connection between table cells), and cross-page continuation relationship (such as stitching data rows across tables).

[0091] The system does not limit itself to using only a single type of relationship when matching. Instead, it allows multiple relationship types to be combined in the same subgraph pattern. For example, a structural pattern can simultaneously include "the right adjacent edge between the field name node and the field value node" and "the horizontal same-table edge between the field value node and other cells in its table row", thus more completely depicting the field context in complex layouts.

[0092] The essence of the matching operation is to perform a subgraph isomorphic search within the node and edge sets of the LayoutGraph. The system iterates through all nodes, starting from each node and expanding based on its type label and the edge type of its adjacent edges, exhaustively listing all subgraph combinations that meet the conditions. Each matched subgraph structure is recorded as a structural pattern template for subsequent steps to perform frequency statistics and pattern filtering.

[0093] As an example, in the aforementioned invoice's LayoutGraph, the system matched a subgraph structure consisting of node 001 ("Invoice Number", field name) and node 002 ("No.88765432", field value) and their adjacent right edge. It also matched the same structure consisting of node 013 ("Invoice Date", field name) and node 014 ("2026-07-01", field value) and their adjacent right edge. Furthermore, it matched a two-dimensional grid subgraph in the table area composed of multiple cell nodes connected by horizontal and vertical table edges. All these subgraphs were recorded as candidate structure patterns.

[0094] After enumerating all candidate structural patterns in step S121, in step S122 the system normalizes each structural pattern, abstracting it into a generic pattern signature independent of specific node IDs (e.g., the signature can be represented as [node type A]4-[edge type X]->4 [node type B]), and merging subgraphs with the same signature into the same structural pattern class. Subsequently, the system counts the total frequency of each structural pattern class in the current LayoutGraph.

[0095] Understandably, frequency statistics are not simply a matter of counting. Instead, they distinguish between the number of times the same pattern appears within a single page and the number of times it appears across multiple pages, recording both separately to provide a more refined basis for subsequent threshold determination. For example, in an invoice, the structure pattern "field name - right adjacent - field value" might appear 5 times within a single page (invoice number, invoice date, amount, tax, buyer's name); in a cross-page contract, the pattern "title - parent-child hierarchy edge - paragraph" might appear on every page, and the system will summarize its total cross-page frequency.

[0096] Step S123 performs pattern filtering, filtering the statistical results of S122 based on a preset frequency threshold. The system determines structural patterns that occur more frequently than the preset threshold as "recurring structural patterns," retains them, and sends them to the next step as candidate sources for rule generation; patterns that do not reach the threshold are considered occasional or noisy structures and do not participate in subsequent rule generation.

[0097] Understandably, the threshold setting can be dynamically adjusted based on document type and layout complexity. In this embodiment, a typical value of 5 times is suggested, meaning that a structural pattern is considered statistically significant and valuable for rule extraction only when it appears at least 5 times in the document. The technical logic behind this threshold setting is that frequently occurring structural patterns reflect stable layout rules in the document (such as repeated key-value pairs in the upper right corner of an invoice, or repeated cell connections in each row and column of a table), and abstracting them into rules has high generalization and low false trigger rate; while occasional structures that only appear 1-2 times (such as the seal annotation at the end of a contract) may be outliers or special cases, and are not suitable for being solidified into general extraction rules.

[0098] Step S124 is a crucial step in converting the filtered structural patterns into executable JSON-formatted extraction rules (Rule Spec). For each recurring structural pattern retained in S123, a "condition-action" rule abstraction is executed.

[0099] Regarding rule conditions, the spatial relationships between nodes in the structural pattern (such as the specific geometric tolerance range of "right adjacency" and "bottom adjacency") and hierarchical relationships (such as "whether the field value node belongs to a certain title or table container") are extracted and encoded as the triggering conditions of the rule (i.e., the "IF" part). Regarding rule actions, the system extracts the extraction methods of "field value" type nodes in the structural pattern, including how to extract the target value from the text field (such as using regular expressions to extract 8-digit numbers), how to select the best match from multiple value candidates (such as taking the nearest adjacent text node), and how to format and transform the extracted results (such as unifying the date format to YYYY-MM-DD)—and encodes them as the action logic for rule execution (i.e., the "THEN" part).

[0100] The rules include coordinate rules (such as "amount value must be present within 50px to the right of the field name"), semantic rules (such as "date field matches YYYY year MM month DD day format"), hierarchy rules (such as "subheading must belong to the parent heading"), table rules (such as "cells in the same table column"), formatting rules (such as "standardized date format"), and business validation rules (such as "total amount = sum of all detailed amounts"). When generating rules, the system automatically determines the most suitable rule category for the structural pattern based on the node and edge types, and generates the corresponding rule specifications.

[0101] Ultimately, each generated rule is output as an independent Rule Spec in structured JSON format, containing the rule type, triggering conditions (including spatial constraints, hierarchical constraints, and type constraints), execution actions (including extraction logic, format conversion, and failure alternatives), and priority identifier, which are then combined by S125 into a rule extraction strategy.

[0102] Step S125 combines the multiple independent Rule Specs generated above into a complete rule extraction strategy. Specifically, the system encapsulates each Rule Spec into a rule execution processing unit (i.e., an independently callable rule node), and arranges multiple rule execution processing units into an ordered processing chain according to the dependencies between fields and the execution order.

[0103] The orchestration logic follows these principles: First, rule units without dependencies can be executed in parallel to improve efficiency (for example, the "invoice number" extraction rule and the "invoice date" extraction rule are independent and can run simultaneously); Second, rule units with data dependencies must be executed sequentially (for example, first execute the "table area positioning rule" to obtain the table container, and then execute the "cell row and column parsing rule" to extract the specific data from it); Third, business validation rules (such as "total amount = sum of all detailed amounts") are placed after all field extraction rules to ensure that all input data is ready.

[0104] The final output rule extraction strategy is a collection of at least one rule execution class processing unit, with each unit having an execution order identifier, a parallel / serial flag, and alternative actions in case of failure (such as marking a rule node failure as "pending skill fallback").

[0105] In step S126, for each page element in the LayoutGraph, a model inference processing unit (i.e., Skill) is automatically generated based on its type tag and text content. Each Skill contains an input / output schema and a prompt word template.

[0106] Regarding input / output schema generation, the system automatically determines the input parameter structure and output format specifications that the Skill needs to receive based on the target field's type label and layout context. For example, for a field value with type= and the context containing a "table" container, the system-generated input schema includes not only the field value text itself, but also all cell text in the row it belongs to, the column header label of the column it belongs to, and page continuity markers. For a date field with type=, the system mandates that the date format in the output schema be a standardized YYYY-MM-DD structure, and appends the original text evidence field.

[0107] Regarding prompt word template generation, the system selects or synthesizes the most suitable LLM prompt word template from a pre-built prompt word template library based on the field's semantic type (such as "date", "amount", "address", "unified social credit code"). It also dynamically fills the template with contextual information from the LayoutGraph (such as the chapter title and the semantics of preceding and following paragraphs) as contextual prompts. For example, for the "contract amount" field, the system generates the prompt word template as: "Extract the contract amount value from the following text, in numeric format, unit: yuan. Text content: [Dynamically filled into the paragraph and context of the field]", and adds constraints on the amount format to the template (such as "exclude lowercase Chinese amounts" and "prefer Arabic numerals").

[0108] In addition, the Skill includes tool adapters (such as table-specific parsing tools and stamp recognition tools), field standardization logic, failure handling strategies, and historical performance metrics. When generating the Skill Spec, the system automatically binds the corresponding tool adapter based on whether the LayoutGraph contains special nodes such as tables, stamps, or signatures. Each generated Skill is output as a Skill Spec in structured JSON format, including input / output schemas, prompt word templates, tool adapter configurations, degradation strategies, and historical performance record slots.

[0109] Ultimately, multiple Skill Specs are combined into a model inference strategy, which consists of at least one model inference class processing unit and will be compiled into Skill extraction nodes in the TaskDAG during the workflow orchestration phase (S140).

[0110] Step S127 merges the rule extraction strategy (composed of rule execution class processing units) generated in S125 with the model inference strategy (composed of model inference class processing units) generated in S126 to form a unified extraction strategy.

[0111] The merged extraction strategy comprises two complementary processing units: rule-based deterministic extraction units (suitable for scenarios with stable field positions and clear formats, such as fixed fields like invoice numbers and amounts), and model-based semantic understanding units (suitable for scenarios with complex semantics or large format variations, such as contract risk clauses and non-standard forms). These two units do not exist in isolation but work collaboratively through a rule-based and Skill-based dual-path cross-validation mechanism. In subsequent workflow execution phases, the system can choose to execute only the rule-based extraction strategy, only the model-based inference strategy, or both with cross-validation, depending on the field risk level and layout stability.

[0112] The merged extraction strategy serves as one of the core inputs to S140 (Automatic Workflow Orchestration and TaskDAG Generation), and together with field risk levels and business configurations, participates in the compilation of Workflow Spec and the generation of TaskDAG topology.

[0113] Example 3 Compared with Embodiments 1 and 2, the difference in this embodiment is that step S130 includes: S131, analyze the processing logic categories of each processing unit in the extraction strategy and the dependencies between each processing unit. The processing logic categories include rule execution and model reasoning.

[0114] S132, for each processing unit, determine the execution mode according to the processing logic category of the processing unit and the preset field risk level; the execution mode includes at least one of single-path execution mode, dual-path parallel execution mode and hybrid execution mode.

[0115] S133, Configure the corresponding processing node for the processing unit according to the execution mode.

[0116] S134, Configure the dependencies between processing nodes according to the dependencies between processing units.

[0117] S135, generate an executable workflow based on the processing nodes and the dependencies between them.

[0118] In step S131, the extraction strategy output in step S127 is used as input. This strategy contains multiple processing units, including rule execution units and model inference units. The system parses the processing logic category carried within each processing unit one by one. The rule execution units correspond to Rule Spec and are used to perform field extraction and validation based on deterministic conditions such as coordinates, semantics, hierarchy, tables, formats, and business validation. The model inference units correspond to Skill Spec and are used to call the large language model to perform semantic understanding and content extraction based on input and output data format specifications, prompt word templates, and tool adapters. At the same time, the system parses the dependencies between processing units. The sources of dependencies include two types: one is data dependency, that is, the output of one processing unit must be used as the input data of another processing unit, for example, the output of the "table area positioning" rule unit is the input of the "table cell parsing" rule unit; the other is control dependency, that is, the execution result of one processing unit determines the direction of subsequent branches or whether alternative paths are triggered. After parsing, the system organizes each processing unit, its logical category, input / output ports, and dependencies into a structured metadata list, which serves as the basis for subsequent steps to determine the execution mode and configure nodes.

[0119] Step S132 determines the execution mode for each processing unit, based on two dimensions. The first dimension is the processing logic category of the processing unit. Rule execution units execute with their inherent deterministic logic, and model reasoning units execute with their inherent model reasoning logic. However, the two are not pre-limited to a single mode at the execution mode level, but need to be comprehensively determined in combination with the second dimension.

[0120] The second dimension is the pre-defined field risk level. This risk level is pre-set and includes four aspects: business importance, downstream dependence, verification rigor, and error correction cost. For example, field types requiring high-risk identification include: core financial fields (such as amount, tax amount, invoice number), identity identification fields (such as name, ID number, tax number), and legal validity fields (such as signing date, legal representative, signature). For these high-risk fields, even if the system already distinguishes between rule execution and model reasoning logic categories, it will still specify a dual-path parallel verification mode at the execution mode level. For low-risk fields with stable layout and clear field positions, a single-path mode is specified to reduce computational costs.

[0121] Furthermore, when a field has high layout complexity and frequent format changes, even if its risk level is medium, the system may specify that it adopts a hybrid mode with model reasoning units as the main component and rule execution units as the auxiliary component. The system uses the execution mode determination result of each processing unit as the configuration basis for S133 and writes it into the execution attribute of the processing unit.

[0122] Based on the execution mode determined in S132, step S133 creates a corresponding processing node for each processing unit and configures its node attributes. The core content of the node attributes includes: node type (rule execution node, skill execution node, cross-validation node, manual confirmation node), node execution logic (calling a specific rule engine to execute rule condition judgments and action execution, or calling a specific model inference interface to execute large language model inference calls), input port mapping (obtaining the required data from upstream nodes or the global context), output port definition (specifying the data structure produced by this node), timeout time and retry strategy (both rule nodes and skill nodes need to be configured with timeout and retry parameters), and failure alternative actions (such as whether to switch to a skill node if a rule node fails, whether the rule node should fall back on a skill node failure or proceed to manual review).

[0123] For high-risk fields designated as "dual-path parallel mode" in step S132, the system creates two processing nodes simultaneously in this step: a rule execution node and a model inference node, configuring them to execute in parallel. In subsequent step S134, both nodes are configured to merge into the cross-validation node. After configuration, each processing node becomes an independent vertex in the TaskDAG.

[0124] After S133 completes the creation of each processing node, the system maps the dependencies between processing units parsed in S131 to directed edges between the corresponding processing nodes. Specifically, the system iterates through the dependency list generated in S131. For each record where "Unit A depends on Unit B," a directed edge is added to the node graph from node B to node A, indicating that node A can only begin execution after node B has successfully completed its execution. During the mapping process, the system also adds corresponding control attributes to the edges based on the nature of the dependency: data dependency edges (downstream nodes require the output data of upstream nodes as input) must simultaneously pass a data reference identifier to ensure that the upstream output result can be correctly injected into the downstream input port; control dependency edges (the execution result of the upstream node determines whether the downstream is triggered) must record a conditional expression on the edge.

[0125] For the two processing nodes designated as "dual-path parallel mode" in step S132, the system does not configure them as mutually exclusive in this step, but rather as parallel execution nodes, and configures each node with a convergence edge pointing to the cross-validation node. For nodes configured with failure alternative actions, the system adds an additional alternative trigger edge from that node to its alternative node, and marks the trigger condition as the node's execution failure. After all dependency edges are configured, the system obtains a complete directed graph containing all processing nodes and their dependencies, providing a complete structure for topology sorting and DAG generation in step S135. Thus, steps S131 to S135 dynamically generate extraction strategies based on the inductive analysis results by automatically extracting and inductively analyzing structural patterns.

[0126] Example 4 The difference between this embodiment and Embodiments 1, 2, and 3 is that step S140 includes: S141, parse the executable workflow and determine the processing nodes contained in the executable workflow and the dependencies between the processing nodes.

[0127] S142, based on dependencies, schedule each processing node to execute sequentially or in parallel.

[0128] S143, when executing each processing node, execute the corresponding processing logic according to the processing logic category corresponding to the processing node to obtain the field extraction results output by the processing node.

[0129] S144, record the execution status information after the processing node has finished executing. The execution status information includes at least one of the following: whether the processing node has executed successfully, execution time, input data integrity, and output data integrity.

[0130] S145, summarize the field extraction results output by each processing node into field extraction results, and summarize the execution status information of each processing node into execution status information.

[0131] Step S141 is the preparation phase before the scheduling system begins actual execution. The scheduling system receives the executable workflow, i.e., TaskDAG, generated in step S135. The system parses the TaskDAG, extracting all processing nodes and the dependencies between them. The parsing includes: traversing the node list to obtain the unique identifier of each node, node type (rule execution node, Skill execution node, cross-validation node, manual confirmation node, etc.), execution logic configuration, input port definition, output port definition, timeout setting, retry strategy configuration, and failure alternative actions; traversing the edge list to obtain the source node identifier and target node identifier of each directed edge, distinguishing between data dependency edges and control dependency edges, and extracting the conditional expressions (if any) on the edges. After parsing, the system constructs a traversable execution plan object in memory. This object retains all the topological structure information of the original DAG and performs preliminary hierarchical classification of all nodes according to dependency depth, providing a complete execution blueprint for the subsequent scheduling execution in S142.

[0132] Taking the contract processing scenario as an example, after parsing the TaskDAG, the scheduling system determines six processing nodes and their dependencies: the contract number rule node and the signing date rule node have no dependency relationship and can be executed in parallel; the amount rule node and the risk clause Skill node both point to the fusion verification node; the Markdown generation node depends on the completion of the fusion verification node.

[0133] After parsing the execution plan, step S142 proceeds to the actual scheduling and execution phase. Based on the inter-node dependencies parsed in step S141, the scheduler determines the start time for each processing node: for nodes without any prerequisite dependencies (i.e., nodes with an in-degree of zero), they are marked as nodes that can be started immediately and added to the ready queue; for nodes with prerequisite dependencies, the node is only transitioned from the waiting state to the ready state and added to the ready queue after all its prerequisite nodes have successfully completed execution. The scheduler retrieves nodes from the ready queue, allocates appropriate execution resources based on the node's resource tags (e.g., rule execution nodes require lower computing power, Skill execution nodes require GPU resources), and triggers execution according to the configured timeout and retry policy. For multiple independent nodes belonging to the same parallel layer (such as the contract number rule node and signing date rule node belonging to the first parallel layer in the example of step S135), the scheduler allocates them to different execution threads or execution containers for parallel execution to make full use of computing resources and shorten the overall processing time. For nodes with dependency chains (such as the fusion verification node depending on the completion of the amount rule node and the risk clause Skill node), the scheduler serializes them to ensure that the dependency relationship is strictly satisfied. The scheduler also continuously monitors the execution status of each node. When it detects that a node is in a waiting state for a long time due to resource contention or abnormal reasons, the system can adjust the resource allocation according to the preset priority strategy.

[0134] When the scheduler triggers the execution of a processing node, that node actually runs according to the execution logic configured in step S133. The system first identifies the processing logic category corresponding to the node: if it is a rule execution node, the system calls the rule engine to load the corresponding Rule Spec, retrieves the target node and its adjacency relationships from the LayoutGraph based on the rule conditions defined in the Rule Spec, performs deterministic logic such as coordinate determination, regular expression matching, format validation, and business validation, and outputs field values ​​and their confidence scores; if it is a model inference node, the system calls the model inference interface to load the corresponding Skill. The system extracts the required data (field name text, context paragraphs, tables, images, etc.) from the LayoutGraph according to the input data format specifications. After replacing placeholders in the prompt word template with actual data, it submits the data to the large language model. It receives the model's return results and performs format validation and standardization according to the output data format specifications, outputting field values ​​and their confidence scores. For cross-validation nodes, the system loads the dual-path input configured for that node (rule path output and Skill path output), performs field value consistency comparison, evidence chain integrity verification, and confidence score difference analysis, and outputs the final field values ​​and fused confidence scores according to the fusion arbitration strategy (high confidence priority, rule priority, Skill priority, or manual confirmation). For manual confirmation nodes, the system suspends execution and generates a review task, waiting for human input before continuing. After each node is completed, a structured field extraction result is output, including field values, confidence scores, and evidence citations.

[0135] After each processing node completes its execution (including success, failure, timeout, or skipping), the scheduling system immediately collects and structures the node's execution status information. The execution status information includes at least the following dimensions: node execution effectiveness. Timeliness score Input data integrity (Check whether the data required by the node input port has been completely obtained, and calculate the ratio of the number of complete data items to the number of data items that should be obtained), and ensure the compliance of the output data. (Check if the node output conforms to the schema defined in the output data format specification), retry stability. Evidence coverage .

[0136] When a node executes successfully When node execution fails When a node is suspended .

[0137] Once all processing nodes in the TaskDAG have completed execution (including successful execution, triggering alternative nodes after failure, or suspending pending manual confirmation after failure), the system initiates the aggregation program. The aggregation program performs two core tasks: First, it collects and integrates the field extraction results output by all processing nodes, merging individual field values, confidence levels, and evidence references by field name or field identifier to form a complete set of field extraction results. Second, it collects and aggregates the execution status information of each processing node recorded in S144, compiling the execution status, execution time, input data integrity, output data compliance, retry count, evidence coverage, and error type of each node into a structured execution status information table. The aggregated field extraction results are then passed to the Markdown structured result generation module as raw data for generating auditable Markdown intermediate results; the aggregated execution status information is then passed to the execution score module for calculating the execution score of each node. ) and workflow execution rate ( The original basis for ( ).

[0138] Example 5 The difference between this embodiment and Embodiments 1, 2, 3, and 4 is that step S150 includes: S151. Based on the field extraction results, execution status information, and unified structured page information, determine at least two dimensions among layout positioning confidence, rule matching confidence, model extraction confidence, cross-validation consistency, field value validity, and historical performance confidence, and perform weighted calculations on at least two dimensions to obtain field-level confidence.

[0139] S152. Based on the execution status information and field extraction results, determine at least one dimension of each processing node's execution status score, input integrity score, output compliance score, and evidence coverage score, and perform weighted calculation on at least one dimension to obtain the process-level execution quality.

[0140] Step S151 calculates the field-level confidence score based on the field extraction results summarized in step S145, the execution status information of each processing node, and the LayoutGraph output in step S117, using the following formula. :

[0141] in, To assess the reliability of layout positioning, a weighted calculation is performed based on coordinate alignment, page consistency, area integrity, and table ownership clarity in the LayoutGraph, reflecting the geometric positioning quality of the element containing the field value on the layout. To assess the reliability of rule matching, a weighted calculation is performed based on rule trigger rate, matching accuracy, tolerance compliance, and historical rule accuracy, reflecting the reliability of the rule execution processing unit in extracting this field. The model extraction credibility is calculated by weighting the model confidence, output structure integrity, evidence chain quality, and historical accuracy of the model reasoning processing unit, reflecting the reliability of the model reasoning processing unit in extracting this field.

[0142] To ensure consistency of cross-validation, the difference rate between the rule extraction results and the model inference results is mapped to the range of 0 to 1. When the results of the two paths are consistent, the score of this dimension is close to 1, and the greater the difference, the lower the score.

[0143] The validity of field values ​​is calculated by weighting the results of format validation, value range validation, business validation, and unit validation, reflecting whether the extracted field values ​​conform to the data specifications that the field type should have.

[0144] The historical performance confidence is calculated based on the historical accuracy and time decay factor of the rule processing unit and the model inference processing unit. This is used to suppress the excessive influence of processing units with unstable historical performance or recent decline in accuracy on the current confidence.

[0145] The system assigns weights to each dimension based on field type and business scenario (the sum of all weights is 1), and then calculates the weighted sum to obtain the final field-level confidence score. This confidence score ranges from 0 to 1 and serves as one of the core inputs for subsequent lake entry decisions. All are weighting coefficients, and the sum of the weights is 1. For core financial fields such as amount, tax amount, and invoice number, cross-validation is performed for consistency. The weight is set between 0.25 and 0.35, and the field value validity is... The weight is set to 0.20 to 0.30; for format-sensitive fields such as dates and addresses, the validity of field values ​​is... The weight is set to 0.25 to 0.35; for fields with complex and varied layouts, the reliability of the layout positioning is... and model extraction credibility The weight can be appropriately increased to 0.20 to 0.30. Each weight coefficient can be dynamically adjusted during system operation based on the feedback from manual review to adapt to different document types and business scenarios. This is only an example and is not intended to limit the scope.

[0146] Field-level confidence The credibility of field values ​​is comprehensively evaluated from six dimensions: layout positioning quality, rule matching reliability, model inference confidence, dual-path consistency, field value validity, and historical performance. This also assesses the process-level execution quality. The reliability of workflow execution is comprehensively evaluated from six dimensions: node execution status, input integrity, output compliance, timeliness, retry stability, and evidence coverage. These two dimensions together constitute a dual guarantee for the quality of data lake writes, encompassing both result credibility and process reliability. Only field values ​​with high confidence generated in high-quality workflows, and whose evidence is traceable, are allowed to be written to the data lake, thus ensuring the accuracy and auditability of data stored in the data lake from the source.

[0147] Step S152, based on the execution status information and field extraction results of each processing node summarized in step S145, calculates the process-level execution quality using the following formula. :

[0148] in, This is the node status value. It is 1 when the node executes successfully, 0 when it fails, and 0.5 when the node is manually suspended. Score the completeness of the input data; To output data compliance scores; Score based on timeliness; The score is for retry stability. Score the evidence coverage. All are weight coefficients, and the sum of the weights is 1. For rule enforcement nodes, output a compliance score. The weighting was set between 0.30 and 0.40 for the evidence coverage score. The weights are set to 0.20 to 0.30; for model inference nodes, the input integrity score is... and evidence coverage score The weights are set to 0.25 to 0.35 respectively, which are only examples and are not limited here.

[0149] As a baseline score, the system calculates a timeliness score by comparing the ratio of the execution time of a node to the expected execution time, and a retry stability score by comparing the ratio of the number of retries to the preset retry limit. The status value, timeliness score, and retry stability score are then weighted and combined to form an execution status score. Regarding input completeness scoring, the system checks whether the data required by the node's input ports has been completely obtained, calculating the ratio of the number of complete data items to the number of data items that should have been obtained. Regarding output compliance scoring, the system checks whether the node's output conforms to the schema requirements defined in the output data format specification, calculating the ratio of the number of compliant output fields to the total number of output fields. Regarding evidence coverage scoring, the system calculates the ratio of the number of pieces of evidence actually cited during node execution to the number of pieces of evidence that should have been cited.

[0150] Based on this, the system further assigns the execution degree of each node according to the node importance weight. Perform a weighted summation and multiply by the directed acyclic graph topological integrity penalty factor. and retry penalty factor To obtain the process-level execution quality of the entire workflow. :

[0151] Among them, topological integrity penalty factor The value is reduced when there are missing critical dependencies, circular dependencies, or skipped critical nodes. The initial value is 1.0. When there are no anomalies, the value is 1.0. When there are missing critical dependencies, the value is 0.6 to 0.8. When there are circular dependencies, the value is 0.5. When the critical node is skipped, the value is 0.7 to 0.9. When multiple anomalies are superimposed, the larger value between the product of the penalty values ​​and 0.3 is taken.

[0152] Retry penalty factor The initial value is 1.0, calculated based on the number of retries. The value is 1.0 when there are no retries, 0.95 when a non-critical node is retried once, 0.85 to 0.90 when a non-critical node is retried two or more times, 0.85 when a critical node is retried once, 0.70 to 0.75 when a critical node is retried two or more times, and 0.5 when a critical node fails after retrying.

[0153] Node importance weight Based on the field risk level setting, the processing node corresponding to the high-risk field. Values ​​range from 1.0 to 1.2, representing the processing nodes corresponding to low-risk fields. The values ​​range from 0.8 to 1.0. Each weight coefficient can also be dynamically adjusted based on historical performance; this is merely an example and not a limitation.

[0154] In conjunction with the first aspect, step S160 includes: S161, when the field-level confidence level is greater than or equal to the preset confidence level threshold, and the process-level execution quality is greater than or equal to the execution quality threshold, a first control instruction is generated. The first control instruction is used to instruct the field extraction results to be written to the target storage system.

[0155] S162, when the field-level confidence is less than the confidence threshold, or the process-level execution quality is less than the execution quality threshold, a second control instruction is generated. The second control instruction is used to instruct the field extraction results to be diverted to manual processing.

[0156] Will Compared with the preset confidence threshold Compare with, and at the same time Compared with the preset execution quality threshold Compare them.

[0157] when and When the system determines that the field meets the automatic lake entry conditions, it generates a first control command. This command instructs the field extraction results to be written to the target storage system. In this embodiment, The value is 0.85. The value is 0.90.

[0158] In this embodiment, before generating the first control command, the system also needs to verify the integrity of the evidence chain (Evidence(f)) for the field, that is, whether the field value can be traced back to evidence such as page number, coordinates, text fragments, or image regions in the LayoutGraph. The integrity of the evidence chain is a necessary condition for the lake entry determination. If the evidence is missing, even if the confidence level and execution quality meet the standards, the first control command will not be triggered. After all conditions are met, the system encapsulates the field extraction results and its accompanying information such as field name, field value, source page number, source coordinates, evidence text, extraction mode, rule version, skill version, field confidence level, workflow execution level, and review status. The first control command instructs to write this structured data into the target storage system. High-confidence data that meets the threshold is directly written into the Iceberg official main table, which stores complete information such as business primary key, field value, evidence reference, confidence level, execution level, lake entry time, rule version, skill version, and review status.

[0159] In this embodiment, while generating the first control instruction, the system also generates the corresponding Markdown structured result as an intermediate product for manual review, audit tracing, and version management.

[0160] when , If any of the conditions occurs, step S162 determines that the field or the overall output of the workflow does not meet the quality requirements for automatic lake entry and generates a second control instruction. This instruction instructs that the field extraction results be diverted to manual processing. The second control instruction does not directly write the data into the formal table of the target storage system, but instead adopts a differentiated diversion strategy based on the specific reason for non-compliance: when However, if the deviation is not serious (e.g., the confidence level is in the range of 0.75 to 0.85), the second control instruction instructs the data of the corresponding field to be written into the confirmation table or the sampling queue, and enters the manual sampling review process; When field-level confidence Significantly low (e.g., below 0.7) or When there is a severe deficiency (e.g., the execution degree is below 0.7 due to the failure of a critical node), the second control instruction indicates that automatic entry into the lake should be blocked, and the complete workflow execution context (including all field extraction results, execution status information, LayoutGraph slices and error logs) should be suspended to the manual review task queue for manual confirmation item by item.

[0161] In addition, the second control instruction also includes generating a visual interface data for manual review, which clearly displays the candidate values, source evidence, confidence scores, detailed scores for each dimension, and execution status logs for each field in the form of Markdown structured results, enabling reviewers to quickly locate the cause of the problem and make corrective decisions.

[0162] Through the aforementioned hierarchical output control mechanism, only field results that simultaneously meet three conditions—field-level confidence threshold, process-level execution quality threshold, and evidence chain integrity—can be written to the target storage system. All other data is effectively intercepted and directed to manual processing. Therefore, this method jointly ensures the write quality of the data lake from three dimensions: field value confidence, execution process reliability, and evidence traceability, avoiding the data pollution problem caused by directly storing low-confidence data in traditional solutions.

[0163] In conjunction with the first aspect, the method also includes: S170, Obtain the results of manual processing; S180, based on the results of manual processing, update the configuration parameters of the corresponding rules or models in the extraction strategy.

[0164] In step S170, the system receives the processing result submitted after manual review. The manual reviewer examines the fields to be confirmed on a visual interface and makes a decision to accept, correct, or reject candidate values ​​based on the chain of evidence. The system obtains the manual processing result through a callback interface, which includes information such as the standard value of the field, review status, and review remarks.

[0165] Subsequently, step S180 uses the manual processing results obtained in S170 as a high-quality supervision feedback signal to update the configuration parameters of the corresponding rule execution processing unit or model reasoning processing unit in the extraction strategy, so that the system can continuously improve the accuracy and avoid repeated errors when processing similar documents in the future.

[0166] Regarding rule parameter updates, the system precisely adjusts the RuleSpec based on the error types noted in the manual review notes. If the error is due to coordinate positioning deviation (such as the actual position of a field value deviating from the rule's preset tolerance range), the system automatically expands or shifts the field's coordinate tolerance parameter based on the actual coordinates corresponding to the manually confirmed standard value. If the error is due to format matching failure (such as a variant of the date format not covered by the regular expression), the system extracts the format pattern of the manually corrected value and automatically expands the matching range of the existing regular expression. If the error is due to business validation conflict (such as a minor error in summing the total amount and the detailed amount), the system reverse-calibrates the tolerance threshold of the business validation rule based on the manually confirmed standard value.

[0167] Regarding model parameter updates, the system optimizes SkillSpec based on the types of model inference defects marked in the manual review notes. If the error is due to the model output format not conforming to the schema specification, the system adds additional format cleaning rules to the output data format specification validation layer. If the error is due to the model failing to locate the correct answer in a given context (i.e., insufficient prompt word coverage), the system automatically adds the manually confirmed standard values ​​and their context as positive examples to the prompt word template's Few-shot example library, or adjusts the instruction descriptions regarding location clues in the prompt words. If the error is due to a systematic failure in a specific format scenario, the system adds a mark to the scenario type in the Skill's historical performance record, guiding subsequent calls to prioritize switching to rule fallback or requesting manual intervention in advance.

[0168] Regarding weight parameter updates, the system feeds back the manual confirmation results to the field-level confidence calculation formula in step S151, updating the historical performance credibility of the relevant dimensions. Specifically, if manual confirmation confirms that the output of a rule execution unit is correct, the execution is recorded as a positive example, and the dynamic historical accuracy of the rule increases; if manual confirmation corrects the output of a rule execution unit, it is recorded as a negative example, and the dynamic historical accuracy of the rule decreases. Similarly, the historical accuracy of model inference units is also updated synchronously based on the manual confirmation results, and the time-degradation factor is adjusted according to recent performance. If there are consecutive manual corrections recently, the system accelerates the degradation of the historical performance weight of the processing unit, reducing its positive contribution to subsequent field-level confidence.

[0169] In this embodiment, all the above update operations take effect incrementally and hotly—the system directly modifies the corresponding configuration items in the rule base, skill base, and weight base. The updated Rule Spec, Skill Spec, and weight coefficients take effect automatically when the next document task is processed, without the need for a system restart or manual redeployment.

[0170] Through the aforementioned manual review result write-back mechanism, the system can continuously accumulate positive and negative example samples for various document types and field scenarios, dynamically adjusting rule tolerance, prompt word templates, and confidence weights, thereby gradually improving the field-level confidence and process-level execution quality of subsequent processing. This closed-loop optimization mechanism ensures continuous improvement in the data lake's write quality. As the number of processed documents increases, the accuracy of automatic data entry into the lake continuously improves, while the manual review rate gradually decreases. The quality of structured data stored in the data lake shows a continuous converging upward trend, forming a complete quality management closed loop of execution-evaluation-feedback-optimization.

[0171] Secondly, embodiments of this application provide a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0172] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0173] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0174] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0176] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A document structuring method based on multimodal data fusion and dynamic rule-driven approach, characterized in that, The method includes: The multimodal document is parsed to extract the attribute information of each page element and the relationship between page elements, and to construct a unified structured page information. The structural patterns in the page information are identified and summarized, and an extraction strategy is dynamically generated based on the summarization and analysis results; the extraction strategy is used to locate the target business field from the page information and extract the field value; The extraction strategy is subjected to dependency resolution and task orchestration to generate an executable workflow containing multiple processing nodes and the dependencies between the processing nodes. The executable workflow is executed to drive each of the processing nodes to run according to the dependencies, and the field extraction results and execution status information output by each of the processing nodes are obtained. Based on the field extraction results, the execution status information, and the unified structured page information, the field-level confidence and process-level execution quality are determined. The field-level confidence is determined based on the reliability of the field value itself and the integrity of the evidence, and the process-level execution quality is determined based on the stability and compliance of the workflow execution process. Based on the comparison results of the field-level confidence level and the process-level execution quality with the preset threshold, a hierarchical output control instruction is generated. The hierarchical output control instruction is used to write data that meets the threshold conditions into the target storage system and to divert data that does not meet the threshold conditions to manual processing.

2. The method according to claim 1, characterized in that, The steps of parsing a multimodal document, extracting attribute information of each page element and the relationships between page elements, and constructing a unified, structured page information include: Obtain the text content, page coordinates, image features, and type labels of each page element in the multimodal document; the type labels include at least one of the following: title, paragraph, field name, field value, table, cell, image, and page number; Multimodal fusion is performed based on the page coordinates, text content, and image features of each page element to determine the spatial adjacency relationship between each page element; Based on the page coordinates of each page element, determine the spatial containment relationship between the page elements; The hierarchical relationship between the page elements is determined based on the type tags of each page element and the spatial inclusion relationship. Based on the page coordinates, type tags, and text content of each page element, determine the same-table edge relationships between cells belonging to the same table; Based on the page coordinates and text content of each page element, determine the cross-page succession relationship between elements belonging to different pages but belonging to the same table or the same paragraph; Based on the spatial adjacency relationship, the hierarchical affiliation relationship, the same table edge relationship, and the cross-page continuity relationship, the unified structured page information is constructed.

3. The method according to claim 2, characterized in that, The steps for constructing the unified structured page information based on the spatial adjacency relationship, the hierarchical affiliation relationship, the same-table edge relationship, and the cross-page continuity relationship include: A graph structure is constructed using each page element as a node and the spatial adjacency relationship, the hierarchical affiliation relationship, the same table edge relationship, and the cross-page continuity relationship as edges. The graph structure is used as the unified structured page information.

4. The method according to claim 1, characterized in that, The steps of identifying and summarizing structural patterns in the page information, and dynamically generating extraction strategies based on the summarization and analysis results, include: Based on the type tags of each page element in the unified structured page information, a subgraph structure consisting of at least two nodes and the relationships between nodes is matched in the unified structured page information, and the matched subgraph structure is used as the structural pattern; wherein, the relationships include at least one of spatial adjacency, hierarchical affiliation, same table edge relationship and cross-page continuation relationship; Statistical analysis of the frequency of the same structural pattern appearing in the unified structured page information; Structural patterns that appear more frequently than a preset threshold are identified as recurring structural patterns. The spatial positional relationship and hierarchical affiliation relationship between nodes in the recurring structural pattern are used as rule conditions, and the extraction method of field value nodes in the recurring structural pattern is used as rule actions. Extraction rules are generated based on the rule conditions and rule actions. The extraction rules are used as rule execution class processing units to generate a rule extraction strategy consisting of at least one of the rule execution class processing units. Based on the type tags and text content of each page element in the unified structured page information, input / output data format specifications and prompt word templates are generated for the target extraction fields indicated by each type tag; the input / output data format specifications and the prompt word templates are encapsulated into model inference processing units; and model inference strategies are generated by combining at least one of the model inference processing units. The rule extraction strategy and the model inference strategy are used as the extraction strategy.

5. The method according to claim 1, characterized in that, The steps of performing dependency resolution and task orchestration on the extraction strategy include: The processing logic categories of each processing unit in the extraction strategy and the dependencies between the processing units are analyzed. The processing logic categories include rule execution and model reasoning. For each processing unit, an execution mode is determined based on the processing logic category of the processing unit and a preset field risk level; the execution mode includes at least one of single-path execution mode, dual-path parallel execution mode, and hybrid execution mode. According to the execution mode, configure the corresponding processing node for the processing unit; Configure the dependencies between the processing nodes according to the dependencies between the processing units; The executable workflow is generated based on the processing nodes and the dependencies between them.

6. The method according to claim 1, characterized in that, The steps of executing the executable workflow, driving each of the processing nodes to run according to the dependencies, and obtaining the field extraction results and execution status information output by each of the processing nodes include: The executable workflow is parsed to determine each processing node contained in the executable workflow and the dependencies between each processing node; Based on the dependencies, each of the processing nodes is scheduled to execute sequentially or in parallel. When executing each of the aforementioned processing nodes, the corresponding processing logic is executed according to the processing logic category corresponding to the processing node to obtain the field extraction results output by the processing node; Record the execution status information after the processing node has finished executing. The execution status information includes at least one of the following: whether the processing node has been successfully executed, execution time, input data integrity, and output data integrity. The field extraction results output by each processing node are summarized into the field extraction results, and the execution status information of each processing node is summarized into the execution status information.

7. The method according to claim 1, characterized in that, The steps for determining field-level confidence and process-level execution quality based on the field extraction results, the execution status information, and the unified structured page information include: Based on the field extraction results, the execution status information, and the unified structured page information, at least two dimensions are determined from the following: layout positioning confidence, rule matching confidence, model extraction confidence, cross-validation consistency, field value validity, and historical performance confidence. The at least two dimensions are then weighted to obtain the field-level confidence. Based on the execution status information and the field extraction results, at least one dimension of each processing node is determined, including execution status score, input integrity score, output compliance score, and evidence coverage score. The at least one dimension is then weighted and calculated to obtain the process-level execution quality.

8. The method according to claim 1, characterized in that, The step of generating hierarchical output control instructions based on the comparison results of the field-level confidence level and the process-level execution quality with a preset threshold includes: When the field-level confidence level is greater than or equal to a preset confidence threshold, and the process-level execution quality is greater than or equal to a preset execution quality threshold, a first control instruction is generated. The first control instruction is used to instruct the field extraction result to be written into the target storage system. When the field-level confidence level is less than the confidence threshold, or the process-level execution quality is less than the execution quality threshold, a second control instruction is generated. The second control instruction is used to instruct the field extraction results to be diverted to manual processing.

9. The method according to claim 1, characterized in that, The method further includes: Obtain the results of manual processing; Based on the results of the manual processing, update the configuration parameters of the corresponding rules or models in the extraction strategy.

10. A storage medium, characterized in that, The storage medium stores computer program instructions, which, when read and executed by a processor, perform the method described in any one of claims 1 to 9.

Images