Paperless office method and system based on intelligent workflow and multi-modal data fusion
By correcting the tilt of multilingual and multi-format documents and dividing the layout into blocks, a semantic graph of the layout is generated and currency is normalized. Combined with the incremental decision tree model training, an intelligent workflow routing decision tree is generated, which solves the problems of mismatch in multilingual document identification and cold start of cross-node routing models, and realizes efficient paperless office.
Patent Information
- Application Number
- CN202611211004.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-11-10
AI Technical Summary
In the process of paperless office in large groups, there are problems such as mismatch and extraction deviation in field recognition of multilingual and multi-format documents, insufficient capacity of cross-node workflow routing models during the cold start period, and difficulty in capturing common workflow patterns caused by data isolation.
By generating text block coordinate sequences with language labels through tilt correction and page segmentation, and using a multilingual and multimodal pre-trained model to output embedding vectors, a page semantic graph is generated and currency normalized. Combined with an incremental decision tree model, an intelligent workflow routing decision tree is generated to achieve cross-node data fusion and approval route recommendation.
It improved the accuracy of field extraction from multimodal documents, shortened the cold start period of approval routing, met data isolation requirements, and enabled cross-subsidiary workflow collaborative training and recommendation accuracy.
Smart Images

Figure CN123070899A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of paperless office technology, specifically to a paperless office method and system based on intelligent workflow and multimodal data fusion. Background Technology
[0002] In the process of implementing paperless office practices, large corporations and their subsidiaries face a variety of document formats, including multilingual invoices, scanned contracts, electronic forms, and mobile-captured images of receipts, as well as expense reports, contract approvals, and seal applications submitted by employees daily. Each subsidiary deploys a workflow engine at its local office node to perform OCR recognition, field extraction, and approval routing processing on these documents.
[0003] Existing technologies typically employ a single-modal OCR approach to extract fields from office documents and train an approval routing model based on local historical samples. However, this approach suffers from the following problems: First, in multilingual and multi-format documents, the positions of fields such as currency symbols, amounts, taxes, and signatures are randomly distributed across different formats, making robust extraction difficult with single-modal OCR and prone to field recognition mismatches and extraction biases. Second, the paperless office samples from a single subsidiary are sparse, resulting in insufficient routing capabilities for the approval routing model during the cold start period. Third, due to data compliance and privacy requirements, office data from various subsidiaries cannot be centrally aggregated across organizations, making it impossible to capture common workflow patterns across subsidiaries while meeting data isolation requirements, further exacerbating the problem of an excessively long cold start period. Summary of the Invention
[0004] This invention provides a paperless office method and system based on intelligent workflow and multimodal data fusion, which solves the technical problems in related technologies such as the difficulty in accurately extracting structured entities from multimodal document images, the inability of cross-node workflow decision-making to collaboratively train while protecting data privacy, and the insufficient accuracy of approval route recommendation.
[0005] This invention discloses a paperless office method based on intelligent workflow and multimodal data fusion, including: performing tilt correction and page block division on multimodal document images, labeling the language of each text block, and generating a text block coordinate sequence with language labels;
[0006] The text block coordinate sequence is input into a multilingual multimodal pre-trained model, which outputs an embedding vector for each text block with semantic type labels.
[0007] Based on the spatial adjacency relationship of the embedded vector, a semantic graph of the layout is generated, semantic binding and currency normalization are performed, and a normalized structured entity is output.
[0008] The incremental decision tree model is trained using the structured entities and historical approval results to obtain local workflow sub-decision trees and extract shareable metadata;
[0009] The coordinating node performs weighted histogram aggregation on the shareable metadata of each local node to generate a global skeleton workflow decision tree;
[0010] Each local node performs node-level fusion of the global skeleton workflow decision tree and the local workflow sub-decision tree to generate an intelligent workflow routing decision tree;
[0011] The structured entity of the application to be processed is input into the intelligent workflow routing decision tree, and the recommended approval chain and confidence level are output.
[0012] Furthermore, the layout semantic graph is a directed graph structure, where nodes are text block embedding vectors with semantic type labels, and edges are semantic binding relationships between text blocks;
[0013] The generated layout semantic graph includes: calculating the spatial adjacency distance between adjacent text blocks based on the rectangular boundary coordinates of each text block; when the shortest distance between the rectangular boundaries of two text blocks is less than the adjacency threshold, an adjacency edge is added between the two nodes.
[0014] Traverse the nodes with adjacent edges. When the semantic type label of a node is currency symbol and the semantic type label of its adjacent node is amount value, add a currency amount binding edge between the two.
[0015] When a node's semantic type label is "signature" and its adjacent node's semantic type label is "term text", add a signature / term binding edge between the two.
[0016] The adjacency threshold ranges from 2% to 8% of the short side length of the document image.
[0017] Furthermore, the currency normalization includes: for the amount nodes with currency amount binding edges in the semantic graph of the layout, obtaining the currency symbol associated with the amount node, querying historical exchange rate data, and converting the amount value into a value under the base currency.
[0018] Each entity in the normalized structured entity contains field names, field values, and binding relationships between fields.
[0019] Furthermore, the weighted histogram aggregation includes: the coordinating node receiving shareable metadata uploaded by each local node, and aligning the sample distribution histograms of each local node according to the feature index of the split node;
[0020] For each local histogram under the same feature index, the weight value is calculated according to the total number of samples in each local node. : ,in For the first The total number of training samples for each local node. This is the sum of the total number of training samples across all local nodes. This represents the total number of local nodes. Multiply the sample count of the same interval in each local histogram by the corresponding weight value. Then, accumulate the results to obtain the global weighted histogram; traverse each candidate split point in the global weighted histogram and select the candidate split point with the largest information gain as the global split candidate;
[0021] The global skeleton workflow decision tree is generated layer by layer from top to bottom using global split candidates, where each node contains only feature index and split threshold.
[0022] Furthermore, the node-level fusion includes: traversing each layer of the global skeleton workflow decision tree from top to bottom, and for each skeleton node, searching for a corresponding node with the same feature index in the local workflow sub-decision tree;
[0023] When the split threshold of the skeleton node is inconsistent with that of the corresponding node, the split threshold of the skeleton node is used to replace the split threshold of the corresponding node, and the subtree structure of the corresponding node is preserved.
[0024] When there is no node corresponding to the skeleton node in the local workflow sub-decision tree, the skeleton node and its subtree are directly inserted into the corresponding level of the local workflow sub-decision tree.
[0025] For each leaf node after fusion, the approval route label and confidence level are recalculated using local training samples.
[0026] Furthermore, the multilingual multimodal pre-trained model is a Transformer architecture model, whose input is a sequence of triples. Each triple includes an image region cropping of a text block, a one-hot encoding of the language label, and a normalized four-dimensional vector of rectangular boundary coordinates.
[0027] The rectangular boundary coordinates are normalized to mean based on range before input, scaling the coordinate values to a uniform range.
[0028] The multilingual, multimodal pre-trained model runs in inference mode and does not perform parameter updates.
[0029] Furthermore, the shareable metadata includes the feature index, split threshold, and sample distribution histogram of each split node, but does not include original document images, field details, or employee information.
[0030] The sample distribution histogram refers to a two-dimensional histogram formed by dividing the training samples arriving at each split node into several intervals according to the candidate feature values, and counting the number of samples of different approval result categories in each interval.
[0031] Furthermore, the tilt correction includes: detecting straight line edges in the image through Hough transform, calculating the principal direction deflection angle, and performing an affine transformation on the image to make the text lines horizontally aligned;
[0032] For ticket images taken on mobile devices, perspective correction is performed before affine transformation to eliminate trapezoidal distortion caused by the shooting angle.
[0033] Furthermore, it also includes: after the approval process is completed, each local node obtains the actual approval result, compares the actual approval result with the recommended approval chain, and generates a feedback difference record; the feedback difference record includes feature vectors, the recommended approval chain and the actual approval result, but does not include the original document image or field details;
[0034] Each local node uploads the feedback differential records to the coordinating node according to the preset update cycle. The coordinating node re-executes the weighted histogram aggregation based on the accumulated feedback differential records, updates the global skeleton workflow decision tree, and sends it to each local node to trigger re-fusion.
[0035] This invention provides a paperless office system based on intelligent workflow and multimodal data fusion, comprising: a document preprocessing module, used to perform tilt correction and page segmentation on multimodal document images, label the language of each text block, and generate a text block coordinate sequence with language labels;
[0036] The semantic embedding module is used to input the text block coordinate sequence into the multilingual multimodal pre-trained model and output the embedding vector of each text block with semantic type label;
[0037] The graph generation and entity extraction module is used to generate a layout semantic graph based on the spatial adjacency relationship of the embedded vector, perform semantic binding and currency normalization, and output normalized structured entities.
[0038] The local decision tree training module is used to train an incremental decision tree model with the structured entities and historical approval results, obtain local workflow sub-decision trees, and extract shareable metadata.
[0039] The global aggregation module is used to perform weighted histogram aggregation on the shareable metadata of each local node to generate a global skeleton workflow decision tree;
[0040] The decision tree fusion module is used to perform node-level fusion of the global skeleton workflow decision tree and the local workflow sub-decision tree to generate an intelligent workflow routing decision tree.
[0041] The routing output module is used to input the structured entity of the application to be processed into the intelligent workflow routing decision tree, and output the recommended approval chain and confidence level.
[0042] This invention transforms multimodal and multilingual heterogeneous office documents into normalized structured entities using a layout semantic graph. By using currency-amount binding edges and signature / term binding edges, fields with random positions in different layouts are mapped to fixed node relationships, making the field extraction results independent of the absolute position of the fields, thereby reducing the occurrence of field extraction mismatches in multi-layout documents.
[0043] This invention uses a weighted histogram aggregation algorithm to aggregate the sample distribution histograms of each local node at the coordination node. It only transmits feature indexes, split thresholds, and sample counts, without transmitting original document images and detail fields, thus meeting the data isolation requirements between subsidiaries.
[0044] This invention integrates the global skeleton workflow decision tree with the local workflow sub-decision tree at the execution node level. The global skeleton workflow decision tree provides local nodes with a cross-subsidiary common split structure in the feature dimension of sparse samples, while retaining the approval routing label of the local leaf nodes. This allows each local node to obtain effective workflow routing capabilities without waiting for sufficient local samples to accumulate, thereby shortening the cold start period of approval routing. Attached Figure Description
[0045] Figure 1 This is a flowchart of a paperless office method based on intelligent workflow and multimodal data fusion provided in an embodiment of the present invention;
[0046] Figure 2 This is a schematic diagram comparing the global weighted count and information gain of each candidate split point according to an embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram of the training sample size and weight distribution of the three subsidiaries provided in an embodiment of the present invention;
[0048] Figure 4 This is a schematic diagram of the normalized coordinate distribution of Japanese taxi invoice text blocks provided in an embodiment of the present invention;
[0049] Figure 5 This is a schematic diagram comparing the thresholds of split nodes before and after fusion, provided in an embodiment of the present invention.
[0050] Figure 6 This is a schematic diagram of the normalized result of the amount of the semantic graph node in the layout provided in the embodiment of the present invention;
[0051] Figure 7 This is a schematic diagram comparing the feature values of the routing path with the threshold of the intelligent workflow routing decision tree provided in this embodiment of the invention;
[0052] Figure 8 This is a schematic diagram of the distribution of incremental decision tree splitting nodes of subsidiary A provided in an embodiment of the present invention;
[0053] Figure 9 This is a schematic diagram illustrating the trend of data processing volume changes in each step provided in the embodiments of the present invention. Detailed Implementation
[0054] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.
[0055] Example
[0056] According to an embodiment of the present invention, a paperless office method based on intelligent workflow and multimodal data fusion is provided, such as... Figure 1 As shown, it includes the following steps:
[0057] Step 1: Obtain the multimodal document image and generate a text block coordinate sequence with language labels.
[0058] Images of multimodal documents submitted paperlessly are acquired at local office nodes of each subsidiary. Skew correction processing is performed on these images to obtain corrected document images. The corrected document images are then divided into text blocks. Language detection is performed on each text block, and a language label is generated for each block. Based on the layout segmentation results and language labels, a sequence of text block coordinates with language labels is generated. Each element in the language-labeled text block coordinate sequence contains the rectangular boundary coordinates of the text block and its corresponding language label.
[0059] It should be noted that the aforementioned tilt correction process refers to correcting geometric distortions generated during mobile shooting or scanning. Specifically, Hough transform is used to detect straight line edges in the image, the principal direction deflection angle is calculated, and an affine transformation is performed on the image to horizontally align the text lines. For ticket images captured by mobile devices, perspective correction is also performed before the affine transformation to eliminate trapezoidal distortion caused by the shooting angle.
[0060] It should be noted that the above-mentioned layout segmentation refers to dividing the corrected document image into text blocks based on connected component analysis. Each text block corresponds to a continuous text area in the document, such as the invoice header area, amount area, or signature area. The segmentation granularity is based on including semantically complete field information, and text content of the same field is grouped into the same text block.
[0061] Step 2: Fuse visual features and location information to generate a semantic embedding vector.
[0062] The coordinate sequence of text blocks with language labels is input into a multilingual multimodal pre-trained model. The model takes image region cropping, language labels, and rectangular boundary coordinates of each text block as input, fuses visual features, language labels, and positional coordinates, and outputs an embedding vector with semantic type labels for each text block. The semantic type labels identify the semantic category to which the text block belongs, including currency symbols, monetary values, taxes, signatures, and contractual text.
[0063] It should be noted that the aforementioned multilingual multimodal pre-trained model is a pre-trained Transformer architecture model. The input to the multilingual multimodal pre-trained model is a sequence of triples, each triple including an image region crop of the text block, a one-hot encoding of the language label, and a normalized four-dimensional vector of rectangular boundary coordinates; the output is a fixed-dimensional embedding vector and a classification probability distribution of the semantic type label. Specifically, the rectangular boundary coordinates undergo range-based mean normalization before being input into the multilingual multimodal pre-trained model, scaling the coordinate values to a uniform numerical range and eliminating the difference in coordinate dimensions between document images of different resolutions. In this invention, the multilingual multimodal pre-trained model operates in inference mode and does not perform parameter updates.
[0064] Step 3: Generate a semantic graph of the layout based on the embedded vectors and obtain normalized structured entities.
[0065] Based on the spatial adjacency relationships of the text block embedding vectors, a layout semantic graph is generated. The layout semantic graph is a directed graph structure, where nodes are text block embedding vectors with semantic type labels, and edges represent the semantic binding relationships between text blocks.
[0066] Step 3 may specifically include:
[0067] Step 301: Calculate the spatial adjacency distance between adjacent text blocks based on the rectangular boundary coordinates of each text block. When the shortest distance between the rectangular boundaries of two text blocks is less than the adjacency threshold, add an adjacency edge between the two nodes.
[0068] Step 302: Traverse the nodes with adjacent edges and perform semantic binding based on the semantic type label of the nodes. When the semantic type label of a node is a currency symbol and the semantic type label of its adjacent node is a monetary value, add a currency-amount binding edge between them to associate the currency symbol with the corresponding monetary value. When the semantic type label of a node is a signature and the semantic type label of its adjacent node is a clause text, add a signature-clause binding edge between them to associate the signature with the corresponding clause content.
[0069] Step 303: For the amount nodes with currency-amount binding edges in the semantic graph of the layout, obtain the currency symbol associated with the amount node, query historical exchange rate data, convert the amount value into the value under the base currency, and obtain the normalized amount value.
[0070] Step 304: Based on the semantic type labels, embedding vectors, and normalized monetary values of each node in the layout semantic graph, output normalized office document structured entities. Each normalized office document structured entity includes field names, field values, and binding relationships between fields.
[0071] It should be noted that the above adjacency threshold is a preset distance parameter used to control whether adjacent edges are formed between text blocks. The adjacency threshold can be set according to the average field spacing of the document, and the value range is from two percent to eight percent of the shorter side length of the document image.
[0072] In this embodiment, to ensure comparability of multilingual and multi-format documents from different subsidiaries within the same semantic space, the layout semantic graph unifies visual location features and semantic type features into a graph structure. By using currency-amount binding edges and signature / term binding edges, fields whose positions are originally random due to layout differences are mapped to fixed node relationships. This makes subsequent workflow routing independent of the absolute position of the fields, thereby reducing the impact of layout changes on the accuracy of field extraction.
[0073] Step 4: Generate a local workflow sub-decision tree using structured entities and historical approval results.
[0074] At each local office node, the normalized office document structured entity and its corresponding historical approval results are used as training samples and input into the incremental decision tree model for training. The incremental decision tree model takes the feature vector composed of the values of each field in the normalized office document structured entity as input, uses the historical approval results as labels, uses information gain as the splitting criterion, and incrementally expands the tree structure to output the local workflow sub-decision tree. After training, the feature index, splitting threshold, and sample distribution histogram of each splitting node are extracted from the local workflow sub-decision tree as shareable metadata.
[0075] It should be noted that the aforementioned feature vectors are composed of the values of each field in the normalized office document structured entity, arranged in a uniform field order. For numeric fields, the normalized values are directly used; for categorical fields, the category codes corresponding to the semantic type labels are used, that is, integer encoding is performed on categorical fields, converting non-numeric semantic type labels into corresponding integer indices. The feature vectors of all subsidiaries have the same dimension, and the field order follows a unified field index table.
[0076] It should be noted that the above sample distribution histogram refers to a two-dimensional histogram formed at each split node by dividing the training samples arriving at that node into several intervals according to the candidate feature values, and counting the number of samples of different approval result categories within each interval. The sample distribution histogram only includes the sample count of each interval and does not include any original document content.
[0077] In this embodiment, the shared metadata only includes feature indexes, split thresholds, and sample distribution histograms, and does not include original document images, field details, or employee information, thereby meeting data isolation requirements when participating in cross-subsidiary aggregation.
[0078] Step 5: Aggregate local metadata to generate a global skeleton workflow decision tree.
[0079] Each local office node can share metadata uploaded to the coordination node. The coordination node uses a weighted histogram aggregation algorithm to process the sample distribution histograms from each local node, calculates global split candidates, and generates a global skeleton workflow decision tree.
[0080] Step 5 may specifically include:
[0081] Step 501: The coordinating node receives the shareable metadata uploaded by each local office node and aligns the sample distribution histogram of each local node according to the feature index of the split node.
[0082] Step 502: For each local histogram under the same feature index, calculate the weight value according to the total number of samples of each local node. :
[0083]
[0084] in, For the first The weight value of each local node. For the first The total number of training samples for each local node. This is the local node's ID. The sum of the total number of training samples across all local nodes, i.e. , This represents the total number of local nodes. The sample count for the same interval in each local histogram is multiplied by the corresponding weight value. Then sum them up to obtain the global weighted histogram.
[0085] Step 503: Traverse each candidate split point in the global weighted histogram, calculate the information gain corresponding to each candidate split point, and select the candidate split point with the largest information gain as the global split candidate.
[0086] Step 504: Using global split candidates as the splitting basis for each layer of nodes, a global skeleton workflow decision tree is generated layer by layer from top to bottom. Each node in the global skeleton workflow decision tree only contains feature indexes and splitting thresholds, and does not contain specific approval route labels for leaf nodes.
[0087] Step 6: Integrate the global skeleton workflow decision tree with the local workflow sub-decision tree to generate an intelligent workflow routing decision tree.
[0088] The coordinating node distributes the global skeleton workflow decision tree to each local office node. Each local office node performs node-level fusion of the global skeleton workflow decision tree and its local workflow sub-decision trees to generate a fused intelligent workflow routing decision tree.
[0089] Step 6 may specifically include:
[0090] Step 601: Traverse the nodes of each layer of the global skeleton workflow decision tree from top to bottom. For each skeleton node, find the corresponding node with the same feature index in the local workflow sub-decision tree.
[0091] Step 602: When the splitting threshold of the skeleton node in the global skeleton workflow decision tree is inconsistent with the corresponding node in the local workflow sub-decision tree, replace the splitting threshold of the corresponding node in the local workflow sub-decision tree with the splitting threshold of the skeleton node in the global skeleton workflow decision tree, and retain the subtree structure of the corresponding node in the local workflow sub-decision tree. When there is no node corresponding to the skeleton node in the local workflow sub-decision tree, directly insert the skeleton node and its subtree into the corresponding level of the local workflow sub-decision tree.
[0092] Step 603: For each leaf node of the fused intelligent workflow routing decision tree, recalculate the approval route label and confidence level using local training samples to obtain the fused intelligent workflow routing decision tree.
[0093] In this embodiment, through node-level fusion, the global skeleton workflow decision tree provides a common splitting structure across subsidiaries, allowing local nodes to obtain global splitting candidates in the feature dimension where samples are sparse. Simultaneously, the approval routing labels of local leaf nodes are retained, enabling the fused intelligent workflow routing decision tree to take into account the unique approval rules of each subsidiary, thereby shortening the cold start period.
[0094] Step 7: Input the structured entity of the new application and output the recommended approval chain and confidence level.
[0095] When a new paperless office application is received, the normalized office document structure entity of the application is obtained according to the processing flow of steps 1 to 3. The feature vector corresponding to the normalized office document structure entity is input into the fused intelligent workflow routing decision tree, and the tree is judged layer by layer along the split nodes. After reaching the leaf node, the recommended approval chain and confidence score are output. The recommended approval chain includes the approval level sequence and the role identifier of the approver at each level. The confidence score is the proportion of historical samples in the leaf node that are consistent with the current recommendation result.
[0096] In this embodiment, to continuously optimize the global skeleton workflow decision tree, each local office node obtains the actual approval result of the application after the approval process is completed, including approval, rejection, endorsement, or transfer. The actual approval result is compared with the recommended approval chain to generate a feedback difference record. The feedback difference record includes three items: feature vector, recommended approval chain, and actual approval result, but does not include the original document image or field details. Each local node uploads the feedback difference record to the coordination node according to a preset update cycle. The coordination node re-executes the weighted histogram aggregation process in step 5 based on the accumulated feedback difference records, updates the global skeleton workflow decision tree, and distributes the updated global skeleton workflow decision tree to each local node to trigger the re-fusion in step 6.
[0097] This implementation transforms multimodal and multilingual heterogeneous office documents into unified, normalized structured entities using a layout semantic graph. Currency and amount binding edges, as well as signature and clause binding edges, map randomly positioned fields with varying layouts to fixed node relationships, ensuring that field extraction results are independent of their absolute positions. Therefore, paperless office samples from different subsidiaries are comparable within the same feature semantic space, reducing mismatches in field extraction across multiple document formats.
[0098] This implementation uses a weighted histogram aggregation algorithm to aggregate the sample distribution histograms of each local node at the coordination node. It only transmits feature indices, split thresholds, and sample counts, without transmitting original documents or detail fields, thus meeting data isolation requirements. The global skeleton workflow decision tree provides local nodes with a common splitting structure across subsidiaries on feature dimensions where samples are sparse. This allows each local node to obtain effective workflow routing capabilities without waiting for sufficient local sample accumulation, thereby shortening the cold start period for approval routing.
[0099] The following is an example of an application of the present invention, such as Figure 2-9 As shown, the implementation process is as follows:
[0100] A group has three subsidiaries (subsidiary A, subsidiary B, and subsidiary C), located in different regions. Their daily operations involve invoices and contracts in Chinese, English, and Japanese. In a certain quarter of 20XX, an employee of subsidiary A submitted a cross-border travel expense reimbursement application. The document included a Japanese taxi receipt (photographed on a mobile device) and a Chinese electronic accommodation invoice (scanned copy). Subsidiary A's local office nodes have a small sample size and are in a cold start phase. They need to utilize the global workflow decision tree of the coordination nodes to complete route recommendation.
[0101] The execution process of step 1;
[0102] The local node receives two document images submitted by an employee. The Japanese taxi invoice image has an approximately 8-degree tilt and trapezoidal distortion due to being handheld. The node first performs perspective correction to eliminate the trapezoidal distortion, then uses a Hough transform to detect the main direction of the text lines, calculates the tilt angle, and performs an affine transformation to horizontally align the text lines. The Chinese accommodation fee invoice is a scanned copy with only about 2 degrees of tilt, so an affine transformation is directly performed for correction.
[0103] After correction, connected component analysis was performed on both documents to divide the image regions into several text blocks. Taking a Japanese taxi invoice as an example, a total of 6 text blocks were divided, covering the invoice header, date, amount, tax, signature, and terms and conditions. After language detection was performed on each text block, a sequence of text block coordinates with language labels was generated.
[0104] Table 1. Coordinate sequence of Japanese taxi invoice text blocks
[0105] Text block number Rectangle boundary coordinates (top left and bottom right pixels) Language tags Preliminary semantic description Block 1 P(18,12) to P(310,48) Japanese Invoice header Block 2 P(18,55) to P(200,85) Japanese date Block 3 P(220,55) to P(380,85) Japanese Amount Block 4 P(220,92) to P(380,118) Japanese Taxes and Fees Block 5 P(18,125) to P(160,155) Japanese Signature area Block 6 P(170,125) to P(380,155) Japanese Terms and Conditions
[0106] The execution process of step 2;
[0107] Input the coordinate sequences of the above 6 text blocks into the multilingual multimodal pre-trained model. Taking block 3 (amount value) as an example, its original rectangular boundary coordinates are the top left corner. , bottom right corner , The image resolution is 398 pixels wide and 168 pixels high. Range-based mean normalization was used to scale the coordinates to a uniform numerical range. This, along with image region cropping of block 3 and one-hot encoding of the Japanese language label, forms a triple, which is then input into the model. The model outputs the embedding vector of block 3 and the classification probability distribution of the semantic type label, with the highest probability category being "amount". Embedding vectors were generated for all six text blocks, and the semantic type labels cover categories such as invoice header, date, amount, tax, signature, and clause text.
[0108] Table 2 Text block embedding vectors and semantic type label outputs
[0109] Text block number Language tags Semantic type tags Normalized coordinates ( ) Embedded vector dimension Block 1 Japanese Invoice header (0.05,0.07,0.78,0.29) 768 dimensions Block 2 Japanese date (0.05,0.33,0.50,0.51) 768 dimensions Block 3 Japanese Amount (0.55,0.33,0.95,0.51) 768 dimensions Block 4 Japanese Taxes and Fees (0.55,0.55,0.95,0.70) 768 dimensions Block 5 Japanese signature (0.05,0.74,0.40,0.92) 768 dimensions Block 6 Japanese Terms and Conditions (0.43,0.74,0.95,0.92) 768 dimensions
[0110] The execution process of step 3;
[0111] Based on the rectangular boundary coordinates of each text block, the spatial adjacency distance between adjacent text blocks is calculated. The shorter side of this invoice image is 168 pixels, and the adjacency threshold is 5% of the shorter side, approximately 8 pixels. The shortest distance between the rectangular boundaries of block 3 (amount value) and block 4 (tax) is 7 pixels, less than the threshold, so an adjacent edge is added. The shortest distance between the rectangular boundaries of block 5 (signature) and block 6 (term text) is 10 pixels, greater than the threshold, so no adjacent edge is added (this is only for illustrative purposes; in practice, the binding of signatures and terms is handled through a semantic binding step).
[0112] After traversing the nodes with adjacent edges, semantic binding is performed. In this invoice, a currency amount binding edge is added between the block containing the Japanese symbol "¥" (the adjacent block of block 3, detected as a currency symbol) and block 3 (amount value); a signature and clause binding edge is added between block 5 (signature) and block 6 (clause text).
[0113] For the currency symbol "¥" associated with block 3, historical exchange rate data was queried to convert the Japanese yen amount into the base currency (Chinese yuan). The invoice amount is 8500 Japanese yen, the applicable exchange rate is 0.048, and the result is 408 yuan.
[0114] Table 3. Semantic Graph Nodes and Binding Relationships
[0115] Node number Semantic type tags Original field value Bound edge type Bind target node Normalized amount (RMB) Node 1 Currency symbols ¥ Currency and Amount Binding Node 2 — Node 2 Amount 8500 yen Currency and Amount Binding Node 1 408.00 yuan Node 3 Taxes and Fees 850 yen — — 40.80 yuan Node 4 signature (Image of a seal) Signature and contract binding Node 5 — Node 5 Terms and Conditions Taxi Operating Terms Signature and contract binding Node 4 —
[0116] Output normalized office document structured entity: fields include document type (taxi invoice), amount (408.00 yuan), tax (40.80 yuan), language (Japanese), signature status (valid), and clause binding (bound). The binding relationship between fields is fully recorded in the page semantic graph.
[0117] The execution process of step 4;
[0118] Subsidiary A's local office node uses the structured entities of normalized office documents approved this quarter, along with their corresponding historical approval results, as training samples and inputs them into the incremental decision tree model. Feature vectors are arranged according to a unified field index table, including fields such as document type code, normalized amount, tax, language code, and signature status code. Numerical fields such as amount are directly represented by normalized values, while categorical fields such as language tags are encoded using integers (1 for Chinese, 2 for English, and 3 for Japanese).
[0119] Using information gain as the splitting criterion, the model performs the first-level split on the "normalized amount" field with a splitting threshold of 500 yuan, and the second-level split on the "language encoding" field, outputting a local workflow sub-decision tree. The feature index, splitting threshold, and sample distribution histogram of each splitting node are extracted from the sub-decision tree as shareable metadata.
[0120] Table 4 shows the shared metadata (split node information) of Subsidiary A.
[0121] Split node number Feature Index Feature Name Split threshold Total local sample size Number of samples in the left child node Number of samples in the right child node Node A Feature 2 Normalized amount 500 yuan 87 54 33 Node B Feature 4 Language Encoding 1.5 54 38 16
[0122] The execution process of step 5;
[0123] The coordinating node receives shareable metadata uploaded by the three subsidiaries. Each subsidiary has split nodes on feature 2 (normalized amount), and weighted histogram aggregation is performed after alignment by this feature index. The total training samples for the three subsidiaries are 87, 142, and 63 respectively. The weight values are respectively , , .
[0124] The sample counts of the same interval in each local histogram are multiplied by their corresponding weight values and then summed to obtain the global weighted histogram. Each candidate split point in the global weighted histogram is traversed, the information gain is calculated, and the candidate split point with the largest information gain is selected. For feature 2, the globally optimal split threshold is 600 yuan, which is inconsistent with the 500 yuan threshold of subsidiary A in its local histogram. For feature 4, the globally optimal split threshold remains consistent with that of each local node, maintaining a constant value of 1.5.
[0125] Table 5 Weighted histogram aggregation results (Feature 2: Normalized amount)
[0126] Candidate split points (element) Weighted count of subsidiary A Weighted average of subsidiary B Weighted average of subsidiary C Global weighted counting Information gain 300 8.40 18.62 6.38 33.40 0.12 500 16.20 28.91 11.44 56.55 0.31 600 14.70 35.28 9.02 59.00 0.47 800 9.60 22.54 7.26 39.40 0.28
[0127] The global optimal split threshold is 600 yuan. Based on this, a global skeleton workflow decision tree is generated layer by layer from top to bottom. Each node only contains feature index and split threshold, and does not contain leaf node approval route label.
[0128] The execution process of step 6;
[0129] The coordinating node distributes the global skeleton workflow decision tree to the local node of subsidiary A. The node traverses the skeleton nodes from top to bottom. At the node corresponding to Feature 2 (normalized amount), the skeleton node's split threshold is 600 yuan, while the local sub-decision tree's split threshold is 500 yuan; these two are inconsistent. The global threshold of 600 yuan is replaced with the local threshold of 500 yuan, preserving the subtree structure of the corresponding local node (including the feature 4 language-encoded split node and all leaf nodes). The threshold for the node corresponding to Feature 4 is consistent and does not require replacement.
[0130] After the replacement is completed, the approval route label and confidence level are recalculated for each leaf node of the merged intelligent workflow routing decision tree using the local training samples of subsidiary A.
[0131] Table 6 Comparison of split nodes before and after fusion
[0132] Feature Index Feature Name Local sub-decision tree threshold Global skeleton threshold After fusion threshold Subtree structure processing Feature 2 Normalized amount 500 yuan 600 yuan 600 yuan Preserve local subtree Feature 4 Language Encoding 1.5 1.5 1.5 No changes required
[0133] The execution process of step 7;
[0134] After processing through steps 1 to 3, employee-submitted travel expense reimbursement applications yield a normalized office document structured entity: document type code 3 (reimbursement form), normalized amount 408.00 yuan, tax 40.80 yuan, language code 3 (Japanese), and signature status code 1 (valid). Feature vectors are constructed using a unified field index table and input into the fused intelligent workflow routing decision tree.
[0135] At the split node of feature 2 (normalized amount), 408 yuan is less than the global threshold of 600 yuan, so it enters the left subtree; at the split node of feature 4 (language encoding), the Japanese encoding 3 is greater than the threshold of 1.5, so it enters the right subtree; upon reaching the leaf node, the recommendation approval chain and confidence level are output.
[0136] Table 7 Recommended Approval Chain Output Results
[0137] Fields content Application type Travel expense reimbursement (cross-border) Normalized amount 408.00 yuan Recommended approval level First tier: Department Head; Second tier: Finance Specialist Role identifiers for approvers at each level Role R01 (Department Head), Role R07 (Finance and Foreign Currency Specialist) Confidence 0.81 leaf node historical sample count 16
[0138] After approval, the local node obtains the actual approval result (passed, approval chain consistent with recommendation), generates a feedback differential record, including feature vector, recommended approval chain, and actual approval result, but excluding original document images or field details. This record is uploaded to the coordination node according to a preset update cycle, triggering the next round of weighted histogram aggregation and global skeleton workflow decision tree update.
[0139] Throughout the process, data starts with Japanese invoice images captured by a mobile device. After tilt correction and layout segmentation, text block coordinate sequences with language tags are generated. These are then fused using a multilingual, multimodal pre-trained model to generate semantic embedding vectors. Adjacency calculation, semantic binding, and currency conversion using the layout semantic graph yield normalized office document structured entities. Incremental decision tree training extracts shareable metadata. Weighted histogram aggregation of coordinating nodes generates a global skeleton workflow decision tree. Finally, this is merged with the local sub-decision tree to output a recommended approval chain for new applications. The output of each step strictly serves as the input for the next step. The data format gradually transforms from the original image into structured feature vectors, and then into routing decisions. No original document content is transmitted throughout the process, meeting data isolation requirements.
[0140] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.
Claims
1. A paperless office method based on intelligent workflow and multimodal data fusion, characterized in that, include: Perform tilt correction and layout segmentation on multimodal document images, label the language of each text block, and generate a sequence of text block coordinates with language labels; The text block coordinate sequence is input into a multilingual multimodal pre-trained model, which outputs an embedding vector for each text block with semantic type labels. Based on the spatial adjacency relationship of the embedded vector, a semantic graph of the layout is generated, semantic binding and currency normalization are performed, and a normalized structured entity is output. The incremental decision tree model is trained using the structured entities and historical approval results to obtain local workflow sub-decision trees and extract shareable metadata. The coordinating node performs weighted histogram aggregation on the shareable metadata of each local node to generate a global skeleton workflow decision tree; Each local node performs node-level fusion of the global skeleton workflow decision tree and the local workflow sub-decision tree to generate an intelligent workflow routing decision tree; The structured entity of the application to be processed is input into the intelligent workflow routing decision tree, and the recommended approval chain and confidence level are output.
2. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 1, characterized in that, The layout semantic graph is a directed graph structure, where nodes are text block embedding vectors with semantic type labels, and edges are semantic binding relationships between text blocks; The generated layout semantic graph includes: calculating the spatial adjacency distance between adjacent text blocks based on the rectangular boundary coordinates of each text block; when the shortest distance between the rectangular boundaries of two text blocks is less than the adjacency threshold, an adjacency edge is added between the two nodes. Traverse the node pairs with adjacent edges. When the semantic type label of a node is currency symbol and the semantic type label of its adjacent node is amount value, add a currency amount binding edge between the two. When a node's semantic type label is "signature" and its adjacent node's semantic type label is "term text", add a signature / term binding edge between the two. The adjacency threshold ranges from 2% to 8% of the short side length of the document image.
3. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 2, characterized in that, The currency normalization includes: for the amount nodes with currency amount binding edges in the semantic graph of the layout, obtaining the currency symbol associated with the amount node, querying historical exchange rate data, and converting the amount value into a value under the base currency. Each entity in the normalized structured entity contains field names, field values, and binding relationships between fields.
4. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 1, characterized in that, The weighted histogram aggregation includes: the coordinating node receiving shareable metadata uploaded by each local node, and aligning the sample distribution histograms of each local node according to the feature index of the split node; For each local histogram under the same feature index, the weight value is calculated according to the total number of samples in each local node. : ,in For the first The total number of training samples for each local node. This is the sum of the total number of training samples across all local nodes. This represents the total number of local nodes. Multiply the sample count of the same interval in each local histogram by the corresponding weight value. Then, accumulate the results to obtain the global weighted histogram; traverse each candidate split point in the global weighted histogram and select the candidate split point with the largest information gain as the global split candidate; The global skeleton workflow decision tree is generated layer by layer from top to bottom using global split candidates, where each node contains only feature index and split threshold.
5. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 1, characterized in that, The node-level fusion includes: traversing the nodes of each layer of the global skeleton workflow decision tree from top to bottom, and for each skeleton node, searching for a corresponding node with the same feature index in the local workflow sub-decision tree; When the split threshold of the skeleton node is inconsistent with that of the corresponding node, the split threshold of the skeleton node is used to replace the split threshold of the corresponding node, and the subtree structure of the corresponding node is preserved. When there is no node corresponding to the skeleton node in the local workflow sub-decision tree, the skeleton node and its subtree are directly inserted into the corresponding level of the local workflow sub-decision tree. For each leaf node after fusion, the approval route label and confidence level are recalculated using local training samples.
6. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 1, characterized in that, The multilingual multimodal pre-trained model is a Transformer architecture model, whose input is a sequence of triples. Each triple includes an image region cropping of the text block, one-hot encoding of the language label, and a normalized four-dimensional vector of the rectangular boundary coordinates. The rectangular boundary coordinates are normalized to mean based on range before input, scaling the coordinate values to a uniform range. The multilingual, multimodal pre-trained model runs in inference mode and does not perform parameter updates.
7. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 1, characterized in that, The shareable metadata includes the feature index, split threshold and sample distribution histogram of each split node, but does not include the original document image, field details or employee information. The sample distribution histogram refers to a two-dimensional histogram formed by dividing the training samples arriving at each split node into several intervals according to the candidate feature values, and counting the number of samples of different approval result categories in each interval.
8. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 1, characterized in that, The tilt correction includes: detecting straight line edges in the image through Hough transform, calculating the principal direction deflection angle, and performing an affine transformation on the image to make the text lines horizontally aligned; For ticket images taken on mobile devices, perspective correction is performed before affine transformation to eliminate trapezoidal distortion caused by the shooting angle.
9. The paperless office method based on intelligent workflow and multimodal data fusion according to claim 1, characterized in that, Also includes: After the approval process is completed, each local node obtains the actual approval result, compares the actual approval result with the recommended approval chain, and generates a feedback difference record. The feedback differential record includes feature vectors, recommended approval chains and actual approval results, but does not include original document images or field details. Each local node uploads the feedback differential records to the coordinating node according to the preset update cycle. The coordinating node re-executes the weighted histogram aggregation based on the accumulated feedback differential records, updates the global skeleton workflow decision tree, and sends it to each local node to trigger re-fusion.
10. A paperless office system based on intelligent workflow and multimodal data fusion, used to execute the paperless office method based on intelligent workflow and multimodal data fusion as described in any one of claims 1 to 9, characterized in that, include: The document preprocessing module is used to perform tilt correction and layout segmentation on multimodal document images, label the language of each text block, and generate a sequence of text block coordinates with language labels. The semantic embedding module is used to input the text block coordinate sequence into the multilingual multimodal pre-trained model and output the embedding vector of each text block with semantic type label; The graph generation and entity extraction module is used to generate a layout semantic graph based on the spatial adjacency relationship of the embedded vector, perform semantic binding and currency normalization, and output normalized structured entities. The local decision tree training module is used to train an incremental decision tree model with the structured entities and historical approval results, obtain local workflow sub-decision trees, and extract shareable metadata. The global aggregation module is used to perform weighted histogram aggregation on the shareable metadata of each local node to generate a global skeleton workflow decision tree; The decision tree fusion module is used to perform node-level fusion of the global skeleton workflow decision tree and the local workflow sub-decision tree to generate an intelligent workflow routing decision tree. The routing output module is used to input the structured entity of the application to be processed into the intelligent workflow routing decision tree, and output the recommended approval chain and confidence level.
