A method and system for intelligent splitting and instance segmentation of multiple types of document layout regions

Abstract

This invention discloses a method and system for intelligent segmentation and instance division of multi-type regions in document layout, belonging to the field of computer vision technology. The invention employs a "quality-driven adaptive process scheduling" mechanism, dynamically allocating computing power and selecting processing branches based on document health; designs a "four-modal cross-attention fusion network" to achieve macro-meta-micro three-level refined region segmentation; proposes a "segmentation-repair bidirectional iterative coupling" model to solve the problem of insufficient segmentation accuracy for low-quality documents; adopts a "segmentation-desensitization integrated network" to eliminate the risk of data leakage during data transfer; constructs an "anchor-free multi-objective parallel segmentation network" to support batch processing of multiple mixed documents; and establishes a "region-level D-S evidence theory verification system" to ensure the authenticity and legality of the segmentation results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for intelligent segmentation and instance division of multiple types of regions in a document layout. Background Technology

[0002] Document segmentation is the process of dividing a complete, potentially long document into multiple smaller, independent text fragments according to certain logic or rules. Traditional document segmentation algorithms and systems suffer from the following problems: Traditional document segmentation systems employ a fixed-parameter, fixed-process model, failing to differentiate between input document quality variations and performing the same computational steps on all images. For high-quality scanned documents, overly complex algorithms lead to computational redundancy; for low-quality, damaged documents, the fixed-parameter segmentation algorithm directly fails, generating numerous erroneous results and wasting significant computational resources. Furthermore, traditional systems lack a process circuit breaker mechanism, continuing to perform full-process calculations on severely damaged, unrepairable documents, resulting in ineffective consumption of computational resources.

[0003] Traditional document segmentation algorithms treat document restoration and segmentation as two independent sequential modules, first restoring the entire image and then performing segmentation. The restoration process does not consider the needs of subsequent segmentation, performing unnecessary restoration on non-segmentable regions, increasing computational load. Simultaneously, the segmentation results are affected by restoration errors; artifacts introduced during restoration lead to inaccurate segmentation boundaries. For severely damaged documents (such as incomplete or perspective-distorted documents), traditional methods cannot effectively restore the document structure, resulting in a sharp decline in segmentation accuracy.

[0004] Traditional document segmentation algorithms can only achieve coarse-grained region division (such as text areas and image areas), which cannot meet the needs of government documents for fine-grained segmentation. On the one hand, they lack multimodal feature fusion capabilities and rely solely on visual features, making it impossible to effectively distinguish regions with similar topological structures but different semantics (such as signatures and ordinary images); on the other hand, the segmentation granularity is limited, and it cannot identify minute elements such as table cells, checkboxes, QR codes, and sensitive fields.

[0005] Traditional document segmentation systems only support single-document processing. For scenarios with multiple documents mixed together or multiple pages stacked, manual splitting and processing of each document is required, which is extremely inefficient. In addition, the sorting of multi-page documents depends on page number recognition. When page numbers are blurry, missing, or out of order, sorting errors cannot be automatically corrected. Spread tables and front and back pages of documents require manual piecing together, making standardized automatic output impossible.

[0006] Traditional data anonymization processes are executed independently after document segmentation, requiring the transfer of sensitive data between different modules, which poses a risk of data leakage. Furthermore, the anonymization strategy is fixed and cannot adaptively adjust the anonymization intensity based on region type and sensitivity level; it also lacks a traceability mechanism for the anonymization operation, making it impossible to track the anonymization time, operator, and other information, thus failing to meet the security and compliance requirements of government systems.

[0007] Traditional document segmentation systems only output segmentation masks and do not provide verification of the results' authenticity. They cannot identify copied, altered, or forged documents, posing legal risks. Furthermore, the system lacks a closed-loop learning mechanism, failing to utilize feedback from human review to continuously optimize model parameters. Under new layouts or new types of damage scenarios, performance degrades rapidly, requiring frequent model retraining. Summary of the Invention

[0008] To address the aforementioned shortcomings in existing technologies, this invention provides a method and system for intelligent splitting and instance segmentation of multiple document layout regions. This method overcomes the limitations of traditional document segmentation technologies, which suffer from "single defect, single input, and lack of security guarantees." It constructs a full-link, global, unidirectional closed-loop segmentation process that includes "input quality pre-inspection - damaged document repair - multi-granularity instance segmentation - multi-document batch processing - sensitive information desensitization - region-level authenticity verification."

[0009] To achieve the aforementioned objectives, the present invention employs the following technical solution: a method for intelligent splitting and instance segmentation of multiple types of regions in a document layout, comprising the following steps: Based on the quality features of the original document image, a document health score is calculated through quality verification. This score is then combined with a quality threshold to adaptively call the corresponding processing branch and pass algorithm parameters, including: Process circuit break / degradation branch: Trigger process circuit break, label the original document image and adjust the quality threshold and quality feature weights, and resample the image. If resampling is not possible, perform degradation processing by using a lightweight segmentation model to generate the basic segmentation result of the labeled damaged area. Damaged document restoration branch: Perform macro-level multi-document instance segmentation on the original document image, automatically select the restoration operation for each independent document obtained by segmentation according to the passed restoration intensity parameter, and start the segmentation-restoration bidirectional iterative coupling model to output the restored document image and the optimized coarse-grained localization mask; after restoration, perform secondary quality verification on the document image, and call the corresponding processing branch and pass the algorithm parameters according to the quality verification result; Multimodal multigranularity segmentation branch: Start a four-modal cross-attention fusion network to extract four types of features from the input document image: visual, text, material and topology. Feature fusion is performed through a topological prior attention mechanism. The fused feature map is split into macro-meta-micro three-level progressive segments to generate a full-granularity region mask. Based on macro-level multi-document boundaries and masks, an anchorless multi-objective parallel segmentation network is used to generate independent segmentation masks and global coordinate mapping matrices for all documents from the fused feature map. The documents are sorted based on topological constraints, and then the full-granularity region masks are transformed and spliced, and standardized and spliced ​​to obtain standardized documents and aligned region masks. Based on standardized documents and aligned region masks, a segmentation-desensitization integrated network is adopted. The global coordinate mapping matrix is ​​used to perform post-coordinate transformation on the full-granularity region mask and the fused feature map. Based on the transformed shared features, sensitive information is located and desensitized in the document copy, and desensitized document is output. A pixel-level coordinate mapping table is established between the encrypted original document image and the desensitized document. The credibility score of each segmented region in the fused feature map is performed based on the DS evidence theory. A Merkel hash tree credibility certificate with digital signature is generated, and the instance segmentation process is closed-loop learned through error attribution.

[0010] Furthermore, a document health score is calculated through quality verification, including: The weights of each quality feature in the original document image are calculated using the dynamic entropy weighting method updated by Bayesian posterior, and the document health score is obtained by weighting. When calling the processing branch, the quality threshold is determined based on the optimal cutoff point of the ROC curve of the historical document annotation dataset, and updated based on the quality assessment error of closed-loop feedback, and supports scenario-based threshold configuration.

[0011] Furthermore, in the damaged document repair branch, the bidirectional coupling iterative process of the segmentation-repair bidirectional iterative coupling model includes: In the forward iteration process, the repaired image is used to generate an initial single-document coarse-grained boundary positioning mask. In the reverse iteration process, the mask is used to constrain and repair the boundary. In addition, the mask confidence is calculated in real time during the iteration process, and a general prior template for document correction is introduced when the mask confidence is lower than a preset confidence threshold.

[0012] Furthermore, in the damaged document repair branch, the variational bidirectional coupling objective function of the segmentation-repair bidirectional iterative coupling model... for: In the formula, In probability distribution The mathematical expectation operator below, In probability distribution The mathematical expectation operator below, Given an original document image and coarse-grained positioning mask At that time, the document image after repair The conditional probability distribution, Given an original document image At that time, coarse-grained positioning mask The conditional probability distribution, Coarse-grained positioning mask The prior distribution, For a given repaired document image At that time, coarse-grained positioning mask The conditional probability distribution, for divergence, For the repaired document image, The original document image, Coarse-grained positioning masks for segmenting each region; The update and iteration rules are as follows: In the formula, For the first The next iteration yields the optimized coarse-grained localization masks for each region. For the first The next iteration yields the repaired document image. At that time, the corresponding coarse-grained positioning mask The conditional probability distribution, Given an original document image and optimal coarse-grained positioning mask At that time, the document image after repair The conditional probability distribution, For the first The repaired document image obtained in the next iteration, index For the number of iterations, This is a set of coarse-grained location masks.

[0013] Furthermore, in the multimodal, multi-granularity segmentation branch, the formula for feature fusion using the topological prior attention mechanism is as follows: In the formula, To fuse feature maps, This is a topological prior attention function. For visual feature maps, For text feature maps, This is a material feature map. For the input feature map, This is a document topological structure feature map. For learnable query projection matrix, The learnable bond projection matrix. The projection matrix is ​​the value of the learnable value. Let be the dimension of the key vector. For activation functions, superscript It is the transpose operator; Three-level cascaded loss when performing macro-meta-micro three-level progressive splitting on the fused feature map for: In the formula, For macro-level segmentation loss, For meso-level segmentation loss, For micro-level segmentation loss, , and They are respectively , and The weight, These are the weighting coefficients for cross-granularity consistency constraints. For macro-level segmentation mask, For meso-level segmentation mask, For micro-level segmentation mask, This is the upsampling operator.

[0014] Furthermore, the documents are sorted based on topological constraints, including: Topological features are extracted from each document segmented by the anchorless multi-objective parallel segmentation network, and document transition probabilities are constructed. The Markov chain algorithm based on topological constraints sorts the documents and tags them based on the topological constraints. The topological constraints, from highest to lowest priority, are: page number order, date order, text direction, and content similarity.

[0015] Furthermore, in the segmentation-desensitization integrated network, a corresponding desensitization strategy is matched according to the region type and sensitivity level, and an invisible hash traceability watermark is added to each desensitized region; In the segmentation-desensitization integrated network, feature sharing multi-task loss for: In the formula, For multi-granularity segmentation loss, For the loss of desensitization mask generation, The weighting coefficients for the desensitization mask loss are... The weight coefficients for the privacy regularization term. To grade the desensitization intensity, The area is classified as sensitive. For the length of the region, This is the desensitization threshold.

[0016] Furthermore, a credibility score based on DS evidence theory is applied to each segmented region in the fused feature map to generate a digitally signed Merkle hash tree credibility certificate, including: Based on a pixel-level coordinate mapping table, each segmented region in the fused feature map is independently verified for authenticity. By integrating the segmentation confidence, authenticity identification results, and region integrity of each segmented region, the credibility score of each segmented region is calculated based on the DS evidence theory. For the segmented regions corresponding to the verified credibility scores, generate Merkel hash tree trusted credentials for digital signature.

[0017] Furthermore, the unified update framework for closed-loop learning during instance segmentation through error attribution is represented as follows: In the formula, For all learnable parameters throughout the entire process, To provide an incorrect attribution function for using source attribution, For the first Closed-loop feedback data of the wheel, For the parameter version number, To update the function, gradient descent is used for differentiable parameters, and statistical learning and rule-based iterative updates are used for non-differentiable parameters.

[0018] A document layout multi-type region intelligent splitting and instance segmentation system, comprising: Input pre-detection and adaptive scheduling module: used to perform quality verification on the original document image, and adaptively call the corresponding processing module and pass algorithm parameters based on the document health score of the quality verification; Process Circuit Breaker / Degradation Module: Used to trigger process circuit breaking, label the original document image and adjust the quality threshold and quality feature weights, and resample the image. When resampling is not possible, it performs degradation processing by generating a basic segmentation result of the labeled damaged area using a lightweight segmentation model. Damaged document repair module: It performs macro-level multi-document instance segmentation on the original document image, automatically selects repair operation for each independent document obtained by segmentation according to the passed repair intensity parameter, and outputs the repaired document image and the optimized coarse-grained localization mask through the deployed bidirectional iterative coupling model of segmentation-repair; and performs secondary quality verification on the document image after repair, and calls the corresponding processing module and passes the algorithm parameters according to the quality verification result. Multimodal multigranularity segmentation module: Deploys a four-modal cross-attention fusion network to simultaneously extract four types of features from the input document image: visual, text, material, and topological features. It performs feature fusion through a topological prior attention mechanism and performs a three-level progressive split of the fused feature map: macro-meta-micro, to generate a full-granularity region mask. Anchor-free multi-objective parallel segmentation module: This is an extension of the multimodal multi-granularity segmentation module in batch scenarios. It is used to generate independent segmentation masks and global coordinate mapping matrices for all documents from the fused feature map through the anchor-free multi-objective parallel segmentation network, and to sort each document based on topological constraints. Then, it performs coordinate transformation and splicing on the original region masks, and performs standardized splicing to obtain standardized documents and aligned region masks. The segmentation and desensitization module is coupled with the multimodal and multigranular segmentation module to achieve synchronous execution of segmentation and desensitization in the same thread. During desensitization, the segmentation-desensitization integrated network is used to perform post-coordinate transformation on the full-granularity region mask and fused feature map. Based on the transformed shared features, sensitive information is located and desensitized in the document copy, and the desensitized document is output. Regional-level authenticity verification and trusted output module: used to verify each segmented region in the fused feature map based on DS evidence theory and generate trusted credentials using Merkle hash trees, and to perform closed-loop learning of each module of the instance segmentation system through error attribution.

[0019] The beneficial effects of this invention are as follows: (1) Full-scene adaptive input processing and optimal allocation of computing power By employing a document health assessment based on Bayesian dynamic entropy weights and a continuously configurable scheduling mechanism, the system achieves full-scenario input adaptation, ranging from high-quality scanned documents to severely damaged historical archives. It dynamically allocates computing power based on input quality, triggering process circuit breakers for severely damaged documents to avoid wasting computing power. A lightweight model is used for high-quality documents, significantly improving processing speed. It also provides degradation processing capabilities for historical archives that cannot be re-collected, covering more practical application scenarios.

[0020] (2) High-precision segmentation capability for low-quality damaged documents This innovative approach proposes a bidirectional iterative coupling model for segmentation and repair, deeply integrating the repair and segmentation processes. It uses a segmentation mask to constrain the repair boundaries, filling in only the missing portions within the segmented region, significantly reducing the computational load for repair and avoiding the impact of repair artifacts on the segmentation results. Furthermore, a priori template correction mechanism is introduced during the iteration process, effectively addressing the error amplification problem and significantly improving the segmentation accuracy of low-quality documents.

[0021] (3) Macro-Meso-Micro Three-Level Multi-Granularity Instance Decomposition A four-modal cross-attention fusion network is designed, integrating visual, textual, material, and topological features. The strong topological constraints of government documents enhance feature fusion accuracy. A three-level progressive segmentation mechanism achieves full-granularity segmentation from document instances down to minute elements such as table cells and checkboxes. It also supports the recognition and segmentation of handwritten signatures and annotations, comprehensively meeting the refined processing needs of government documents. It supports small-sample adaptive learning, enabling rapid adaptation to new document layouts.

[0022] (4) Efficient batch processing and standardized output of multiple documents An anchor-free, multi-objective parallel segmentation network is employed to achieve simultaneous segmentation of multiple document instances within the same scene, significantly improving batch processing speed. A topology-constrained Markov chain sorting algorithm is proposed, combining preliminary page number consistency detection with subsequent DS tampering verification. This algorithm enables automatic sorting and anti-tampering error correction of multi-page documents without relying on page numbers, effectively solving sorting errors caused by blurred, missing, or tampered page numbers. Standardized output functions such as automatic front-and-back stitching of ID cards and automatic splicing of cross-page tables are supported, fully meeting the format requirements of government archives.

[0023] (5) End-to-end data security and compliance anonymization A segmentation-desensitization integrated network was constructed, achieving synchronous feature sharing through global coordinate mapping. This enabled the simultaneous execution of sensitive information location and desensitization within the same thread, eliminating the risk of leakage during data transfer. Original images were transmitted and stored with end-to-end encryption throughout the process, further ensuring data security. A tiered desensitization strategy was adopted, adaptively adjusting the desensitization intensity based on region type and sensitivity level. An invisible hash-based traceability watermark was added to each desensitized region, supporting full-process traceability of the desensitization operation and fully complying with the security and compliance requirements of government systems.

[0024] (6) Regional-level trusted verification and closed-loop self-optimization mechanism A regional-level authenticity verification system based on DS evidence theory is established, independently verifying the authenticity of each segmented region. This effectively identifies documents that have been copied, altered, or forged. A Merkel hash tree-based trusted credential with digital signatures is generated, supporting regional-level batch verification and serving as evidence of data integrity and source authenticity for judicial acceptance. Through a hybrid architecture and a closed-loop learning mechanism, employing source-based error attribution and utilizing manual review results and negative samples to continuously optimize end-to-end parameters, the system's adaptability to new scenarios is significantly improved, eliminating the need for frequent model retraining. Attached Figure Description

[0025] Figure 1 The flowchart of the document layout multi-type region intelligent splitting and instance segmentation method provided by the present invention. Detailed Implementation

[0026] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0027] Example 1:

[0028] This invention provides a method for intelligent segmentation and instance division of multiple types of document layout regions. By introducing a hybrid architecture bidirectional closed-loop high-reliability segmentation algorithm, a "full-link information-driven" document processing architecture is constructed, unifying six stages—input quality assessment, damage repair, instance division, batch processing, desensitization, and verification—into a single framework. Gradient backpropagation is used to update differentiable parameters, while statistical learning and rule iterative updates are used for non-differentiable parameters, achieving closed-loop learning in the segmentation process.

[0029] refer to Figure 1 A method for intelligent splitting and instance segmentation of multiple types of document layouts, comprising the following steps: Based on the quality features of the original document image, a document health score is calculated through quality verification. This score is then combined with a quality threshold to adaptively call the corresponding processing branch and pass algorithm parameters, including: Process circuit break / degradation branch: Trigger process circuit break, label the original document image and adjust the quality threshold and quality feature weights, and resample the image. If resampling is not possible, perform degradation processing by using a lightweight segmentation model to generate the basic segmentation result of the labeled damaged area. Damaged document restoration branch: Perform macro-level multi-document instance segmentation on the original document image, automatically select the restoration operation for each independent document obtained by segmentation according to the passed restoration intensity parameter, and start the segmentation-restoration bidirectional iterative coupling model to output the restored document image and the optimized coarse-grained localization mask; after restoration, perform secondary quality verification on the document image, and call the corresponding processing branch and pass the algorithm parameters according to the quality verification result; Multimodal multigranularity segmentation branch: Start a four-modal cross-attention fusion network to extract four types of features from the input document image: visual, text, material and topology. Feature fusion is performed through a topological prior attention mechanism. The fused feature map is split into macro-meta-micro three-level progressive segments to generate a full-granularity region mask. Based on macro-level multi-document boundaries and masks, an anchorless multi-objective parallel segmentation network is used to generate independent segmentation masks and global coordinate mapping matrices for all documents from the fused feature map. The documents are sorted based on topological constraints, and then the full-granularity region masks are transformed and spliced, and standardized and spliced ​​to obtain standardized documents and aligned region masks. Based on standardized documents and aligned region masks, a segmentation-desensitization integrated network is adopted. The global coordinate mapping matrix is ​​used to perform post-coordinate transformation on the full-granularity region mask and the fused feature map. Based on the transformed shared features, sensitive information is located and desensitized in the document copy, and desensitized document is output. A pixel-level coordinate mapping table is established between the encrypted original document image and the desensitized document. The credibility score of each segmented region in the fused feature map is performed based on the DS evidence theory. A Merkel hash tree credibility certificate with digital signature is generated, and the instance segmentation process is closed-loop learned through error attribution.

[0030] In this embodiment of the invention, for the input original document image, four quality features are calculated: blurriness, exposure, damage rate, and ROI integrity. Blurriness is calculated based on Laplacian variance, with a value range of [0, 100], where lower values ​​indicate greater blurriness. Exposure is calculated based on grayscale histogram distribution, with a value range of [0, 100], where deviations from 50 indicate more abnormal exposure. Damage rate is calculated based on edge detection and connected component analysis, with a value range of [0, 100], where higher values ​​indicate more severe damage. ROI integrity is calculated based on document boundary detection, with a value range of [0, 100], where lower values ​​indicate more missing document elements.

[0031] In this embodiment of the invention, calculating a document health score through quality verification includes: The weights of each quality feature in the original document image are calculated using the dynamic entropy weighting method updated by Bayesian posterior, and the weighted average is used to obtain the document health score.

[0032] Specifically, a dynamic entropy weight method with Bayesian posterior update replaces the traditional fixed weights, supporting closed-loop self-optimization in the segmentation process. During quality assessment, based on statistical analysis of a dataset of 100,000 labeled government documents, the quality assessment error approximately follows a normal distribution, and its update formula is: In the formula, For a given number After the closed-loop feedback data of the wheel, the weights The posterior probability distribution, For the first The weights of each quality feature For the first Feedback dataset for closed-loop learning The document quality assessment error approximately follows a normal distribution with a mean of 0 and a variance equal to the historical error variance. To predict health status for the model, This represents the true health status as manually labeled. The variance of historical quality assessment error. For the first The first round of closed-loop learning The weights of each quality feature For the first Mathematical expectation operator under the posterior probability distribution after closed-loop learning.

[0033] In the above formula, the initial prior is calculated using the entropy weight method of a dataset of 100,000 labeled government documents, where the ambiguity is 0.35, the exposure is 0.25, the damage rate is 0.25, and the ROI integrity is 0.15.

[0034] This yields a document health score that incorporates uncertainty quantification. for: In the formula, For quality feature weight vector, For quality feature vectors, A quantitative value for the uncertainty of document health score. Weight vector The covariance matrix, superscript This is the transpose operator.

[0035] In this embodiment of the invention, when calling the processing branch, the quality threshold is determined based on the optimal cutoff point of the ROC curve of the historical document annotation dataset, and is updated based on the quality assessment error of the closed-loop feedback, and supports scenario-based threshold configuration.

[0036] Specifically, the initial quality threshold (30 / 70) is determined by the optimal cutoff point of the ROC curve of the dataset of 100,000 government documents. After every 100 labeled circuit breaker samples, the ROC curve is recalculated and the global quality threshold is updated. The step size of each adjustment does not exceed 1, and the upper and lower limits of the global threshold are limited to [20,40]. At the same time, the ROC curve is recalculated and the global quality threshold (70) is updated every quarter based on the quality assessment error of the closed-loop feedback. The step size of each adjustment does not exceed 5 (based on the step size sensitivity analysis of 100,000 samples, a step size >5 will lead to a computing power imbalance of >10%).

[0037] Furthermore, for specific scenarios (such as historical archives and handwritten documents), it supports scenario-based quality threshold configuration, including: Configuration permissions: System administrator; Scope of application: Specified business scenarios (does not affect the overall system); Closed-loop learning: The update of the scenario-based threshold is independent of the global threshold, and uses the feedback data of the corresponding scenario; Version management: Scenario-specific parameters use independent version branches, the global parameter version number records the core parameters, and the trusted credential contains both the global version number and the scenario-specific version number.

[0038] Based on the quality thresholds determined above, a piecewise linear soft scheduling function for handling branch calls is constructed. (In conjunction with hard thresholds, to achieve a smooth transition of computing power), it is represented as: Based on the document health score and quality threshold determined above, the parameters for passing the algorithm include: when : Repair strength is The edge detection threshold is Use the standard segmentation model; when : Using the minimum repair strength and the highest edge detection threshold (100), switch to the lightweight segmentation model; when : Linearly decreasing repair intensity and edge detection threshold with a smooth transition, using a standard segmentation model.

[0039] In this embodiment of the invention, for The original document image is called in the circuit breaker / degradation branch of the process, directly triggering the circuit breaker, terminating all subsequent calculations, and popping up an image resampling prompt; the circuit breaker sample needs to be associated with the manually labeled "whether it can be repaired" tag. If labeled "repairable", the sample is added to the repair module training set; if labeled "unrepairable", the sample is added to the negative sample set of the quality assessment module, and the weight of the corresponding quality feature is optimized; a degradation processing entry is provided for documents that cannot be resampled: a lightweight segmentation model is used to generate basic segmentation results with labeled damaged areas. The degradation results are not entered into automatic credibility verification and are forcibly marked as "awaiting manual review", and their credibility score is uniformly set to 0.5.

[0040] In this embodiment of the invention, for For low-quality images, the damaged document repair branch is invoked. By constructing a segmentation-repair bidirectional iterative coupling model (SRBIM), the problem of insufficient segmentation accuracy of low-quality documents caused by the disconnect between repair and segmentation in traditional methods is solved.

[0041] In this embodiment, for each individual document obtained from segmentation, a repair operation is automatically selected based on the transmitted repair strength parameter, including: Minor damage ( ): Only perform mid-slit shadow removal and minor exposure correction; Moderate damage ( Added surface text flattening and edge super-resolution reconstruction; Severe damage ( ): Add semantic completion for incomplete areas and 3D perspective distortion correction.

[0042] In this embodiment, the bidirectional coupling iterative process of the segmentation-repair bidirectional iterative coupling model includes: In the forward iteration process, the repaired image is used to generate an initial single-document coarse-grained boundary positioning mask. In the reverse iteration process, the mask is used to constrain and repair the boundary, and only the missing parts within the positioning area are filled in. In addition, the mask confidence is calculated in real time during the iteration process, and a general prior template for document correction is introduced when the mask confidence is lower than the preset confidence threshold.

[0043] The bidirectional iterative process of the segmentation-repair bidirectional iterative coupling model is modeled as a probabilistic inference problem, with its variational bidirectional coupling objective function. for: In the formula, In probability distribution The mathematical expectation operator below, In probability distribution The mathematical expectation operator below, Given an original document image and coarse-grained positioning mask At that time, the document image after repair The conditional probability distribution, Given an original document image At that time, coarse-grained positioning mask The conditional probability distribution, Coarse-grained positioning mask The prior distribution, For a given repaired document image At that time, coarse-grained positioning mask The conditional probability distribution, for divergence, For the repaired document image, The original document image, Coarse-grained positioning masks for segmenting each region; The update and iteration rules are as follows: In the formula, For the first The next iteration yields the optimized coarse-grained localization masks for each region. For the first The next iteration yields the repaired document image. At that time, the corresponding coarse-grained positioning mask The conditional probability distribution, Given an original document image and optimal coarse-grained positioning mask At that time, the document image after repair The conditional probability distribution, For the first The restored document image obtained in the next iteration, index For the number of iterations, This is a set of coarse-grained location masks.

[0044] The convergence criterion for the above model with an uncertainty threshold is: In the formula, For the first In the next iteration, given the original document image Under these conditions, granular positioning mask The variational approximation of the posterior probability distribution.

[0045] This proves that when the objective function is convex, the iterative process will inevitably converge; for the non-convex case, the system stability is guaranteed by setting the maximum number of iterations.

[0046] The model is iterated until convergence (KL change rate is less than 1% for three consecutive iterations) or the maximum number of iterations (default 20) is reached, and the final repaired image and optimized coarse-grained localization mask are output. If convergence is not achieved after 20 iterations, the result of the last iteration is output and marked as "awaiting manual review".

[0047] After the repair is completed, a second quality check is performed (standard: single document health S'≥70). If the standard is met, the result is passed to the multimodal multigranular segmentation branch for fine-grained region splitting. If the standard is not met, enhanced repair (adding generative semantic completion and multi-scale repair branches) or process circuit breaking is triggered.

[0048] In this embodiment of the invention, for For high-quality images or restored images that meet the standards, a multimodal, multi-granularity segmentation branch is invoked. Through a four-modal cross-attention fusion network and a three-level progressive splitting mechanism, document region instance segmentation with full granularity at the macro, meso, and micro levels is achieved, solving the problems of coarse segmentation granularity and low accuracy of traditional algorithms.

[0049] In this embodiment, four types of features are extracted simultaneously using a four-modal cross-attention fusion network (QMCAN). Specifically, visual features of the image are extracted based on ResNet, including edge, texture, and color features; text features of printed and handwritten text lines are extracted based on CRNN; material features of paper, including texture and reflectivity, are extracted based on gray-level co-occurrence matrix; and topological features of the document layout structure are extracted based on graph neural network.

[0050] In this embodiment, the formula for feature fusion using the topological prior attention mechanism in the multimodal, multi-granularity segmentation branch is as follows: In the formula, To fuse feature maps, This is a topological prior attention function that uses topological features as the query guide and applies weighted enhancements to the basic features. Visual feature maps (based on ResNet to extract edge, texture and color features). This is a text feature map (based on CRNN to extract semantic features of printed and handwritten text lines). This is a material feature map (based on gray-level co-occurrence matrix to extract paper texture and reflective features). For the input feature map, This is a document topological structure feature map. For learnable query projection matrix, The learnable bond projection matrix. The projection matrix is ​​the value of the learnable value. Let be the dimension of the key vector. For activation functions, superscript This is the transpose operator.

[0051] In this embodiment, a three-level progressive split is performed on the fused feature map, including: Macro level: Identify multiple document instances in the same scene and generate document bounding boxes; Intermediate level: Each document is divided into large areas such as title area, body text area, table area, signature area, portrait area, and handwritten annotation area; Microscopic level: Identifies tiny elements such as table cells, checkboxes, QR codes, and sensitive fields.

[0052] Furthermore, during the splitting process, if a new, unseen format is encountered, a small-sample adaptive learning mechanism is activated: only 10-20 fully labeled samples of the same format are needed, and adaptation can be achieved by fine-tuning the last two layers of the model. After adaptation, the segmentation accuracy is no less than 90%. The new branch is automatically integrated into the full system parameter update system. If there are fewer than 10 samples of the same format, a general model is used and it is forcibly marked as "awaiting manual review". Once the number of samples reaches 10, small-sample adaptation is automatically triggered.

[0053] In this embodiment, the three-level cascaded loss is used when performing a macro-meta-micro three-level progressive split on the fused feature map. for: In the formula, For macro-level segmentation loss, For meso-level segmentation loss, For micro-level segmentation loss, , and They are respectively , and The weight, These are the weighting coefficients for cross-granularity consistency constraints. For macro-level segmentation mask, For meso-level segmentation mask, For micro-level segmentation mask, This is the upsampling operator.

[0054] in, As an upsampling operator, it upsamples the low-resolution mask to the same dimension as the high-resolution mask, thus solving the problem of cross-granularity tensor dimension mismatch. , and weight , and The changes in document health score are as follows: when : (Prioritize ensuring the accuracy of macroscopic boundaries); when : (Balanced Level 3 Precision); when : (Prioritize ensuring the accuracy of microscopic elements).

[0055] In this embodiment, for the output of multimodal multi-granularity segmentation branches, an anchorless multi-objective parallel segmentation network and a topological constraint sorting algorithm are used to achieve efficient processing and standardized output of multi-document mixed scenarios, thus solving the problem of low batch processing efficiency in traditional systems.

[0056] In this embodiment, an anchorless multi-object parallel segmentation network (AF-MTSN) is used to simultaneously generate independent segmentation masks and global coordinate mapping matrices for all documents based on the fused feature map. For regions with document overlap greater than 50%, mask separation is performed based on edge features and topology. If separation fails, it is marked as "awaiting manual review". For regions with document overlap greater than 80%, manual splitting is triggered directly, and automatic mask separation is not performed.

[0057] In this embodiment, sorting documents based on topological constraints includes: Topological features are extracted from each document segmented by the anchorless multi-objective parallel segmentation network, and document transition probabilities are constructed. The Markov chain algorithm based on topological constraints sorts the documents and tags them based on the topological constraints. Among them, the priority of topological constraints from high to low is as follows: page number order, date order, text direction, and content similarity.

[0058] Specifically, the topological features extracted for each document include page number, date, document orientation, and content similarity. A document transition probability matrix is ​​constructed, and its calculation formula is as follows: In the formula, For document The document that follows The transition probability, For document With Documents Content similarity between them For document With Documents Similarity between them This is the topological feature weight vector. For document With Documents The topological feature vector.

[0059] The formula for sorting documents using the Markov chain algorithm based on topological constraints is as follows: In the formula, This is the optimal document arrangement sequence. For the first The location document is immediately followed by the first The transition probability of a document. This is an indicator function that takes the value 1 if the condition is true and 0 otherwise. For the first in the sorting The document number corresponding to each location and Sort the documents respectively and documents The corresponding document number, For a set of topological constraints, the subscript is... The position index in the sorted sequence, subscript This represents the total number of pages in the documents to be sorted divided by the total number of instances.

[0060] In this embodiment, a Markov chain algorithm with topological constraints is used to solve for the optimal document sorting sequence, correcting the multi-page disorder problem and supplementing the sorting strategy with blank pages / pure image pages as fallback. When constraints conflict, the higher priority constraint takes precedence and is marked as "awaiting manual review". A preliminary page number consistency check (number continuity, font consistency) is first performed. If an anomaly is found, it is temporarily marked as "sorting to be verified". After subsequent DS verification confirms that the page number has been tampered with, the sorting is automatically backtracked and corrected to "date + content similarity" and upgraded to a high-risk mark.

[0061] In this embodiment, during the document sorting process, for multi-page documents without page numbers, dates, or content cosine similarity less than a threshold, a "physical location sorting + manual confirmation" mode is adopted. Furthermore, by default, the documents are sorted according to their order from top to bottom and from left to right in the original image and marked as "awaiting manual review". The results of manual review are incorporated into closed-loop learning to optimize the topological feature extraction algorithm.

[0062] In this embodiment, based on the generated coordinate mapping matrix and document sorting results, coordinate transformation and splicing are performed on the original region mask to ensure that the region mask of the spliced ​​document is completely aligned with the pixel coordinates; when splicing cross-page tables, the corresponding cells of the cross-page are determined by "header feature matching + row coordinate alignment"; the masks of the corresponding cells are merged, and the sensitive field markings are inherited from the original cells. If multiple original cells are sensitive fields, the merged cell is marked as sensitive.

[0063] Furthermore, during standardized assembly, the segmented documents are automatically assembled into a standard format; for example, according to government archiving standards, standardized assembly includes: ID card / business license: Identify the front and back of the same document by matching the document number features and automatically combine them into an A4 size; Multi-page documents: Generate multi-page TIFF / PDF files based on sorted results; Split-page tables: Automatically stitch together data using header recognition, row and column alignment, and merged cell matching; supports exporting to Excel.

[0064] In this embodiment of the invention, a segmentation-desensitization integrated network (SDN) is adopted. Based on the unified coordinate system after multi-target assembly, the features of multi-modal and multi-granularity segmentation branches are synchronously transformed to the unified coordinate system through a global coordinate mapping matrix, and the sensitive information is located synchronously by sharing features.

[0065] Specifically, in the segmentation-de-identification integrated network (SDN), the original encrypted image is transmitted and stored with end-to-end encryption using AES-256 throughout the entire process. The de-identification operation only applies to the document copy, and the original encrypted image is completely preserved for subsequent authenticity verification.

[0066] The corresponding de-identification strategy is matched according to the region type and sensitivity level, and an invisible hash traceability watermark is added to each de-identified region; for example: Highly sensitive areas (ID card number, face, bank card number): Forced irreversible pixelation desensitization (τ=1); Medium-sensitive areas (address, seal, phone number): partially hidden and desensitized (the hiding ratio is calculated by a formula); Low-sensitivity areas (plain text, headings): No desensitization (τ=0).

[0067] Furthermore, when adding an invisible hash traceability watermark to each desensitized area, the watermark adopts a frequency domain embedding method, embedding only the redundant pixels of the desensitized area, which is in a different frequency domain from the original anti-counterfeiting watermark of the document and does not interfere with each other; at the same time, information such as desensitization time, operator ID, and device ID is recorded.

[0068] In this embodiment, in the segmentation-desensitization integrated network, the multi-task loss of feature sharing is used. for: In the formula, For multi-granularity segmentation loss, For the loss of desensitization mask generation, The weighting coefficients for the desensitization mask loss are... The weight coefficients for the privacy regularization term. To grade the desensitization intensity, The area is classified as sensitive. For the length of the region, The desensitization threshold is used. For example, the phone number T=11, the address T=20, and the seal T=the diagonal length of the seal's bounding rectangle.

[0069] In this embodiment, a pixel-level coordinate mapping table is established between the original encrypted image and the de-identified document. A credibility score based on the DS evidence theory is applied to each segmented region in the fused feature map to generate a Merkel hash tree trusted credential with a digital signature, including: Based on a pixel-level coordinate mapping table, each segmented region in the fused feature map is independently verified for authenticity. By integrating the segmentation confidence, authenticity identification results, and region integrity of each segmented region, the credibility score of each segmented region is calculated based on the DS evidence theory. For the segmented regions corresponding to the verified credibility scores, generate Merkel hash tree trusted credentials for digital signature.

[0070] For example, when performing independent authenticity verification on segmented regions, the process includes: Document area: Photocopy recognition, anti-counterfeiting watermark verification; Signature area: Seal authenticity verification and proxy signature detection; Text area: Image tampering detection and splicing detection.

[0071] In calculating the credibility score of each segmented region based on the DS evidence theory, the formula for the DS evidence theory basic probability allocation (BPA) is as follows: In the formula, and The first The basic probability of each region being true or false. To include two assumptions, "true" and "false", For the first The degree of uncertainty in each region and This is the evidence weighting coefficient. , and They are respectively , and The weight, To determine the confidence level for authenticity, To segment confidence levels, Confidence level for regional integrity.

[0072] The DS evidence fusion formula is: In the formula, Assuming after fusion The basic probability distribution, , and These are the hypotheses for the first type of evidence (authenticity verification evidence), the second type of evidence (separation confidence evidence), and the third type of evidence (regional integrity evidence). , and The BPA (Best Aspects) are categorized into three types of evidence: authenticity verification, segmentation confidence, and regional integrity. The coefficient of evidence conflict.

[0073] When the coefficient of evidence conflict In this case, the weighted DS evidence fusion method is used instead of the standard DS fusion, with the weight being the historical accuracy of each piece of evidence.

[0074] This yields the credibility score for the segmented region. for: In the formula, This is the basic probability distribution that is assumed to be true after fusion. This represents the basic probability distribution of the assumptions being true and false after fusion.

[0075] Furthermore, while calculating the credibility score, the marked "sorting to be verified" documents are subjected to final page number tampering verification. If tampering is confirmed, the sorting results are backtracked and corrected, and marked as high risk.

[0076] In this embodiment, based on credibility scoring Hierarchical processing is performed, including: Automatic approval, generating trusted credentials; Marked as suspicious, triggering manual review; Marked as high risk, directly intercepted and alerted, and simultaneously added to the negative sample database.

[0077] For verified results, a Merkle hash tree trusted credential with digital signature is generated, which includes information such as image hash, operation log, timestamp, device ID, global parameter version number, and scenario-specific parameter version number. It supports regional-level batch verification and can serve as proof of data integrity and authenticity of source.

[0078] In this embodiment, a Merkle hash tree trusted certificate with a digital signature is generated. Represented as: In the formula, The SHA-256 hash function is used. For string concatenation operators, The hash value of the left child node of the current node. The hash value of the right child node of the current node. For the first Image pixel hash values ​​for each region For the first The complete image pixel matrix of each region For the first Operation logs for each region For the first Credibility scores for each region.

[0079] In this embodiment, the process for supporting region-level batch verification of the aforementioned trusted credentials is as follows: Generate the Merkle proof corresponding to the region to be verified (containing all hash values ​​from the leaf node to the root node). The digital signature object is the root hash. ; During verification, the root hash is recalculated using Merkle proof and compared with the signed root hash to verify the integrity of a single region.

[0080] In this embodiment, the manual review results, high-risk intercepted samples, and quality assessment errors in the above process are independently attributed to their causes. After locating the source of the error, the error is synchronously fed back to the corresponding processing branch for online learning and parameter self-optimization.

[0081] In this embodiment, the unified update framework for closed-loop learning during instance segmentation through error attribution is represented as follows: In the formula, For all learnable parameters throughout the entire process, To provide an incorrect attribution function for using source attribution, For the first Closed-loop feedback data of the wheel, For the parameter version number, To update the function, gradient descent is used for differentiable parameters, and statistical learning and rule-based iterative updates are used for non-differentiable parameters.

[0082] Among them, the parameter version number It adopts the format "major version number.minor version number.revision number", including: Update only non-differentiable parameters: revision number +1; Regular update of differentiable parameters only: minor version number +1; Major update to differentiable parameters (such as model architecture change): major version number +1, minor version number and revision number reset to 0; Simultaneously update two types of parameters: major version number +1, minor version number and revision number reset to 0; Global core parameters share a unified version number, while scenario-specific parameters use independent branch version numbers.

[0083] In this embodiment, the process of attributing errors based on manual review results (correct / incorrect + error type), the original input image, and the intermediate outputs of each processing branch is as follows: Error type mapping, including: quality assessment errors (mapped to the input quality pre-inspection and adaptive process scheduling module, with an error value of...). Errors include: repair errors (mapped to the bidirectional iterative coupling module of segmentation-repair, with the error value being the IoU of the repair region), segmentation errors (first check the repair quality and input quality, locate the root cause and then update the corresponding module, with the error value being the IoU of the segmentation region), sorting errors (mapped to the multi-objective parallel processing and standardization assembly module, with the error value being the sorting accuracy), and verification errors (mapped to the region-level authenticity verification and trusted output module, with the error value being the verification accuracy).

[0084] We adopt a source-tracing attribution approach, investigating the root cause of errors layer by layer from downstream to upstream, rather than prioritizing downstream. If the same error is mapped to multiple processing branches, we quantify the contribution ratio of each branch using the control variable method and allocate the update amplitude proportionally.

[0085] Output the parameter update instructions and quantization error values ​​for the corresponding processing branch.

[0086] Furthermore, the above error attribution process also includes a negative sample error correction mechanism: all high-risk intercepted samples and samples that are manually reviewed and determined to be misjudged must be manually confirmed a second time before they can be used for model training; if a misjudged sample is found, it is automatically removed from the negative sample library, and the parameters of the corresponding processing branch are corrected in reverse.

[0087] The document layout multi-type region intelligent segmentation and instance segmentation method provided in the embodiments adopts a "quality-driven adaptive process scheduling" mechanism, which dynamically allocates computing power and selects processing branches according to the document health; designs a "four-modal cross-attention fusion network" to achieve macro-meta-micro three-level fine-grained region segmentation; proposes a "segmentation-repair bidirectional iterative coupling" model to solve the problem of insufficient segmentation accuracy of low-quality documents; adopts a "segmentation-desensitization integrated network" to eliminate the risk of data leakage during data transfer; constructs an "anchor-free multi-objective parallel segmentation network" to support batch processing of multiple mixed documents; and establishes a "region-level DS evidence theory verification system" to ensure the authenticity and legality of the segmentation results.

[0088] Example 2:

[0089] This embodiment is a further limitation based on Embodiment 1. Its purpose is to provide a document layout multi-type region intelligent splitting and instance segmentation system, which is implemented based on the document layout multi-type region intelligent splitting and instance segmentation method in Embodiment 1. Other parts not mentioned refer to the description in Embodiment 1 or the prior art.

[0090] The document layout multi-type region intelligent splitting and instance segmentation system in this embodiment includes: Input pre-detection and adaptive scheduling module: used to perform quality verification on the original document image, and adaptively call the corresponding processing module and pass algorithm parameters based on the document health score of the quality verification; Process Circuit Breaker / Degradation Module: Used to trigger process circuit breaking, label the original document image and adjust the quality threshold and quality feature weights, and resample the image. When resampling is not possible, it performs degradation processing by generating a basic segmentation result of the labeled damaged area using a lightweight segmentation model. Damaged document repair module: It performs macro-level multi-document instance segmentation on the original document image, automatically selects repair operation for each independent document obtained by segmentation according to the passed repair intensity parameter, and outputs the repaired document image and the optimized coarse-grained localization mask through the deployed bidirectional iterative coupling model of segmentation-repair; and performs secondary quality verification on the document image after repair, and calls the corresponding processing module and passes the algorithm parameters according to the quality verification result. Multimodal multigranularity segmentation module: Deploys a four-modal cross-attention fusion network to simultaneously extract four types of features from the input document image: visual, text, material, and topological features. It performs feature fusion through a topological prior attention mechanism and performs a three-level progressive split of the fused feature map: macro-meta-micro, to generate a full-granularity region mask. Anchor-free multi-objective parallel segmentation module: This is an extension of the multimodal multi-granularity segmentation module in batch scenarios. It is used to generate independent segmentation masks and global coordinate mapping matrices for all documents from the fused feature map through the anchor-free multi-objective parallel segmentation network, and to sort each document based on topological constraints. Then, it performs coordinate transformation and splicing on the original region masks, and performs standardized splicing to obtain standardized documents and aligned region masks. The segmentation and desensitization module is coupled with the multimodal and multigranular segmentation module to achieve synchronous execution of segmentation and desensitization in the same thread. During desensitization, the segmentation-desensitization integrated network is used to perform post-coordinate transformation on the full-granularity region mask and fused feature map. Based on the transformed shared features, sensitive information is located and desensitized in the document copy, and the desensitized document is output. Regional-level authenticity verification and trusted output module: used to verify each segmented region in the fused feature map based on DS evidence theory and generate trusted credentials using Merkle hash trees, and to perform closed-loop learning of each module of the instance segmentation system through error attribution.

[0091] In this embodiment, the input pre-detection and adaptive scheduling module is the "brain center" of the entire system. By quantifying the quality characteristics of the input documents, it achieves optimal allocation of computing power and adaptive scheduling of the process, solving the problems of computing power waste and low-quality document failure caused by fixed parameters in traditional systems.

[0092] Specifically, the input pre-detection and adaptive scheduling module is the "entry point" for all subsequent modules, determining the processing efficiency and resource consumption of the entire system. Its online learning mechanism receives manual review feedback from the trusted verification module and dynamically adjusts the weights and circuit breaker thresholds of quality features through statistical methods to improve the accuracy of quality assessment in different scenarios. If input quality leads to subsequent segmentation / repair errors, the error attribution module will feed back to this module to optimize the weights of the corresponding quality features.

[0093] In this embodiment, the damaged document repair module is a "low-quality input dedicated branch" of the input pre-inspection and adaptive scheduling module. By constructing a bidirectional iterative coupling model of segmentation and repair, it solves the problem of insufficient accuracy in low-quality document segmentation caused by the disconnect between repair and segmentation in traditional methods. The coarse-grained positioning mask output by the damaged document repair module provides high-quality input for the multi-granularity segmentation module, which greatly improves the accuracy of subsequent fine segmentation.

[0094] In this embodiment, the multimodal, multi-granularity segmentation module is the "core engine" of the entire system. Through a four-modal cross-attention fusion network and a three-level progressive splitting mechanism, it achieves full-granularity document region instance segmentation at the macro, meso, and micro levels, solving the problems of coarse segmentation granularity and low accuracy in traditional algorithms. The refined region mask output by the multimodal, multi-granularity segmentation module forms the basis for the region adaptive desensitization module, region-level authenticity verification, and trusted output module's region-level authenticity verification.

[0095] In this embodiment, the anchorless multi-objective parallel segmentation module is a "batch processing extension" of the multimodal multi-granularity segmentation module. Through the anchorless multi-objective parallel segmentation network and the topology constraint sorting algorithm, it achieves efficient processing and standardized output of multi-document mixed scenarios, solving the problem of low batch processing efficiency in traditional systems.

[0096] Specifically, the anchorless multi-objective parallel segmentation module is an extension of the multimodal multi-granularity segmentation module in batch processing scenarios, improving the single-document segmentation capability to multi-document mixed scenarios; its standardized output and aligned region mask provide a unified data format for subsequent desensitization, verification and archiving; if the sorting error causes the subsequent trusted verification to fail, the error attribution module will feed back to this module to optimize the weights of the transition probability matrix.

[0097] In this embodiment, the segmentation region desensitization module serves as the system's "data security guarantee." Through an integrated segmentation-desensitization network and hierarchical desensitization technology, it achieves real-time and secure desensitization of sensitive information, mitigating the data leakage risk caused by the disconnect between segmentation and desensitization in traditional methods. The segmentation region desensitization module is deeply coupled with the multimodal, multi-granularity segmentation module, enabling synchronous execution of segmentation and desensitization in the same thread. This eliminates the risk of leakage during data transfer. Its desensitization results are a crucial component of the region-level authenticity verification and trusted output module.

[0098] In this embodiment, the regional-level authenticity verification and trusted output module is the "result authenticity guarantee" of the system. An independent error attribution module is configured as the "closed-loop self-optimization entry point". Through regional-level DS evidence theory verification and Merkle hash tree trusted credentials, the authenticity and legality of the segmentation results are guaranteed, and a closed-loop learning mechanism for the entire system is formed.

[0099] Specifically, the regional-level authenticity verification and trusted output module is the "last line of defense" of the entire system, ensuring the authenticity and legitimacy of the segmentation results. The independent error attribution module forms a closed loop for the entire system, continuously improving the overall performance of the system. This module receives all intermediate results from the preceding modules, providing complete data support for error attribution.

[0100] This invention proposes for the first time a hybrid architecture bidirectional closed-loop high-reliability segmentation framework, integrating a differentiable neural network module with a non-differentiable rule / statistic module to construct a full-link bidirectional information transmission mechanism, achieving joint optimization of all module parameters and system closed-loop self-learning. Based on this, the seven core modules mentioned above are designed, with deep coupling and collaborative operation to form an adaptive, full-granularity, highly secure, and highly reliable next-generation document layout segmentation system.

[0101] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0102] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for intelligent splitting and instance segmentation of multiple types of regions in a document layout, characterized in that, Includes the following steps: Based on the quality features of the original document image, a document health score is calculated through quality verification. This score is then combined with a quality threshold to adaptively call the corresponding processing branch and pass algorithm parameters, including: Process circuit break / degradation branch: Trigger process circuit break, label the original document image and adjust the quality threshold and quality feature weights, and resample the image. If resampling is not possible, perform degradation processing by using a lightweight segmentation model to generate the basic segmentation result of the labeled damaged area. Damaged document restoration branch: Perform macro-level multi-document instance segmentation on the original document image, automatically select the restoration operation for each independent document obtained by segmentation according to the passed restoration intensity parameter, and start the segmentation-restoration bidirectional iterative coupling model to output the restored document image and the optimized coarse-grained localization mask; after restoration, perform secondary quality verification on the document image, and call the corresponding processing branch and pass the algorithm parameters according to the quality verification result; Multimodal multigranularity segmentation branch: Start a four-modal cross-attention fusion network to extract four types of features from the input document image: visual, text, material and topology. Feature fusion is performed through a topological prior attention mechanism. The fused feature map is split into macro-meta-micro three-level progressive segments to generate a full-granularity region mask. Based on macro-level multi-document boundaries and masks, an anchorless multi-objective parallel segmentation network is used to generate independent segmentation masks and global coordinate mapping matrices for all documents from the fused feature map. The documents are sorted based on topological constraints, and then the full-granularity region masks are transformed and spliced, and standardized and spliced ​​to obtain standardized documents and aligned region masks. Based on standardized documents and aligned region masks, a segmentation-desensitization integrated network is adopted. The global coordinate mapping matrix is ​​used to perform post-coordinate transformation on the full-granularity region mask and the fused feature map. Based on the transformed shared features, sensitive information is located and desensitized in the document copy, and desensitized document is output. A pixel-level coordinate mapping table is established between the encrypted original document image and the desensitized document. The credibility score of each segmented region in the fused feature map is performed based on the DS evidence theory. A Merkel hash tree credibility certificate with digital signature is generated, and the instance segmentation process is closed-loop learned through error attribution.

2. The method for intelligent splitting and instance segmentation of multiple types of document layout regions according to claim 1, characterized in that, Document health scores are calculated through quality verification, including: The weights of each quality feature in the original document image are calculated using the dynamic entropy weighting method updated by Bayesian posterior, and the document health score is obtained by weighting. When calling the processing branch, the quality threshold is determined based on the optimal cutoff point of the ROC curve of the historical document annotation dataset, and updated based on the quality assessment error of closed-loop feedback, and supports scenario-based threshold configuration.

3. The method for intelligent splitting and instance segmentation of multiple types of document layout regions according to claim 1, characterized in that, In the damaged document repair branch, the bidirectional coupling iterative process of the segmentation-repair bidirectional iterative coupling model includes: In the forward iteration process, the repaired image is used to generate an initial single-document coarse-grained boundary positioning mask. In the reverse iteration process, the mask is used to constrain and repair the boundary. In addition, the mask confidence is calculated in real time during the iteration process, and a general prior template for document correction is introduced when the mask confidence is lower than a preset confidence threshold.

4. The intelligent splitting and instance segmentation method for multiple types of document layout regions according to claim 1, characterized in that, In the damaged document repair branch, the variational bidirectional coupling objective function of the segmentation-repair bidirectional iterative coupling model... for: In the formula, In probability distribution The mathematical expectation operator below, In probability distribution The mathematical expectation operator below, Given an original document image and coarse-grained positioning mask At that time, the document image after repair The conditional probability distribution, Given an original document image At that time, coarse-grained positioning mask The conditional probability distribution, Coarse-grained positioning mask The prior distribution, For a given repaired document image At that time, coarse-grained positioning mask The conditional probability distribution, for divergence, For the repaired document image, The original document image, Coarse-grained positioning masks for segmenting each region; The update and iteration rules are as follows: In the formula, For the first The next iteration yields the optimized coarse-grained localization masks for each region. For the first The next iteration yields the repaired document image. At that time, the corresponding coarse-grained positioning mask The conditional probability distribution, Given an original document image and optimal coarse-grained positioning mask At that time, the document image after repair The conditional probability distribution, For the first The restored document image obtained in the next iteration, index For the number of iterations, This is a set of coarse-grained location masks.

5. The method for intelligent splitting and instance segmentation of multiple types of document layout regions according to claim 1, characterized in that, In the multimodal, multi-granularity segmentation branch, the formula for feature fusion using the topological prior attention mechanism is as follows: In the formula, To fuse feature maps, For topological prior attention functions, For visual feature maps, For text feature maps, This is a material feature map. For the input feature map, This is a document topological structure feature map. For learnable query projection matrix, The learnable bond projection matrix. The projection matrix is ​​the learnable value. Let be the dimension of the key vector. For activation functions, superscript It is the transpose operator; Three-level cascaded loss when performing macro-meta-micro three-level progressive splitting on the fused feature map for: In the formula, For macro-level segmentation loss, For meso-level segmentation loss, For micro-level segmentation loss, , and They are respectively , and The weight, These are the weighting coefficients for cross-granularity consistency constraints. For macro-level segmentation mask, For meso-level segmentation mask, For micro-level segmentation mask, This is the upsampling operator.

6. The method for intelligent splitting and instance segmentation of multiple types of document layout regions according to claim 1, characterized in that, The documents are sorted based on topological constraints, including: Topological features are extracted from each document segmented by the anchorless multi-objective parallel segmentation network, and document transition probabilities are constructed. The Markov chain algorithm based on topological constraints sorts the documents and tags them based on the topological constraints. The topological constraints, from highest to lowest priority, are: page number order, date order, text direction, and content similarity.

7. The method for intelligent splitting and instance segmentation of multiple types of document layout regions according to claim 1, characterized in that, In the segmentation-desensitization integrated network, the corresponding desensitization strategy is matched according to the region type and sensitivity level, and an invisible hash traceability watermark is added to each desensitized region; In the segmentation-desensitization integrated network, feature sharing multi-task loss for: In the formula, For multi-granularity segmentation loss, For the loss of desensitization mask generation, The weighting coefficients for the desensitization mask loss are... The weight coefficients for the privacy regularization term. To grade the desensitization intensity, The area is classified as sensitive. For the length of the region, This is the desensitization threshold.

8. The method for intelligent splitting and instance segmentation of multiple types of document layout regions according to claim 1, characterized in that, For each segmented region in the fused feature map, a credibility score based on DS evidence theory is applied to generate a digitally signed Merkle hash tree credibility credential, including: Based on a pixel-level coordinate mapping table, each segmented region in the fused feature map is independently verified for authenticity. By integrating the segmentation confidence, authenticity identification results, and region integrity of each segmented region, the credibility score of each segmented region is calculated based on the DS evidence theory. For the segmented regions corresponding to the verified credibility scores, generate Merkel hash tree trusted credentials for digital signature.

9. The method for intelligent splitting and instance segmentation of multiple types of document layout regions according to claim 1, characterized in that, The unified update framework for closed-loop learning in the instance segmentation process based on error attribution is represented as follows: In the formula, For all learnable parameters throughout the entire process, To provide an incorrect attribution function for using source attribution, For the first Closed-loop feedback data of the wheel, For the parameter version number, To update the function, gradient descent is used for differentiable parameters, and statistical learning and rule-based iterative updates are used for non-differentiable parameters.

10. A document layout multi-type region intelligent splitting and instance segmentation system, implemented based on the document layout multi-type region intelligent splitting and instance segmentation method according to any one of claims 1 to 9, characterized in that, include: Input pre-detection and adaptive scheduling module: used to perform quality verification on the original document image, and adaptively call the corresponding processing module and pass algorithm parameters based on the document health score of the quality verification; Process Circuit Breaker / Degradation Module: Used to trigger process circuit breaking, label the original document image and adjust the quality threshold and quality feature weights, and resample the image. When resampling is not possible, it performs degradation processing by generating a basic segmentation result of the labeled damaged area using a lightweight segmentation model. Damaged document repair module: It is used to perform macro-level multi-document instance segmentation on the original document image. For each independent document obtained by segmentation, it automatically selects the repair operation according to the passed repair intensity parameter, and outputs the repaired document image and the optimized coarse-grained positioning mask through the deployed bidirectional iterative coupling model of segmentation-repair. And after the repair is completed, a second quality check is performed on the document image, and based on the quality check result, the corresponding processing module is called and the algorithm parameters are passed; Multimodal multigranularity segmentation module: Deploys a four-modal cross-attention fusion network to simultaneously extract four types of features from the input document image: visual, text, material, and topological features. It performs feature fusion through a topological prior attention mechanism and performs a three-level progressive split of the fused feature map: macro-meta-micro, to generate a full-granularity region mask. Anchor-free multi-objective parallel segmentation module: This is an extension of the multimodal multi-granularity segmentation module in batch scenarios. It is used to generate independent segmentation masks and global coordinate mapping matrices for all documents from the fused feature map through the anchor-free multi-objective parallel segmentation network, and to sort each document based on topological constraints. Then, it performs coordinate transformation and splicing on the original region masks, and performs standardized splicing to obtain standardized documents and aligned region masks. The segmentation and desensitization module is coupled with the multimodal and multigranular segmentation module to achieve synchronous execution of segmentation and desensitization in the same thread. During desensitization, the segmentation-desensitization integrated network is used to perform post-coordinate transformation on the full-granularity region mask and fused feature map. Based on the transformed shared features, sensitive information is located and desensitized in the document copy, and the desensitized document is output. Regional-level authenticity verification and trusted output module: used to verify each segmented region in the fused feature map based on DS evidence theory and generate trusted credentials using Merkle hash trees, and to perform closed-loop learning of each module of the instance segmentation system through error attribution.

Images