Structured document forgery detection method and system based on cross-modal consistency learning

Abstract

The application belongs to the technical field of text forgery detection, and particularly relates to a structured document forgery detection method and system based on cross-modal consistency learning, which comprises two major core innovations of a hierarchical consistency perception fusion module HCAF and a field-level comparison and ranking learning FCR strategy. The HCAF module performs cross-modal feature fusion and consistency calculation layer by layer at three granularities of Token level, field level and global level, wherein the consistency score simultaneously serves as a fusion guide signal and a detection output, so as to realize collaborative optimization of fusion and detection. The FCR strategy is constrained by three losses of cross-modal alignment loss, field-level ranking loss and true-false separation loss, and utilizes the structural characteristics that the forged sample is partially tampered, so as to improve the field-level detection accuracy under the weak supervision condition of only image-level labels. The method is designed with a three-stage progressive training strategy to ensure stable convergence of the model.

Description

Technical Field

[0001] This application belongs to the field of text forgery detection technology, specifically involving a structured document forgery detection method and system based on cross-modal consistency learning. Background Technology

[0002] With the popularization of digital office work and the advancement of paperless processes, structured documents (such as VAT invoices, contracts, certificates, bank statements, insurance policies, etc.) are widely used in scenarios such as corporate financial management, financial risk control, document auditing, and administrative approval. Although electronic documents can achieve original document anti-counterfeiting through cryptographic means such as digital signatures and blockchain notarization, in actual business processes, a large number of documents are stored and transmitted in image form (such as scanned copies of photos, converted screenshots, fax copies, photos transmitted via WeChat, etc.), rendering the original cryptographic protection mechanisms ineffective.

[0003] Existing document forgery detection methods, such as OCR-based text comparison methods and image-based visual detection methods, rely entirely on OCR recognition results without considering the visual information of the image itself. This makes them unable to determine if the image has been edited. Even if the OCR result matches the original record, the image may have undergone completely different rearrangement or synthesis. Image-based visual detection methods only focus on whether the image has been edited, without understanding whether the edited content is correct—even if edits are detected, the authenticity of the edited content cannot be determined. Existing multimodal fusion methods typically separate feature fusion from consistency detection, performing feature fusion first in the intermediate layers of the network and then consistency judgment in the output layer. This separate architecture fails to achieve coordinated optimization between the fusion and detection processes: the fusion module doesn't know which features are more important for detection, and the detection module cannot guide the fusion strategy to focus on key areas, resulting in limited overall detection accuracy. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a structured document forgery detection method and system based on cross-modal consistency learning, the technical solution of which is as follows: A structured document forgery detection method based on cross-modal consistency learning includes the following steps: S1. Input multimodal data; Receive a document image to be detected and text annotation information from a trusted source, wherein the text annotation information includes the text content of each structured field and its bounding box coordinates in the image; S2. Multimodal feature extraction; Visual features are extracted from document images using a visual encoder, and text features that combine semantic and spatial information are extracted using a text encoder combined with layout position embedding. S3. Hierarchical consistency perception fusion; Through the hierarchical consistency-aware fusion module, cross-modal feature fusion and consistency calculation are performed layer by layer at three granularities: Token level, field level, and global level. Among them, the field level fusion calculates the consistency score of each field, and the consistency score serves as both a guiding signal for the fusion process and a component of the detection result, thereby achieving collaborative optimization of fusion and detection. S4. Field-level comparison and sorting learning; By jointly optimizing the training through the triple constraints of cross-modal alignment loss, field-level ranking loss, and true / false separation loss, and by taking advantage of the structural characteristics that some fields in the fake sample are tampered with while the rest are normal, the field-level detection accuracy is improved under the weak supervision of only image-level labels. S5. Forgery detection output; By fusing visual features, text features, fusion features, and consistency statistics, the system outputs image-level forgery probability and field-level consistency score.

[0005] Preferably, hierarchical consistency-aware fusion includes token-level fusion, field-level fusion, and global-level fusion; Token-level fusion: Using text features as queries and visual features as keys and values, a fine-grained semantic association between text tokens and image patches is established through a multi-head cross-modal attention mechanism, and the fused token sequence and attention weight matrix are output. Field-level fusion: For each structured field, the fusion token features corresponding to that field are aggregated to obtain the field text vector. The image is cropped from the field region and a field visual vector is obtained through an independent visual encoder. The field text vector and field visual vector are mapped to the alignment space through a projection head, and the cosine similarity is calculated as the consistency score for that field. At the same time, a consistency-aware gating mechanism is introduced to dynamically adjust the model's attention to real and fake regions. ; in To represent field-level text features, Represents field-level visual features. The corresponding consistency score, The sigmoid activation function is used, and LayerNorm is the layer normalization operation. This is element-wise multiplication. The mechanism adaptively scales the fused features based on a consistency score: for fields with high consistency... Approaching larger values ​​preserves more fused feature information; for fields with low consistency, This reduces unreliable features and highlights discriminative information for anomalous regions.

[0006] Global-level fusion: A consistency-weighted aggregation strategy is adopted, which uses a negative correlation weighting mechanism to give higher aggregation weights to fields with lower consistency scores, guiding the model to focus on suspicious regions. ; ; in Temperature coefficient; global-level fusion outputs a global fusion feature vector. .

[0007] Preferably, the consistency score has a dual function, serving as both a guiding signal for the fusion process and a component of the detection result: Fusion guidance: The fused features are weighted and modulated using consistency scores and then adaptively scaled. Specifically, fields with high consistency retain more fused feature information, while fields with low consistency suppress fused features, thereby suppressing unreliable features and highlighting the discriminative information of abnormal regions. Detection output: Field-level consistency scores are directly used for field-level forgery location, with the field with the lowest score being the most suspicious tampered field; Field consistency scores are further used to assist in image-level forgery detection after statistical feature extraction. The feature fusion process can perceive the detection target, and the detection results can guide the fusion strategy in reverse, forming an end-to-end collaborative optimization closed loop.

[0008] Preferably, the field-level comparative ranking learning includes three loss components: Cross-modal alignment loss: For real samples, constrain the consistency scores of all fields to approach the target high value: ; in The consistency score is given by N, which represents the total number of fields in the current sample. For forged samples, a minimum selection strategy is used to identify the most suspicious field, constraining the consistency score of that field to approach a target low value: ; The final cross-modal alignment loss is the sum of the two: ; Field-level ranking loss: Calculated only for forged samples, constraining the consistency score of normal fields within the same forged sample to be higher than the score of potentially tampered fields, using a soft-margin ranking loss form: ; in This is the sorting interval hyperparameter. This is a collection of normal fields. For a set of potentially tamperable fields, , These represent the consistency scores of the corresponding fields; True / False Separation Loss: Construct true / false sample pairs within a batch, constraining the overall average consistency score of true samples to be higher than that of fake samples. ; in This represents the set of paired real and fake samples constructed in the current batch. Represents a real sample. This indicates a forged sample. and The average consistency scores for real samples and fake samples are respectively. This is the separation interval hyperparameter.

[0009] Preferably, the identification of potential tampered fields in the field-level ranking loss employs one or a combination of the following heuristic strategies: Fixed k-value strategy: Select the k fields with the lowest consistency scores in the current sample as the set of potential tampered fields, and the remaining fields as the set of normal fields, where k is a preset fixed value; Dynamic threshold strategy: Calculate the mean of the consistency scores of all fields in the current sample. and standard deviation The score is lower than The fields are marked as potentially tamperable fields, where As an adjustable hyperparameter, this strategy can adaptively determine the number of fields to be tampered with based on the characteristics of the sample.

[0010] Preferably, the field-level ranking loss does not require field-level forgery labeling, but only image-level true / false labels to learn field-level discrimination ability under weak supervision. Its design principle is: to utilize the natural difference in consistency scores between tampered and untampered fields in forged samples, and to transform the structural characteristics of this sample into fine-grained supervision signals through ranking constraints.

[0011] The preferred joint optimization objective for the triple loss is: ; ; in For classifying losses, , , These are the weighting coefficients for each loss component, used to balance the contributions of the three losses. This represents the overall weight of the FCR loss.

[0012] Preferably, joint optimization training is performed using three constraints: cross-modal alignment loss, field-level ranking loss, and true / false separation loss. A progressive training strategy is adopted, dividing the training process into three stages: The first stage is the visual warm-up stage: the text encoder parameters are frozen, and the visual encoder and multimodal fusion module are trained using only classification loss and alignment loss; The second stage is the joint optimization stage: unfreeze some layers of the text encoder, introduce field-level ranking loss, establish cross-modal alignment and learn the relative relationships between fields; The third stage is the fine-tuning stage: full parameter fine-tuning, introducing the complete FCR loss, and using a complete joint optimization objective that includes cross-modal alignment loss, field-level sorting loss and true / false separation loss to complete end-to-end fine-tuning; The learning rate decreases sequentially at each stage, and loss components are gradually introduced to ensure stable convergence of multi-objective optimization.

[0013] Preferably, in step S5, the forgery detection output includes: Consistency statistical feature extraction: The consistency scores of all fields are aggregated into a fixed-dimensional statistical feature vector, including the minimum value, mean value, and the k lowest scores, to reflect the score distribution patterns of different forgery types; single-field tampering is manifested as the minimum value being less than the threshold but the mean value being normal, and multi-field tampering is manifested as multiple low scores and a decrease in the mean value; Field-level output: Outputs the consistency score and sorting of each field, used to locate suspected tampered fields and provide auditors with interpretable detection evidence.

[0014] A structured document forgery detection system based on cross-modal consistency learning includes: Multimodal input module: responsible for receiving the document image to be detected and text annotation information from trusted sources; Visual feature extraction module: Extracts visual features from document images and outputs local feature sequences and global visual vectors; Text feature extraction module: Combines layout position embedding to extract text features that integrate semantic and spatial location information, and outputs token-level feature sequence and global text vector; The hierarchical consistency-aware fusion module performs cross-modal feature fusion and consistency calculation at three granularities: token level, field level, and global level. It includes a token-level cross-modal attention submodule, a field-level consistency-aware gating fusion submodule, and a global-level consistency weighted aggregation submodule. The consistency score serves as both a fusion guidance signal and a detection output. The fusion and detection are optimized collaboratively through gating mechanisms and weighted aggregation strategies. Field-level comparison and ranking learning module: Calculates cross-modal alignment loss, field-level ranking loss, and true / false separation loss, constructs a joint optimization objective, and improves field-level detection accuracy under weak supervision by utilizing the internal structural characteristics of fake samples; Classification Decision Module: It integrates multi-source features to output image-level forgery probability, and also outputs field-level consistency score for forgery localization.

[0015] Compared with the prior art, the beneficial effects of this application are as follows: (1) Co-optimization of fusion and detection: This invention uses consistency scores to guide the fusion process and output detection results simultaneously, overcoming the limitation of existing methods that separate fusion and detection. The consistency score participates in field-level feature fusion through a gating mechanism and guides global-level feature aggregation through a weighted aggregation strategy, enabling the fusion process to perceive the detection target and the detection results to guide the fusion strategy in reverse, forming an end-to-end co-optimization closed loop, which significantly improves detection accuracy.

[0016] (2) Weakly supervised field-level learning: The FCR strategy of this invention can effectively improve the field-level detection accuracy by utilizing the structural characteristics of "partial tampering" within the forged sample and providing fine-grained supervision signals through ranking loss under weak supervision conditions with only image-level labels. Compared with strong supervision methods that require field-level annotation, this invention significantly reduces the annotation cost, making large-scale application possible.

[0017] (3) Hierarchical multi-granularity fusion: This invention constructs a three-level fusion architecture of Token-Field-Global, establishing cross-modal associations layer by layer from fine-grained to coarse-grained. Token-level fusion captures the local correspondence between characters and pixels, field-level fusion establishes cross-modal consistency of semantic units, and global-level fusion integrates all field information to form an overall judgment. This hierarchical design can fully capture the image-text correspondence at different semantic levels and has a stronger feature expression capability.

[0018] (4) Hierarchical Collaborative Optimization: This invention designs a triple constraint of cross-modal alignment, field sorting, and true / false separation, forming a hierarchical learning objective from feature alignment → field sorting → sample separation. The three loss components each have their own focus and complement each other, enabling the model to learn forgery detection capabilities from multiple perspectives.

[0019] (5) Interpretable field-level location: This invention can output the consistency score of each field, accurately locating the specific field that has been tampered with. Auditors can not only know the judgment result of "whether the document is forged", but also know the specific information of "which fields have been tampered with". This interpretable detection result meets the requirements of traceability for detection in scenarios such as financial auditing and tax inspection.

[0020] (6) Not dependent on visual forgery traces: This invention detects forgery by measuring the consistency between the visual content of an image and a credible text record, rather than relying on forgery traces in the image itself. Even if a forger uses professional tools to create a visually flawless altered image, it can be detected as long as the altered content does not match the original record from a credible source.

[0021] (7) Progressive stable training: The present invention is designed with a three-stage progressive training strategy, which introduces each loss component in the order of visual warm-up → joint optimization → fine-tuning to ensure stable convergence of the model and give full play to the role of the FCR strategy, and avoids the problem of training instability caused by the simultaneous introduction of multiple losses. Attached Figure Description

[0022] Figure 1 This is an overall network structure diagram of an embodiment of the present invention; Figure 2 This is a diagram of the Hierarchical Consistency Awareness Fusion (HCAF) module architecture according to an embodiment of the present invention; Figure 3 This is a flowchart of the field-level consistency detection process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the field-level comparison and sorting learning strategy in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The present invention is illustrated using the detection of forged value-added tax invoices as a specific implementation scenario.

[0024] like Figure 1 As shown, a structured document forgery detection method based on cross-modal consistency learning is presented, with the following specific steps:

[0025] S1. Receive multimodal input data: The system receives images of VAT invoices to be detected, along with text annotation information from trusted sources (such as tax system databases). The text annotation information includes the text content and bounding box coordinates of fields such as invoice code, invoice number, invoice date, buyer's name, buyer's taxpayer identification number, seller's name, seller's taxpayer identification number, amount, tax rate, tax amount, and total price including tax. Simultaneously, during the training phase, it receives image-level forgery labels y∈{0,1}.

[0026] The input image is then preprocessed: scaled to a uniform resolution to preserve document details, and pixel values ​​are normalized and standardized. The text information is then segmented into words.

[0027] S2. Multimodal feature extraction: (1) Visual feature extraction: The preprocessed document image is input into the visual encoder. This embodiment employs a hierarchical visual Transformer architecture, extracting multi-scale visual features through multi-stage encoding. The output features of the final stage are taken and linearly projected to obtain a 256-dimensional patch-level feature sequence. The patch sequence is then averaged and pooled along the spatial dimensions to obtain a 256-dimensional global visual vector, which represents the visual information of the entire document image.

[0028] (2) Extraction of semantic features from text: Textual information from a trusted source is input into a pre-trained language model to extract semantic features, which are then linearly projected to obtain 256-dimensional token features.

[0029] The bounding box coordinates corresponding to each token are encoded into a layout position embedding through an embedding layer. The four coordinate values ​​(top-left x, top-left y, bottom-right x, bottom-right y) are mapped through independent embedding layers and then concatenated and fused to obtain a 256-dimensional layout position embedding. The semantic features are then added element-wise to the position embedding to obtain a token representation that integrates semantic and spatial location information.

[0030] The vector at the first special marker position of the sequence is taken as the 256-dimensional global text feature.

[0031] S3 hierarchical consistency-aware fusion: like Figure 2 As shown, the overall architecture of the Hierarchical Consistency-Aware Fusion (HCAF) module comprises three levels of sub-modules: a token-level fusion sub-module, a field-level fusion sub-module, and a global-level fusion sub-module. This module takes visual feature sequences and text feature sequences as input and achieves cross-modal feature fusion and consistency modeling through a progressively layered approach. Details are as follows: Token-level fusion (Level 1): Using text tokens as queries and image patches as keys and values, a fine-grained image-text correspondence is established through a multi-head cross-modal attention mechanism. The output is a fused token sequence and an attention weight matrix. The attention weight matrix can be reconstructed into a spatial heatmap to visualize the image region focused on by each text token, providing interpretable visual evidence for the detection results.

[0032] Field-level merging (Level 2): like Figure 3As shown, the field-level fusion process includes three steps: field feature extraction, consistency score calculation, and post-fusion gating modulation based on the consistency score. Field text vector extraction: Based on the token index range corresponding to the field, relevant tokens are extracted from the token sequence output by token-level fusion, and a 256-dimensional field text vector is obtained through mean pooling aggregation. .

[0033] Field visual vector extraction: Regions of interest (ROIs) are cropped from the original document image based on the bounding box coordinates of the fields, fed into a separate visual encoder, and linearly projected to obtain 256-dimensional field visual vectors. .

[0034] Consistency score calculation: The field text vector and field visual vector are mapped to the alignment space through the projection head, and the cosine similarity of the L2 normalized vectors is calculated as the consistency score. ; The consistency score ranges from -1 to 1. For normal fields, the text and image content are consistent, and the score is close to 1; for tampered fields, the text and image content are inconsistent, and the score is lower.

[0035] Consistency-aware gating fusion mechanism: After the initial fusion of textual and visual features, a consistency score-based gating mechanism is introduced to modulate the fused features. Specifically defined as: ; in To represent field-level text features, Represents field-level visual features. The corresponding consistency score, The sigmoid activation function is used, and LayerNorm is the layer normalization operation. This is element-wise multiplication. The mechanism adaptively scales the fused features based on a consistency score: for fields with high consistency... Approaching larger values ​​preserves more fused feature information; for fields with low consistency, This reduces unreliable features and highlights discriminative information for anomalous regions.

[0036] Global-level fusion (Level 3): A consistent weighted aggregation strategy is adopted, and a negative correlation weighting mechanism is designed so that fields with lower scores receive higher weights. ; ; Temperature coefficient This mechanism controls the sharpness of the weight distribution. It allows the model to focus on suspicious regions with low consistency, creating a closed-loop detection logic from identifying suspicious fields to focusing on those regions.

[0037] S4. Field-level comparison and sorting learning: like Figure 4 As shown, the field-level contrastive ranking learning strategy includes three loss components: Cross-modal alignment loss ( ): For real samples ( Constrain all field scores to approach 1: ; Forged samples ( The score of the most suspicious field in the constraint approaches 0. ; Cross-modal alignment loss:

[0038] Field-level sorting loss ( ): Calculations are performed only on forged samples. A heuristic strategy is used to identify potentially tampered fields: the field with the lowest score is selected. The fields are a set of potential tampering fields. The remaining fields are treated as a normal field set. .

[0039] The sorting loss uses a soft-interval method, requiring that the score of a normal field is at least one interval higher than the score of a tampered field. : ; The interval This loss function leverages the structural characteristics of forged samples—partially altered and partially normal—to provide a more granular supervisory signal than image-level labels, even under weak supervision with only image-level labels. It constrains the relative ordering relationships between fields within the sample.

[0040] True / False Separation Loss ( ): Construct pairs of real and fake samples within a batch for comparison: ; and These are the average consistency scores for real samples and fake samples, respectively. This is the separation interval hyperparameter, with a default value of 0.5.

[0041] Joint optimization objective: ; ; in For classifying losses, , , These are the weighting coefficients for each loss component, used to balance the contributions of the three losses. This represents the overall weight of the FCR loss.

[0042] Progressive training strategy: Phase 1 (Visual Warm-up): Freeze the text encoder and use classification loss and alignment loss. The goal is to allow the visual encoder to learn basic visual feature representations of the document.

[0043] The second stage (joint optimization) involves unfreezing some layers of the text encoder and introducing field-level ranking loss. The goal is to establish cross-modal alignment and learn the relative relationships between fields.

[0044] The third stage (fine-tuning): fine-tuning all parameters using the full FCR loss. The goal is to fully leverage the full effectiveness of the FCR strategy.

[0045] The learning rate decreases sequentially at each stage, and loss components are gradually introduced to ensure stable convergence of multi-objective optimization.

[0046] S5 forgery detection output: Global visual features, global text features, global fusion features, and consistency statistical features (including minimum, mean, and lowest k scores) are concatenated into a total feature vector. This vector is then processed by a two-layer MLP classifier and activated by a sigmoid function to output the forgery probability. The consistency scores of each field are output to locate suspicious fields.

[0047] Generate a structured detection report, including: image-level forgery probability, field-level consistency score ranking table, suspicious field markers and bounding box locations, attention heatmap visualization, etc., providing auditors with comprehensive and interpretable detection basis.

[0048] A structured document forgery detection system based on cross-modal consistency learning includes: Multimodal input module: responsible for receiving the document image to be detected and text annotation information from trusted sources; Visual feature extraction module: Extracts visual features from document images and outputs local feature sequences and global visual vectors; Text feature extraction module: Combines layout position embedding to extract text features that integrate semantic and spatial location information, and outputs token-level feature sequence and global text vector; The hierarchical consistency-aware fusion module performs cross-modal feature fusion and consistency calculation at three granularities: token level, field level, and global level. It includes a token-level cross-modal attention submodule, a field-level consistency-aware gating fusion submodule, and a global-level consistency weighted aggregation submodule. The consistency score serves as both a fusion guidance signal and a detection output. The fusion and detection are optimized collaboratively through gating mechanisms and weighted aggregation strategies. Field-level comparison and ranking learning module: Calculates cross-modal alignment loss, field-level ranking loss, and true / false separation loss, constructs a joint optimization objective, and improves field-level detection accuracy under weak supervision by utilizing the internal structural characteristics of fake samples; Classification Decision Module: It integrates multi-source features to output image-level forgery probability, and also outputs field-level consistency score for forgery localization.

[0049] The present invention also provides a computer-readable storage medium storing a program for a structured document forgery detection method based on cross-modal consistency learning, wherein the program, when executed by a processor, implements the steps of the structured document forgery detection method based on cross-modal consistency learning as described above.

[0050] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0051] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A structured document forgery detection method based on cross-modal consistency learning, characterized in that, Includes the following steps: S1. Input multimodal data; Receive the document image to be detected and text annotation information from a trusted source; S2. Multimodal feature extraction; Visual features are extracted from document images using a visual encoder, and text features that combine semantic and spatial information are extracted using a text encoder combined with layout position embedding. S3. Hierarchical consistency perception fusion; Through the hierarchical consistency-aware fusion module, cross-modal feature fusion and consistency calculation are performed layer by layer at three granularities: Token level, field level, and global level. Among them, the field level fusion calculates the consistency score of each field, and the consistency score serves as both a guiding signal for the fusion process and a component of the detection result, thereby achieving collaborative optimization of fusion and detection. S4. Field-level comparison and sorting learning; Joint optimization training is performed using triple constraints: cross-modal alignment loss, field-level sorting loss, and true / false separation loss. S5. Forgery detection output; By fusing visual features, text features, fusion features, and consistency statistics, the system outputs image-level forgery probability and field-level consistency score.

2. The structured document forgery detection method based on cross-modal consistency learning according to claim 1, characterized in that, The hierarchical consistency-aware fusion module includes token-level fusion, field-level fusion, and global-level fusion; Token-level fusion: Using text features as queries and visual features as keys and values, a fine-grained semantic association between text tokens and image patches is established through a multi-head cross-modal attention mechanism, and the fused token sequence and attention weight matrix are output. Field-level fusion: For each structured field, the fusion token features corresponding to that field are aggregated to obtain the field text vector. The image is cropped from the field region and a field visual vector is obtained through an independent visual encoder. The field text vector and field visual vector are mapped to the alignment space through a projection head, and the cosine similarity is calculated as the consistency score for that field. At the same time, a consistency-aware gating mechanism is introduced to dynamically adjust the model's attention to real and fake regions, specifically defined as: ； in, To represent field-level text features, Represents field-level visual features. The corresponding consistency score, The sigmoid activation function is used, and LayerNorm is the layer normalization operation. This is element-wise multiplication. The mechanism adaptively scales the fused features based on a consistency score: for fields with high consistency... Approaching larger values ​​preserves more fused feature information; for fields with low consistency, This reduces unreliable features and highlights discriminative information for abnormal regions; Global-level fusion: A consistency-weighted aggregation strategy is adopted, which uses a negative correlation weighting mechanism to give higher aggregation weights to fields with lower consistency scores, guiding the model to focus on suspicious regions. ； ； in Temperature coefficient; global-level fusion outputs a global fusion feature vector. .

3. The structured document forgery detection method based on cross-modal consistency learning according to claim 1, characterized in that, The consistency score serves as both a guiding signal for the fusion process and a component of the detection results. Fusion guidance: The fused features are weighted and modulated using consistency scores and then adaptively scaled. Specifically, fields with high consistency retain more fused feature information, while fields with low consistency suppress fused features, thereby suppressing unreliable features and highlighting the discriminative information of abnormal regions. Detection output: Field-level consistency scores are directly used for field-level forgery location, with the field with the lowest score being the most suspicious tampered field; Field consistency scores are further used to assist in image-level forgery detection after statistical feature extraction. The feature fusion process can perceive the detection target, and the detection results can guide the fusion strategy in reverse, forming an end-to-end collaborative optimization closed loop.

4. The structured document forgery detection method based on cross-modal consistency learning according to claim 1, characterized in that, The field-level comparative ranking learning includes three loss components: Cross-modal alignment loss: For real samples, constrain the consistency scores of all fields to approach the target high value: ； in The consistency score is given by N, which represents the total number of fields in the current sample. For forged samples, a minimum selection strategy is used to identify the most suspicious field, constraining the consistency score of that field to approach a target low value: ； The final cross-modal alignment loss is the sum of the two: ； Field-level ranking loss: Calculated only for forged samples, constraining the consistency score of normal fields within the same forged sample to be higher than the score of potentially tampered fields, using a soft-margin ranking loss form: ； in This is the sorting interval hyperparameter. This is a collection of normal fields. For a set of potentially tamperable fields, , These represent the consistency scores of the corresponding fields; True / False Separation Loss: Construct true / false sample pairs within a batch, constraining the overall average consistency score of true samples to be higher than that of fake samples. ； Where P represents the set of real and fake sample pairs constructed in the current batch, r represents a real sample, and f represents a fake sample. and The average consistency scores for real samples and fake samples are respectively. This is the separation interval hyperparameter.

5. The structured document forgery detection method based on cross-modal consistency learning according to claim 4, characterized in that, The identification of potential tampered fields in the field-level ranking loss employs one or a combination of the following heuristic strategies: Fixed k-value strategy: Select the k fields with the lowest consistency scores in the current sample as the set of potential tampered fields, and the remaining fields as the set of normal fields, where k is a preset fixed value; Dynamic threshold strategy: Calculate the mean of the consistency scores of all fields in the current sample. and standard deviation The score is lower than The fields are marked as potentially tamperable fields, where As an adjustable hyperparameter, this strategy can adaptively determine the number of fields to be tampered with based on the characteristics of the sample.

6. The structured document forgery detection method based on cross-modal consistency learning according to claim 4, characterized in that, The field-level ranking loss does not require field-level forgery annotation; it only requires image-level true and false labels to learn field-level discrimination ability under weak supervision. Its design principle is to utilize the natural difference in consistency scores between the tampered and untampered fields in the forged sample, and to transform the structural characteristics of this sample into fine-grained supervision signals through sorting constraints.

7. The structured document forgery detection method based on cross-modal consistency learning according to claim 6, characterized in that, The joint optimization objective of the triple loss is: ； ； in For classifying losses, , , These are the weighting coefficients for each loss component, used to balance the contributions of the three losses. This represents the overall weight of the FCR loss.

8. The structured document forgery detection method based on cross-modal consistency learning according to claim 1, characterized in that, Joint optimization training is performed using three constraints: cross-modal alignment loss, field-level ranking loss, and true / false separation loss. A progressive training strategy is adopted, dividing the training process into three stages: The first stage is the visual warm-up stage: freeze the text encoder parameters, and train the visual encoder and multimodal fusion module using only classification loss and alignment loss; The second stage is the joint optimization stage: unfreeze some layers of the text encoder, introduce field-level ranking loss, establish cross-modal alignment and learn the relative relationships between fields; The third stage is the fine-tuning stage: full parameter fine-tuning, introducing the complete FCR loss, and using a complete joint optimization objective that includes cross-modal alignment loss, field-level sorting loss and true / false separation loss to complete end-to-end fine-tuning; The learning rate decreases sequentially at each stage, and loss components are gradually introduced to ensure stable convergence of multi-objective optimization.

9. The structured document forgery detection method based on cross-modal consistency learning according to claim 1, characterized in that, In step S5, the forgery detection output includes: Consistency statistical feature extraction: The consistency scores of all fields are aggregated into a fixed-dimensional statistical feature vector, including the minimum value, mean value, and the k lowest scores, to reflect the score distribution patterns of different forgery types; single-field tampering is manifested as the minimum value being less than the threshold but the mean value being normal, and multi-field tampering is manifested as multiple low scores and a decrease in the mean value; Field-level output: Outputs the consistency score and sorting of each field, used to locate suspected tampered fields and provide auditors with interpretable detection evidence.

10. A structured document forgery detection system based on cross-modal consistency learning, characterized in that, include: Multimodal input module: responsible for receiving the document image to be detected and text annotation information from trusted sources; Visual feature extraction module: Extracts visual features from document images and outputs local feature sequences and global visual vectors; Text feature extraction module: Combines layout position embedding to extract text features that integrate semantic and spatial location information, and outputs token-level feature sequence and global text vector; The hierarchical consistency-aware fusion module performs cross-modal feature fusion and consistency calculation at three granularities: Token level, field level, and global level. It includes a Token-level cross-modal attention submodule, a field-level consistency-aware gating fusion submodule, and a global-level consistency weighted aggregation submodule. The consistency score serves as both a fusion guidance signal and a detection output, achieving synergistic optimization of fusion and detection through gating mechanisms and weighted aggregation strategies. Field-level comparison and ranking learning module: Calculates cross-modal alignment loss, field-level ranking loss, and true / false separation loss, constructs a joint optimization objective, and improves field-level detection accuracy under weak supervision by utilizing the internal structural characteristics of fake samples; Classification Decision Module: It integrates multi-source features to output image-level forgery probability, and also outputs field-level consistency score for forgery localization.

Images