Certificate tampering detection method, device, equipment, medium and program

Abstract

The application relates to the technical field of image forgery detection, in particular to a voucher tampering detection method, device, equipment, medium and program, wherein the method comprises the following steps: acquiring a voucher image to be verified; inputting the voucher image to be verified into a pre-trained target detection model; the target detection model outputs a pixel-by-pixel tampering probability graph and a true-false judgment result; generating a tampering region heat map and tampering region positioning information according to the pixel-by-pixel tampering probability graph; and pushing the tampering region heat map, the tampering region positioning information and the true-false judgment result to a business review interface. Therefore, the technical problems of low detection accuracy, high missed detection rate and the like caused by the fact that the detection means in the related art is difficult to cover multiple tampering means are solved.

Description

Technical Field

[0001] This application relates to the field of image forgery detection technology, and in particular to a method, apparatus, equipment, medium and program for detecting credential tampering. Background Technology

[0002] In corporate lending by commercial banks, pre-loan due diligence requires verifying the authenticity of numerous documents, including VAT invoices, bank statements, contracts, audit reports, financial statements, and tax payment certificates. These documents, in the form of scanned copies or photos taken with a mobile phone, circulate throughout the business process, involving core aspects such as credit approval, trade finance, and risk control. Their authenticity is directly related to the security of credit assets and the stability of the financial order.

[0003] With the widespread use of digital image editing tools, various forgery methods such as counterfeiting, photocopying, splicing, and digital tampering have emerged.

[0004] For image tampering detection, the main approaches include: first, locating tampered areas such as splicing and copy-pasting by analyzing the discontinuities in image noise distribution; second, detecting frequency domain anomalies introduced by re-photographing and secondary compression through spectral analysis; and third, performing end-to-end detection based on deep learning-trained segmentation or classification networks. However, these methods are all insufficient to cover multiple tampering techniques, resulting in low detection accuracy and high false negative rates. Summary of the Invention

[0005] This application provides a method, apparatus, equipment, medium, and procedure for detecting credential tampering, in order to solve the technical problems in related technologies, such as the difficulty of detection methods to cover multiple types of tampering methods, resulting in low detection accuracy and high false negative rate.

[0006] The first aspect of this application provides a method for detecting credential tampering, comprising the following steps: acquiring an image of a credential to be verified; inputting the image of the credential to be verified into a pre-trained target detection model, wherein the target detection model outputs a pixel-by-pixel tampering probability map and a authenticity determination result; generating a tampering area heatmap and tampering area location information based on the pixel-by-pixel tampering probability map; and pushing the tampering area heatmap, the tampering area location information, and the authenticity determination result to a business review interface.

[0007] Further, the training method of the target detection model is as follows: Real desensitized samples, rule-synthesized samples, and generative adversarial samples are acquired. The rule-synthesized samples generate tampering regions based on voucher format rules and anti-counterfeiting element rules, and simultaneously generate pixel-level tampering annotations corresponding to the tampering regions. The real desensitized samples, rule-synthesized samples, and generative adversarial samples are mixed in a set ratio to form a training set. The real desensitized samples are used to anchor the real data distribution, and the rule-synthesized samples are used to supplement sparse tampering patterns. The target detection model is trained using the training set, and the training results output by the target detection model are obtained. A joint loss value is calculated based on the segmentation loss of the tampering localization head and the classification loss of the true / false classification head in the training results. The parameters of the shared visual backbone, the tampering localization head, and the true / false classification head in the target detection model are updated together based on the joint loss value until a preset training stopping condition is met.

[0008] Furthermore, the formula for calculating the joint loss value based on the segmentation loss of the tampered localization head and the classification loss of the true and false classification heads in the training results is as follows: ; in, For balance coefficient, To tamper with the segmentation loss of the positioning head, The classification loss is the difference between true and false classification heads.

[0009] Furthermore, the target detection model includes: an input layer, a shared visual backbone layer, a tampering localization head, and a authenticity classification head; wherein, the input layer extracts noise residual features, frequency domain features, and semantic consistency features from the image of the voucher to be verified, encodes them into feature maps with the same spatial resolution and number of channels through intra-path encoding, and then outputs fused features through channel attention reweighting and cross-path channel compression; the shared visual backbone layer performs sampling operations on the fused features step by step to extract multi-scale features, saves intermediate features at each level, and outputs deep feature representations; the tampering localization head performs encoding and decoding operations on the deep feature representations, wherein the decoding side upsamples step by step and splices them with intermediate features at each level of the shared visual backbone through skip connections, restores the input image resolution, and outputs a pixel-by-pixel tampering probability map; the authenticity classification head performs global pooling on the deep feature representations and then maps them through a fully connected layer to output the voucher authenticity classification result.

[0010] Further, the extraction of noise residual features, frequency domain features, and semantic consistency features from the image of the voucher to be verified includes: the noise residual features are generated by high-pass filtering and convolution of the input image to obtain a noise response map, followed by encoding; the frequency domain features are generated by performing a block-wise discrete cosine transform on the input image, rearranging the frequency domain coefficients corresponding to each transformed image block along the channel dimension, and then encoding; the semantic consistency features are generated by rasterizing the verification results of the voucher fields into a feature map aligned with the input image and then encoding.

[0011] Further, the step of generating a heatmap of the tampered region and tampered region location information based on the pixel-by-pixel tampering probability map includes: binarizing the pixel-by-pixel tampering probability map according to a set threshold to obtain a tampering mask; performing connected component analysis on the tampering mask to extract at least one connected component and filtering out noise regions with an area smaller than a preset lower limit; calculating the bounding rectangle based on the connected components to generate the tampered region location information; and rendering the pixel-by-pixel tampering probability map as the tampered region heatmap.

[0012] A second aspect of this application provides an apparatus for detecting credential tampering, comprising: an acquisition module for acquiring an image of a credential to be verified; a processing module for inputting the image of the credential to be verified into a pre-trained target detection model, wherein the target detection model outputs a pixel-by-pixel tampering probability map and a authenticity determination result; and a generation module for generating a tampering region heatmap and tampering region location information based on the pixel-by-pixel tampering probability map, and pushing the tampering region heatmap, the tampering region location information, and the authenticity determination result to a business review interface.

[0013] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to perform the credential tampering detection method as described in the above embodiments.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to perform the credential tampering detection method as described in the above embodiments.

[0015] The fifth aspect of this application provides a computer program product, including a computer program or instructions, which, when executed, implement the credential tampering detection method as described in the above embodiments.

[0016] Therefore, this application has at least the following beneficial effects: This application embodiment can input the image of the voucher to be verified into the target detection model. The network can simultaneously output a pixel-by-pixel tampering probability map and the authenticity judgment result in a single forward propagation, realizing integrated processing from image input to detection result. Compared with the method of needing to run multiple independent models in sequence, it significantly improves detection efficiency. It also allows pixel-level positioning information and image-level classification results to share the same set of backbone features and mutually verify each other. With the skip connection and upsampling operation of the positioning head, the probability map is output after restoring to the original resolution, so that the tampered area can be identified with pixel precision. This avoids the limitation that the tampering location cannot be indicated when relying solely on image-level classification results. Then, based on the probability map, a visual heat map and positioning information are generated and pushed to the review interface, allowing business personnel to intuitively locate suspected tampered areas and conduct targeted key reviews in combination with the authenticity judgment results. This improves the interpretability of voucher verification and makes up for the shortcomings of traditional methods that only output judgment conclusions without positioning evidence.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a credential tampering detection method provided according to an embodiment of this application; Figure 2 This is a flowchart of the algorithm for procedurally generating samples according to the rules provided in the embodiments of this application; Figure 3 This is a structural diagram of the shared backbone target detection model provided according to the embodiments of this application; Figure 4 This is a tensor flow graph of the target detection model provided according to the embodiments of this application; Figure 5 This is a schematic diagram of signal fusion according to an embodiment of this application; Figure 6 This is a schematic diagram of the overall process of the credential tampering detection method provided in the embodiments of this application; Figure 7 This is a schematic diagram of a credential tampering detection device provided according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0019] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0020] The specific values ​​involved in the following embodiments are merely illustrative examples, and those skilled in the art can perform adaptive calibrations accordingly.

[0021] In corporate lending at commercial banks, pre-loan due diligence requires verifying the authenticity of numerous documents, including VAT invoices, bank statements, contracts, audit reports, financial statements, and tax payment certificates. These documents are circulated in the form of scanned copies or photos taken with a mobile phone, and there are various methods of forgery, such as forgery by copying, forgery by splicing, and forgery by tampering with numbers, which are difficult for account managers to reliably identify with the naked eye.

[0022] For image tampering detection, existing technologies mainly include the following approaches: (1) Noise residual-based methods: by analyzing the discontinuity of image noise distribution, the splicing, copying and pasting areas are located; (2) Frequency domain feature-based methods: by using spectrum analysis to detect frequency domain anomalies introduced by re-photographing, secondary compression and resampling; (3) Deep learning-based methods: by using supervised learning to train segmentation or classification networks, the tampered areas are located end-to-end or the authenticity is determined.

[0023] The above methods have achieved certain results on general tamper detection public datasets, but when applied to the scenario of Bank of China credit certificates, they have the following insurmountable defects.

[0024] (1) Real forged samples are extremely scarce, making it difficult for supervised learning to converge. In actual banking operations, the number of forged documents that are detected and retained is extremely small, resulting in a severe shortage of positive samples (forged samples). Conventional supervised learning is difficult to train a usable model due to the extreme imbalance between positive and negative samples.

[0025] (2) Real samples are subject to strict anonymization constraints, limiting data availability. The credentials contain a large amount of sensitive corporate and personal information and are subject to regulatory and compliance requirements. Real samples must be anonymized and the data must not leave the domain, further reducing the scale of real data available for training.

[0026] (3) The general model differs significantly from the distribution of credit documents in China. The general tampering detection model is trained on public datasets and has significant distribution differences from the anti-counterfeiting elements of Chinese value-added tax invoices, the format features of bank statements, and the format features of audit materials. Direct deployment does not meet expectations and there is a serious distribution drift problem.

[0027] (4) Single signal sources are prone to missing complex tampering. Different forgery methods such as photocopying, re-photographing, splicing, and digital tampering leave different traces on the image. Relying solely on noise residuals or single signals with frequency domain characteristics is insufficient to cover them and is prone to missing detection.

[0028] Therefore, this application provides a training method for a three-source collaborative credential tampering detection model under small sample constraints. By systematically constructing three types of complementary samples and mixing them in proportion, and combining a target detection model with a shared backbone with three-way signal fusion, a robust detection model against multiple tampering methods is trained under the constraints of scarce real forged samples and strong desensitization. The trained detection model is then used to output a heat map of the tampered area and the results of authenticity determination.

[0029] The following description, with reference to the accompanying drawings, describes a credential tampering detection method, apparatus, electronic device, storage medium, and program product according to embodiments of this application.

[0030] Specifically, Figure 1 This is a flowchart illustrating a credential tampering detection method provided in an embodiment of this application.

[0031] like Figure 1 As shown, the credential tampering detection method includes the following steps: In step S101, the image of the voucher to be verified is obtained.

[0032] It is understood that this embodiment can acquire the image of the voucher to be verified and use the image as input data for subsequent dual-task network processing, providing the original image basis for pixel-by-pixel tampering probability analysis of the entire voucher, thereby enabling subsequent processing steps to simultaneously perform pixel-level positioning of the tampered area and authentication of the voucher as a whole on the image.

[0033] It should be noted that the images of the vouchers to be verified include financial credit documents such as VAT invoices, bank statements, audit reports, financial statements, and tax payment certificates. These voucher images circulate in the business process in the form of scanned copies or mobile phone photos, involving core links such as credit approval, trade finance, and risk control. Their authenticity is directly related to the security of credit assets and the stability of the financial order.

[0034] In step S102, the image of the certificate to be verified is input into the pre-trained target detection model, and the target detection model outputs a pixel-by-pixel tampering probability map and a authenticity determination result.

[0035] It is understood that this embodiment can input the image of the voucher to be verified into a pre-trained target detection model. This model can simultaneously output a pixel-by-pixel tampering probability map and a authenticity determination result in a single forward propagation, realizing end-to-end processing from image input to tampering probability and authenticity determination. This eliminates the need to run multiple independent models or perform multiple forward calculations for tampering location and authenticity classification, significantly simplifying the detection process and improving inference efficiency. Furthermore, the two types of outputs share the same input image and model features in the same forward process, making the tampering probability distribution and authenticity determination result inherently consistent at the data level, avoiding potential decision conflicts caused by different feature extraction methods when processing separately. At the same time, the pixel-by-pixel tampering probability map retains the spatial location information of the tampered area, giving the detection result spatial orientation in pixel precision, while the authenticity determination result provides a qualitative conclusion from the overall image level. The two complement each other within the same framework, jointly providing a detection basis with both spatial precision and overall discriminative power for subsequent visual verification, thereby improving the intelligence level of voucher tampering detection and the completeness of output information.

[0036] In this embodiment, the training method for the target detection model is as follows: Real desensitized samples, rule-synthesized samples, and generative adversarial samples are acquired. The rule-synthesized samples generate tampering regions based on voucher format rules and anti-counterfeiting element rules, and simultaneously generate pixel-level tampering annotations corresponding to the tampering regions. A training set is formed by mixing real desensitized samples, rule-synthesized samples, and generative adversarial samples in a set ratio. Real desensitized samples are used to anchor the real data distribution, and rule-synthesized samples are used to supplement sparse tampering patterns. The target detection model is trained using the training set, and the training results output by the target detection model are obtained. A joint loss value is calculated based on the segmentation loss of the tampering localization head and the classification loss of the true / false classification head in the training results. The parameters of the shared visual backbone, tampering localization head, and true / false classification head in the target detection model are updated jointly based on the joint loss value until a preset training stopping condition is met.

[0037] It is understood that this embodiment can obtain real de-identified samples, rule-synthesized samples, and generative adversarial samples. The rule-synthesized samples generate tampering areas and simultaneously generate pixel-level tampering annotations based on the voucher format rules and anti-counterfeiting element rules. This ensures that the training set can still obtain sufficient tampering samples with pixel-level annotations even when real forged samples are scarce. Furthermore, the tampering methods of these samples strictly adhere to the voucher's own format constraints and anti-counterfeiting logic, thus distinguishing it from blind random data augmentation. This allows the network to learn tampering patterns consistent with real business scenarios during training. By mixing the three types of samples in a set ratio to form the training set, real de-identified samples anchor the real data distribution, rule-synthesized samples supplement sparse tampering patterns, and generative adversarial samples cover unknown tampering methods. The synergy of these three elements ensures that the training set has a balanced sample distribution. To ensure authenticity and diversity, this approach alleviates the constraints on model performance caused by the scarcity and distribution drift of genuine forged samples. The target detection model is trained using this training set, and a joint loss value is calculated based on the segmentation loss of the tampered localizer and the classification loss of the genuine / fake classification heads. This ensures that the localization and classification tasks are mutually constrained under the same optimization objective, avoiding the tendency of the shared backbone to favor one task under single-task supervision. The parameters of the shared visual backbone, the tampered localizer, and the genuine / fake classification heads are updated jointly based on the joint loss value, allowing the gradients of both tasks to simultaneously act on the shared parts. Pixel-level supervision of the localization task forces the backbone to encode spatial evidence of tampering, while global supervision of the classification task guides the backbone to focus on discriminative regions that contribute to the final judgment. These two methods are mutually regularizing, enabling the model to converge stably under small sample conditions and improving detection accuracy and robustness.

[0038] In this embodiment of the application, the formula for calculating the joint loss value based on the segmentation loss of the tampered localization head and the classification loss of the true and false classification heads in the training results is as follows: ; in, For balance coefficient, To tamper with the segmentation loss of the positioning head, The classification loss is the difference between true and false classification heads.

[0039] Understandably, this embodiment can construct a formula for calculating the joint loss value, weighting and summing the segmentation loss of the tampered localization head and the classification loss of the true and false classification heads in the same objective function. This allows the network to be subject to dual constraints of pixel-level localization supervision and image-level classification supervision during training, avoiding the problem of the network being biased towards one task and neglecting the other due to relying on only a single loss function. Here, α is a balancing coefficient used to adjust the weight ratio of segmentation loss and classification loss in joint training, allowing the two tasks to flexibly adjust their optimization focus according to the actual training effect, and reasonably allocate the contribution of supervision signals in different training stages or under different data conditions, thereby achieving a balance between localization accuracy and classification accuracy in joint optimization. Through the calculation of this joint loss value, the two tasks form a mutually restrictive and mutually reinforcing optimization relationship on the shared backbone. The gradient of the localization task forces the backbone to encode spatial evidence of tampering, while the gradient of the classification task guides the backbone to focus on the discriminative region. The two work together to update the parameters of the shared visual backbone, thereby improving the overall performance of the model on both tasks.

[0040] Specifically, (1) such as Figure 2 As shown, to address the dual challenges of scarce and drifting distribution of genuine forgery samples, three types of complementary samples are systematically constructed. These three types of samples differ in their source, generation method, and function: Genuine de-identified samples originate from genuine vouchers from partner banks. After de-identification and differential privacy processing, the data remains within the domain. Their function is to anchor the distribution of real data, but their scarcity means they serve only as a distribution benchmark and cannot undertake the main training task. Rule-synthesized samples are generated procedurally based on voucher format rules and anti-counterfeiting element rules, creating areas for photocopying, re-photographing, splicing, and digital tampering. Their function is to supplement the sparse tampering patterns that genuine samples cannot cover. Generative adversarial samples utilize generative adversarial networks or diffusion models to generate highly realistic and difficult samples, covering unknown and highly realistic tampering methods, thereby improving the model's generalization and robustness.

[0041] The generation mechanism of rule-based synthetic samples does not employ simple data augmentation. Instead, it encodes knowledge from areas such as the anti-counterfeiting elements of VAT invoices and the format rules of bank statements into the sample generator. It then programmatically applies tampering to a real sample according to the rules, simultaneously recording the precise pixel areas that have been tampered with. The resulting samples not only closely resemble the distribution of real documents but also naturally carry pixel-level tampering annotations, directly providing monitoring signals for subsequent tampering location tasks. This effectively solves the dual problems of scarcity of real tampering samples and the lack of pixel-level annotations.

[0042] In its specific implementation, the procedural generation process of rule-synthesized samples is as follows: First, the format rules and anti-counterfeiting element rules are loaded onto the actual certificate base. Then, the tampering type, such as photocopying, re-photographing, splicing, or digital alteration, is selected. Corresponding tampering operators are applied to the specified area according to the rules, and the coordinates of the tampered pixel area are recorded. This automatically generates pixel-level annotations paired with the tampered area, and the entire process requires no manual intervention. When the number of samples is insufficient, generation can be cyclical, and the samples are ultimately incorporated into the three-source training set. The essence of this process lies in solidifying domain knowledge into executable sample generation steps, enabling the synthesized samples to automatically carry pixel-level supervision signals while possessing realistic distribution characteristics. This is also the substantial feature that distinguishes this application from conventional data augmentation.

[0043] Through the complementary synergy of the above three types of samples, the real desensitized samples are anchored to the real distribution to prevent the model from deviating from the actual scenario. The rule-synthesized samples systematically supplement various tampering modes and come with pixel-level supervision signals through a procedural generation method driven by domain knowledge. The generative adversarial samples cover unknown tampering methods with highly realistic and difficult examples. The three work together to alleviate the three core problems of sample scarcity, lack of pixel-level annotation and distribution drift, laying a data foundation for the joint training of the subsequent dual-task network.

[0044] (2) Mix the three source samples according to a set ratio to form a training set. This ratio can be expressed as the sampling proportion of the three types of samples in each training batch. , , The sum of the three is 1. Since the number and distribution characteristics of the three types of samples are significantly different, a fixed sampling ratio is difficult to balance the model's full learning of the tampering pattern and accurate fitting of the true distribution throughout the training process. Therefore, it is necessary to design a reasonable matching strategy to balance the contribution of each sample source.

[0045] As a preferred implementation, this ratio can be dynamically adjusted as the training process progresses. In the early stages of training, the rule-synthesized samples are plentiful and have pixel-level annotations, providing dense supervisory signals. Therefore, increasing their sampling ratio helps the tamper detection head quickly learn the basic characteristics of various tampering patterns. In the later stages of training, although the number of real desensitized samples is small, their distribution is closest to the actual business scenario. Increasing their sampling ratio helps correct the distribution bias learned by the model on the synthetic samples, allowing the model to gradually transition from fitting synthetic traces to data distributions that are closer to real vouchers, thereby alleviating the distribution drift problem.

[0046] This application addresses the dilemma of using a dynamically adjustable three-source ratio strategy. Under a fixed ratio, if the proportion of rule-synthesized samples is too high, the model is prone to overfitting to the artificial traces in the synthesized samples and failing to generalize to real-world scenarios. Conversely, if the proportion of real, desensitized samples is too high, the model struggles to fully learn tampering patterns due to insufficient labeled samples. The dynamic ratio strategy effectively avoids this predicament: initially, training is dominated by synthetic samples, enabling rapid model convergence; later, real samples are used to correct the distribution, bringing the model closer to actual business scenarios. These two approaches are seamlessly integrated and synergistic throughout the training process, systematically mitigating the performance constraints imposed by sample imbalance and distribution drift.

[0047] It should be noted that the fusion of the three signals is carried out in two stages. The design logic is to first encode the three heterogeneous signals into a unified feature format, and then integrate the cross-channel information. The first stage is intra-channel encoding alignment. Each of the three signals is first processed by a shallow convolutional encoding layer to unify them into feature maps with the same spatial resolution and number of channels as the input image. This allows signals with different formats and scales to be processed by the subsequent network in the same feature space. The noise residual route is obtained by convolving the original image with a set of high-pass filter kernels to obtain a noise response map, and then convolves it to extract the discontinuous information of noise distribution related to splicing and copying / pasting in the image. The frequency domain route obtains a frequency domain feature map by performing a block discrete cosine transform on the image and rearranging the coefficients of each block into channel dimensions, and then convolves it to capture frequency domain statistical anomalies introduced by re-photographing, secondary compression, and resampling. The semantic consistency route rasterizes the page field verification results into a feature map aligned with the image and then convolves it to integrate the business logic judgment results such as field position and whether the cross-validation is passed into the feature expression in a spatially aligned manner.

[0048] The second stage is inter-path fusion, where the feature maps after the three-path encoding are aligned are concatenated along the channel dimension to obtain joint features. These are then compressed and cross-path information is exchanged through a 1×1 convolution layer to output fused features. As a preferred implementation, a channel attention mechanism is introduced before the 1×1 convolution. Channel-wise weights are obtained through global average pooling, two fully connected layers, and sigmoid activation. These weights are then reweighted on the concatenated joint features, enabling the network to adaptively adjust the contribution of each signal based on the type of tampering in the input image—for example, increasing the weight of the frequency domain path on a reproduced image to enhance the response to frequency domain anomalies, and increasing the weight of the noise residual path on a concatenated image to enhance the perception of discontinuous noise distribution—thus achieving adaptive signal selection based on the type of tampering.

[0049] This application employs an input-level early fusion strategy, where the three signals are fused before entering the shared visual backbone. This allows the backbone network to encode on a unified fusion feature, rather than encoding each signal separately before decision-level fusion. This design enables the dual task heads to share the same fusion representation. During training, the gradients of the tampered localization head and the true / false classification head work together on the fusion layer and the backbone network, allowing the supervision signals of the two tasks to interact during the fusion phase. The localization task forces the backbone to encode tampered spatial evidence, while the classification task guides the backbone to focus on discriminative regions. The two tasks mutually constrain and regularize each other, thereby improving the model's generalization ability and detection robustness under small sample conditions.

[0050] (3) Train a network with a shared visual backbone and dual output heads. This network achieves multi-dimensional tampering perception of voucher images by fusing three signals: noise residual, frequency domain features, and semantic consistency. For example... Figure 3 and Figure 4 As shown, the input image first extracts heterogeneous features through three signal branches, then encodes them into deep feature representations through a shared backbone, and then connects to the tamper location head and the true / false classification head for parallel output. The specific tensor dimensions of the entire network are labeled at each stage to facilitate engineering reproduction.

[0051] like Figure 5 As shown, the three signals complement each other against different tampering methods. The noise residual branch extracts the noise fingerprint of the image to capture the discontinuity in noise distribution caused by splicing and copying / pasting; the frequency domain feature branch captures frequency domain anomalies introduced by re-photographing, secondary compression, and resampling through spectrum analysis; and the semantic consistency branch captures the discrepancies in monetary amounts and field logic caused by digital tampering by verifying field and layout logic. After attention weighting and feature-level splicing fusion, the network can adaptively rely on different signal paths according to the tampering type of the input image, thereby significantly improving the coverage of complex tampering and overcoming the inherent defect of easy missed detection by a single signal source.

[0052] The fused features share a visual backbone, which can employ a Swing Transformer or ResNet structure. This backbone encodes the fused features into a unified shared feature representation, which is then fed into two output heads. The tampering localization head uses a semantic segmentation network to output pixel-by-pixel tampering probabilities to locate the tampered region, corresponding to a segmentation loss. The authenticity classification head outputs the overall binary classification result of the credentials, corresponding to a classification loss. Both tasks share the same backbone and are jointly optimized. The joint loss can be expressed as a weighted sum of the segmentation loss and the classification loss, where a balancing coefficient is used to adjust the contribution ratio of each during training.

[0053] The dual-task shared backbone ensures that tamper localization and true / false classification are mutually regularized during training: pixel-level supervision in the localization task forces the backbone to encode spatial evidence of tampering, while global supervision in the classification task guides the backbone to focus on discriminative regions. Both jointly constrain the learning direction of the shared representation, effectively improving the model's accuracy and robustness under small sample conditions. Simultaneously, the fusion of three heterogeneous signals specifically covers different forgery methods such as photocopying, re-photographing, splicing, and digital tampering, overcoming the limitation of a single signal source's incomplete coverage of complex tampering. The combination of these two approaches enables the network to achieve robust tamper detection even under sample scarcity.

[0054] It should be noted that the processing of the shared backbone, using the Swin Transformer as an example, is as follows: The fused feature map is first divided into non-overlapping image blocks, which are then linearly mapped to a label sequence via block embedding. Subsequently, it passes through Transformer blocks consisting of alternating window self-attention and shift window self-attention. Window self-attention calculates attention within a local window to control computation, while shift windows enable information exchange between adjacent windows. Downsampling is performed between each level through block merging, with the spatial resolution halved and the number of channels doubled at each level. The final output is a deep feature representation with a spatial resolution of one-sixteenth of the input image and D channels. Intermediate features at each level are retained for use in the localization head's skip connections. When using ResNet as the backbone, the fused features are processed through initial convolution and several residual stages. Each stage consists of multiple residual blocks, each containing convolution, batch normalization, and activation functions, and superimposed with identity shortcut connections. Similarly, downsampling is performed at each level, and intermediate features at each level are retained. Both types of backbones output multi-scale features. This application does not limit the type of backbone; any visual backbone that can output multi-scale feature representations can be used equivalently.

[0055] The localization head employs an encoder-decoder structure. The intermediate features at each level of the backbone are used as encoding features, while the decoding side upsamples progressively from the deepest features output by the backbone. After each upsampling, intermediate features of the same resolution as the encoding side are concatenated via skip connections, thus preserving low-level spatial details while restoring spatial resolution. After progressively restoring to the input image resolution, the final layer undergoes a 1×1 convolution to compress the channels to 1, followed by Sigmoid activation to obtain a pixel-by-pixel tampering probability map. The value of each pixel in this probability map represents the probability that its location belongs to a tampered region. The classification head performs global average pooling on the deepest features output by the backbone to obtain a global feature vector. This vector is mapped to two-dimensional logits through a fully connected layer and then normalized using Softmax to obtain the overall probabilities of the credentials being true or false. The fully connected layer may contain a hidden layer and incorporates ReLU activation and Dropout regularization to prevent overfitting.

[0056] During the joint training of the two tasks, the localization head uses segmentation loss with pixel-level labeled masks as supervision, while the classification head uses cross-entropy loss with sample-level true / false labels as supervision. The total loss is a weighted sum of the two losses, with a balancing coefficient used to adjust the weight ratio of the two types of losses in the joint training. The gradients of the two tasks are backpropagated through their respective heads to the shared backbone and fusion layer. This allows the localization supervision to help the classification task learn spatial evidence of tampering, while the classification supervision helps the localization task focus on discriminative regions. Under small sample conditions, the two are mutually regularized and mutually constrained, thereby effectively improving the model's generalization ability and detection robustness in situations where labeled data is scarce.

[0057] U-Net employs a symmetrical encoder-decoder structure: the encoder side consists of multiple levels of convolution and downsampling operations, extracting semantic features step by step; the decoder side consists of multiple levels of upsampling and convolution operations, restoring spatial resolution step by step; skip connections are set between layers of the same resolution on the encoder and decoder sides to concatenate the features from the encoder side to the decoder side, preserving low-level spatial details while restoring resolution, thus achieving pixel-level precise localization. DeepLabV3+, on the other hand, sets a dilated spatial pyramid pooling module after the backbone network. This module consists of multiple dilated convolutions with different dilation rates operating in parallel, used to capture multi-scale contextual information, which is then fused with low-level features by the decoder and upsampled for output, thereby restoring spatial resolution while maintaining the receptive field.

[0058] (4) In the output stage of the entire training process, the results of the two output heads of the dual-task network are transformed into forms that can be used by the business. The pixel-by-pixel tampering probability output by the tampering location head is rendered into a tampering area heatmap after post-processing. This heatmap visually indicates the spatial distribution and confidence level of suspected tampering areas in the image, supporting business personnel to conduct visual verification of the model detection results; the authenticity judgment result output by the authenticity classification head serves as a qualitative conclusion on the authenticity of the voucher, providing a direct basis for credit approval. Both output detection conclusions from the two dimensions of spatial location and overall judgment, respectively, so that the business review process can see both the tampering location and obtain the overall judgment, improving the comprehensiveness and interpretability of the verification.

[0059] As a preferred implementation, the training process of this application is conducted under the constraint that data does not leave the data domain, in order to meet the regulatory compliance requirements of the financial industry. Real, anonymized samples participate in training through differential privacy or federated learning, ensuring that real data can still contribute to model training without leaving the bank's data domain. Rule-synthesized samples and generative adversarial samples are generated locally on demand, without relying on external data input, thus avoiding the risk of data leakage at the source. This compliant training mechanism enables this application to be legally deployed in the actual business environment of banks, solving the bottleneck of general models being difficult to implement in financial scenarios due to data compliance restrictions.

[0060] Furthermore, this application employs efficient parameter fine-tuning techniques to fine-tune the pre-trained multimodal document understanding model. Specifically, using methods such as LoRA or AdaLoRA, only a small number of low-rank adaptation parameters are updated while freezing most of the parameters of the pre-trained model. This allows the model to inherit the general document understanding capabilities of the pre-trained model while adapting to the specific task of credential tampering detection, significantly reducing the computational cost and storage overhead required for training while ensuring detection performance. This strategy complements the data-without-domain training method; the former addresses compliance issues, while the latter addresses resource constraints, together enabling this application to be deployable and sustainable in actual banking business environments.

[0061] In this embodiment, the target detection model includes: an input layer, a shared visual backbone layer, a tampering localization head, and a authenticity classification head. The input layer extracts noise residual features, frequency domain features, and semantic consistency features from the image of the voucher to be verified. After intra-path encoding into feature maps with the same spatial resolution and number of channels, it performs channel attention reweighting and cross-channel compression to output fused features. The shared visual backbone layer performs sampling operations on the fused features at each level to extract multi-scale features, saves intermediate features at each level, and outputs a deep feature representation. The tampering localization head performs encoding and decoding operations on the deep feature representation. The decoding side upsamples at each level and concatenates it with intermediate features at each level of the shared visual backbone through skip connections, restoring the resolution to the input image and outputting a pixel-by-pixel tampering probability map. The authenticity classification head performs global pooling on the deep feature representation and maps it through a fully connected layer to output the voucher authenticity classification result.

[0062] It is understood that this embodiment can extract noise residual features, frequency domain features, and semantic consistency features from the image of the certificate to be verified through the input layer. These three features are sensitive to different forgery methods such as splicing and digital tampering. After being unified into feature maps with the same spatial resolution and number of channels through intra-channel encoding, the three features are fused, enabling the network to simultaneously perceive the periodic anomalies in the frequency domain of the image's low-level noise fingerprint and the logical contradictions in the high-level fields, overcoming the deficiency of a single signal source in fully covering complex tampering. Channel attention reweighting enables the network to adaptively adjust the contribution weights of each signal according to the type of tampering contained in the input image. Cross-channel compression then enables information interaction between different signals, thereby improving the ability of the fused features to represent different tampering methods. The shared visual backbone layer performs downsampling operations on the fused features step by step to improve... Multi-scale features are extracted, intermediate features at each level are preserved, and deep feature representations are output, so that shallow high-resolution spatial details and deep semantic information are preserved simultaneously, providing support for subsequent localization and resolution restoration. The tampering localization head adopts an encoding and decoding structure. The decoding side upsamples level by level and splices it with intermediate features at each level of the backbone through skip connections, so that deep semantic information and shallow spatial details are combined to restore the original input resolution, thereby outputting a pixel-by-pixel tampering probability map, realizing the precise localization of tampered areas at the pixel level. The authenticity classification head performs global pooling on the deep feature representation and maps it through a fully connected layer, outputting the overall authenticity classification result of the certificate. This allows the network to simultaneously provide pixel-level tampering spatial distribution and image-level authenticity qualitative conclusions within the same framework. The two complement each other to improve the interpretability of the detection results and the completeness of decision information.

[0063] In this embodiment, the noise residual features, frequency domain features, and semantic consistency features of the voucher image to be verified are extracted, including: the noise residual features are generated by high-pass filtering and convolution of the input image to obtain a noise response map and then encoding it; the frequency domain features are generated by performing a block discrete cosine transform on the input image, rearranging the frequency domain coefficients corresponding to each image block along the channel dimension, and then encoding them; the semantic consistency features are generated by rasterizing the voucher field verification results into a feature map aligned with the input image and then encoding it.

[0064] Understandably, this embodiment can use a high-pass filter kernel to convolve the input image to obtain a noise response map, and then encode it to generate noise residual features. This allows the network to capture the discontinuities in the image noise distribution caused by operations such as splicing and copy-pasting. This is because such tampering methods often introduce noise breaks at splicing boundaries, and the high-pass filter kernel specifically responds to high-frequency changes in the image, thus explicitly extracting these noise differences as features. By performing a block-wise discrete cosine transform on the input image and rearranging the coefficients of each block into channel dimensions, and then encoding them to generate frequency domain features, the network can perceive the periodic anomalies introduced by re-photographing and resampling in the frequency domain. This is because after the discrete cosine transform converts spatial domain information to the frequency domain, re-photographing and secondary compression will leave identifiable patterns on specific frequency components. The network calculates the deviation; by rasterizing the verification results of the voucher fields into feature maps aligned with the input image and then encoding them to generate semantic consistency features, the network can capture the contradictions in monetary amount and field logic caused by digital tampering. This is because the field verification results directly reflect whether the internal business logic of the voucher has passed or not. After rasterizing into feature maps, the spatial correspondence of this information on the image is preserved. The three features express the voucher image from three different dimensions: the frequency domain statistical characteristics of the low-level noise fingerprint and the high-level semantic logic. Each of the extraction methods is designed for the trace generation mechanism of specific tampering methods, so that the subsequent fusion steps can obtain complementary multi-dimensional feature inputs, providing a feature basis for the network to cover various forgery methods such as photocopying, re-photographing, splicing, and digital tampering.

[0065] In step S103, a heat map of the tampered area and the location information of the tampered area are generated based on the pixel-by-pixel tampering probability map, and the heat map of the tampered area, the location information of the tampered area and the authenticity judgment result are pushed to the business review interface.

[0066] Understandably, this embodiment can generate a heatmap of tampered areas based on a pixel-by-pixel tampering probability map, transforming the tampering probability of each pixel, represented by numerical values, into an image form that is visually expressed by color depth or warmth / coolness. This allows the originally abstract probability data to intuitively present the spatial distribution and confidence level of suspected tampering, enabling business personnel to quickly locate high-probability tampering areas visually without needing to interpret the data. By generating tampering area location information, the discrete tampered pixel areas in the probability map are transformed into bounding rectangles with clear location boundaries and geometric ranges, allowing the tampering location to be output in a regularized, structured identifier form, facilitating precise comparison and coordinate association with the actual field areas on the voucher. The tampering area heatmap, tampering area location information, and authenticity determination results are pushed to the business review interface, presenting the determination conclusion, tampering spatial distribution, and tampering location identifier on the same screen. Business personnel can directly compare and cross-verify the overall authenticity conclusion with the specific tampering location, completing a comprehensive review of the detection results without switching between different views or systems, thereby improving the intuitiveness and efficiency of the voucher verification process.

[0067] In this embodiment of the application, generating a heatmap of the tampered region and tampered region location information based on a pixel-by-pixel tampering probability map includes: binarizing the pixel-by-pixel tampering probability map according to a set threshold to obtain a tampering mask; performing connected component analysis on the tampering mask to extract at least one connected component and filtering out noise regions with an area smaller than a preset lower limit; calculating the bounding rectangle based on the connected components to generate tampered region location information; and rendering the pixel-by-pixel tampering probability map into a heatmap of the tampered region.

[0068] The threshold can be set according to actual needs. The threshold is the probability judgment boundary value used to convert the continuous probability map into a binary mask. The connected region is the continuous region formed by pixels that are adjacent to each other and are all marked as tampered in the binary tampering mask. The preset lower limit is the minimum retention size set for the area of ​​the connected region, without specific limitation.

[0069] It is understood that this embodiment can binarize the pixel-by-pixel tampering probability map according to a set threshold, converting the continuous probability value of each pixel into a binary judgment result of 0 or 1. This transforms the probability map from a continuous distribution state into a binary mask with a clear attribution of whether or not tampering has occurred, providing a definite basis for pixel attribution in subsequent region analysis. By performing connected component analysis on the tampering mask and extracting at least one connected component, scattered isolated tampered pixels are aggregated into independent regions with a holistic structure according to spatial adjacency. Noise regions with an area smaller than a preset lower limit are filtered out, eliminating false alarms caused by single-point or small-area false detections. This process ensures that the preserved connected components possess the spatial continuity and a certain size required for the actual tampered area. By calculating the circumscribed rectangle based on the connected components, irregularly shaped connected components are transformed into rectangular boxes with clear boundary positions and standard geometric shapes. This allows the tampered area location information to be output in a regularized identifier form, facilitating precise coordinate correspondence and comparison with specific field areas on the voucher image. Simultaneously, the pixel-by-pixel tampering probability map is rendered as a tampering area heatmap, preserving the degree of tampering suspicion expressed by continuous probability values ​​for each pixel. This enables business personnel to intuitively judge the distribution of tampering confidence in each area from the heatmap during review. Binarization and connected component analysis enable the transformation from continuous probability distribution to discrete connected regions. Area filtering removes isolated noise points with low confidence to ensure the reliability of the location area. Circumscribed rectangle calculation standardizes irregular connected regions into regular location boxes to enhance the clarity of location information. Heatmap rendering preserves the continuous information of probability values ​​through color mapping to help business personnel understand the confidence distribution. The combination of these four features ensures that the tampered location information pushed to the business review interface has both regular geometric boundaries and a refined confidence expression.

[0070] Specifically, the calculation process for the tampering probability is as follows: The localization head outputs a real value for each spatial location in the feature map, which is the logistic regression response for that location. A Sigmoid activation function is applied to this real value, mapping it to the interval between 0 and 1, yielding the probability that the pixel belongs to the tampered region. A value closer to 1 indicates a higher probability of tampering at that location, while a value closer to 0 indicates a higher probability that the location is normal. During the training phase, the pixel-level tampering label mask inherent in the rule-synthesized samples is used as the ground truth supervision signal. A binary cross-entropy loss is employed to measure the difference between the predicted probability and the true label. Simultaneously, a Dice loss can be superimposed to address the class imbalance problem where tampered pixels are far fewer than normal pixels, enabling the model to effectively learn tampering features even when the tampered region accounts for a very small proportion. Both of these factors together constitute the segmentation loss of the localization head.

[0071] The location of the tampered region during the inference phase is determined from the probability map using the following steps. First, the pixel-by-pixel tampering probability map is binarized according to a set threshold. Pixels with probability values ​​greater than or equal to the threshold are marked as tampered, while pixels with probability values ​​less than the threshold are marked as normal. This transforms the continuous probability map into a binary tampering mask. This threshold can be flexibly adjusted based on the business's tolerance for missed and false detections. Next, connected component analysis is performed on the binary mask to extract all interconnected tampered pixel regions. Noise regions with areas smaller than a preset lower limit are filtered out to eliminate false alarms caused by single-point or extremely small area false detections. For each retained connected component, its bounding rectangle is calculated as the location box of the tampered region on the original image. Simultaneously, the mean of the probability values ​​of all pixels within this region is output as the region-level confidence level to reflect the reliability of the tampering region determination. Finally, the original pixel-by-pixel probability map is rendered into a heatmap of the tampered area through color mapping. This heatmap is then overlaid on the original voucher image in a semi-transparent manner and output to the business review interface along with each location bounding box. This allows business personnel to intuitively view the distribution of suspected tampered areas, the gradual change in confidence level, and the specific boundaries of each area, thereby achieving a visual review of the model detection results.

[0072] The classification header's determination process is as follows: The two-dimensional logits output by the classification header are normalized using Softmax to obtain two probabilities: the document is genuine and the document is counterfeit. The sum of these two probabilities is 1. If the probability of being counterfeit is not lower than the determination threshold, the document is determined to be counterfeit; otherwise, it is determined to be genuine. This probability value is also output as the determination confidence level. This determination threshold can be flexibly adjusted according to business risk preferences. For scenarios more sensitive to missed detections, the threshold can be appropriately lowered to improve the recall rate of counterfeit documents; for scenarios more sensitive to false detections, the threshold can be raised to reduce false alarms.

[0073] The basis for the judgment is rooted in the supervised design during the training phase. During training, sample-level true / false labels serve as the supervision signal: real desensitized samples are labeled as true, while rule-synthesized samples and generative adversarial samples are labeled as false. Cross-entropy loss is used to train the classification head. Because the classification head and the localization head share the same visual backbone and fusion layer, pixel-level supervision in the localization task forces the shared feature representation to encode spatial evidence of tampering. Therefore, the classification head learns to make judgments based on the analysis of tampering trace features, rather than shallow correlations of surface textures, thereby improving the reliability and generalization ability of classification decisions.

[0074] As a preferred implementation, the classification head can be linked with the positioning head to further reduce the risk of misjudgment. Specifically, the global statistics of the tampering probability map output by the positioning head can be concatenated to the input features of the classification head for joint judgment. A consistency verification mechanism is also set: when the classification head determines it as false while the positioning head does not detect a significantly tampered area, or when the classification head determines it as true while the positioning head detects a significantly tampered area, it indicates a contradiction between the conclusions of the two output heads. In this case, the overall output confidence is reduced and a manual review process is triggered. The final output includes the true / false judgment result, the judgment confidence, and the heatmap of the tampered area. Figure 3 The results are pushed to the business review interface, providing both qualitative conclusions and credibility levels, as well as visual evidence to support manual review decisions.

[0075] According to the credential tampering detection method proposed in this application, by inputting the image of the credential to be verified into the target detection model, the network synchronously outputs a pixel-by-pixel tampering probability map and authenticity judgment results in a single forward propagation, realizing integrated processing from image input to detection results. Compared with the method that requires the serial running of multiple independent models, it significantly improves detection efficiency and allows pixel-level positioning information and image-level classification results to share the same set of backbone features for mutual verification. By relying on the skip connections and upsampling operations of the positioning head, the probability map is output after restoring to the original resolution, so that the tampered area can be identified with pixel precision. This avoids the limitation that the tampering location cannot be indicated when relying solely on image-level classification results. Then, based on the probability map, a visual heat map and positioning information are generated and pushed to the review interface, allowing business personnel to intuitively locate suspected tampered areas and conduct targeted key reviews in combination with the authenticity judgment results. This improves the interpretability of credential verification and makes up for the shortcomings of traditional methods that only output judgment conclusions without positioning evidence.

[0076] The following is combined Figure 6 The implementation flowchart shown illustrates the training and inference application methods of the target detection model in the credential tampering detection method, as detailed below: (1) Training phase (model building and parameter learning) The core objective of the training phase is to train a dual-task network with both tamper location capability and authenticity discrimination capability under the conditions of scarce real forged samples and strong desensitization constraints, using a three-source collaborative strategy.

[0077] Step 1: Construct three complementary sample sources. Authentic de-identified samples are taken from genuine vouchers of partner banks. After de-identification and differential privacy processing, the data remains within the domain. These samples are scarce and primarily used to anchor the distribution of real data, serving as a distribution benchmark. Rule-synthesized samples, based on voucher format rules and anti-counterfeiting element rules, programmatically apply tampering operations such as photocopying, photographing, splicing, and digital alteration to the authentic voucher, simultaneously generating pixel-level tampering annotations. These samples are driven by domain knowledge, supplementing sparse tampering patterns that authentic samples cannot cover, and also contain built-in supervisory signals. Generative adversarial samples use generative adversarial networks or diffusion models to generate highly realistic and difficult samples, covering unknown and highly realistic tampering methods, improving the model's generalization ability. These three types of samples respectively play complementary roles in anchoring distribution, supplementing sparse patterns, and covering unknown methods, systematically alleviating the problems of sample scarcity and missing annotations.

[0078] Step 2 involves mixing the three types of samples according to a predetermined ratio to form a training set. The ratio is represented by the sampling proportion of the three types of samples, ensuring that the sum of the three is 1. As a preferred implementation, the ratio is dynamically adjusted during the training process: in the early stages of training, the proportion of rule-synthesized samples is increased to enable the localization head to quickly learn basic tampering patterns; in the later stages of training, the proportion of real desensitized samples is increased to correct the model distribution and approximate the real-world scenario. This strategy balances the two objectives of rapidly learning tampering patterns and approximating the real-world distribution.

[0079] Step 3: Train a network with a shared visual backbone and dual output heads using the training set. The input voucher image is extracted and fused through three signal branches. The noise residual path uses an SRM (Spatial Rich Model) high-pass filter kernel to extract noise fingerprints, capturing the discontinuous noise distribution caused by splicing and copy-pasting. The frequency domain feature path extracts spectral features through block-based discrete cosine transform, capturing frequency domain anomalies introduced by re-photographing and recompression. The semantic consistency path rasterizes the field verification results into feature maps, capturing the discrepancies between monetary amounts and field logic caused by digital tampering. The three signals are aligned by intra-path encoding and spliced ​​along the channel dimension, then reweighted by SE channel attention and compressed with 1×1 convolution to output fused features, enabling the network to adaptively rely on different signal paths according to the tampering type. The fused features are input to the shared visual backbone, which uses a SwinTransformer or ResNet structure, progressively downsampling to extract multi-scale features, retaining intermediate features at each level, and finally outputting a deep feature representation with a spatial resolution of one-sixteenth of the input image. The deep feature representations are fed into two output heads respectively. The tampering localization head adopts an encoder-decoder structure. The decoder side upsamples step by step and splices it with the backbone intermediate features through a skip connection. After restoring to the input resolution, it is activated by Sigmoid and outputs a pixel-by-pixel tampering probability map. The true and false classification head performs global average pooling on the deep features, maps them through a fully connected layer, and then normalizes them through Softmax to output the true and false binary classification probabilities.

[0080] A joint loss function is constructed, where the segmentation loss is a combination of binary cross-entropy loss and Dice loss, supervised by pixel-level annotations of rule-synthesized samples, and the classification loss is cross-entropy loss, supervised by sample-level labels. The weighted sum of the two forms the joint loss. The gradients of the tampered localization head and the gradients of the true and false classification heads are jointly backpropagated to the shared backbone and fusion layer, achieving mutual regularization: the localization task forces the backbone to encode spatial evidence of tampering, and the classification task guides the backbone to focus on discriminative regions. The two constrain each other, improving generalization ability under small sample conditions. Training employs LoRA (Low-Rank Adaptation) or AdaLoRA (Adaptive Low-Rank Adaptation) parameter efficient fine-tuning techniques on the pre-trained multimodal document understanding model, updating only the low-rank adaptation parameters. Real desensitized samples participate in training through differential privacy or federated learning, ensuring the data does not leave the domain and meeting financial compliance requirements.

[0081] Step 4: After training is completed, output the parameters of the trained dual-task network model. This model has the ability to output a pixel-by-pixel tampering probability map and authenticity judgment result for the input voucher, which can be deployed and called in the inference stage.

[0082] (2) Reasoning Application Stage During the inference phase, the trained model is deployed in the bank's credit business system to detect tampering and verify the authenticity of voucher images in actual business transactions.

[0083] Step 1: Obtain the image of the credit document to be verified. This document includes financial credit documents such as VAT invoices, bank statements, audit reports, financial statements, and tax payment certificates. Input the document into the system as a scanned copy or a photo taken with a mobile phone.

[0084] Step 2: Input the image of the voucher to be verified into the pre-trained dual-task network. After extracting noise residuals, frequency domain features and semantic consistency features through the three signal input stages, the images are fused and then connected to the tampering location head and the authenticity classification head after being encoded by a shared visual backbone. In a single forward propagation, the pixel-by-pixel tampering probability map and the authenticity judgment result are output synchronously.

[0085] Step 3: Post-process the output pixel-by-pixel tampering probability map: adjust the probability according to a set threshold. Figure 2 Values ​​are converted into a tampering mask. Connectivity analysis is performed on the tampering mask, and noise regions with areas smaller than a preset lower limit are filtered out. The bounding rectangles of the retained connected regions are calculated as the tampering region location information. At the same time, the pixel-by-pixel tampering probability map is rendered into a tampering region heatmap through color mapping.

[0086] Step 4: Push the heat map of the tampered area, the location information of the tampered area, the authenticity judgment result and the judgment confidence level to the business review interface, so that the judgment conclusion, the spatial distribution of tampering and the location of tampering are displayed on the same screen, for credit approval personnel to conduct visual review.

[0087] Step 5, as a preferred implementation, when the classification is false but the positioning head does not detect a significantly tampered area, or when the classification is true but the positioning head detects a significantly tampered area, the confidence level of the judgment is reduced and a manual review is recommended to further reduce the risk of misjudgment.

[0088] In summary, this application constructs three complementary sample types: real de-identified samples, rule-synthesized samples, and generative adversarial samples. The rule-synthesized samples programmatically generate tampering areas based on the certificate format and anti-counterfeiting rules, simultaneously outputting pixel-level annotations. Real samples anchor the distribution, rule-synthesized samples supplement sparse tampering patterns, and adversarial samples cover unknown methods. These three elements work synergistically to systematically alleviate the problems of scarce real forgery samples and missing pixel-level annotations. Furthermore, by anchoring real data distribution with real samples and employing a dynamic allocation strategy, the application emphasizes rule-synthesized samples in the early stages of training to quickly learn tampering patterns, and increases the proportion of real samples in the later stages of training to correct the distribution, effectively mitigating distribution drift when transferring general models to financial certificate scenarios. Based on this, by fusing noise residuals, frequency domain features, and semantic consistency signals, respectively... Complementary responses are employed to address the trace features of different forgery methods, such as splicing and digital tampering, thereby covering multiple tampering techniques and reducing false negatives. Simultaneously, a dual-task architecture with a shared backbone enables tamper location and true / false classification to share feature representations and be mutually regularized. Pixel-level supervision in the location task forces the backbone to encode spatial evidence of tampering, while global supervision in the classification task guides the backbone to focus on discriminative regions. These two aspects mutually constrain each other during joint optimization, further improving the model's accuracy and robustness under small sample conditions. Furthermore, the training process utilizes real samples through differential privacy or federated learning methods under the constraint of data not leaving the domain. Combined with efficient parameter fine-tuning techniques, only a small number of adaptation parameters are updated. This significantly reduces training computational costs while meeting financial regulatory compliance requirements, making the overall solution deployable in actual banking business environments.

[0089] Figure 7 This is a block diagram of a credential tampering detection device according to an embodiment of this application.

[0090] like Figure 7 As shown, the credential tampering detection device 20 includes: an acquisition module 210, a processing module 220, and a generation module 230.

[0091] The acquisition module 210 is used to acquire the image of the voucher to be verified; the processing module 220 is used to input the image of the voucher to be verified into a pre-trained target detection model, and the target detection model outputs a pixel-by-pixel tampering probability map and a authenticity determination result; the generation module 230 is used to generate a tampering area heat map and tampering area location information based on the pixel-by-pixel tampering probability map, and push the tampering area heat map, tampering area location information and authenticity determination result to the business review interface.

[0092] It should be noted that the foregoing explanation of the embodiment of the credential tampering detection method also applies to the credential tampering detection device of this embodiment, and will not be repeated here.

[0093] According to the credential tampering detection device proposed in this application, by inputting the image of the credential to be verified into the target detection model, the network synchronously outputs a pixel-by-pixel tampering probability map and authenticity judgment results in a single forward propagation, realizing integrated processing from image input to detection results. Compared with the method of needing to run multiple independent models serially, it significantly improves detection efficiency and allows pixel-level positioning information and image-level classification results to share the same set of backbone features for mutual verification. By relying on the jump connection and upsampling operation of the positioning head, the probability map is output after restoring to the original resolution, so that the tampered area can be identified with pixel precision. This avoids the limitation that the tampering location cannot be indicated when relying solely on image-level classification results. Then, based on the probability map, a visual heat map and positioning information are generated and pushed to the review interface, allowing business personnel to intuitively locate suspected tampered areas and conduct targeted key reviews in combination with the authenticity judgment results, thereby improving the interpretability of credential verification and making up for the shortcomings of traditional methods that only output judgment conclusions without positioning basis.

[0094] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0095] When the processor 802 executes the program, it implements the credential tampering detection method provided in the above embodiments.

[0096] Furthermore, electronic devices also include: Communication interface 803 is used for communication between memory 801 and processor 802.

[0097] The memory 801 is used to store computer programs that can run on the processor 802.

[0098] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0099] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0100] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0101] The processor 802 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0102] This application also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the above-described credential tampering detection method.

[0103] This application also provides a computer program product, including a computer program or instructions, which, when executed, implement the above-described credential tampering detection method.

[0104] In the description of this specification, the references to the terms "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0106] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0107] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0108] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

Claims

1. A method for detecting credential tampering, characterized in that, Includes the following steps: Obtain the image of the voucher to be verified; The image of the certificate to be verified is input into the pre-trained target detection model, and the target detection model outputs a pixel-by-pixel tampering probability map and a authenticity determination result; Based on the pixel-by-pixel tampering probability map, a heat map of the tampered area and the location information of the tampered area are generated. The heat map of the tampered area, the location information of the tampered area, and the authenticity determination result are pushed to the business review interface.

2. The credential tampering detection method according to claim 1, characterized in that, The training method for the target detection model is as follows: Obtain real desensitized samples, rule-synthesized samples, and generative adversarial samples. The rule-synthesized samples generate tampering areas based on voucher format rules and anti-counterfeiting element rules, and simultaneously generate pixel-level tampering annotations corresponding to the tampering areas. The real desensitized samples, rule-synthesized samples, and generative adversarial samples are mixed according to a set ratio to form a training set. The real desensitized samples are used to anchor the real data distribution, and the rule-synthesized samples are used to supplement the sparse tampering patterns. The target detection model is trained using the training set, and the training results output by the target detection model are obtained. A joint loss value is calculated based on the segmentation loss of the tampered localizer and the classification loss of the true / false classifier in the training results. The parameters of the shared visual backbone, the tampered localizer, and the true / false classifier in the target detection model are updated together based on the joint loss value until a preset training stopping condition is met.

3. The credential tampering detection method according to claim 2, characterized in that, The formula for calculating the joint loss value based on the segmentation loss of the tampered localization head and the classification loss of the true and false classification heads in the training results is as follows: ； in, For balance coefficient, To tamper with the segmentation loss of the positioning head, The classification loss is the difference between true and false classification heads.

4. The credential tampering detection method according to claim 1, characterized in that, The target detection model includes: an input layer, a shared visual backbone layer, a tamper localization head, and a true / false classification head; The input layer extracts noise residual features, frequency domain features, and semantic consistency features of the voucher image to be verified. After being encoded into feature maps with the same spatial resolution and the same number of channels within the channel, the feature maps are fused by channel attention reweighting and cross-channel compression. The shared visual backbone layer performs sampling operations on the fused features step by step to extract multi-scale features, saves intermediate features at each level, and outputs deep feature representations. The tampering localization head uses encoding and decoding operations on the deep feature representation. The decoding side upsamples the data level by level and splices it with the intermediate features of each level of the shared visual backbone through skip connections. After restoring the resolution of the input image, it outputs a pixel-by-pixel tampering probability map. The authenticity classification head performs global pooling on the deep feature representation and then maps it through a fully connected layer to output the authenticity classification result of the credentials.

5. The credential tampering detection method according to claim 4, characterized in that, The extraction of noise residual features, frequency domain features, and semantic consistency features from the image of the voucher to be verified includes: The noise residual feature is generated by encoding the noise response map obtained by convolving the input image with a high-pass filter kernel; The frequency domain features are generated by performing a block-based discrete cosine transform on the input image, rearranging the frequency domain coefficients corresponding to each image block along the channel dimension, and then encoding them. The semantic consistency feature generates the voucher field verification result by rasterizing it into a feature map aligned with the input image and then encoding it.

6. The credential tampering detection method according to claim 1, characterized in that, The step of generating a heatmap of the tampered region and location information of the tampered region based on the pixel-by-pixel tampering probability map includes: The pixel-by-pixel tampering probability map is binarized according to a set threshold to obtain a tampering mask; Perform connected component analysis on the tampered mask, extract at least one connected component, and filter out noise regions with an area smaller than a preset lower limit; The location information of the tampered region is generated by calculating the bounding rectangle based on the connected components. The pixel-by-pixel tampering probability map is rendered as a heatmap of the tampered region.

7. A device for detecting credential tampering, characterized in that, include: The acquisition module is used to acquire images of the vouchers to be verified. The processing module is used to input the image of the voucher to be verified into the pre-trained target detection model, and the target detection model outputs a pixel-by-pixel tampering probability map and a authenticity determination result; The generation module is used to generate a heat map of the tampered area and the location information of the tampered area based on the pixel-by-pixel tampering probability map, and push the heat map of the tampered area, the location information of the tampered area and the authenticity determination result to the business review interface.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the credential tampering detection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they are used to implement the credential tampering detection method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the credential tampering detection method as described in any one of claims 1-6.

Images