Document image tampering detection model training method, tampering detection method and device

Abstract

The invention provides a training method of a document image tampering detection model and a tampering detection method and device.The training method of the document image tampering detection model comprises the steps that multi-scale visual domain features are extracted from a sample document image, and multi-scale frequency domain compressed sensing features are extracted from frequency domain information; acquiring tampered area edge mask data from the document image; fusing the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features; performing semi-supervised training on the multi-scale sensing network by taking the multi-scale visual domain feature as a sample feature of a first prediction head, taking the multi-modal fusion feature as a sample feature of a second prediction head, taking a real label or a pseudo label as a sample label and taking joint loss as a loss function to obtain a document image tampering detection model; according to the method provided by the invention, document image tampering pixel-level detection under low labeling cost is realized, and the detection precision of a document image tampering detection model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a document image tampering detection model training method, tampering detection method and device. BACKGROUND

[0002] With the rapid development of Internet technology and social economy, the number of digital document images has increased rapidly, and the types are diverse; a large amount of key private information is contained in the document images, such as invoices, contract documents, and proof materials, and the change of individual characters will cause a large semantic difference, and such images are more likely to be tampered with; at the same time, image editing technology is constantly improving, and various image editing tools can easily modify the content in the digital document image, and can specifically tamper with important sensitive information to generate a large number of high-simulation fake images; therefore, it is particularly important to design an effective image tampering detection method to ensure the completeness and authenticity of the content of the digital document image.

[0003] In related technologies, the traditional document image tampering detection technology detects features by hand, focuses on the internal inconsistency of a type of tampering method or a type of document, and this method lacks universality and is difficult to cope with the constantly updated image tampering technology; to solve this problem, the image tampering detection algorithm based on deep learning uses a supervised learning paradigm to learn tampering features from large-scale labeled data and realize tampering detection; this method requires a large amount of data, and it is particularly difficult to collect and label tampered document images due to the involvement of a large amount of sensitive information, which leads to the fact that the detection model cannot learn effective information, and further leads to low text tampering detection precision of the detection model in complex scenes. SUMMARY

[0004] The present application provides a document image tampering detection model training method, tampering detection method and device, which solves the defect that when the prior art samples the image tampering detection model based on deep learning to detect text tampering, the model lacks high-quality labeled samples, which leads to the fact that the detection model cannot learn effective information, and further leads to low text tampering detection precision of the detection model in complex scenes; the method of the present application realizes pixel-level detection of document image tampering under low labeling cost, and improves the detection precision of the document image tampering detection model.

[0005] The present application provides a document image tampering detection model training method, comprising: extracting multi-scale visual domain features from the RGB visual information of the sample document image, extracting multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions, and extracting tampering region edge mask data from the label data of the document image based on morphological operation; wherein the multi-scale visual domain features are used to represent the texture characteristics and spatial relationship characteristics of the text image at different feature scales; perform feature fusion on the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features; The multi-scale visual domain features are used as sample features of a first prediction head of a multi-scale perception network, the multi-modal fusion features are used as sample features of a second prediction head of the multi-scale perception network, a real label or a pseudo label corresponding to the sample document image is used as a sample label, and a joint loss is used as a loss function to perform semi-supervised training on the multi-scale perception network to obtain a document image tampering detection model; the joint loss is determined based on a visual domain supervision loss, a multi-modal supervision loss and a consistency loss, the visual domain supervision loss is determined based on a binary cross-entropy loss and an edge supervision loss, and the edge supervision loss is determined based on the tampering region edge mask data.

[0006] According to the training method of the document image tampering detection model provided by the application, the multi-scale visual domain features are extracted from the RGB visual information of the sample document image, including: The RGB visual information is subjected to stage-wise deep feature extraction based on an EfficientViT network to obtain multi-scale visual domain features corresponding to different stages. The multi-scale linear attention layer and the feedforward network layer; the feedforward network layer includes a deep convolution layer, and the multi-scale linear attention layer includes a small-kernel deep separable convolution kernel.

[0007] According to the training method of the document image tampering detection model provided by the application, the multi-scale frequency domain compressed sensing features are extracted from the frequency domain information corresponding to the document image according to different resolutions, including: The spatial domain image information of the document image is converted into the frequency domain information through a discrete cosine transform algorithm; The frequency domain information is encoded to obtain a binary statistical volume, a convolution operation based on frequency separation is performed on the binary statistical volume to obtain a frequency domain feature, and the target compression quantization table is expanded and copied according to the size of the document image to obtain a processed compression quantization table. The processed compression quantization table and the frequency domain feature are multiplied element by element to obtain a new compression quantization feature, and the compression quantization feature and the frequency domain feature are subjected to frequency component separation operations respectively, and the frequency domain features after frequency separation and the compression quantization features after frequency separation are spliced to obtain frequency domain compressed sensing features. The frequency domain compressed sensing features are subjected to three-resolution multi-scale feature extraction to obtain the multi-scale frequency domain compressed sensing features.

[0008] According to the training method of the document image tampering detection model provided by the present invention, the feature fusion of the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multi-modal fused features includes: Based on the spatial attention mechanism, feature fusion is performed on the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain a spatial attention feature map; based on the channel attention mechanism, feature fusion is performed on the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain a channel attention feature map. The spatial attention feature map and the channel attention feature map are concatenated and dimensionality reduced to obtain the multimodal fusion feature.

[0009] According to the training method of the document image tampering detection model provided by the present invention, before extracting multi-scale visual domain features from the RGB visual information of the sample document image, the method further includes: A new sample document image is obtained by performing a target data augmentation operation on the sample document image; wherein the target data augmentation operation includes at least one of random compression, color dithering, random blurring, and grayscale augmentation.

[0010] The present invention also provides a tamper detection method, comprising: Acquire the image to be detected; The image to be detected is processed based on the document image tampering detection model to obtain two types of tampering detection results; wherein, the document image tampering detection model is trained by the training method of the document image tampering detection model.

[0011] Based on the confidence levels corresponding to the two types of detection results, the target tampering detection result is determined.

[0012] The present invention also provides a training apparatus for a document image tampering detection model, comprising: The feature extraction module is used to extract multi-scale visual domain features from the RGB visual information of the sample document image, extract multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions, and extract tampered region edge mask data from the label data of the document image based on morphological operations; wherein, the multi-scale visual domain features are used to characterize the texture characteristics and spatial relationship characteristics of the text image at different feature scales; The feature fusion module is used to fuse the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multi-modal fused features; The training module is used to use the multi-scale visual domain features as sample features for the first prediction head of the multi-scale perception network, and the multi-modal fusion features as sample features for the second prediction head of the multi-scale perception network. It uses the real or pseudo-labels corresponding to the sample document image as sample labels and the joint loss as the loss function to perform semi-supervised training on the multi-scale perception network to obtain a document image tampering detection model. The joint loss is determined based on visual domain supervision loss, multi-modal supervision loss, and consistency loss; the visual domain supervision loss is determined based on binary crossover loss and edge supervision loss; and the edge supervision loss is determined based on the edge mask data of the tampered region.

[0013] The present invention also provides a tamper detection device, comprising: The image acquisition module is used to acquire the image to be detected; The detection module is used to process the image to be detected based on the document image tampering detection model to obtain two types of tampering detection results; wherein, the document image tampering detection model is trained by the training method of the document image tampering detection model.

[0014] The filtering module is used to filter out the target tampering detection results based on the confidence level of the two types of detection results.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a training method or a tampering detection method for a document image tampering detection model as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method or a tampering detection method for a document image tampering detection model as described above.

[0017] The document image tampering detection model training method, tampering detection method, and apparatus provided by this invention extract multi-scale visual domain features from the RGB visual information of sample document images, extract multi-scale frequency domain compressed sensing features from the corresponding frequency domain information of the document images according to different resolutions, and extract tampering region edge mask data from the label data of the document images based on morphological operations. Then, the multi-scale visual domain features and multi-scale frequency domain compressed sensing features are fused to obtain multi-modal fusion features. Finally, the multi-scale visual domain features are used as sample features for the first prediction head of the multi-scale sensing network, and the multi-modal fusion features are used as sample features for the second prediction head of the multi-scale sensing network. Using the real or pseudo labels corresponding to the sample document images as sample labels, and using joint loss as the loss function, the multi-scale sensing network is semi-supervised to obtain a document image tampering detection model. This achieves pixel-level detection of document image tampering with low annotation costs and improves the detection accuracy of the document image tampering detection model. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the training method for the document image tampering detection model provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the working mechanism of the multi-scale prediction head provided by the present invention.

[0021] Figure 3 This is a flowchart illustrating the visual domain feature extraction process provided by the present invention.

[0022] Figure 4 This is a flowchart illustrating the multi-scale frequency domain compressed sensing feature extraction process provided by the present invention.

[0023] Figure 5 This is the second flowchart illustrating the training method for the document image tampering detection model provided by this invention.

[0024] Figure 6 This is a flowchart illustrating the tamper detection method provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of the training device for the document image tampering detection model provided by the present invention.

[0026] Figure 8This is a schematic diagram of the tamper detection device provided by the present invention.

[0027] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The following is combined Figures 1-8 This invention describes the training method, tamper detection method, and apparatus for the document image tampering detection model.

[0030] Figure 1 This is one of the flowcharts illustrating the training method of the document image tampering detection model provided by the present invention, such as... Figure 1 As shown, the method includes the following steps: Step 110: Extract multi-scale visual domain features from the RGB visual information of the sample document image, extract multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions, and extract the tampered region edge mask data from the label data of the document image based on morphological operations; wherein, the multi-scale visual domain features are used to characterize the texture characteristics and spatial relationship characteristics of the text image at different feature scales.

[0031] In this step, the sample document images can be obtained from a public dataset or directly photographed.

[0032] Specifically, the content of the sample document image can include symbols, Chinese characters, numbers, or letters.

[0033] In this step, the multi-scale visual domain features include visual features of different sizes (width, height, and number of channels).

[0034] In this embodiment, the multi-scale visual domain features include, in addition to capturing features such as consistent color and texture and whether there are abnormal splicing edges in the background, for document images, features such as whether the text font and font size are consistent and whether the text lines are horizontally aligned.

[0035] In this embodiment, a convolutional neural network can be used to perform convolution operations on RGB visual information to obtain the corresponding visual domain features; alternatively, a separable convolution operation can be used to perform staged separation of the visual domain features to obtain multi-scale visual domain features.

[0036] In this step, frequency domain features can be extracted in parallel from the perspective of DCT compression artifacts.

[0037] For example, the spatial information in the sample document image is first converted into frequency domain information, then the image is quantized and compressed for storage to obtain the corresponding digital image. Then, the digital image is subjected to binary statistical processing and frequency separation using DCT coefficients to obtain the corresponding frequency domain features. Then, the frequency domain compressed sensing features are obtained by combining the quantization table auxiliary module with the frequency domain features. Finally, feature extraction operations at different resolutions are performed to obtain the corresponding multi-scale frequency domain compressed sensing features.

[0038] In this embodiment, by introducing an additional supervision mechanism for the edge of the tampered region during the visual domain detection process, and by guiding the model to focus on the edge information of the tampered region, the model's ability to capture visual details can be effectively improved.

[0039] Specifically, this embodiment first uses morphological knowledge to analyze the label image. M I A dilation operation is performed to expand the label edge region. Then, the image is inverted, and a dilation operation is performed again to remove internal connected regions. Finally, edges are extracted by calculating the difference between the dilated and eroded images. A 0-1 tampering region edge mask is generated. M e ,Right now M e The generation is shown in the following formula: ; in, B ( k ) represents a size of (2 k +1)×(2 k +1) cross matrix, the matrix contains only the first k Column and number k The value of the row is 1, and the values ​​of the rest are 0. k It can roughly represent the width of the region edge in the boundary mask.

[0040] Step 120: Perform feature fusion on multi-scale visual domain features and multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features.

[0041] In this step, in order to further explore and fuse the potential relationship between the two modal features, an attention mechanism can be used to further optimize and fuse the multimodal features.

[0042] For example, a spatial attention mechanism can be used to fuse multi-scale visual domain features and multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features; a channel attention mechanism can also be used to fuse multi-scale visual domain features and multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features.

[0043] In this embodiment, a combination of spatial attention mechanism and channel attention mechanism can be used to fuse multi-scale visual domain features and multi-scale frequency domain compressed sensing features respectively, and then the fused features are spliced ​​together to obtain the final multimodal fusion features, which improves the representation ability of image features and thus improves the accuracy of tamper detection.

[0044] Step 130: Using multi-scale visual domain features as the sample features of the first prediction head of the multi-scale perception network, and multi-modal fusion features as the sample features of the second prediction head of the multi-scale perception network, the real or pseudo labels corresponding to the sample document images are used as the sample labels, and the multi-scale perception network is semi-supervised trained with joint loss as the loss function to obtain the document image tampering detection model; wherein, the joint loss is determined based on visual domain supervision loss, multi-modal supervision loss and consistency loss, the visual domain supervision loss is determined based on binary cross loss and edge supervision loss, and the edge supervision loss is determined based on the edge mask data of the tampered region.

[0045] In this step, in order to simplify the network and improve computational efficiency, a multi-scale sensing prediction head (i.e., a multi-scale sensing network) is used to complete pixel-level tampering prediction.

[0046] Figure 2 This is a schematic diagram of the working mechanism of the multi-scale prediction head provided by the present invention. Figure 2 In the illustrated embodiment, multi-scale visual domain features are sequentially subjected to multi-scale feature fusion, MBConv residual convolution, and segmentation mapping. First, multi-scale visual domain features of three resolutions are fused using dilated convolution and 1×1 convolution, with operations to align each channel and unify the resolution. Then, the features of the three resolutions are merged by element-wise addition. The fused multi-scale features are upsampled to the maximum resolution (corresponding to multi-scale upsampling and downsampling operations). The feature maps of the current resolution are then concatenated sequentially (corresponding to same-scale convolution concatenation). Subsequently, the nonlinear expressive power of the model is enhanced by three layers of MBConv convolutional blocks. Finally, the feature maps are mapped into two-class segmentation prediction probability maps.

[0047] In this embodiment, the multi-scale perception network adopts a dual-head prediction structure, in which the first prediction head and the second prediction head are used to process visual domain features and multimodal fusion features, respectively. The first prediction head only takes in visual domain features, while the other prediction head fuses multimodal features from the visual domain and the frequency domain.

[0048] It should be noted that the joint training process using the dual-head prediction structure, as part of the semi-supervised framework, uses the dual prediction head outputs to provide additional supervision signals. Consistency regularization promotes the model to learn effectively even when labeled data is scarce. The dual-head prediction design not only enhances the model's performance in semi-supervised learning but also allows for flexible selection of high-confidence prediction results based on different types of tampered samples, thereby improving overall prediction performance.

[0049] To improve the performance of document image tampering detection under low annotation requirements, this embodiment proposes a semi-supervised training strategy for training a multi-scale perceptual network, including a staged pseudo-label screening strategy and a dual-head prediction consistency regularization strategy; specifically including: Step S1: Supervised training: Train a multi-scale perceptual network on a small number of labeled document tampering images to obtain an initial document image tampering detection model; Step S2, pseudo-label generation: Using the model that has been fully trained in the previous step, generate tamper prediction pseudo-labels for the unlabeled image; Step S3, Retraining: The generated pseudo-labels are subjected to reliability screening. Labeled images and highly reliable unlabeled images and their pseudo-labels are mixed as a new training set to retrain the initial document image tampering detection model. It should be noted that the reliability of pseudo-labels is evaluated by their stability during training; specifically, this can be achieved by calculating the stability of the same unlabeled image before, during, and after training. The reliability of the pseudo-tags is evaluated by the average crossover ratio between the initial pseudo-tags and the final pseudo-tags, denoted as . , The calculation is as follows: ; Among them, the training process was measured. K The number of training parameters at different times. M ij Unlabeled image In the saved number Pseudo-labels generated on each training parameter; M iK Then it refers to its position in the first place. K The pseudo-labels generated on the training parameters are usually the final pseudo-labels.

[0050] In this embodiment, after ranking the unlabeled images by reliability, the high-quality pseudo-labels with the highest rankings are selected for training. This strategy, by gradually introducing high-quality pseudo-labels, ensures that the model can achieve optimal performance at each stage with less interference from erroneous pseudo-labels.

[0051] Step S4, Iterative Update: Iterate through steps S2 and S3, continuously updating the model and pseudo-labels until all unlabeled data is added.

[0052] In this embodiment, the two-head prediction consistency regularization strategy includes: Two identical but parameter-independent prediction heads are used to independently predict visual domain features and multimodal fusion features, respectively. Despite the different features input to the two prediction heads, maintaining prediction consistency across different modalities allows the model to generalize better and improves performance when labeled data is scarce. During training, L2 consistency loss is utilized. Loss c The constraint is calculated using the following formula: in, This represents the first detection head. This is a representation of the second detection head. Features of the multi-scale visual domain; For multimodal fusion features; || ||2 represents the 2-norm operation.

[0053] To ensure that the two prediction heads can be optimized independently and to fully leverage the complementary advantages of the multi-scale perception network in different modal features, this embodiment calculates the loss of each output prediction separately, ensuring that each prediction head can be fully trained. Therefore, the total loss during the training of the multi-scale perception network can be expressed by the following formula: ; ; ; in, For visual domain supervision loss, For multimodal supervision loss, The consistency loss weights are used; for labeled images, This refers to its corresponding label data; for unlabeled images, Refers to its corresponding pseudo-tag; The edge supervision loss is denoted as ; BCE (Binary Cross Entropy) is denoted as .

[0054] During the testing phase, the two-head prediction results will be... and After high-confidence screening, the final pixel-level segmentation prediction is generated. That is, when two predictions disagree on a certain pixel, the prediction with higher confidence will be adopted to ensure that the output results are more stable and reliable.

[0055] The training method for the document image tampering detection model provided in this invention extracts multi-scale visual domain features from the RGB visual information of sample document images, extracts multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document images according to different resolutions, and extracts tampering region edge mask data from the label data of the document images based on morphological operations. Then, it fuses the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features. Finally, it uses the multi-scale visual domain features as the sample features of the first prediction head of the multi-scale sensing network, and the multi-modal fusion features as the sample features of the second prediction head of the multi-scale sensing network. Using the real or pseudo labels corresponding to the sample document images as sample labels, and the joint loss as the loss function, it performs semi-supervised training on the multi-scale sensing network to obtain the document image tampering detection model. This achieves pixel-level detection of document image tampering with low annotation costs and improves the detection accuracy of the document image tampering detection model.

[0056] In some embodiments, extracting multi-scale visual domain features from the RGB visual information of sample document images includes: performing a staged deep feature extraction operation on the RGB visual information based on the EfficientViT network to obtain multi-scale visual domain features corresponding to different stages; wherein, a multi-scale linear attention layer and a feedforward network layer are included; the feedforward network layer includes a deep convolutional layer, and the multi-scale linear attention layer includes a small kernel depth separable convolutional kernel.

[0057] In this embodiment, the present invention uses the EfficientViT network as the visual domain feature extraction network. The EfficientViT network consists of a multi-scale linear attention layer and a feedforward network layer. A deep convolution kernel is set in the middle of the feedforward network for convolution operations.

[0058] In this embodiment, the multi-scale linear attention layer uses a small kernel depth-separable convolution to aggregate nearby Q, K, and V token information to generate multi-scale tokens. Then, ReLU linear attention is applied to the multi-scale tokens to combine the global receptive field with multi-scale learning.

[0059] The linear attention layer in this embodiment is an improvement on the SoftMax attention mechanism in the traditional ViT architecture, which can reduce computational complexity and memory consumption; specifically, the ReLU linear attention mechanism expression is as follows: ; in, Representation matrix The OK; , , , , and It is a learnable linear projection matrix.

[0060] Figure 3 This is a flowchart illustrating the visual domain feature extraction process provided by the present invention. Figure 3 In the illustrated embodiment, the EfficientViT network includes: an input feature stem layer and convolutional modules corresponding to four stages (Stage 1-4) (inverted residual MBConv structure, MBConv+EfficientViT layer); specifically, the EfficientViT module is used in stages 3 and 4; the backbone structure of the EfficientViT network adopts MBConv, and the features are up-dimensionalized through 1×1 convolution, then processed by batch normalization and the Swish activation function, followed by a depthwise separable convolutional layer of DWConv with a kernel size of k×k, and then a squeeze activation module (SE Module) is used to enhance the relationship and importance between channels; the EfficientViT network processes RGB color document images (corresponding to RGB visual information) After being input into the visual domain EfficientViT feature extraction network, the features are extracted through staged deep feature extraction to form multi-scale visual domain features, denoted as . It includes three scales. , and .

[0061] The training method for the document image tampering detection model provided in this embodiment of the invention uses the EfficientViT network to perform staged deep feature extraction on RGB visual information to obtain multi-scale visual domain features corresponding to different stages. When processing high-resolution images and complex multi-scale tampering, EfficientViT shows significant advantages in scalability and adaptability. At the same time, the reduced computational cost also enables the model to be deployed on edge devices, improving the model deployment efficiency.

[0062] In some embodiments, extracting multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions includes: converting the spatial domain image information of the document image into frequency domain information using a discrete cosine transform algorithm; encoding the frequency domain information to obtain a binary statistical volume; performing a frequency-separated convolution operation on the binary statistical volume to obtain frequency domain features; expanding and copying the target compressed quantization table according to the size of the document image to obtain a processed compressed quantization table; multiplying the processed compressed quantization table element-wise with the frequency domain features to obtain new compressed quantization features; performing frequency component separation operations on the compressed quantization features and the frequency domain features respectively; concatenating the frequency domain features after frequency separation and the compressed quantization features after rate separation to obtain frequency domain compressed sensing features; and performing a three-resolution multi-scale feature extraction operation on the frequency domain compressed sensing features to obtain multi-scale frequency domain compressed sensing features.

[0063] This embodiment performs frequency domain feature extraction in parallel from the perspective of DCT compression artifacts. The feature extraction process is as follows: During the acquisition process, the sample document image is first divided into 8×8 pixel blocks, and the spatial domain image information is converted into frequency domain information through Discrete Cosine Transform (DCT). Then, an 8×8 quantization table is used to quantize and compress the image for storage, saving image information with less storage space and completing the acquisition of the digital image. Subsequently, in the image tampering operation, the original digital image is first decompressed, read, and edited. The edited digital image is then quantized and compressed again through an 8×8 DCT grid for storage, finally obtaining the tampered image.

[0064] It should be noted that during image tampering, regardless of the specific tampering operation, the image will undergo double DCT mesh compression for storage. Since block mesh compression itself can cause discontinuities or artifacts at block boundaries, for the tampered area, the quantization table during the initial compression may differ from the original. Figure 1 The alignment of the 8×8 grids in the two compressions will cause differences in the distribution of DCT coefficients, making the inconsistency of block boundary effects more obvious. This inconsistency is likely to reveal traces of image tampering in the frequency domain. The DCT coefficient distribution histogram can prove that there are differences in the statistical distribution of DCT coefficients between the tampered area and the original area.

[0065] To address the above issues, this embodiment employs a binary statistical preprocessing method for DCT coefficients; the specific steps of this method are as follows: (1) Input RGB color document image Convert to YCbCr color encoding format to concentrate information in the Y channel. Then, regarding The channel is divided into 8×8 blocks for DCT transformation, each 8×8 image block DCT coefficients The calculation formula is as follows: ; in, and The standardized coefficient is defined as follows: , ; (2) By integrating it into the original image location, we can obtain... DCT coefficients Then, take The absolute value, and set a threshold. The data is truncated, preserving detailed information while reducing unnecessary redundancy; then, it undergoes one-hot encoding... Processed as binary statistical volume The calculation expression is as follows: Among them, the aforementioned binary statistical volume It can be compared to the expanded form of the DCT coefficient histogram that preserves location information.

[0066] (3) Since each 8×8 DCT block has 64 different positions representing different frequency components, in order to prevent the subsequent convolutional network from aliasing different frequency components, this invention uses frequency-separated convolutional layers, which consist of 8×8 dilated convolutional layers with holes of 8 and 1×1 convolutional layers, to further extract the binary statistical volume of DCT. Features .

[0067] (4) Considering that most current image transmission systems use lossy compression to transmit images, the tampered document image may lose some detailed information during transmission due to compression, which may cover up or change the tampering traces; this embodiment introduces a quantization table auxiliary module to improve the robustness of the algorithm under different image compression levels. The compression quantization table is denoted as .

[0068] Specifically, first, the 8×8 quantization table is repeatedly stretched to the image size, and then... After binary statistical preprocessing and frequency-separated convolution, the extracted frequency domain features Element-by-element multiplication, and finally, summation. Similarly, after frequency separation, the data are spliced ​​together to obtain the final frequency domain compressed sensing features. .

[0069] (5) In order to obtain multi-scale frequency domain tampering clues, considering the size of the frequency domain feature mapping and the advantage of HRNet in maintaining maximum resolution, this embodiment adopts a three-resolution variant of HRNet to enhance the perception capability of frequency domain compression artifact features; after three-resolution multi-scale feature extraction, three-scale frequency domain compression sensing features can be obtained, denoted as ,include , and .

[0070] Figure 4 This is a flowchart illustrating the multi-scale frequency domain compressed sensing feature extraction process provided by the present invention. Figure 4 In the illustrated embodiment, the DCT coefficients corresponding to the original document image are preprocessed using binary statistics to obtain a binary statistical volume. This volume is then convolved using frequency-separated dilated convolution to obtain the corresponding frequency domain features. Simultaneously, an 8×8 quantization table is copied and expanded to the size of the original document image and multiplied with the frequency domain features to obtain compressed quantization features. The compressed quantization features and frequency domain features are then subjected to frequency component separation. The frequency domain features after frequency separation and the compressed quantization features after frequency separation are then concatenated to obtain frequency domain compressed sensing features. Finally, the frequency domain compressed sensing features are subjected to multi-scale feature extraction using an HRNet three-resolution variant to obtain multi-scale frequency domain compressed sensing features (corresponding to multi-scale frequency domain features).

[0071] The training method for the document image tampering detection model provided in this invention converts the spatial domain image information of the document image into frequency domain information using the discrete cosine transform algorithm; encodes the frequency domain information, retaining detailed information while reducing unnecessary redundant information; performs frequency-separation-based convolution operation on the binary statistical volume to obtain frequency domain features; expands and copies the target compressed quantization table according to the size of the document image to obtain a processed compressed quantization table; multiplies the processed compressed quantization table element-wise with the frequency domain features, and performs frequency component separation and feature concatenation operations on the compressed quantization features and frequency domain features respectively to obtain frequency domain compressed sensing features; finally, performs three-resolution multi-scale feature extraction on the frequency domain compressed sensing features to obtain multi-scale frequency domain compressed sensing features, which improves the robustness of the document image tampering detection model under different image compression levels and further improves the model's prediction accuracy.

[0072] In some embodiments, feature fusion of multi-scale visual domain features and multi-scale frequency domain compressed sensing features to obtain multimodal fused features includes: feature fusion of multi-scale visual domain features and multi-scale frequency domain compressed sensing features based on a spatial attention mechanism to obtain a spatial attention feature map; feature fusion of multi-scale visual domain features and multi-scale frequency domain compressed sensing features based on a channel attention mechanism to obtain a channel attention feature map; and concatenation and dimensionality reduction of the spatial attention feature map and the channel attention feature map to obtain multimodal fused features.

[0073] In this embodiment, in order to further explore and fuse the potential relationship between the two modal features, the scSE attention module is used to further optimize and fuse the multimodal features.

[0074] It should be noted that the scSE attention module used in this embodiment combines spatial attention mechanism and channel attention mechanism, which can simultaneously enhance and integrate information in the feature space dimension and channel dimension.

[0075] Specifically, the scSE attention module concatenates visual and frequency domain features of the same scale. In the spatial dimension, a 1×1 convolution kernel is used, and a convolutional layer with one output channel is set to compress the feature information, resulting in a new weight matrix with dimension 1. The weight matrix is ​​then normalized using a sigmoid function, and the normalized matrix is ​​multiplied element-wise with the original multimodal feature map in the spatial dimension, thereby assigning different attention weights to feature points at different spatial locations.

[0076] Meanwhile, in the channel dimension of the feature map, the multimodal feature map is first reduced in dimensionality by a global average pooling layer; then, two 1×1 convolutional layers are used to further reduce and increase the dimensionality of the features, finally obtaining a 1-channel weight vector, and then the vector is normalized by sigmoid; the normalized channel weight vector is multiplied with the original multimodal feature map in the channel dimension to obtain the channel attention feature map.

[0077] In this embodiment, the spatial attention feature map and the channel attention feature map are concatenated, and a 1×1 convolutional layer is applied to compress the dimensionality of the fused features to obtain the final multimodal fused features, denoted as . , which includes , and .

[0078] The training method for the document image tampering detection model provided in this invention fuses multi-scale visual domain features and multi-scale frequency domain compressed sensing features using a spatial attention mechanism to obtain a spatial attention feature map; it then fuses multi-scale visual domain features and multi-scale frequency domain compressed sensing features using a channel attention mechanism to obtain a channel attention feature map; finally, it concatenates and reduces the dimensionality of the spatial and channel attention feature maps to obtain a multimodal fusion feature. This method can weight features in both spatial and channel dimensions, adaptively enhance and suppress important information in multimodal features, optimize the representation effect of the feature map, and thus better extract discriminative tampering trace features, thereby improving the model's detection performance.

[0079] In some embodiments, before extracting multi-scale visual domain features from the RGB visual information of the sample document image, the training method of the document image tampering detection model further includes: obtaining a new sample document image by performing a target data augmentation operation on the sample document image; wherein the target data augmentation operation includes at least one of random compression, color dithering, random blurring, and grayscale augmentation.

[0080] In this embodiment, in order to improve the performance of document image tampering detection under low labeling requirements, during the semi-supervised training stage of the multi-scale perceptual network, random augmentation is applied to the learning process of unlabeled images, so that the same image presents different input forms in different training rounds.

[0081] The strong data augmentation strategy proposed in this embodiment includes four methods: random compression, color dithering, random blurring, and grayscale augmentation.

[0082] Specifically, (1) random compression includes: to address the problem of image compression masking tampering traces, a compression coefficient is randomly selected within the range of 75~100 to perform lossy compression on the image, and the quantization table of the compression coefficient is loaded in the quantization table auxiliary module; (2) color jitter includes: to simulate the color differences of images under different lighting conditions or under different shooting devices, and randomly adjust the image color attributes, such as brightness, contrast, saturation and hue, to achieve data augmentation; (3) random blurring includes: to simulate the situation where the shooting device is not clear due to inaccurate focus, motion blur and other reasons, and apply a Gaussian filter to make the image details blurry, thereby improving the model's ability to recognize images with low image quality or blurry images; (4) grayscale augmentation includes: to convert the red, green and blue color channels of the image into a single grayscale value. Color information is removed, while image brightness information is retained, weakening the model's dependence on color features.

[0083] The training method for the document image tampering detection model provided in this invention obtains new sample document images by performing random compression, color jitter, random blurring, and grayscale augmentation operations on sample document images. The above augmentation methods are randomly introduced with a certain probability, so that the model does not rely on the fixed input-output mapping of unlabeled images, effectively alleviating the overfitting phenomenon of the model to erroneous pseudo-labels and promoting the model to learn more diverse representation information.

[0084] Figure 5 This is the second flowchart illustrating the training method for the document image tampering detection model provided by this invention. Figure 5 In the illustrated embodiment, during the extraction of multi-scale frequency domain features, the input image is first processed by a visual feature extraction network to obtain multi-scale visual domain features. During the extraction of multi-scale frequency domain compressed sensing features, the input image is first converted to RGB→YCbCr format, and then an 8×8 DCT transform is performed on the converted image to obtain the DCT coefficients corresponding to the input image. The frequency domain features are then obtained through the frequency domain feature extraction network based on the DCT coefficients. Finally, the compressed quantization table processed by extended replication is multiplied with the frequency domain features, and frequency component separation, concatenation, and multi-scale feature extraction are performed to obtain the multi-scale frequency domain compressed sensing features. The multi-scale visual domain features are then input to the multi-scale sensing prediction head 1 (corresponding to the first prediction head), and the output visual domain features are obtained. The system performs visual prediction and constructs an edge supervision loss by combining the edge mask data of the tampered region. The multi-scale frequency domain compressed sensing features are input into the multi-scale sensing prediction head 2 (corresponding to the second prediction head), and the multi-modal prediction is output. The consistency loss between the two prediction heads is determined based on the visual prediction and the multi-modal prediction. Then, the joint loss prediction sensing result is determined by the visual domain supervision loss corresponding to the multi-scale sensing prediction head 1, the multi-modal supervision loss corresponding to the scale sensing prediction head 2, and the above consistency loss. The prediction probabilities of the two types of pseudo labels are obtained respectively. Finally, the pseudo label corresponding to the prediction probability with high confidence is determined as the final pseudo label. The above-mentioned edge mask data of the tampered region is obtained by performing the operation of extracting the edge of the tampered region from the label corresponding to the original document image or the pseudo label.

[0085] The tampering detection method provided by the present invention will be described below. The tampering detection method described below can be referred to in correspondence with the training method of the document image tampering detection model described above.

[0086] Figure 6 This is a flowchart illustrating the tamper detection method provided by the present invention, as shown below. Figure 6 As shown, the tampering detection method includes the following steps: Step 610: Obtain the image to be detected.

[0087] In this step, the image to be detected can be obtained from a public dataset or it can be captured directly.

[0088] Specifically, the content of the image to be detected can include symbols, Chinese characters, numbers, or letters.

[0089] Step 620: Process the image to be detected based on the document image tampering detection model to obtain two types of tampering detection results; wherein, the document image tampering detection model is trained using the document image tampering detection model training method.

[0090] In this step, the document image tampering detection model is trained through the following steps: (1) Extract multi-scale visual domain features from the RGB visual information of the sample document image, extract multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions, and extract the tampered region edge mask data from the label data of the document image based on morphological operations; among them, multi-scale visual domain features are used to characterize the texture characteristics and spatial relationship characteristics of the text image at different feature scales. (2) Feature fusion is performed on multi-scale visual domain features and multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features; (3) Using multi-scale visual domain features as the sample features of the first prediction head of the multi-scale perception network, and multi-modal fusion features as the sample features of the second prediction head of the multi-scale perception network, the real or pseudo labels corresponding to the sample document images are used as the sample labels, and the multi-scale perception network is semi-supervised with joint loss as the loss function to obtain the document image tampering detection model; wherein, the joint loss is determined based on visual domain supervision loss, multi-modal supervision loss and consistency loss, the visual domain supervision loss is determined based on binary cross loss and edge supervision loss, and the edge supervision loss is determined based on the edge mask data of the tampered region.

[0091] In this embodiment, the implementation of steps (1)-(3) corresponds one-to-one with the implementation of steps 110-130 in the above embodiment, and will not be repeated in this embodiment.

[0092] Step 630: Determine the target tampering detection result based on the confidence level of the two types of detection results.

[0093] In this step, during the testing phase, the final dual-head prediction results are filtered for high confidence to generate the final pixel-level segmentation prediction; that is, when the two predictions make inconsistent judgments on a certain pixel, the prediction category with higher confidence will be adopted to ensure that the output results are more stable and reliable.

[0094] The document image tampering detection model training method provided in this embodiment of the invention processes the image to be detected by the document image tampering detection model obtained by the document image tampering detection model training method, obtains two types of tampering detection results, and determines the target tampering detection result according to the confidence level of the two types of detection results, thereby improving the accuracy of document image tampering detection in complex scenarios.

[0095] The training apparatus for the document image tampering detection model provided by the present invention will be described below. The training apparatus for the document image tampering detection model described below can be referred to in correspondence with the training method for the document image tampering detection model described above.

[0096] Figure 7 This is a schematic diagram of the structure of the training device for the document image tampering detection model provided by the present invention, as shown below. Figure 7 As shown, the training device for the document image tampering detection model includes: a feature extraction module 710, a feature fusion module 720, and a training module 730.

[0097] The feature extraction module 710 is used to extract multi-scale visual domain features from the RGB visual information of the sample document image, extract multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions, and extract the tampered region edge mask data from the label data of the document image based on morphological operations; wherein, the multi-scale visual domain features are used to characterize the texture characteristics and spatial relationship characteristics of the text image at different feature scales. The feature fusion module 720 is used to fuse multi-scale visual domain features and multi-scale frequency domain compressed sensing features to obtain multi-modal fused features; Training module 730 is used to use multi-scale visual domain features as sample features for the first prediction head of the multi-scale perception network, multi-modal fusion features as sample features for the second prediction head of the multi-scale perception network, real or pseudo labels corresponding to sample document images as sample labels, and joint loss as the loss function to perform semi-supervised training on the multi-scale perception network to obtain a document image tampering detection model; wherein, the joint loss is determined based on visual domain supervision loss, multi-modal supervision loss and consistency loss, the visual domain supervision loss is determined based on binary cross loss and edge supervision loss, and the edge supervision loss is determined based on the edge mask data of the tampered region.

[0098] The training device for the document image tampering detection model provided in this invention extracts multi-scale visual domain features from the RGB visual information of sample document images, extracts multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document images according to different resolutions, and extracts tampering region edge mask data from the label data of the document images based on morphological operations. Then, it fuses the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features. Finally, it uses the multi-scale visual domain features as the sample features of the first prediction head of the multi-scale sensing network, and the multi-modal fusion features as the sample features of the second prediction head of the multi-scale sensing network. Using the real or pseudo labels corresponding to the sample document images as sample labels, and the joint loss as the loss function, it performs semi-supervised training on the multi-scale sensing network to obtain the document image tampering detection model. This achieves pixel-level detection of document image tampering with low annotation costs and improves the detection accuracy of the document image tampering detection model.

[0099] The tamper detection device provided by the present invention is described below. The tamper detection device described below can be referred to in correspondence with the tamper detection method described above.

[0100] Figure 8 This is a schematic diagram of the tamper detection device provided by the present invention, as shown below. Figure 8 As shown, the tamper detection device includes: an image acquisition module 810, a detection module 820, and a screening module 830.

[0101] Image acquisition module 810 is used to acquire the image to be detected; The detection module 820 is used to process the image to be detected based on the document image tampering detection model to obtain two types of tampering detection results; wherein, the document image tampering detection model is trained by the document image tampering detection model training method.

[0102] The filtering module 830 is used to filter out the target tampering detection results based on the confidence level of the two types of detection results.

[0103] The tampering detection device provided in this embodiment of the invention processes the image to be detected by a document image tampering detection model obtained by a training method based on the document image tampering detection model, obtains two types of tampering detection results, and determines the target tampering detection result based on the confidence level of each of the two types of detection results, thereby improving the accuracy of document image tampering detection in complex scenarios.

[0104] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a training method for a document image tampering detection model. This method includes: extracting multi-scale visual domain features from the RGB visual information of the sample document image; extracting multi-scale frequency domain compressed sensing features from the corresponding frequency domain information of the document image at different resolutions; and extracting tampering region edge mask data from the label data of the document image based on morphological operations. The multi-scale visual domain features are used to characterize the texture and spatial relationship characteristics of the text image at different feature scales. Feature fusion is performed on the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features. The multi-modal fusion features are obtained by combining the multi-scale visual domain features as the sample features of the first prediction head of the multi-scale perception network, and the multi-modal fusion features as the sample features of the second prediction head of the multi-scale perception network. The real or pseudo labels corresponding to the sample document images are used as the sample labels. The multi-scale perception network is semi-supervised and trained with joint loss as the loss function to obtain the document image tampering detection model. Among them, the joint loss is determined based on visual domain supervision loss, multi-modal supervision loss and consistency loss, the visual domain supervision loss is determined based on binary cross loss and edge supervision loss, and the edge supervision loss is determined based on the edge mask data of the tampered region.

[0105] Alternatively, a tampering detection method may be implemented, which includes: acquiring an image to be detected; processing the image to be detected based on a document image tampering detection model to obtain two types of tampering detection results; wherein the document image tampering detection model is trained using a document image tampering detection model training method; and determining the target tampering detection result based on the confidence level of each of the two types of detection results.

[0106] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the document image tampering detection model provided by the above methods. This method includes: extracting multi-scale visual domain features from the RGB visual information of a sample document image; extracting multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions; and extracting tampering region edge mask data from the label data of the document image based on morphological operations. The multi-scale visual domain features are used to characterize the texture properties of the text image at different feature scales. Spatial relationship characteristics; feature fusion of multi-scale visual domain features and multi-scale frequency domain compressed sensing features to obtain multi-modal fusion features; using multi-scale visual domain features as sample features of the first prediction head of the multi-scale sensing network, and multi-modal fusion features as sample features of the second prediction head of the multi-scale sensing network, using the real or pseudo labels corresponding to the sample document images as sample labels, and using joint loss as the loss function to perform semi-supervised training on the multi-scale sensing network to obtain a document image tampering detection model; wherein, the joint loss is determined based on visual domain supervision loss, multi-modal supervision loss and consistency loss, the visual domain supervision loss is determined based on binary cross loss and edge supervision loss, and the edge supervision loss is determined based on the edge mask data of the tampered region.

[0108] Alternatively, a tampering detection method may be implemented, which includes: acquiring an image to be detected; processing the image to be detected based on a document image tampering detection model to obtain two types of tampering detection results; wherein the document image tampering detection model is trained using a document image tampering detection model training method; and determining the target tampering detection result based on the confidence level of each of the two types of detection results.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a document image tampering detection model, characterized in that, include: Multi-scale visual domain features are extracted from the RGB visual information of the sample document image. Multi-scale frequency domain compressed sensing features are extracted from the frequency domain information corresponding to the document image according to different resolutions. The tampered region edge mask data is extracted from the label data of the document image based on morphological operations. The multi-scale visual domain features are used to characterize the texture characteristics and spatial relationship characteristics of the text image at different feature scales. The multi-scale visual domain features and the multi-scale frequency domain compressed sensing features are fused to obtain multi-modal fusion features; Using the multi-scale visual domain features as the sample features of the first prediction head of the multi-scale perception network, and the multi-modal fusion features as the sample features of the second prediction head of the multi-scale perception network, the real or pseudo labels corresponding to the sample document images are used as sample labels, and the multi-scale perception network is semi-supervised trained using joint loss as the loss function to obtain a document image tampering detection model; wherein, the joint loss is determined based on visual domain supervision loss, multi-modal supervision loss and consistency loss, the visual domain supervision loss is determined based on binary cross loss and edge supervision loss, and the edge supervision loss is determined based on the edge mask data of the tampered region.

2. The training method for the document image tampering detection model according to claim 1, characterized in that, The extraction of multi-scale visual domain features from the RGB visual information of the sample document image includes: Based on the EfficientViT network, a staged deep feature extraction operation is performed on the RGB visual information to obtain multi-scale visual domain features corresponding to different stages. The multi-scale linear attention layer and the feedforward network layer are included; the feedforward network layer includes a deep convolutional layer, and the multi-scale linear attention layer includes a small kernel depth-separable convolutional kernel.

3. The training method for the document image tampering detection model according to claim 1, characterized in that, The step of extracting multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions includes: The spatial domain image information of the document image is converted into the frequency domain information using the discrete cosine transform algorithm; The frequency domain information is encoded to obtain a binary statistical volume. A frequency-separated convolution operation is performed on the binary statistical volume to obtain frequency domain features. The target compressed quantization table is expanded and copied according to the size of the document image to obtain the processed compressed quantization table. The processed compressed quantization table is multiplied element-wise with the frequency domain feature to obtain a new compressed quantization feature. The compressed quantization feature and the frequency domain feature are then subjected to frequency component separation. The frequency domain feature after frequency separation and the compressed quantization feature after rate separation are then concatenated to obtain the frequency domain compressed sensing feature. The frequency domain compressed sensing features are subjected to a three-resolution multi-scale feature extraction operation to obtain the multi-scale frequency domain compressed sensing features.

4. The training method for the document image tampering detection model according to claim 1, characterized in that, The feature fusion of the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multimodal fused features includes: Based on the spatial attention mechanism, feature fusion is performed on the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain a spatial attention feature map; based on the channel attention mechanism, feature fusion is performed on the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain a channel attention feature map. The spatial attention feature map and the channel attention feature map are concatenated and dimensionality reduced to obtain the multimodal fusion feature.

5. The training method for the document image tampering detection model according to claim 1, characterized in that, Before extracting multi-scale visual domain features from the RGB visual information of the sample document image, the method further includes: A new sample document image is obtained by performing a target data augmentation operation on the sample document image; wherein the target data augmentation operation includes at least one of random compression, color dithering, random blurring, and grayscale augmentation.

6. A tamper detection method, characterized in that, include: Acquire the image to be detected; The image to be detected is processed based on the document image tampering detection model to obtain two types of tampering detection results; wherein the document image tampering detection model is trained by the training method of the document image tampering detection model as described in any one of claims 1-5. Based on the confidence levels corresponding to the two types of detection results, the target tampering detection result is determined.

7. A training device for a document image tampering detection model, characterized in that, include: The feature extraction module is used to extract multi-scale visual domain features from the RGB visual information of the sample document image, extract multi-scale frequency domain compressed sensing features from the frequency domain information corresponding to the document image according to different resolutions, and extract tampered region edge mask data from the label data of the document image based on morphological operations; wherein, the multi-scale visual domain features are used to characterize the texture characteristics and spatial relationship characteristics of the text image at different feature scales; The feature fusion module is used to fuse the multi-scale visual domain features and the multi-scale frequency domain compressed sensing features to obtain multi-modal fused features; The training module is used to use the multi-scale visual domain features as sample features for the first prediction head of the multi-scale perception network, and the multi-modal fusion features as sample features for the second prediction head of the multi-scale perception network. It uses the real or pseudo-labels corresponding to the sample document image as sample labels and the joint loss as the loss function to perform semi-supervised training on the multi-scale perception network to obtain a document image tampering detection model. The joint loss is determined based on visual domain supervision loss, multi-modal supervision loss, and consistency loss; the visual domain supervision loss is determined based on binary crossover loss and edge supervision loss; and the edge supervision loss is determined based on the edge mask data of the tampered region.

8. A tamper detection device, characterized in that, include: The image acquisition module is used to acquire the image to be detected; The detection module is used to process the image to be detected based on the document image tampering detection model to obtain two types of tampering detection results; wherein the document image tampering detection model is trained by the training method of the document image tampering detection model as described in any one of claims 1-5. The filtering module is used to filter out the target tampering detection results based on the confidence level of the two types of detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the training method of the document image tampering detection model as described in any one of claims 1 to 5 or the tampering detection method as described in claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the document image tampering detection model as described in any one of claims 1 to 5 or the tampering detection method as described in claim 6.

Images