A self-supervised document image tampering detection method and device and medium
By generating training samples through self-supervised learning and using a multimodal Transformer model, the problem of poor robustness and generalization in document image tampering detection methods is solved. This achieves efficient detection of tampering without visual traces, saves human resources, and has wide applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2023-04-14
- Publication Date
- 2026-05-12
AI Technical Summary
Existing document image tampering detection methods have poor robustness and generalization, making it difficult to efficiently utilize unlabeled documents for self-supervised training, and they are also unable to detect document image tampering with little or no visual traces.
Training samples are generated using a self-supervised method, and a document image tampering detection model and a frequency domain attack detection model based on multimodal Transformer are constructed. Tampering detection is performed using visual and frequency domain information, and tampering detection is performed by fusing visual and frequency domain features through the multimodal Transformer model.
It achieves self-supervised document image tampering detection without manual annotation, can efficiently detect tampering without visual traces, has strong generalization ability, saves human resources, and can be widely applied.
Smart Images
Figure CN116524294B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision, and in particular to a self-supervised document image tampering detection method, apparatus and medium. Background Technology
[0002] Document images are an important medium for information dissemination and are widely present in people's daily lives. However, the rapid development of digital image processing technology has made document images increasingly vulnerable to tampering by criminals, providing opportunities for fraud, spreading rumors, and other illegal activities, thus posing a threat to social security. Therefore, society urgently needs high-performance document tampering detection systems to determine whether document images have been tampered with and to locate the tampered areas.
[0003] Traditional tampering detection methods often suffer from poor robustness and generalization, and their detection capabilities in complex scenarios are weak. While deep neural network-based methods offer better detection capabilities, they are highly dependent on labeled training data. Manually altering and labeling images is time-consuming, labor-intensive, and extremely costly, hindering the application of conventional deep neural network-based tampering detection methods. Society urgently needs a text tampering detection system capable of efficiently utilizing massive amounts of unlabeled documents for self-supervised training.
[0004] On the other hand, because the font and background of text in document images are usually consistent, and the text area is typically very small, document image tampering is generally difficult to detect, leaving visual traces. How to efficiently detect document images with few or no visual traces has become a significant problem in the field of information security applications. Summary of the Invention
[0005] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a self-supervised document image tampering detection method, device and medium.
[0006] The technical solution adopted in this invention is:
[0007] A self-supervised document image tampering detection method includes the following steps:
[0008] Get document images;
[0009] Preprocess the document images;
[0010] Based on the preprocessed document images, a self-supervised training sample is generated using a selective tampering generation method.
[0011] Construct and train a document image tampering detection model based on multimodal Transformer;
[0012] Construct and train a frequency domain attack detection model;
[0013] Based on the trained document image tampering detection model and frequency domain attack detection model, a document image tampering detection system is constructed to predict input samples.
[0014] Furthermore, the preprocessing of the document image includes:
[0015] Obtain the foreground mask of the document image using a document image binarization algorithm;
[0016] Obtain the text detection bounding box of the document image;
[0017] The document image erasure algorithm is used to erase each text region, and the erasure result is saved as a background image.
[0018] Furthermore, the method of generating self-supervised training samples using selective tampering includes:
[0019] The foreground and background colors of each text element are obtained using the foreground mask and text detection bounding box of the document image.
[0020] The text in the image is clustered based on the size of the detection box and the foreground and background colors of the text as font features.
[0021] For each text instance in each cluster, the first and second order gradient changes of the replacement position image edge are performed based on the pairwise position swaps of each text instance (i.e., each text instance in each cluster). Some text instance pairs with gradient changes less than the threshold are randomly selected for image text region swapping. After post-processing, copy-move and splicing type document image tampering samples are generated.
[0022] A text instance is randomly selected from a cluster. The area where the text box of the instance (i.e., the text instance in the randomly selected cluster) is located in the original image is replaced with the corresponding area in the background image. The size and color of the electronic font are set according to the size of the text box and the foreground color. The electronic font is used to print electronic characters in the replaced area. After post-processing, a Generation type document image tampering sample is generated.
[0023] Furthermore, the document image tampering detection model based on multimodal Transformer is trained using a course learning method based on dynamic compression. In the initial stage of training, random light dynamic image compression augmentation is performed with a relatively high probability, and then gradually transitions to uniform random dynamic image compression augmentation in the later stage of training.
[0024] Furthermore, the document image tampering detection model includes a frequency domain sensing head, a visual sensing head, a multimodal interaction module, and a multi-visual iterative decoder module;
[0025] The frequency domain sensing head consists of a learnable quantization table embedding layer, a frequency domain orthogonal basis embedding layer, a convolutional layer, a normalization layer, an activation function layer, and a position coding layer;
[0026] The visual perception head consists of convolutional layers, pooling layers, normalization layers, and activation function layers;
[0027] The multimodal interaction module consists of a spatial and channel attention layer, a convolutional layer, and a Transformer layer.
[0028] The multi-vision iterative decoder module consists of convolutional layers, normalization layers, and activation function layers.
[0029] Furthermore, the frequency domain attack detection model randomly performs dynamic frequency domain attacks on tampered or untampered samples during the training process, and assigns binary classification labels to input samples based on whether a frequency domain attack has been performed.
[0030] Furthermore, the frequency domain attack detection model includes a frequency domain sensing head, a visual sensing head, a multimodal interaction module, and a classifier module;
[0031] The frequency domain sensing head consists of a learnable quantization table embedding layer, a frequency domain orthogonal basis embedding layer, a convolutional layer, a normalization layer, an activation function layer, and a position coding layer;
[0032] The visual perception head consists of convolutional layers, pooling layers, normalization layers, and activation function layers;
[0033] The multimodal interaction module consists of an attention layer, a convolutional layer, and a Transformer layer;
[0034] The classifier module consists of a global pooling layer, a fully connected layer, and an activation function layer.
[0035] Furthermore, the document image tampering detection system predicts the input sample in the following way:
[0036] The trained frequency domain attack detection model is used to detect frequency domain attacks on the input samples.
[0037] If no frequency domain attack is detected, the input sample is fed into the trained document image tampering detection model for tampering detection, and the prediction result is output.
[0038] If a frequency domain attack is detected, the detection is rejected, and the user is prompted to re-enter the original image as required. The original image is not processed except for compression; alternatively, the input sample is input into another preset pure visual tampering detection model to obtain the prediction result.
[0039] Another technical solution adopted in this invention is:
[0040] A self-supervised document image tampering detection device, comprising:
[0041] At least one processor;
[0042] At least one memory for storing at least one program;
[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.
[0044] Another technical solution adopted in this invention is:
[0045] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the method described above.
[0046] The beneficial effects of this invention are: the document image tampering detection system of this invention is trained using only unlabeled document images, saving a lot of manpower and realizing self-supervised document image tampering detection; in addition, the model of this invention is based on a multimodal Transformer and uses both visual and frequency domain information for tampering detection, which can efficiently detect tampering without visual traces and has strong generalization ability, and can be widely used. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a flowchart of a self-supervised document image tampering detection method according to an embodiment of the present invention;
[0049] Figure 2 This is a structural diagram of the document image tampering detection model based on multimodal Transformer in an embodiment of the present invention;
[0050] Figure 3 This is a structural diagram of the frequency domain sensing head in an embodiment of the present invention;
[0051] Figure 4 This is a structural diagram of the multi-view iterative decoder in an embodiment of the present invention;
[0052] Figure 5 This is a flowchart of the document image tampering detection system predicting output samples in an embodiment of the present invention;
[0053] Figure 6 This is a prediction result diagram of the document image tampering detection system in an embodiment of the present invention. Detailed Implementation
[0054] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0055] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0056] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0057] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0058] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0059] like Figure 1 As shown, this embodiment provides a self-supervised document image tampering detection method, including the following steps:
[0060] S1. Obtain the document image.
[0061] In this embodiment, 50,000 images of contracts, notices, invoices, and receipts were collected.
[0062] S2. Preprocess the document images.
[0063] Specifically, step S2 includes steps S21-S23:
[0064] S21. Obtain the foreground mask of the document image using a document image binarization algorithm.
[0065] Specifically, the document image binarization algorithm SAUVOLA is used to obtain the foreground mask of the document image, where the window size of the SAUVOLA algorithm is selected as 75 and the hyperparameter k is selected as 0.2.
[0066] S22. Obtain the text detection box of the document image.
[0067] Specifically, the text detection bounding boxes (x1, y1, x2, y2) of the document image are obtained using open-source OCR tools PaddleOCRv3 and TesseractOCR or other OCR tools, where x1 represents the x-coordinate of the upper left corner of the text box, y1 represents the y-coordinate of the upper left corner of the text box, x2 represents the x-coordinate of the lower right corner of the text box, and y2 represents the y-coordinate of the lower right corner of the text box.
[0068] S23. Use a document image erasure algorithm to erase each text area and save the erasure result as a background image.
[0069] Specifically, the document image erasure algorithm is used to erase each text region and the result is saved as a background image. The erasure algorithm can be the image restoration algorithm LaMa. The document image foreground mask obtained in step S21 is dilated using a square dilation kernel with a radius of 15 and then used as the mask for the specified erasure region.
[0070] S3. Based on the preprocessed document images, generate self-supervised training samples using a selective tampering generation method.
[0071] Specifically, step S3 includes steps S31-S34:
[0072] S31. Use the foreground mask of the document image and the text detection box to obtain the foreground color and background color of each text.
[0073] Specifically, the foreground and background colors of each text are obtained using a document image foreground mask and text detection boxes. For each text box region, the foreground of the original document image is obtained using the mask obtained in step S21, and the average foreground color is used as the foreground color C of that text. f Invert the mask to obtain the background of the original document image, and use the average background color as the background color C of the text. b .
[0074] S32. Using the size of the detection box of the text and the foreground and background colors of the text as font features, cluster the text in the text image according to these font features.
[0075] As an optional implementation method, clustering is performed in the following way:
[0076] S32.1. Randomly select one text from all incomplete clustering texts, with height h = (x2 - x1), width w = (y2 - y1), and foreground color C. f Background color C b For each other text that has not yet completed clustering, its height is h`, its width is w`, and its foreground color is C. f` The background color is C. b` When |hh`| < 0.1*h and |ww`| < 0.1*w and |C f -C f` |<12 and|C b -C b` When | < 10, the two texts are grouped into one category.
[0077] S32.2 Repeat step S32.1 until all texts have been clustered.
[0078] S33. For each text instance in each cluster, perform first-order and second-order gradient changes on the replacement position image edges after pairwise position swapping of these text instances. Randomly select some text instance pairs with gradient changes less than the threshold, perform image text region swapping, and generate Copy-move and Splicing type document image tampering samples after post-processing.
[0079] Specifically, for each text instance in each cluster, the Sobel and Laplacian operators are used to determine the first and second order gradient changes of the image edges after pairwise position swapping. Text instances with first-order gradient increases less than 1 / 8 of the original first-order gradient and second-order gradient increases less than 1 / 8 of the original second-order gradient are randomly selected. These instances are then swapped to generate copy-move and splicing type document image tampering samples. If the two texts originally belonged to the same image, a copy-move type tampering sample is generated; if the two texts did not originally belong to the same image, a splicing type tampering sample is generated. The tampered region of the sample is automatically labeled as the text box region where the swapping operation occurred.
[0080] S34. Randomly select text instances in the cluster, replace the area where the text box of the selected text instance is located in the original image with the corresponding area in the background image, and set the size and color of the electronic font according to the size of the text box and the foreground color. Use the electronic font to print electronic characters in the replaced area, and generate a Generation type document image tampering sample after post-processing.
[0081] Specifically, text instances are randomly selected from clusters, and the area containing the text box in the original image is replaced with the corresponding area in the background image. The height of the text box, h = (x2 - x1), and the foreground color C are then used as the basis for the replacement. f Set the size and color of the TrueTypeFont (TTF) electronic font, and use this TTF font to print electronic characters in the replaced area to generate a Generation type document image tampering sample. The tampered area of the sample is automatically marked as the text box area where the replacement operation was performed.
[0082] S4. Construct and train a document image tampering detection model based on multimodal Transformer.
[0083] A document image tampering detection model based on multimodal Transformer is constructed and trained using a dynamic compression control course learning method, as detailed below:
[0084] S41, such as Figure 2 As shown, the encoder of the multimodal Transformer document image tampering detection model includes a frequency domain sensing head, a visual sensing head, a multimodal interaction module, and a multi-visual iterative decoder module.
[0085] The decoder is a multi-scale iterative decoder, and its visual perceptron consists of seven sequentially connected convolutional layers, normalization layers, and activation function layers. An input sample with height H and width W is processed by the visual perceptron to extract a feature map F with height H / 4, width W / 4, and 96 channels. f0 And a feature map F with a height of H / 8, a width of W / 8, and 192 channels. v The frequency domain sensing head extracts a feature map F with a height of H / 8, a width of W / 8, and 256 channels from the Y-channel DCT coefficient matrix and quantization table of this input sample. d F v and F d The feature map F is fused together by an scSE module and a 1*1 convolution to form a feature map F with a height of H / 8, a width of W / 8, and 192 channels. f1 Using the last three stages of the SwinTransformer-Small as the backbone network, the input F... f1A feature map F with a height of H / 16, a width of W / 16, and 384 channels was extracted. f2 And a feature map F with height H / 32, width W / 32, and 768 channels. f3 .like Figure 4 As shown, feature map F f0 F f1 F f2 F f3 The input is fed into a multi-view iterative decoder to obtain the final prediction result output.
[0086] S42, such as Figure 3 As shown, the frequency domain sensing head of the document image tampering detection model based on multimodal Transformer consists of orthogonal basis embedding, learnable quantization table embedding, position encoding layer, and several convolutional layers. Each convolutional layer is followed by a normalization layer and an activation function layer. One branch of the frequency domain sensing head extracts the quantization table from the image file and obtains the quantized feature map after learnable quantization embedding and convolutional layers. The other branch extracts the Y-channel DCT resonance coefficient matrix from the image file and obtains the DCT feature map F after orthogonal basis embedding and convolutional layers. p1 F p1 Multiplying by the quantized feature map yields F p2 F p1 With F p2 After concatenating the feature dimensions, the feature map F is obtained by convolution downsampling with a stride and radius of 8 and then passing it through a CoordConv encoding layer. p3 F p3 After passing through three more 3x3 convolutional layers, the final frequency domain feature map F is obtained. d .
[0087] S43. Training is conducted using the dynamic compression control course learning method, as detailed below:
[0088] S43.1, The number of times the input image is dynamically compressed is randomly selected between [1, 3], and the random compression quality is between [B1, 100]. The selection range of B1 is... S represents the iteration number in the current iteration during the training process.
[0089] S43.2 Randomly flip the image horizontally, vertically, or rotate it 90 degrees to augment it.
[0090] S43.3. The model is constrained by cross-entropy loss and optimized using the AdamW optimizer. The learning rate changes monotonically from the initial 3e-4 to 1e-5 according to the cosine curve, and the number of training iterations is 100,000.
[0091] S5. Construct and train a frequency domain attack detection model.
[0092] The frequency domain attack detection model includes a frequency domain sensing head, a visual sensing head, a multimodal interaction module, and a classifier module. Step S5 specifically includes steps S51-S54:
[0093] S51. Construct a frequency domain attack detection model. The encoder part of this model is the same as the encoder part of the model described in S41. The decoder consists of a global pooling layer and a fully connected binary classification layer.
[0094] S52. Randomly perform scaling, rotation, blurring, or noise addition operations on the input image with a probability of 0.5, or a combination thereof, and label them as frequency domain attack type samples. Samples that have not undergone random processing are labeled as non-frequency domain attack samples.
[0095] S53. Use the cross-entropy loss function to constrain model optimization.
[0096] S54. Optimize using the AdamW optimizer, with the learning rate changing monotonically from the initial 3e-4 to 1e-5 according to a cosine curve, and the number of training iterations is 100,000.
[0097] S6. Based on the trained document image tampering detection model and frequency domain attack detection model, construct a document image tampering detection system for predicting input samples.
[0098] Construct a document image tampering detection system and predict input samples, such as... Figure 5 As shown, the specific steps include S61-S63:
[0099] S61. For a sample, first input it into the frequency domain attack detection model that has been trained in step S5 to determine whether a frequency domain attack exists.
[0100] S62. If a frequency domain attack is detected, the detection is rejected and the user is prompted to re-enter the image as required. This means that no image operations are performed on the original image except for compression. Alternatively, the image can be input into another purely visual tampering detection model to obtain the output result.
[0101] S63. If it is determined that there is no frequency domain attack, then input it into the document image tampering detection model based on multimodal Transformer that has been trained in step S4 to obtain the corresponding prediction result, which is returned as the final result.
[0102] like Figure 6 As shown in Figure 6, this is a prediction result diagram of the document image tampering detection system.
[0103] In summary, the method of the present invention has the following advantages and beneficial effects compared with the prior art:
[0104] (1) The present invention uses a self-supervised method to train the tamper detection and frequency domain attack detection model, which does not require manual tampering and labeling of image samples, thus greatly saving human resources.
[0105] (2) The frequency domain sensing head in the document image tampering detection model based on multimodal Transformer proposed in this invention can efficiently extract the frequency domain features of the input sample and accurately locate the tampered area by efficiently capturing the discontinuity of frequency domain block artifacts in the tampered sample.
[0106] (3) The document image tampering detection model based on multimodal Transformer proposed in this invention can efficiently fuse visual domain features and frequency domain features using Transformer to extract better multimodal features to assist in tampering detection tasks.
[0107] (4) The document image tampering detection model based on multimodal Transformer proposed in this invention can effectively locate tampered areas of document images without visual tampering traces.
[0108] (5) The tampering detection system proposed in this invention combines frequency domain attack detection and tampering detection model in an efficient manner, avoiding the impact of frequency domain attacks on the performance of the tampering detection document image tampering detection model, making it more complete and with fewer vulnerabilities.
[0109] This embodiment also provides a self-supervised document image tampering detection device, including:
[0110] At least one processor;
[0111] At least one memory for storing at least one program;
[0112] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.
[0113] This embodiment of the self-supervised document image tampering detection device can execute the self-supervised document image tampering detection method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0114] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.
[0115] This embodiment also provides a storage medium storing instructions or programs that can execute the self-supervised document image tampering detection method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.
[0116] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.
[0117] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.
[0118] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0120] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0121] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0122] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0123] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
[0124] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A self-supervised document image tampering detection method, characterized in that, Includes the following steps: Get document images; Preprocess the document images; Based on the preprocessed document images, a self-supervised training sample is generated using a selective tampering generation method. Construct and train a document image tampering detection model based on multimodal Transformer; Construct and train a frequency domain attack detection model; Based on the trained document image tampering detection model and frequency domain attack detection model, a document image tampering detection system is constructed to predict input samples; The method of generating self-supervised training samples using selective tampering includes: The foreground and background colors of each text element are obtained using the foreground mask and text detection bounding box of the document image. The text in the image is clustered based on the size of the detection box and the foreground and background colors of the text as font features. For each text instance in each cluster, based on the first and second order gradient changes of the image edge of the replacement position after pairwise position swapping, some text instance pairs with gradient changes less than the threshold are randomly selected for image text region swapping. After post-processing, copy-move and splicing type document image tampering samples are generated. Text instances in a cluster are randomly selected, and the area where the text box is located in the original image is replaced with the corresponding area in the background image. The size and color of the electronic font are set according to the size of the text box and the foreground color. Electronic characters are printed in the replaced area using the electronic font. After post-processing, a Generation type document image tampering sample is generated.
2. The self-supervised document image tampering detection method of claim 1, wherein, The preprocessing of the document image includes: Obtain the foreground mask of the document image using a document image binarization algorithm; Obtain the text detection bounding box of the document image; The document image erasure algorithm is used to erase each text region, and the erasure result is saved as a background image.
3. The self-supervised document image tampering detection method according to claim 1, characterized in that, The document image tampering detection model based on multimodal Transformer is trained using a course learning method based on dynamic compression. In the initial stage of training, random light dynamic image compression and augmentation are performed, and then the model gradually transitions to uniform random dynamic image compression and augmentation in the later stages of training.
4. A self-supervised document image tampering detection method according to claim 1 or 3, characterized in that, The document image tampering detection model includes a frequency domain sensing head, a visual sensing head, a multimodal interaction module, and a multi-visual iterative decoder module; The frequency domain sensing head consists of a learnable quantization table embedding layer, a frequency domain orthogonal basis embedding layer, a convolutional layer, a normalization layer, an activation function layer, and a position coding layer; The visual perception head consists of convolutional layers, pooling layers, normalization layers, and activation function layers; The multimodal interaction module consists of a spatial and channel attention layer, a convolutional layer, and a Transformer layer. The multi-vision iterative decoder module consists of convolutional layers, normalization layers, and activation function layers.
5. The self-supervised document image tampering detection method according to claim 1, characterized in that, The frequency domain attack detection model randomly performs dynamic frequency domain attacks on tampered or untampered samples during the training process, and assigns binary classification labels to input samples based on whether a frequency domain attack has been performed.
6. A self-supervised document image tampering detection method according to claim 1 or 5, characterized in that, The frequency domain attack detection model includes a frequency domain sensing head, a visual sensing head, a multimodal interaction module, and a classifier module; The frequency domain sensing head consists of a learnable quantization table embedding layer, a frequency domain orthogonal basis embedding layer, a convolutional layer, a normalization layer, an activation function layer, and a position coding layer; The visual perception head consists of convolutional layers, pooling layers, normalization layers, and activation function layers; The multimodal interaction module consists of an attention layer, a convolutional layer, and a Transformer layer; The classifier module consists of a global pooling layer, a fully connected layer, and an activation function layer.
7. The self-supervised document image tampering detection method according to claim 1, characterized in that, The document image tampering detection system predicts input samples in the following way: The trained frequency domain attack detection model is used to detect frequency domain attacks on the input samples. If no frequency domain attack is detected, the input sample is fed into the trained document image tampering detection model for tampering detection, and the prediction result is output. If a frequency domain attack is detected, the detection is rejected, and the user is prompted to re-enter the original image as required. The original image is not processed except for compression; alternatively, the input sample is input into another preset pure visual tampering detection model to obtain the prediction result.
8. A self-supervised document image tampering detection device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7.
9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-7.