Document image tampering detection method and system based on double-branch feature fusion

Abstract

The invention discloses a document image tampering detection method and system based on double-branch feature fusion, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-detected image; constructing a tampering detection and positioning model, wherein the tampering detection and positioning model comprises a detail enhancement module, a local encoder, a global encoder and a mixed attention module; a to-be-detected image is input into a tampering detection and positioning model, one branch extracts high-frequency information of the to-be-detected image through a detail enhancement module and extracts local features of the high-frequency information through a local encoder, and the other branch extracts global features of the to-be-detected image through a global encoder. And finally, fusing the local features and the global features through a mixed attention module, decoding a fusion result through a decoder, and positioning a tampered region. According to the document image tampering model provided by the invention, the characteristics of the image domain and the frequency domain are integrated, and the detection precision and the performance of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a document image tampering detection method and system based on dual-branch feature fusion. Background Art

[0002] With the development of the computer industry and the imaging field, media information containing images is almost everywhere on the Internet and in the real world. Images have become an important carrier and channel for information transmission and information sharing. In recent years, there has been a phenomenon in which criminals have used fake documents and images to spread rumors, fabricate false news, and illegally obtain economic benefits.

[0003] The text tampering methods in most document images can be roughly divided into three types: (1) splicing, copying and pasting areas from one image into other images; (2) copy-move, that is, moving the spatial position of objects in the image; (3) generation, replacing the original area of ​​the image with other content. In order to promote the application of image forensics technology in a wider range of fields, document image tampering detection algorithms have gradually emerged. To prevent the impact of illegal fake images on all parties, it is necessary to research and invent detection algorithms to authenticate the authenticity of document images. Summary of the invention

[0004] The purpose of the present invention is to provide a document image tampering detection method and system based on dual-branch feature fusion to solve the problems in the prior art.

[0005] The present invention specifically provides the following technical solutions: A document image tampering detection method based on dual-branch feature fusion includes the following steps: Acquire the image to be detected; Constructing a tampering detection and positioning model, wherein the tampering detection and positioning model includes a detail enhancement module, a local encoder, a global encoder, and a hybrid attention module; The image to be detected is input into a trained tampering detection and positioning model. One branch extracts the high-frequency information of the image to be detected through a detail enhancement module, and extracts local features in the high-frequency information through a local encoder. The other branch uses a global encoder to extract the global features of the image to be detected, and the local features and the global features are fused through a hybrid attention module. The fusion result is decoded by a decoder, and the tampering area is located according to the decoding result.

[0006] Preferably, in the constructed tampering detection and positioning model, the detail enhancement module extracts high-frequency information in the image to be detected by performing discrete cosine transform coefficient quantization and Bayar convolution operations on the image to be detected, and performs the initial enhancement for the local encoder; the local encoder is improved based on the ConvNeXt network, the global encoder adopts the ViT structure, and the hybrid attention module includes an enhanced self-attention module and an adaptive cross-attention module.

[0007] Preferably, the local encoder is improved based on the ConvNeXt network, including: Based on ConvNeXt-B, the detailed branch network block is designed to extract features from the image. The network block has 4 layers, and the number of channels in each layer is and number of blocks They are: , ; Every 4 layers of the network constitute a stage, with a total of four stages. The dimensions of the final output of each stage are 128, 256, 512, and 1024 respectively.

[0008] Preferably, the global encoder adopts the ViT structure, and the global encoder construction process includes: The input image is represented as , the true label is represented as , where h and w correspond to the height and width of the image respectively; Fill the image's height and width to and , and As a constant, Pass it to the 12-layer windowed ViT encoder to obtain a global encoder, where every 3 layers retain a complete global attention block; specifically expressed as: ; in, represents the ViT encoder, Represents the encoded feature map.

[0009] Preferably, the hybrid attention module The specific expression is: ; ; in, It is the main fusion feature of the two branches. is the final fusion feature, To enhance the self-attention block, is the adaptive cross-attention block, It is the decoding fusion module (Decoder blending module).

[0010] Preferably, before inputting the image to be detected into the trained tampering detection and positioning model, the method further includes: The tampering detection and positioning model is trained through back propagation of the loss function to obtain a trained tampering detection and positioning model.

[0011] The present invention provides a document image tampering detection system based on dual-branch feature fusion, comprising: An acquisition module, used for acquiring an image to be detected; A model building module for a tampering detection and localization model, wherein the tampering detection and localization model includes a detail enhancement module, a local encoder, a global encoder, and a hybrid attention module; The detection module is used to input the image to be detected into a trained tampering detection and positioning model. One branch extracts the high-frequency information of the image to be detected through the detail enhancement module, and extracts the local features in the high-frequency information through the local encoder. The other branch uses the global encoder to extract the global features of the image to be detected, and the local features and the global features are fused through the hybrid attention module. The fusion result is decoded by the decoder, and the tampering area is located according to the decoding result.

[0012] The present invention provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the above-mentioned document image tampering detection method based on dual-branch feature fusion.

[0013] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the document image tampering detection method based on dual-branch feature fusion are implemented.

[0014] Compared with the prior art, the present invention has the following significant advantages: The present invention designs a tampering detection and positioning model, which uses two branches to extract the local and global aspects of the image to be detected respectively. One branch extracts the high-frequency information of the image to be detected through a detail enhancement module, and extracts the local features in the high-frequency information through a local encoder, which focuses on the image detail information. The other branch uses a global encoder to extract the global features of the image to be detected, which is used to capture the overall structural information, thereby achieving a comprehensive analysis of the overall image information and the details of the text area, promoting the interactive fusion of feature information at different levels, and effectively avoiding the interference of irrelevant information while reducing the loss of detail information, significantly enhancing the representation ability of key text content areas, improving the quality of the tampering area positioning map, and facilitating the authenticity identification of document images. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of the overall steps of a document image tampering detection method based on dual-branch feature fusion provided by the present invention; Figure 2 It is a flowchart of the steps of making and obtaining an image to be detected for document tampering provided by the present invention; Figure 3 It is the overall network model framework diagram provided by the present invention; Figure 4 Schematic diagram of the structure of the enhanced self-attention module provided by the present invention; Figure 5 is a schematic diagram of the structure of the adaptive cross attention module provided by the present invention; Figure 6 This is a diagram showing the effect of detecting tampered areas on document images of different tampered types according to an embodiment of the present invention; wherein: Figure 6 (a1), Figure 6 (a2) and Figure 6 (a3) are the first group of original pictures, the results of the present invention and the true labels, Figure 6 (b1), Figure 6 (b2) and Figure 6 (b3) are the second group of original pictures, the results of the present invention and the true labels, Figure 6 (c1), Figure 6 (c2) and Figure 6 (c3) are the third group of original pictures, the results of the present invention and the true labels. DETAILED DESCRIPTION

[0016] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0017] See also Figure 1 , which shows a flowchart of a method for document image tampering detection based on dual-branch feature fusion provided by an embodiment of the present invention, the method comprising the following steps: Step S1: Acquire the image to be detected.

[0018] Figure 2 As shown, the embodiment of the present invention provides the workflow of making and acquiring a data set of the present invention.

[0019] Obtain document image datasets, the source datasets include invoices, magazines, chat records, PDF images and medical prescriptions; use existing text recognition technology to extract text and mark the location, tamper with high-confidence text matching similar samples, randomly augment low-confidence images, and add noise and adjust the style of the tampered samples.

[0020] First, use existing text recognition technology to extract text information from the image, retrieve and locate the text content in the image, and record the mark information of the text location. Set a text detection threshold. If it exceeds the confidence threshold, perform similarity matching based on color difference and recognition content, and select (similar ones for follow-up) tampering operations / methods.

[0021] For images whose recognized text confidence is lower than the preset threshold, random image processing operations are performed to augment the data set (random tampering method). These operations include random jitter, cropping, compression, etc., which are intended to increase the diversity of data through randomness. For images that exceed the threshold, pairs are made, that is, pairs of text tampering samples are made and saved. In order to further increase the difficulty of tampering detection and the robustness of the model, noise is added to these samples, and the image style is adjusted (post-processing). These operations can effectively improve the generalization ability of the model, making it more stable when processing different scenes or changing image features.

[0022] Step S2: construct a tampering detection and localization model, which includes a detail enhancement module, a local encoder, a global encoder and a hybrid attention module.

[0023] The detail enhancement module extracts high-frequency information from the image to be detected by performing discrete cosine transform coefficient quantization (mapping) and Bayar convolution operations on the image to be detected, and then adds convolution operations to expand the detail information to provide the initial enhancement for the local encoder.

[0024] The local encoder is improved based on the ConvNeXt network, the global encoder adopts the ViT (Vision Transformer) structure, and the hybrid attention module consists of an enhanced self-attention module and an adaptive cross-attention module.

[0025] The local encoder is improved based on the ConvNeXt network, including: Based on ConvNeXt-B, the detailed branch network block is designed to extract features from the image. The network block has 4 layers, and the number of channels in each layer is and number of blocks They are: , ; Every 4 layers of the network constitute a stage, with a total of four stages. The dimensions of the final output of each stage are 128, 256, 512, and 1024 respectively.

[0026] The global encoder adopts the ViT (Vision Transformer) structure, including: The input image is represented as , the true label is represented as , where h and w correspond to the height and width of the image respectively; embed the image positionally and fill the image height and width to and , and As a constant, Pass it to the 12-layer windowed ViT encoder to obtain a global encoder, where every 3 layers retain a complete global attention block; specifically expressed as: ; in, represents the ViT encoder, Represents the encoded feature map.

[0027] The hybrid attention module consists of an enhanced self-attention module and an adaptive cross-attention module, including: Hybrid Attention Module The specific expression is: ; ; in, It is the main fusion feature of the two branches. is the final fusion feature, To enhance the self-attention block, is the adaptive cross-attention block, It is the decoding fusion module (Decoder blending module).

[0028] Step S3: Input the image to be detected into the trained tampering detection and positioning model. One branch extracts the high-frequency information of the image to be detected through the detail enhancement module, and extracts the local features of the high-frequency information through the local encoder. The other branch uses the global encoder to extract the global features of the image to be detected. Finally, the local features and global features are fused through the hybrid attention module, the fusion result is decoded by the decoder, and the tampered area is located by the decoding result.

[0029] like Figure 3 As shown in , the local encoder focuses on extracting information at the detail level of the image, such as texture, edge and other high-resolution features; the global encoder focuses on the overall structure and semantic information. The fused features are input to the enhanced self-attention module (such as Figure 4 ), further optimizing the feature representation. In the enhanced self-attention module, in order to balance computational efficiency and model performance, a pair of asymmetric complementary convolutions (with convolution kernels of 1×3 and 3×1 respectively) are specially designed. By decomposing the two-dimensional convolution operation, the directional clues and texture features in the image can be efficiently extracted. This asymmetric convolution design not only retains the receptive field of the convolution layer, but also effectively reduces the computational complexity, while improving the model's ability to capture subtle forgery traces.

[0030] In addition, the enhanced self-attention module focuses on the edge and detail features of the forged area. In order to further improve the model's ability to focus on the target area, an adaptive cross-attention module is added on its basis as another branch of the hybrid attention mechanism (such as Figure 5 This mechanism enables the model to strike a balance between accuracy and efficiency by adaptively allocating weights to local and global information, thereby more accurately capturing and locating forged areas and significantly improving network performance. Figure 6 shown.

[0031] Before the image to be detected is input into the tampering detection and localization model, it also includes: The tampering detection and positioning model is trained by back propagation of the loss function to obtain a trained tampering detection and positioning model. The final fusion feature is embedded in the classification result after upsampling by the decoder and the real label to perform binary cross entropy and Dice constraints.

[0032] The multi-layer features are fused as the decoder input for the prediction mask. The decoder consists of three decoder blocks, each of which contains an upsampling operation and multiple convolutional layers to gradually restore the spatial resolution and extract higher-level semantic information. The feature map size is gradually restored to the original image size through the upsampling operation, and the features are further refined through the convolutional layers in each decoder block. In the final decoder output, the classification result, i.e., the prediction mask, is obtained.

[0033] Specifically, the predicted mask is compared with the true label, and constraints are imposed on the classification results by calculating the binary cross entropy loss and the Dice loss. Among them, the binary cross entropy loss is used to measure the pixel-level difference between the classification result and the true label, while the Dice loss is used to optimize the shape similarity and overlap rate between the target area and the true label. Finally, by minimizing the loss function, the model parameters are continuously updated using back propagation, thereby improving the accuracy of the model for image tampering detection and positioning. In order to improve the detection effect, the present invention designs a decoder to decode the dual-branch fusion features, and adopts a multi-stage loss supervision strategy to optimize the detection results of the tampered area layer by layer, significantly improving the quality of the tampered area positioning map.

[0034] Based on the above method, the present invention provides a document image tampering detection system based on dual-branch feature fusion, including: an acquisition module, a model building module and a detection module.

[0035] Among them, the acquisition module is used to obtain the image to be detected; the model construction module is used for the tampering detection and positioning model, and the tampering detection and positioning model includes a detail enhancement module, a local encoder, a global encoder and a hybrid attention module; the detection module is used to input the image to be detected into the trained tampering detection and positioning model, one branch extracts the high-frequency information of the image to be detected through the detail enhancement module, and extracts the local features in the high-frequency information through the local encoder, and the other branch uses the global encoder to extract the global features of the image to be detected, and fuses the local features and the global features through the hybrid attention module, and decodes the fusion result through the decoder, and locates the tampered area through the decoding result.

[0036] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a document image tampering detection method based on dual-branch feature fusion.

[0037] In accordance with the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0038] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a document image tampering detection method based on dual-branch feature fusion are implemented.

[0039] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0040] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A document image tampering detection method based on dual-branch feature fusion, characterized in that: include: Acquire the image to be detected; Constructing a tampering detection and localization model, wherein the tampering detection and localization model includes a detail enhancement module, a local encoder, a global encoder, and a hybrid attention module; The image to be detected is input into a trained tampering detection and positioning model. One branch extracts the high-frequency information of the image to be detected through a detail enhancement module, and extracts local features in the high-frequency information through a local encoder. The other branch uses a global encoder to extract the global features of the image to be detected, and the local features and the global features are fused through a hybrid attention module. The fusion result is decoded by a decoder, and the tampering area is located according to the decoding result.

2. A document image tampering detection method based on dual-branch feature fusion as claimed in claim 1, characterized in that: In the tampering detection and positioning model, the detail enhancement module extracts high-frequency information in the image to be detected by performing discrete cosine transform coefficient quantization and Bayar convolution operations on the image to be detected, and performs the initial enhancement for the local encoder; the local encoder is improved based on the ConvNeXt network, the global encoder adopts the ViT structure, and the hybrid attention module includes an enhanced self-attention module and an adaptive cross-attention module.

3. A document image tampering detection method based on dual-branch feature fusion as claimed in claim 2, characterized in that: The local encoder is improved based on the ConvNeXt network, including: Based on ConvNeXt-B, the detailed branch network block is designed to extract features from the image. The network block has 4 layers, and the number of channels in each layer is and number of blocks They are: , ; Every 4 layers of the network constitute a stage, with a total of four stages. The dimensions of the final output of each stage are 128, 256, 512 and 1024 respectively.

4. A document image tampering detection method based on dual-branch feature fusion as claimed in claim 2, characterized in that: The global encoder adopts the ViT structure, and the global encoder construction process includes: The input image is represented as , the true label is represented as , where h and w correspond to the height and width of the image respectively; Fill the image's height and width to and , and As a constant, Pass it to the 12-layer windowed ViT encoder to obtain a global encoder, where every 3 layers retain a complete global attention block; specifically expressed as: ； in, represents the ViT encoder, Represents the encoded feature map.

5. The document image tampering detection method based on dual-branch feature fusion as claimed in claim 2, characterized in that: The hybrid attention module The specific expression is: ； ； in, It is the main fusion feature of the two branches. is the final fusion feature, To strengthen the self-attention block, is the adaptive cross attention block, It is the decoding fusion module.

6. The document image tampering detection method based on dual-branch feature fusion as claimed in claim 1, characterized in that: Before inputting the image to be detected into the trained tampering detection and positioning model, the method further includes: The tampering detection and positioning model is trained through back propagation of the loss function to obtain a trained tampering detection and positioning model.

7. A document image tampering detection system based on dual-branch feature fusion, characterized in that: include: An acquisition module, used for acquiring an image to be detected; A model building module for a tampering detection and localization model, wherein the tampering detection and localization model includes a detail enhancement module, a local encoder, a global encoder, and a hybrid attention module; The detection module is used to input the image to be detected into a trained tampering detection and positioning model. One branch extracts the high-frequency information of the image to be detected through the detail enhancement module, and extracts the local features in the high-frequency information through the local encoder. The other branch uses the global encoder to extract the global features of the image to be detected, and the local features and the global features are fused through the hybrid attention module. The fusion result is decoded by the decoder, and the tampering area is located according to the decoding result.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a document image tampering detection method based on dual-branch feature fusion as described in any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a document image tampering detection method based on dual-branch feature fusion as described in any one of claims 1 to 6 are implemented.

Images