Document image tampering detection and positioning method and device based on feature enhancement

Abstract

The present invention discloses a method and device for document image tampering detection and localization based on feature enhancement. The method comprises the following steps: performing sliding window block preprocessing on a document image; performing secondary tampering and post-processing operations on the document images in a training set after the sliding window block preprocessing, setting the tampered area of ​​the corresponding mask image, and obtaining a data-enhanced image; enhancing the tampering trace features from multiple angles to obtain a feature enhancement map; using the Swin Transformer V2Block to construct an encoder and decoder of the model, obtaining prediction results, calculating the loss, and backpropagating the training model; and inputting the test set into the trained document image tampering detection and localization model to locate the tampered area. The present invention designs a method for constructing a document image tampering detection and localization model that can improve the document image tampering detection and localization capabilities, thereby avoiding the problems of low document image tampering detection accuracy and insufficient localization capabilities in existing forensic technology.

Description

Technical Field

[0001] The present invention relates to the technical field of image digital forensics, and in particular to a method and device for detecting and locating document image tampering based on feature enhancement. Background Art

[0002] With the widespread adoption of data acquisition devices and the booming internet, images have become a crucial medium for acquiring information, playing a vital role. Simultaneously, image processing technology continues to advance and develop, and image editing software continues to grow in functionality and maturity, while also becoming increasingly simple to use. This has lowered the barrier to image tampering, allowing people to easily modify image content, leaving only traces of manipulation that are invisible to the naked eye. This negatively impacts all aspects of life, demonstrating that what you see is not necessarily what you believe. Due to their remote, convenient, and efficient nature, people have become accustomed to using e-government and electronic documents to meet their daily needs. In this process, documents are scanned and uploaded as images. Document images, as a key type of digital image, contain sensitive information such as contracts, academic qualifications, and credentials. Detecting and locating tampering in document images has become a pressing issue.

[0003] Document image tampering detection and localization technology is an important branch of image digital forensics. It is primarily used to detect whether a document image contains forgery or tampering, and if so, locate the corresponding tampered area. Currently, there are two main types of document image tampering detection and localization technologies. One is the traditional feature extraction-based method. The paper "Cruz F, Sidere N, Coustaty M, et al. Local binary patterns for document forgery detection[C] / / 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2017, 1:1223-1228." uses local binary patterns (LBP) to capture forgery traces and uses a support vector machine (SVM) to identify whether each image block is forged. The paper "Sun K, Cao G, Zhao Q, et al. Differential Abnormality-Based Tampering Detection in Digital Document Images[C] / / 2019IEEE / ACIS18th International Conference on Computer and Information Science(ICIS).IEEE, 2019:145-149" computes first-order image differences at row and column levels to capture abnormal differences at image stitching boundaries. It then uses a sliding window-based method to detect and locate forged boundaries by searching for single linear features. Its performance depends on the size of the sliding window. These methods rely on handcrafted features and cannot achieve satisfactory performance in complex forensic scenarios. The other is a method based on deep learning. The paper "Xu W, Luo J, Zhu C, et al. Document images forgery localization using a two-stream network[J]. International Journal of Intelligent Systems, 2022, 37(8): 5272-5289." combines the characteristics of document images and designs a two-stream network using Inception blocks and convolution blocks to capture tampering traces from the spatial information and pixel correlation of document images.

[0004] Most existing document image tampering detection and localization technologies fail to consider the differences between document images and natural images. Compared to natural images, the visual differences between positive and negative document image samples are small because their background textures are simple, mostly solid colors or with regular textures. Furthermore, the images are mostly text, resulting in images with no distinct closed boundaries and regular spacing between objects. Image areas that have been copied, pasted, spliced, or repaired easily blend with the background, leaving only subtle traces of tampering. Therefore, improving detection and localization accuracy and achieving document image tampering detection and localization that is more tailored to document image scenarios is a pressing issue for those skilled in the art. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology, provide a document image tampering detection and positioning model, improve the document image tampering detection and positioning capabilities, and thus avoid the problems of existing forensic technology in document image tampering detection accuracy and insufficient positioning capabilities.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for detecting and locating document image tampering based on feature enhancement, comprising the following steps:

[0008] Perform sliding window block preprocessing on the document image;

[0009] Performing secondary tampering and post-processing operations on the document images in the training set after sliding window block pre-processing, and setting the corresponding tampering area of ​​the mask image to obtain a data-enhanced image; the secondary tampering includes copying and pasting, splicing and repairing; the post-processing operation is used to compress, adjust parameters and process noise on the document image;

[0010] A document image tampering detection and localization model is constructed, and the tampering trace features in the data-enhanced image are enhanced from multiple angles to obtain a feature enhancement map. Specifically, a convolutional layer is used to enhance the tampering trace features in the RGB space; a constrained convolutional layer is used to adaptively learn shallow tampering trace features; a pre-filtering layer is used to obtain the image residual; a channel connection method is used to fuse the convolutional layer, the constrained convolutional layer, and the pre-filtering layer to obtain shallow tampering trace features, and the shallow tampering trace features are convolved and normalized to obtain a feature enhancement map. The document image tampering detection and localization model includes a Swin-Unet improved by using the Transformer V2 Block.

[0011] The model's encoder and decoder are constructed using the Swin Transformer V2 Block, and the feature enhancement graph is used to derive prediction results. A softmax operation is performed on the prediction results, and the loss between the true and predicted results is calculated before backpropagation training of the document image tampering detection and localization model.

[0012] The trained document image tampering detection and localization model is used to detect and locate the tampered area of ​​the document image to be detected.

[0013] As a preferred technical solution, the secondary tampering and post-processing operation further includes the following steps:

[0014] The document images in the training set after sliding window block preprocessing are tampered with twice, including randomly cutting out a certain area from the tampered image and copying it to other areas of the image, moving areas from other images to the tampered image, and deleting a certain area in the image;

[0015] The document image after secondary tampering is subjected to JPEG compression, random adjustment of brightness and contrast, addition of camera sensor noise, and Gaussian noise post-processing operations.

[0016] As a preferred technical solution, the method of enhancing the tampering trace features in the data-enhanced image from multiple angles to obtain a feature enhancement map further includes the following steps:

[0017] The convolution layer is used to enhance the tampering trace features in the RGB space, and several n c ×n c The filter kernel of different sizes performs convolution operation on the data augmented image;

[0018] The constrained convolution layer is used to adaptively learn shallow tampering features. The filter kernel weight of the i-th channel of this layer for:

[0019]

[0020] Among them, (m,n) represents the spatial coordinates of each RGB channel pixel, and the spatial coordinates of the center pixel of the corresponding RGB channel are marked as (0,0);

[0021] The Pre-Filtering layer is used to enhance the tampering trace features of the image residual domain, and several filter kernels of size 3×3 are used to perform convolution operations on the image. The parameters of these filters are trainable, and the following three first-order derivative high-pass filters are used. Initialize the filter kernel parameters:

[0022]

[0023] Use the channel connection method to fuse the convolution layer, constrained convolution layer, and Pre-Filtering layer to obtain shallow tampering trace features;

[0024] A convolution operation is performed on the shallow tampering trace features, and Batch Normalization is used to obtain the normalized feature enhancement map.

[0025] As a preferred technical solution, the encoder uses the Swin Transformer V2 Block to downsample the image and convert it into a low-dimensional feature representation for capturing key features for document image tampering detection and localization; the decoder uses the Swin Transformer V2 Block structure to restore the low-dimensional features to the original image size for generating a mask map for document image tampering detection and localization results;

[0026] As a preferred technical solution, the document image tampering detection and positioning model is trained by backpropagation after calculating the loss of the real result and the predicted result, specifically:

[0027] Calculate the dice loss L d , focal loss L f With the total loss L total :

[0028] L total =αL d +βL f

[0029]

[0030] L f =-(1-p t ) γ log(p t )

[0031]

[0032] Among them, α represents the dice loss L d The weight of β represents the focal loss L f The weight of, N and L represent the number of pixels and the number of categories respectively. Represents the true category of the pixel, represents the predicted probability distribution of the model for the pixel point, ε represents a very small constant, y∈{0,1} represents the true category of the pixel point, y=1 means the pixel point has been tampered with, y=0 means it has not been tampered with, p is the probability that the model predicts the pixel point has been tampered with, p t Measures the difference between the model prediction result and the true value. The larger the value, the closer it is. γ represents the adjustment factor, which smoothly adjusts the speed of weight reduction of simple samples.

[0033] Backpropagation training of document image tampering detection and localization models.

[0034] As a preferred technical solution, the sliding window block preprocessing is specifically to use a sliding window method to crop the document image into N×N blocks with an overlap size of M.

[0035] As a preferred technical solution, the setting of the corresponding tampering area of ​​the mask image is specifically as follows:

[0036] On the corresponding mask image of the document image after the second tampering, the second tampering area is set to white. After the setting, the tampered area of ​​the document image is white and the non-tampered area is black.

[0037] In a second aspect, the present invention provides a document image tampering detection and positioning system based on feature enhancement, which is applied to the document image tampering detection and positioning method based on feature enhancement, comprising a first processing module, a second processing module, a third processing module, a model training module and a test execution module;

[0038] The first processing module performs sliding window block preprocessing on the document image;

[0039] The second processing module performs secondary tampering and post-processing operations on the document images in the training set after the sliding window block pre-processing, and sets the tampering area of ​​the corresponding mask image to obtain a data-enhanced image; the secondary tampering includes copying and pasting, splicing and repairing; the post-processing operation is used to compress, adjust parameters and perform noise processing on the document image;

[0040] The third processing module constructs a document image tampering detection and localization model, enhances the tampering trace features in the data-enhanced image from multiple angles to obtain a feature enhancement map; specifically, uses a convolutional layer to enhance the tampering trace features in the RGB space; uses a constrained convolutional layer to adaptively learn shallow tampering trace features; uses a pre-filtering layer to obtain the image residual; uses a channel connection method to fuse the convolutional layer, the constrained convolutional layer, and the pre-filtering layer to obtain shallow tampering trace features, and convolves and normalizes the shallow tampering trace features to obtain a feature enhancement map; the document image tampering detection and localization model includes a Swin-Unet backbone network improved by using a Transformer V2 Block;

[0041] The model training module uses the Swin Transformer V2 Block to form the model's encoder and decoder, and uses the feature enhancement map to obtain the prediction results; a softmax operation is performed on the prediction results, and the loss between the true result and the predicted result is calculated, and then backpropagation is performed to train the document image tampering detection and localization model;

[0042] The test execution module uses the trained document image tampering detection and positioning model to detect and locate the tampered area of ​​the document image to be detected.

[0043] In a third aspect, the present invention provides an electronic device, comprising:

[0044] at least one processor; and,

[0045] a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores computer program instructions that can be executed by the at least one processor. The computer program instructions are executed by the at least one processor to enable the at least one processor to perform the document image tampering detection and positioning method based on feature enhancement.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the document image tampering detection and positioning method based on feature enhancement.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] (1) Based on the problem of weak tampering traces in document images, the present invention uses a block preprocessing method to retain image tampering traces, and adds a third processing module based on convolution operation, constrained convolution operation, and PF layer to the model to enhance the tampering trace features of document images from multiple angles such as RGB spatial features, residual features, and shallow semantic features. This makes it easier for the model network to extract tampering trace features, thereby improving the detection and positioning accuracy.

[0050] (2) The present invention uses the Swin Transformer V2 Block to construct the encoder and decoder of the document image tampering detection and localization model, improves the Swin-Unet network, ensures the accuracy of tampering detection and localization, and makes the document image tampering detection and localization model more suitable for document image scenarios;

[0051] (3) The document image tampering detection and positioning method provided by the present invention utilizes secondary tampering and post-processing operations to perform data enhancement, and uses focal loss to balance sample weights, effectively solving the problem of imbalance between positive and negative samples in the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 This is a flow chart of a method for detecting and locating document image tampering based on feature enhancement according to an embodiment of the present invention;

[0054] Figure 2 This is a subjective visual effect diagram based on different data in the embodiment of the present invention;

[0055] Figure 3 Schematic diagram of the structure of a document image tampering detection and location system based on feature enhancement according to an embodiment of the present invention;

[0056] Figure 4 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0059] See also Figure 1 The document image tampering detection and location method based on feature enhancement in this embodiment includes the following steps:

[0060] S1. Preprocess the image by sliding window block;

[0061] Optionally, the sliding window block preprocessing is specifically to use a sliding window method to crop the document image into N×N blocks with an overlap size of M;

[0062] Preferably, the block size parameters of the document image cropped into are N=512 and M=128;

[0063] S2. Perform secondary tampering and post-processing operations on the document images in the training set after sliding window block preprocessing, and set the corresponding tampering area of ​​the mask image to obtain the data enhanced image;

[0064] Optionally, the secondary tampering includes copying and pasting, splicing and repairing;

[0065] Optionally, the post-processing operation is used to compress, adjust parameters and perform noise processing on the document image;

[0066] Furthermore, the secondary tampering and post-processing operation further includes the following steps:

[0067] S21. Perform secondary tampering on the document images in the training set after sliding window block preprocessing, including randomly cutting out a region from the tampered image and copying it to other regions of the image, moving regions from other images to the tampered image, and deleting a region from the image;

[0068] S22. Perform JPEG compression on the second tampered document image, randomly adjust brightness and contrast, add camera sensor noise, and Gaussian noise post-processing operations;

[0069] Furthermore, the setting of the corresponding mask image tampering area is specifically,

[0070] S23. On the mask image corresponding to the document image after the second tampering, set the second tampering area to white. After the setting, the tampered area of ​​the document image is white and the non-tampered area is black.

[0071] S3. Build a document image tampering detection and localization model, enhance the tampering trace features in the data-enhanced image from multiple angles, and obtain a feature enhancement map;

[0072] Optionally, the document image tampering detection and localization model includes Swin-Unet improved by using Transformer V2 Block;

[0073] Optionally, the image after data enhancement is a 3-channel RGB image, and step S3 is specifically as follows:

[0074] S31. Use convolutional layers to enhance tampering trace features in RGB space;

[0075] Preferably, f c The size is n c ×n c The filter kernel performs convolution operation on the image after data enhancement; where fc Indicates the number of filter kernels in the convolution layer;

[0076] Furthermore, the number of filter kernels f in the convolution layer of the third processing module is c Set to 3, size n c is 5; the number of filter kernels in the constrained convolution layer is set to 3 and the size is 5; the number of filter kernels in the Pre-Filtering layer is f p Set to 9; the number of channels in the convolution layer after feature fusion is set to 32, and the filter kernel size is set to 3;

[0077] S32. Adaptively learning shallow tampering trace features using constrained convolutional layers;

[0078] Preferably, the constrained convolution layer is used to adaptively learn shallow tampering features, and the filter kernel weight of the i-th channel of this layer is for:

[0079]

[0080] Among them, (m,n) represents the spatial coordinates of each RGB channel pixel, and the spatial coordinates of the center pixel of the corresponding RGB channel are marked as (0,0);

[0081] S33. Use the Pre-Filtering layer to enhance the tampering trace features of the image residual domain;

[0082] Preferably, use f p The image is convolved with a filter kernel of size 3×3. The parameters of these filters are trainable and use the following three first-order derivative high-pass filters: Initialize the filter kernel parameters:

[0083]

[0084]

[0085] S34. Use the channel connection method to fuse the convolution layer, constrained convolution layer, and pre-filtering layer to obtain shallow tampering trace features;

[0086] S35. Convolve and normalize the shallow tampering trace features to obtain a feature enhancement map.

[0087] S4. Use the Swin Transformer V2 Block to construct the network's encoder and decoder to obtain the prediction results. Perform a softmax operation on the prediction results, calculate the loss between the true result and the predicted result, and then perform backpropagation to train the document image tampering detection and localization model.

[0088] Optionally, the encoder uses a Swin Transformer V2 Block to downsample the image and convert it into a low-dimensional feature representation for capturing key features for document image tampering detection and localization; the decoder also uses a Swin Transformer V2Block structure to restore the low-dimensional features to the original image size for generating a document image tampering detection and localization result mask.

[0089] Preferably, the back propagation training of the document image tampering detection and positioning model after calculating the loss of the real result and the predicted result is specifically as follows:

[0090] Calculate the dice loss L d , focal loss L f With the total loss L total :

[0091] L total =αL d +βL f

[0092]

[0093] L f =-(1-p t ) γ log(p t )

[0094]

[0095] Among them, α represents the dice loss L d The weight of β represents the focal loss L f The weight of , N and L represent the number of pixels and the number of categories respectively; Represents the true category of the pixel; Represents the model's predicted probability distribution for pixels; ε is a very small constant to avoid the denominator being 0, y∈{0,1} represents the true category of the pixel, y=1 indicates that the pixel has been tampered with, y=0 indicates that it has not been tampered with, p is the model's predicted probability of the pixel being tampered with, p t It measures the difference between the model prediction result and the true value. The larger the value, the closer it is. γ is the adjustment factor, which smoothly adjusts the speed of weight reduction of simple samples.

[0096] Back-propagation training of document image tampering detection and localization models;

[0097] Furthermore, the parameters are set to α = 0.3, β = 0.7, and γ = 2 to train the document image tampering detection and localization model.

[0098] S5. Use the trained document image tampering detection and localization model to detect and locate the tampered area of ​​the document image to be detected.

[0099] Based on the same concept as the feature-enhanced document image tampering detection and location method in the above-mentioned embodiment, the present invention also provides a feature-enhanced document image tampering detection and location system, which can be used to implement the feature-enhanced document image tampering detection and location method. For ease of explanation, the structural diagram of the embodiment of the feature-enhanced document image tampering detection and location system only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and the device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0100] See also Figure 3 In another embodiment of the present application, a document image tampering detection and location system 10 based on feature enhancement is provided, which includes a first processing module 11, a second processing module 12, a third processing module 13, a model training module 14 and a test execution module 15;

[0101] The first processing module 11 performs sliding window block preprocessing on the document image and divides it into a training set and a test set;

[0102] The second processing module 12 performs secondary tampering and post-processing operations on the document images in the training set after the sliding window block pre-processing, and sets the tampering area of ​​the corresponding mask image to obtain a data enhanced image; the secondary tampering includes copying and pasting, splicing and repairing; the post-processing operation is used to compress, adjust parameters and process noise on the document image;

[0103] The third processing module 13 constructs a document image tampering detection and localization model, enhances the tampering trace features in the data-enhanced image from multiple angles to obtain a feature enhancement map; specifically, uses a convolutional layer to enhance the tampering trace features in the RGB space; uses a constrained convolutional layer to adaptively learn shallow tampering trace features; uses a pre-filtering layer to obtain the image residual; uses a channel connection method to fuse the convolutional layer, the constrained convolutional layer, and the pre-filtering layer to obtain shallow tampering trace features, and convolves and normalizes the shallow tampering trace features to obtain a feature enhancement map; the document image tampering detection and localization model includes a Swin-Unet backbone network improved by using a Transformer V2 Block;

[0104] The model training module 14 uses the Swin Transformer V2 Block to form the encoder and decoder of the model, and uses the feature enhancement map to obtain the prediction result; performs a softmax operation on the prediction result, calculates the loss between the true result and the predicted result, and then performs back propagation to train the document image tampering detection and positioning model;

[0105] The test execution module 15 uses the trained document image tampering detection and positioning model to detect and locate the tampered area of ​​the document image to be detected.

[0106] It should be noted that the feature enhancement-based document image tampering detection and positioning system of the present invention corresponds one-to-one to the feature enhancement-based document image tampering detection and positioning method of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the feature enhancement-based document image tampering detection and positioning method are all applicable to the embodiment of the feature enhancement-based document image tampering detection and positioning method. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0107] In addition, in the implementation of the feature enhancement-based document image tampering detection and positioning system in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the feature enhancement-based document image tampering detection and positioning system is divided into different program modules to complete all or part of the functions described above.

[0108] See also Figure 4 In one embodiment, an electronic device that implements a document image tampering detection and positioning method based on feature enhancement is provided. The electronic device 20 may include a first processor 21, a first memory 22 and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a document image tampering detection and positioning program 23 based on feature enhancement.

[0109] The first memory 22 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as a mobile hard disk of the electronic device 20. In other embodiments, the first memory 22 can also be an external storage device of the electronic device 20, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device 20. Furthermore, the first memory 22 can also include both an internal storage unit of the electronic device 20 and an external storage device. The first memory 22 can not only be used to store application software and various types of data installed in the electronic device 20, such as the code of the feature-enhanced document image tampering detection and location program 23, but can also be used to temporarily store data that has been output or is about to be output.

[0110] In some embodiments, the first processor 21 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The first processor 21 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 22, as well as calling data stored in the first memory 22, to perform various functions of the electronic device 20 and process data.

[0111] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the electronic device 20 , and the electronic device 20 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0112] The document image tampering detection and location program 23 based on feature enhancement stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions. When running in the first processor 21, it can achieve the following:

[0113] Perform sliding window block preprocessing on the document image;

[0114] Performing secondary tampering and post-processing operations on the document images in the training set after sliding window block pre-processing, and setting the corresponding tampering area of ​​the mask image to obtain a data-enhanced image; the secondary tampering includes copying and pasting, splicing and repairing; the post-processing operation is used to compress, adjust parameters and process noise on the document image;

[0115] A document image tampering detection and localization model is constructed, and the tampering trace features in the data-enhanced image are enhanced from multiple angles to obtain a feature enhancement map. Specifically, a convolutional layer is used to enhance the tampering trace features in the RGB space; a constrained convolutional layer is used to adaptively learn shallow tampering trace features; a pre-filtering layer is used to obtain the image residual; a channel connection method is used to fuse the convolutional layer, the constrained convolutional layer, and the pre-filtering layer to obtain shallow tampering trace features, and the shallow tampering trace features are convolved and normalized to obtain a feature enhancement map. The document image tampering detection and localization model includes a Swin-Unet backbone network improved by using the Transformer V2 Block.

[0116] The model's encoder and decoder are constructed using the Swin Transformer V2 Block, and the feature enhancement graph is used to derive prediction results. A softmax operation is performed on the prediction results, and the loss between the true and predicted results is calculated before backpropagation training of the document image tampering detection and localization model.

[0117] The trained document image tampering detection and localization model is used to detect and locate the tampered area of ​​the document image to be detected.

[0118] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0120] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A document image tampering detection and location method based on feature enhancement, characterized in that: The following steps are involved: Perform sliding window block preprocessing on the document image; Performing secondary tampering and post-processing operations on the document images in the training set after sliding window block pre-processing, and setting the corresponding tampering area of ​​the mask image to obtain a data-enhanced image; the secondary tampering includes copying and pasting, splicing and repairing; the post-processing operation is used to compress, adjust parameters and process noise on the document image; Build a document image tampering detection and localization model, enhance the tampering trace features in the data-enhanced image from multiple angles, and obtain a feature enhancement map; Specifically, the convolutional layer is used to enhance the tampering trace features in the RGB space; A constrained convolutional layer is used to adaptively learn shallow tampering trace features; a pre-filtering layer is used to obtain image residuals; a channel connection method is used to fuse the convolutional layer, the constrained convolutional layer, and the pre-filtering layer to obtain shallow tampering trace features, and the shallow tampering trace features are convolved and normalized to obtain a feature enhancement map; the document image tampering detection and localization model includes a Swin-Unet improved by using the Swin Transformer V2 Block; The model's encoder and decoder are constructed using the Swin Transformer V2 Block, and a feature enhancement map is used to derive prediction results. A softmax operation is performed on the prediction results, and after calculating the loss between the true result and the predicted result, backpropagation is performed to train a document image tampering detection and localization model. The encoder uses the Swin Transformer V2 Block to downsample the image into a low-dimensional feature representation for capturing key features for document image tampering detection and localization. The decoder uses the Swin Transformer V2 Block structure to restore the low-dimensional features to the original image size for generating a document image tampering detection and localization result mask. The trained document image tampering detection and localization model is used to detect and locate the tampered area of ​​the document image to be detected.

2. The document image tampering detection and location method based on feature enhancement according to claim 1 is characterized in that: The secondary tampering and post-processing operation further includes the following steps: The document images in the training set after sliding window block preprocessing are tampered with twice, including randomly cutting out a certain area from the tampered image and copying it to other areas of the image, moving areas from other images to the tampered image, and deleting a certain area in the image; The document image after secondary tampering is subjected to JPEG compression, random adjustment of brightness and contrast, addition of camera sensor noise, and Gaussian noise post-processing operations.

3. The document image tampering detection and location method based on feature enhancement according to claim 1 is characterized in that: The method further includes the following steps: enhancing the tampering trace features in the data-enhanced image from multiple angles to obtain a feature enhancement map; The convolution layer is used to enhance the tampering trace features in the RGB space, and several n c ×n c The filter kernel of different sizes performs convolution operation on the data augmented image; The constrained convolution layer is used to adaptively learn the shallow tampering trace features. The filter kernel weight of the i-th channel of the constrained convolution layer for: Among them, (m,n) represents the spatial coordinates of each RGB channel pixel, and the spatial coordinates of the center pixel of the corresponding RGB channel are marked as (0,0); The Pre-Filtering layer is used to enhance the tampering trace features of the image residual domain, and several filter kernels of size 3×3 are used to perform convolution operations on the image. The parameters of these filters are trainable, and the following three first-order derivative high-pass filters are used. Initialize the filter kernel parameters: Use the channel connection method to fuse the convolution layer, constrained convolution layer, and Pre-Filtering layer to obtain shallow tampering trace features; A convolution operation is performed on the shallow tampering trace features, and Batch Normalization is used to obtain the normalized feature enhancement map.

4. The document image tampering detection and location method based on feature enhancement according to claim 1, characterized in that: After calculating the loss between the true result and the predicted result, back propagation training is performed on the document image tampering detection and positioning model, specifically: Calculate the dice loss L d , focal loss L f With the total loss L total : L total =αL d +βL f L f =-(1-p t ) γ log(p t ) Among them, α represents the dice loss L d The weight of β represents the focal loss L f The weight of, N and L represent the number of pixels and the number of categories respectively. Represents the true category of the pixel, represents the predicted probability distribution of the model for the pixel point, ε represents a very small constant, y∈{0,1} represents the true category of the pixel point, y=1 means the pixel point has been tampered with, y=0 means it has not been tampered with, p is the probability that the model predicts the pixel point has been tampered with, p t Measures the difference between the model prediction result and the true value. The larger the value, the closer it is. γ represents the adjustment factor, which smoothly adjusts the speed of weight reduction of simple samples. Backpropagation training of document image tampering detection and localization models.

5. The document image tampering detection and location method based on feature enhancement according to claim 1, characterized in that: The tampering area of ​​the mask image corresponding to the setting is specifically, On the corresponding mask image of the document image after secondary tampering, the secondary tampered area is set to white and the non-tampered area is set to black.

6. Document image tampering detection and location system based on feature enhancement, characterized by: A document image tampering detection and location method based on feature enhancement applied to any one of claims 1-5, comprising a first processing module, a second processing module, a third processing module, a model training module and a test execution module; The first processing module performs sliding window block preprocessing on the document image; The second processing module performs secondary tampering and post-processing operations on the document images in the training set after the sliding window block pre-processing, and sets the tampering area of ​​the corresponding mask image to obtain a data-enhanced image; the secondary tampering includes copying and pasting, splicing and repairing; the post-processing operation is used to compress, adjust parameters and perform noise processing on the document image; The third processing module constructs a document image tampering detection and positioning model, enhances the tampering trace features in the data-enhanced image from multiple angles, and obtains a feature enhancement map; Specifically, the convolutional layer is used to enhance the tampering trace features in the RGB space; A constrained convolutional layer is used to adaptively learn shallow tampering trace features; a pre-filtering layer is used to obtain image residuals; a channel connection method is used to fuse the convolutional layer, the constrained convolutional layer, and the pre-filtering layer to obtain shallow tampering trace features, and the shallow tampering trace features are convolved and normalized to obtain a feature enhancement map; the document image tampering detection and localization model includes a Swin-Unet backbone network improved by using the Swin Transformer V2 Block; The model training module uses the Swin Transformer V2 Block to form the model's encoder and decoder, and uses the feature enhancement map to obtain the prediction results; a softmax operation is performed on the prediction results, and the loss between the true result and the predicted result is calculated, and then backpropagation is performed to train the document image tampering detection and localization model; The test execution module uses the trained document image tampering detection and positioning model to detect and locate the tampered area of ​​the document image to be detected.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the document image tampering detection and positioning method based on feature enhancement as described in any one of claims 1-5.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the document image tampering detection and positioning method based on feature enhancement according to any one of claims 1 to 5 is implemented.

Images