A digital document image tampering positioning method based on character texture perception

Abstract

The present application relates to the technical field of image tamper detection, and in particular to a digital document image tamper positioning method based on character texture perception, which adopts a coding and decoding structure, the encoder includes a character texture feature extractor (CTS) and an image texture feature extractor (ITS), the CTS is used to extract the texture tamper features of the character level in the digital document image, the ITS is used to extract the texture tamper features of the image level in the digital document image, the feature tensors extracted from the two flows are combined in a bit-by-bit addition manner, so that the present application can perceive the tamper target in the digital document image, after the encoding of the encoder, the decoder Localization Network uses the deconvolution manner to up-sample the fused features, so as to meet the demand of accurately positioning the tamper target area, and the application of the digital document image tamper detection method can simultaneously detect three types of tamperings, namely splicing, copy moving and removing, and has a certain robustness, and can still maintain good detection effect when coping with different post-processing operations.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multimedia security technology, and particularly relates to a digital document image tampering positioning method based on character texture perception. BACKGROUND

[0002] With the development of computer science and technology, the process of social digitization is being constantly promoted. Traditional paper documents cannot meet the use requirements of people in the digital society due to their disadvantages such as easy damage, easy aging, large storage space overhead, etc. Digital document images gradually become one of the mainstream information acquisition and transmission media in people's daily life, and are gradually applied to fields such as identity authentication, qualification verification, judicial authentication, bill review, seal authentication, public opinion supervision, etc. However, with the emergence of powerful and convenient image editing software, digital document images are vulnerable to malicious tampering by attackers in the process of use. In addition, the development of deep convolutional neural networks and the application of generative adversarial networks enable the tampering traces left after image editing to be further repaired and concealed, resulting in a significant improvement in the tampered digital document images in terms of visual effect and anti-detection performance. The forged digital document images obtained by this image editing technology, if spread to various fields such as social media, government departments, judicial departments and enterprises by unscrupulous people, will destroy the public opinion order of social media, affect the authority and credibility of government and judicial departments, harm the legitimate interests of enterprises, and cause serious adverse effects to the society.

[0003] Digital document image tampering technology is a computer technology that can verify the integrity, authenticity and originality of digital document images, which can be divided into two categories: active forensics and passive forensics. Active forensics mainly uses some technology to embed key information of the image into the digital document image to prevent and verify possible illegal tampering. Although active forensics can achieve certain anti-counterfeiting effect, it needs to use a specific method to pre-embed or generate auxiliary information, and the verifier needs to know the embedding method in advance. Therefore, the use of active forensics method requires a large cost, which seriously restricts its application and development. Passive forensics is a method of detecting authenticity directly based on the characteristics or rules of digital document images. At present, passive forensics of digital documents can be divided into two directions: traditional methods based on feature extraction and methods based on deep learning. Traditional methods based on feature extraction mainly use pre-designed feature operators to detect inconsistencies in the image to capture the tampered area in the image. Literature "S. Gupta, and M. Kumar, Forensic Document Examination System Using Boosting and Bagging Methodologies, Soft Computing, vol. 24, no. 7, pp. 5409-5426, 2020." extracts multiple features of digital document images, such as KPNF, SURF, ORB, FAST, etc., and then fuses multiple machine learning methods for tampering detection. The detection method based on deep learning is to use deep learning network to automatically learn the features of tampering traces, so that the network has better detection performance. Literature "Y. Wu, W. AbdAlmageed, and P. Natarajan, ManTra-Net: Manipulation Tracing Network for Detection and Localization of Image Forgeries with Anomalous Features, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 9535-9544, 2019." first uses different feature extractors to extract features, then classifies the tampering operations, and finally converts the tampering area detection problem into a local anomaly detection problem to realize the positioning of the tampering area.The document "D. Cozzolino, and L. Verdoliva, Noiseprint: A CNN-based camera model fingerprint, IEEE Transactions on Information Forensics and Security, vol. 15, no. 2, pp. 144-159, 2020." learns the inconsistent noise features by using deep neural networks based on the inconsistency of noise patterns between tampered regions and original regions, so as to achieve the purpose of locating the tampered region. The document "P. Zhuang, H. Li, S. Tan, B. Li, and J. Huang, Image Tampering Localization Using a Dense Fully Convolutional Network, IEEE Transactions on Information Forensics and Security, vol. 16, pp. 2986-2999, 2021." adopts the structure of an encoder and a decoder, designs an end-to-end neural network to extract tampering trace features in the image, and then decodes the features to locate the tampered region. The document "W. Xu, J. Luo, C. Zhu, W. Lu, J. Zeng, S. Shi, and C. Lin, Document Images Forgery Localization using a Two-stream Network, International Journal of Intelligent Systems, vol. 37, no. 8, pp. 5272-5289, 2022." designs a two-stream network for digital documents, respectively extracts spatial information features and co-correlation features of the image, so as to achieve better tampering localization performance.

[0004] Currently, the tamper detection method for digital document images does not consider the difference between digital document images and natural images. Compared with natural images, digital document images have obvious differences in main content, color space and texture complexity. Considering the characteristics of digital document images, the main difference between the present application and the existing work is that: (1) the method designs an encoder composed of a character texture extractor and an image texture extractor, which focuses on learning the texture inconsistency features in the digital document image from the image and character level; (2) the method designs a new inter-layer connection mode of convolution layer-dense addition connection. This mode can make the model maintain the size of the feature map unchanged when reusing the features learned by the previous convolution layer, thereby preventing the exponential increase of the model parameters during training. The present application aims at the specific target of digital document images and proposes a digital document image tamper localization method based on character texture perception. SUMMARY

[0005] The present application aims to overcome the limitations of the prior art and proposes a digital document image tamper localization method based on character texture perception, which learns the features of the tampered image at the character texture and image texture level, deeply mines the tamper traces left by the tampering operation in the digital document image, and thereby improves the positioning accuracy of the tampered area. It mainly includes four contents:

[0006] (1) a character texture feature extractor (CTS) based on a convolutional neural network is proposed;

[0007] (2) an image texture feature extractor (ITS) based on dense connection and dense addition connection is proposed;

[0008] (3) a feature decoder based on deconvolution (Localization Network) is proposed;

[0009] (4) a dense addition inter-layer connection mode of convolution layer is proposed.

[0010] Further, the specific contents are as follows:

[0011] (1) a character texture feature extractor (CTS) based on a convolutional neural network is proposed. CTS is a character texture feature extractor based on convolutional layer and max pooling layer. This structure can effectively extract the texture features of digital document images at the character level, and its detailed structure is as follows:

[0012] The first data flow layer includes a 3x3 convolution layer Conv2d; an activation function RELU;

[0013] The second data flow layer includes a max pooling layer MaxPooling with a kernel of 2x1; a 3x3 convolution layer Conv2d; an activation function RELU;

[0014] The 3rd data flow layer includes a max pooling layer MaxPooling with a kernel of 2x1; a 3x3 convolution layer Conv2d; and a RELU activation function;

[0015] The 4th data flow layer includes a 3x3 convolution layer Conv2d; and a RELU activation function;

[0016] The 5th data flow layer includes a max pooling layer MaxPooling with a kernel of 2x2; a 3x3 convolution layer Conv2d; and a RELU activation function;

[0017] The 6th data flow layer includes a max pooling layer MaxPooling with a kernel of 2x2; a 3x3 convolution layer Conv2d; and a RELU activation function;

[0018] The 7th data flow layer includes a max pooling layer MaxPooling with a kernel of 2x2; and a batch normalization layer BN.

[0019] (2) An image texture feature extractor (ITS) based on dense connection and dense addition connection is proposed. ITS is based on the network structure designed by dense connection (Densely Connection) and its variant dense addition connection, and can achieve good results in learning the background texture features of digital document images. Dense connection is a connection method between convolution layers, which can connect all previous layers with the following layers, and realize feature reuse through the stacking of feature maps in the channel dimension. Dense addition connection connects the output of the previous layer with the output of the following layer through bit-by-bit addition of feature maps, so as to realize feature reuse. The detailed structure of the data flow layer in ITS is as follows:

[0020] The 1st data flow layer is an adding connection block (Adding Connection Block) containing 4 layers of adding connection layers (Adding Connection Layer), which are connected in the form of dense addition between layers. Each adding connection layer includes a 3x3 convolution layer Conv2d; a batch normalization layer BN; and a RELU activation function;

[0021] The 2nd data flow layer is a transition layer (Transition Layer) containing a batch normalization layer BN; a RELU activation function; a 1x1 convolution layer Conv2d; and a max pooling layer MaxPooling with a kernel of 2x1;

[0022] The third data flow layer is a convolution block (Adding Connection Block) including two layers of dense addition connection layers (Adding Connection Layer), and the layers are connected in a dense addition manner. Each dense addition connection layer includes a 3*3 convolution layer Conv2d, a batch normalization layer BN, and a activation function RELU.

[0023] The fourth data flow layer is a feature output control layer (Transition Layer) including a batch normalization layer BN, an activation function RELU, a 1*1 convolution layer Conv2d, and a maximum pooling layer MaxPooling with a kernel of 2*1.

[0024] The fifth data flow layer is a convolution block (Adding Connection Block) including two layers of dense addition connection layers (Adding Connection Layer), and the layers are connected in a dense addition manner. Each dense addition connection layer includes a 3*3 convolution layer Conv2d, a batch normalization layer BN, and a activation function RELU.

[0025] The sixth data flow layer is a feature output control layer (Transition Layer) including a batch normalization layer BN, an activation function RELU, a 1*1 convolution layer Conv2d, and an average pooling layer AvgPooling with a kernel of 2*2.

[0026] The seventh data flow layer is a dense block with dilated convolution (Dense Block with Dilated Conv) including two layers of dense connection layers (Densely Connection Layer), and the layers are connected in a channel stacking manner. Each dense connection layer includes a 3*3 dilated convolution layer Dilated Conv2d, a batch normalization layer BN, and a activation function RELU.

[0027] The eighth data flow layer is a feature output control layer (Transition Layer) including a batch normalization layer BN, an activation function RELU, a 1*1 convolution layer Conv2d, and an average pooling layer AvgPooling with a kernel of 2*2.

[0028] The ninth data flow layer is a dense block with dilated convolution (Dense Block with Dilated Conv) including two layers of dense connection layers (Densely Connection Layer), and the layers are connected in a channel stacking manner. Each dense connection layer includes a 3*3 dilated convolution layer Dilated Conv2d, a batch normalization layer BN, and a activation function RELU.

[0029] The 10th data flow layer is a transition layer, which comprises a batch normalization layer BN, a rectified linear unit RELU, a 1x1 convolution layer Conv2d, and a 2x2 average pooling layer AvgPooling;

[0030] (3) A feature decoder based on deconvolution is proposed, which is called Localization Network. The Localization Network is a decoder composed of a deconvolution layer and a densely connected block, which decodes the features extracted by the encoder to obtain the tampered region prediction map of the digital document image. The detailed structure is as follows:

[0031] The 1st data flow layer comprises a deconvolution layer (ConTranspose) with a convolution kernel of 4x4 and a step of 2, a batch normalization layer BN, and a rectified linear unit RELU.

[0032] The 2nd data flow layer is a convolution block (Dense Block) comprising two densely connected layers (Densely Connection Layer), which are connected in a channel stacking manner. Each densely connected layer comprises a 3x3 convolution layer Conv2d, a batch normalization layer BN, and a rectified linear unit RELU.

[0033] The 3rd data flow layer is a transition layer, which comprises a batch normalization layer BN, a rectified linear unit RELU, a 1x1 convolution layer Conv2d, and a 2x2 max pooling layer MaxPooling.

[0034] The 4th data flow layer comprises a deconvolution layer (ConTranspose) with a convolution kernel of 4x4 and a step of 2, a batch normalization layer BN, and a rectified linear unit RELU.

[0035] The 5th data flow layer is a convolution block (Dense Block) comprising two densely connected layers (Densely Connection Layer), which are connected in a channel stacking manner. Each densely connected layer comprises a 3x3 convolution layer Conv2d, a batch normalization layer BN, and a rectified linear unit RELU.

[0036] The 6th data flow layer is a transition layer, which comprises a batch normalization layer BN, a rectified linear unit RELU, a 1x1 convolution layer Conv2d, and a 2x2 max pooling layer MaxPooling.

[0037] The 7th data flow layer comprises a deconvolution layer (ConTranspose) with a convolution kernel of 4x4 and a step of 2; a batch normalization layer BN; and an activation function RELU;

[0038] The 8th data flow layer comprises a 1x1 convolution layer Conv2d; a batch normalization layer BN; and an activation function Log SoftMax.

[0039] Compared with daily natural images, digital document images have their characteristics in subject content, color space and texture. In digital document images, text is the subject content. Meanwhile, most tampering is mainly concentrated in the text area. Moreover, the color of digital document images is less complex than that of natural images, and the contour features of objects are less. On the contrary, the texture features are the main features of digital document images, including character texture features and background image texture features. Existing works often ignore the characteristics of digital document images and do not make targeted design, but use the network structure for natural image tampering, so that good performance cannot be achieved in digital document image tampering positioning.

[0040] CTS is a feature extractor designed for character texture features. By controlling the number of channels of the feature map and using max pooling for down-sampling, the model can effectively extract character texture features while reducing information loss, so as to identify tampering traces in character texture features. ITS is a feature extractor designed for image texture features, which uses traditional dense connection and improved dense addition connection, so that the model will not increase the parameters exponentially when reusing existing feature maps.

[0041] (4) A dense addition inter-layer convolution connection method is proposed. Define f i (H, W, C) as the output feature map of the i-th layer, where H and W are the length and width of the feature map, and C is the number of channels of the feature map; S i+1 (K, F, f i (H, W, C)) is the i+1-th convolution layer, where K is the size of the convolution kernel, and F is the number of filters in the convolution layer. Then the output feature map f i+1 (H, W, C) of the i+1-th layer is:

[0042] f i+1 (H, W, C) = S i+1 (K, F, f i (H, W, C)) + f i (H, W, C) (1)

[0043] It can be seen that the dense addition connection mode does not cause the number of channels of the feature map to increase exponentially with the increase of the number of layers. At the same time, through this mode, the model can also reuse the features learned in the previous stage when learning high-level features. By fusing the texture features extracted by CTS and ITS, the model can better learn the tampering trace features at the character texture level. In the Localization Network, deconvolution is used for upsampling instead of traditional upsampling methods, so that the model can obtain more detailed feature maps after upsampling. In digital document images, the strokes of characters are also relatively fine. Therefore, the tampering prediction probability map obtained after decoding can better fit the relatively fine stroke area, so that a more accurate tampering area positioning map can be obtained in the digital document image.

[0044] Compared with the prior art, the above technical scheme has the following remarkable effects:

[0045] 1. The present application provides a digital document image tampering positioning method based on character texture perception. In view of the characteristics of characters in digital document images, CTS and ITS are designed to extract tampering features in the character texture level and the image texture level of digital document images respectively, so that the model can achieve better performance in the positioning of digital document image tampering areas.

[0046] 2. The present application provides a digital document image tampering positioning scheme of an encoder-decoder structure, which realizes the learning of tampering features for digital document images and decodes the extracted features to obtain an accurate tampering area prediction map. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The network structure diagram of the present application "a digital document image tampering positioning method based on character texture perception";

[0048] Figure 2 The subjective visual effect diagram of the present application "a digital document image tampering positioning method based on character texture perception" on different data; DETAILED DESCRIPTION

[0049] The present application is a digital document image tampering positioning method based on character texture perception. In order to facilitate explanation, the present embodiment takes the data set (SACP) published by "the fifth period of security AI challenger plan: adversarial attack on forged images" as an example to describe the specific implementation of the present application, but those skilled in the art should know that the technical scheme of the present application does not limit the image format. These embodiments are only used to explain the technical principles of the present application, and are not intended to limit the scope of the present application.

[0050] The present embodiment can be implemented according to the following steps, not limited to any programming language, in the present example, the python programming language is taken as an example, and the model is built on the pytorch deep learning platform, and the specific steps are as follows:

[0051] Step 1: Character texture feature extraction. The input data is a digital document image with a size of (H, W, 3), and the output is a feature map with a size of (H / 8, W / 8, 96). The input data will pass through 7 data flow layers. The first layer outputs a feature map with a size of (H, W, 64), the second layer outputs a feature map with a size of (H, W, 64), the third layer outputs a feature map with a size of (H, W, 128), the fourth layer outputs a feature map with a size of (H, W, 128), the fifth layer outputs a feature map with a size of (H / 2, W / 2, 96), the sixth layer outputs a feature map with a size of (H / 4, W / 4, 96), and the seventh layer outputs a character texture feature representation with a size of (H / 8, W / 8, 96).

[0052] Step 2: Image texture feature extraction. The input data is a digital document image with a size of (H, W, 3), and the output is a feature map with a size of (H / 8, W / 8, 96). The input data passes through 10 data flow layers. The first layer outputs a feature map with a size of (H, W, 11), the second layer outputs a feature map with a size of (H, W, 16), the third layer outputs a feature map with a size of (H, W, 16), the fourth layer outputs a feature map with a size of (H, W, 32), the fifth layer outputs a feature map with a size of (H, W, 32), the sixth layer outputs a feature map with a size of (H / 2, W / 2, 64), the seventh layer outputs a feature map with a size of (H / 2, W / 2, 192), the eighth layer outputs a feature map with a size of (H / 4, W / 4, 96), the ninth layer outputs a feature map with a size of (H / 4, W / 4, 288), and the tenth layer outputs an image texture feature representation with a size of (H / 8, W / 8, 96).

[0053] Step 3: Character texture feature and image texture feature fusion. The character texture feature and the image texture feature are fused in a bit-by-bit addition manner to obtain a fusion feature representation with a size of (H / 8, W / 8, 96).

[0054] Step 4: Decoder construction. The input data is a fusion feature with a size of (H / 8, W / 8, 96), and the output data is a tampering prediction map with a size of (H, W, 1). The input data will pass through 8 data flow layers. The first layer outputs a feature map with a size of (3H / 8, 3W / 8, 64), the second layer outputs a feature map with a size of (3H / 8, 3W / 8, 128), the third layer outputs a feature map with a size of (3H / 8, 3W / 8, 64), the fourth layer outputs a feature map with a size of (H / 2, W / 2, 48), the fifth layer outputs a feature map with a size of (H / 2, W / 2, 112), the sixth layer outputs a feature map with a size of (H / 4, W / 4, 48), the seventh layer outputs a feature map with a size of (H, W, 32), and the eighth layer outputs a tampering area prediction map with a size of (H, W, 1).

[0055] Step 5: Model training. The model is built according to the design shown in steps 1, 2, 3, 4 using the pytorch framework, and end-to-end training is performed. The initial learning rate is set to 0.0001, the learning rate decay coefficient is 0.01, the training is performed for 400 rounds, the batch size is 8, the optimizer is Adam, and the loss function is cross-entropy loss function, whose formula is as follows: Figure 1

[0056]

[0057] Wherein, L is the cross-entropy loss value, N represents the total number of pixel points, C represents the category of pixel points, y represents the label value of pixel points, and p represents the predicted value of pixel points.

[0058] The method of the present application can perform binary classification on each pixel point in the image to be tested, i.e. tampered points and non-tampered points, so as to realize the positioning of the tampered region. As shown in the figure, in the tampered region prediction process, the tampered region will be marked as white, and the non-tampered region will be marked as black. The method of the present application designs CTS and ITS for digital document images to learn the tampering traces therein, which can effectively learn the tampering trace features, so as to be more effective in positioning the tampering of digital document images. Figure 2

[0059] Those skilled in the art can understand that the protection scope of the present application is not limited to the specific embodiments described. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application. It should be noted that the technical solutions after the changes or replacements will fall within the protection scope of the present application.​​

Claims

1. A digital document image tampering location method based on character texture perception, characterized in that: The method comprises: This method adopts an encoder-decoder structure, in which the encoder consists of a character texture extractor (CTS) and an image texture extractor (ITS), and the decoder (Localization Network) consists of a deconvolution network and a densely connected convolution layer. All modules are designed based on convolutional neural networks; in CTS, it contains the 1st to 7th data stream layers; in ITS, it contains the 1st to 10th data stream layers; in Localization Network, it contains the 1st to 8th data layers; in each data stream, the data is connected sequentially in series between each data layer; the image to be detected is input into CTS and ITS respectively, and the character texture features and image texture features are obtained respectively. The two feature maps will be fused in a bitwise addition manner. The fused image will be input into the decoder for decoding to obtain a tampering probability map. Finally, the tampering probability map is converted into a mask map using the Logic Softmax function, thereby obtaining the final digital document image tampering area prediction map; The character texture extractor has a total of 7 data layers, consisting of a two-dimensional convolution layer, a RELU activation function layer, a maximum pooling layer, and a batch normalization layer. Among them, the kernel of the two-dimensional convolution layer corresponds to 3*3, the kernel of the maximum pooling layer in the second and third layers corresponds to 2*1, and the kernel of the maximum pooling layer in the fifth, sixth, and seventh layers corresponds to 2*1. The first and fourth layers do not contain a maximum pooling layer. The character texture feature extractor learns the character texture features in digital document images through multi-layer convolution. At the same time, by embedding a maximum pooling layer in the multi-layer convolution layer to downsample the feature map, remove redundant information in the feature map, and complete the extraction of character texture features. The image texture feature extractor has a total of 10 data layers, which are composed of dense additive convolution blocks, feature output control layers, and dense connected dilated convolution blocks; the 1st, 3rd, and 5th layers are dense additive connection blocks, each of which is composed of a data layer composed of a two-dimensional convolution, a batch normalization layer, and a RELU activation function layer, and the data layers are connected in a dense additive manner; the 7th and 9th layers are densely connected dilated convolution blocks, each of which is composed of a data layer block composed of a two-dimensional dilated convolution, a batch normalization layer, and a RELU activation function layer, and the data layers are connected in a densely connected manner. The 2nd, 4th, 6th, 8th and 10th layers are feature output control layers, which are composed of two-dimensional convolutional layers and pooling layers, and are used to control the output size of each convolution block. The pooling layers in the 2nd and 4th layers are maximum pooling layers, and the pooling layers in the 6th, 8th and 10th layers are average pooling layers. Through dense addition connections, the learned feature maps can be used in subsequent feature learning, so that the convolution blocks can learn multi-scale information. Downsampling is performed through the maximum pooling layer and the average pooling layer to remove redundant information in the feature map and complete the extraction of image texture features.

2. The digital document image tampering location method based on character texture perception according to claim 1 is characterized in that: The LocalizationNetwork specifically includes: The Localization Network has a total of 8 data layers, including a two-dimensional deconvolution layer, a densely connected convolution block, a feature output control layer, and a tampering probability map output layer; the 1st, 4th, and 7th layers are deconvolution layers, and the kernel corresponds to 4*4; the 2nd and 5th layers are densely connected convolution blocks, which include a two-dimensional convolution layer, a batch normalization layer, and a RELU activation function; the 3rd and 6th layers are feature output control layers; the 8th layer is the tampering probability map output layer, which converts the upsampled feature map into a tampering probability map through a convolution layer with a kernel of 1*1 and a Logic SoftMax activation function.

3. The digital document image tampering location method based on character texture perception according to claim 1 is characterized in that: The specific definition of the dense addition connection includes: Define f i (H, W, C) is the feature map output by the i-th layer, where H, W are the length and width of the feature map, and C is the number of channels of the feature map; S i+1 (K, F, f i (H, W, C)) is the i+1th convolution layer, where K is the convolution kernel size and F is the number of filters in the convolution layer. Then the feature map f output by the i+1th layer is i+1 (H, W, C) is: f i+1 (H,W,C)=S i+1 (K,F,f i (H,W,C))+f i (H,W,C) (1)。

Images