A dual-stream U-Net image tampering detection network system and image tampering detection method

Abstract

The invention provides a dual-stream U-Net image tampering detection network system. The network adopts a U-type encoding and decoding network structure. The encoder includes an RGB stream and a Noise stream. The RGB stream is used to extract high- and low-level tampering features in an image, so as to obtain a rough positioning effect. The Noise stream is used as a supplementary stream to reveal the local noise inconsistency in the image. The two streams are combined in a lightweight hierarchical manner, so that the network of the invention can perceive tampering targets with large differences in shape and size at different scales. After encoding by the encoder, the network adopts a decoder to simultaneously fuse encoding features and jump features to obtain rich low-level tampering traces and spatial positioning information, so as to meet the demand for accurately positioning the tampering target area. The image tampering detection method using the dual-stream U-Net image tampering detection network system can simultaneously detect three types of tampering, namely, splicing, copying and moving, and removal. The method has certain robustness and can still maintain a good detection effect in response to JPEG compression attacks with different compression factors.

Description

Technical Field

[0001] The present invention relates to the field of digital image tampering detection, and in particular to a dual-stream U-Net image tampering detection network system and an image tampering detection method thereof. Background Art

[0002] In recent years, with the rapid development of deep learning technology, neural networks have achieved good results in computer vision tasks (such as target detection and semantic segmentation), and researchers in the field of forensics have also begun to seek better solutions in neural network structures. Neural networks have the ability to automatically learn and nonlinear feature mapping, which makes them have great advantages over traditional methods, and their detection performance also greatly exceeds that of traditional detection methods. In past studies, some methods focused on simply judging and classifying whether an image has been tampered with, some methods treated the tampering location problem as a binary classification problem of image blocks, which made it difficult to accurately locate the tampering edge, and some methods only focused on a single type of tampering (such as splicing, copying and moving, removal, and global image enhancement). Although a single type of tampering detection can achieve good results, it is difficult to directly apply it in real scenarios, because the tampering detection in real situations is often the case that the tampering type is unknown and multiple tampering operations may occur at the same time. In response to such scenario requirements, more and more researchers tend to study a general integrated forensic algorithm framework to cope with complex and diverse situations. Based on the long short-term memory network, it explores the inconsistent edges between the original image block and the tampered image block; as an improved version of J-LSTM, it extracts resampled features in the image block through the Laplacian filter, and uses the HilbertCurve route to make decisions on the image block input sequence, and simply merges with the spatial domain features to make tampering positioning decisions. Both are based on the preprocessing of image blocks, which is time-consuming and difficult to apply efficiently. MFCN solves the tampering detection problem by designing a multi-task network. One task branch locates the tampered area, and the other task branch locates the tampered edge. The fusion of multiple tasks improves the tampering detection effect, but the two tasks are carried out simultaneously, and the network training process requires a long convergence time. MT-Net proposes a feature extractor pre-trained in 385 types of tampering traces, and integrates a local anomaly positioning module to perform tampering positioning; based on this feature extractor, SPAN introduces a local self-attention mechanism in the form of a spatial pyramid to improve forensic positioning. Both require millions of data for pre-training to achieve the generalization effect of the real situation, which requires huge resources. RGB-N uses two-stream Fast R-CNN to perform regular bounding-box level tampering target positioning, where one stream is the RGB stream and the other is the Noise stream. The two-stream features are combined in a late-fusion manner through binlinear pooling, but it can only provide a rough target box positioning result.

[0003] The prior art discloses a patent for a dual-stream network image forgery detection method and system based on image block feature extraction, which includes: segmenting the image to be detected into image blocks; inputting the obtained image blocks into a CNN model and a CNN-GRU model to extract internal artifacts of the image blocks and spatial features between image blocks, respectively, and obtaining corresponding forgery scores, wherein the closer the forgery score is to 1, the greater the possibility that the image to be detected is forged; using an attention-based fusion method, the two obtained forgery scores are fused to obtain a judgment score for the authenticity of the image to be detected. However, this method fails to detect the three types of tampering, namely splicing, copying and moving, and removal, and locate the tampering location. Summary of the invention

[0004] The present invention provides a dual-stream U-Net image tampering detection network system to solve the problem of simultaneously making accurate tampering location detection for three tampering types: splicing, copying and moving, and removal.

[0005] Another object of the present invention is to provide an image tampering detection method using the above-mentioned dual-stream U-Net image tampering detection network system.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0007] A dual-stream U-Net image tampering detection network system, the network comprises two parts, an encoder and a decoder, wherein the encoder comprises two data transmission paths: an RGB stream and a Noise stream, each data transmission path comprises the 1st to the 5th data stream layers; the decoder comprises the 1st to the 6th decoding layers; a high-pass filter is arranged between the RGB stream 1st data stream layer and the Noise stream 1st data stream layer, and an input addition jump channel is arranged, leading to the 6th decoding layer, the outputs of the RGB stream 1st data stream layer and the Noise stream 1st data stream layer are arranged with an addition jump channel, leading to the 5th decoding layer, the outputs of the RGB stream 2nd layer and the Noise stream 2nd layer are arranged with an addition jump channel, leading to the 4th decoding layer, the outputs of the RGB stream 3rd layer and the Noise stream 3rd layer are arranged with an addition jump channel, leading to the 3rd decoding layer, the outputs of the RGB stream 4th layer and the Noise stream 4th layer are arranged with an addition jump channel, leading to the 2nd decoding layer, and the outputs of the RGB stream 5th layer and the Noise stream 5th layer are added and input to the 1st decoding layer.

[0008] The network of the present invention adopts a network structure of a U-shaped encoder and a decoder as a whole. The encoder includes an RGB stream and a Noise stream. The RGB stream is used to extract high-level tampering features and low-level tampering features in the image, so as to obtain a rough positioning effect; the Noise stream is used as a supplementary stream to reveal local noise inconsistencies in the image; the two streams are combined in a lightweight hierarchical manner, so that the network of the present invention can perceive tampering targets with large differences in shape and size at multiple different scales; after the encoder, the network of the present invention uses a decoder to simultaneously fuse coding features and jump features to obtain rich low-level tampering traces and spatial positioning information, so as to meet the needs of accurately locating the tampering target area; at the decoder end, the detailed information of the tampering target and the spatial dimension of the image are gradually restored in the global coding features, and finally a high-resolution prediction probability map is generated to reveal the original area and the tampered area.

[0009] Furthermore, the internal coding structure of the RGB stream and the Noise stream includes:

[0010] The first data stream layer of the RGB stream includes the convolution kernel conv; the batch normalization data stream layer norm; the activation function Relu; the first data stream layer of the Noise stream includes the convolution kernel conv; the batch normalization data stream layer norm; the activation function Relu;

[0011] The second data stream layer of the RGB stream includes a maximum pooling data stream layer pool; a dense connection module denseblock; the second data stream layer of the Noise stream includes a maximum pooling data stream layer pool; a dense connection module denseblock;

[0012] The third data stream layer of the RGB stream includes a transposition module transition; a dense link module denseblock; the third data stream layer of the Noise stream includes a transposition module transition; a dense link module denseblock;

[0013] The 4th data stream layer of the RGB stream includes a transposition module transition; a dense link module denseblock; the 4th data stream layer of the Noise stream includes a transposition module transition; a dense link module denseblock;

[0014] The fifth data stream layer of the RGB stream includes a transposition module transition; the fifth data stream layer of the Noise stream includes a transposition module transition.

[0015] As the image content stream, the RGB stream simultaneously and automatically captures high-level and low-level tampering traces. Tampering traces (i.e., forensic features) exist not only in low-level features, but also in high-level features. High-level tampering traces are mainly unnatural and uncommon high contrasts in images (such as color contrast, brightness contrast, tampering edges, and visual distortion effects). This type of contrast helps the network to be proficient in capturing local anomalies. When an image is tampered with by splicing or copying and moving, the tampered area often shows a certain color contrast and brightness contrast without any post-processing operation. When an image is tampered with by removal type, the tampered area will show a certain visual distortion effect and unnatural tampering edges (this edge is easily distinguished from the natural edge). However, this high-level tampering feature is more easily captured by the network, and the network will gradually lose the learning of low-level tampering features in the process of capturing high-level features.

[0016] During the image generation process, a series of processing from hardware and software will be experienced. For each camera, these processing traces are specific and are called "image fingerprints" (such as traces introduced by lens distortion, CFA interpolation, de-mosaicing algorithm, noise properties, JPEG compression properties, etc.). These fingerprint information allow the pixel distribution in the native image to show a certain pattern, that is, the local dependency relationship in the pixels. This relationship is very subtle and delicate, and it is difficult for humans to perceive it subjectively. This relationship has the basic attributes of the image rather than the semantic content information of the image. However, modern neural networks that are too deep tend to capture high-level semantic information in the image and ignore the basic attributes of the image (or, the image content will interfere with the network's learning of low-level attribute features to a considerable extent), while too shallow neural networks do not have enough learning ability to capture local inconsistencies in different image fingerprints. A compromise is to suppress interference from the semantic content of the image at the source of the network, and then continuously deepen the network structure to capture effective features. From this point of view, the present invention applies different preprocessing layers (high-pass filters) to suppress image content and further emphasize high-frequency detail information in the image. The Noise stream strengthens the reuse of fragile low-level tampering traces and acts as a local noise descriptor to reveal the inconsistency of fingerprints in tampered images. After applying the Noise stream, DS-UNet obtains an improved result compared to the RGB stream, proving the effectiveness of applying the Noise stream as a supplementary stream.

[0017] Furthermore, the specific parameters of the internal coding structure of the RGB stream and the Noise stream and the data transmission process include:

[0018] The data of the first data stream layer of the input RGB stream is an image of size N*N*3. After the first data stream layer of the RGB stream, the convolution kernel conv with a width of 7, a height of 7, a number of channels of 64, and a step size of 2; the batch normalization data stream layer norm; the activation function Relu, the final output size is The RGB stream feature map of the input Noise stream first data stream layer is the high-frequency noise residual map of size N*N*3 after being processed by the SRM filter, and the convolution kernel conv with a width of 7, a height of 7, a number of channels of 64, and a step size of 2 passes through the Noise stream first data stream layer; the batch normalization data stream layer norm; the activation function Relu, the final output size is Noise flow feature map;

[0019] The data of the second data stream layer of the input RGB stream is the size of the first data stream layer of the RGB stream. The RGB stream feature map is passed through the RGB stream second data stream layer with a window width of 3, a height of 3, and a maximum pooling data stream layer with a step size of 2; the dense link module denseblock, and the final output size is RGB stream feature map; the data input to the second data stream layer of the Noise stream is the size of the first data stream layer of the Noise stream The Noise stream feature map is passed through the Noise stream second data stream layer with a window width of 3, a height of 3, and a maximum pooling data stream layer with a step size of 2; the dense link module denseblock, and the final output size is Noise flow feature map;

[0020] The data of the third data stream layer of the input RGB stream is the size of the second data stream layer of the RGB stream. The RGB stream feature map is passed through the 3rd data stream layer of the RGB stream, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense block module has a final output size of RGB stream feature map; the data input to the third data stream layer of the Noise stream is the size of the second data stream layer from the Noise stream The Noise stream feature map is passed through the Noise stream third data stream layer, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense block module has a final output size of Noise flow feature map;

[0021] The data of the 4th data stream layer of the input RGB stream is the data from the 3rd data stream layer of the RGB stream. The RGB stream feature map is passed through the 4th data stream layer of the RGB stream, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense link module denseblock, and the final output size is RGB stream feature map; the data input to the 4th data stream layer of the Noise stream is the size of the 3rd data stream layer of the Noise stream. The Noise stream feature map is passed through the Noise stream 4th data stream layer, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense block module has a final output size of Noise flow feature map;

[0022] The data of the 5th data stream layer of the input RGB stream is the size of the 4th data stream layer of the RGB stream. The RGB stream feature map is transformed through the 5th data stream layer of the RGB stream, which reduces the number of feature channels by half and the spatial dimension of the feature map by half. The final output size is RGB stream feature map; the data input to the 5th data stream layer of the Noise stream is the size of the 4th data stream layer of the Noise stream The Noise stream feature map is transformed through the Noise stream 5th data stream layer, which reduces the number of feature channels by half and the spatial dimension of the feature map by half. The final output size is Noise flow feature map.

[0023] Furthermore, the internal decoding structure of the first to sixth decoding layers includes:

[0024] The first decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer, and the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer;

[0025] The second decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer;

[0026] The third decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer;

[0027] The 4th decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer;

[0028] The 5th decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer;

[0029] The sixth decoding layer includes the convolution kernel conv and the Sigmiod activation function layer.

[0030] The present invention uses different high-pass filters for comparative analysis (including SRM, Bayar, and adaptive layer). It can be found that the filter with SRM fixed weights has a better effect. The reasons are: ① It is difficult to reveal the inconsistency between the tampered area and the native area by exploring the texture features only through the adaptive layer; ② The Bayar layer has a forced update mode, which will introduce uncertainty to the data distribution, which can easily disrupt the data distribution and is not conducive to the hierarchical combination of the two streams; ③ The SRM layer has a fixed filtering weight, which makes the training more stable and is conducive to the fusion of the two streams.

[0031] Furthermore, the specific parameters of the internal decoding structure and the data transmission process of the first to sixth decoding layers include:

[0032] The output size is The RGB stream feature map and the Noise stream feature map are added in the channel and input into the upsampling feature extraction module up_conv_block of the first decoding layer. After the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 256, and a step size of 1, the Inplace-ABN layer, the output size is The upsampled feature map; then the size of the RGB stream 4th layer output is The size of the RGB stream feature map and the Noise stream 4th layer output is The Noise stream feature maps are added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a channel number of 1024, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of

[0033] The size of the output of the first decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the second decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 512, and a step size of 1, the Inplace-ABN layer, and the output size is The upsampled feature map; then the size of the RGB stream layer 3 output is The size of the RGB stream feature map and the Noise stream layer 3 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the feature extraction module conv_block with two convolution kernels of width 3, height 3, number of channels 512, and step size 1, and the Inplace-ABN layer. The output size is The feature map of

[0034] The size of the second decoding layer output is The feature map is input into the upsampling feature extraction module up_conv_block of the 3rd decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block is passed through the convolution kernel conv with a width of 3, a height of 3, a number of channels of 256, and a step size of 1. The output size is The upsampled feature map; then the size of the RGB stream second layer output is The size of the RGB stream feature map and the Noise stream layer 2 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a channel number of 256, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of

[0035] The size of the output of the third decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the 4th decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 64, and a step size of 1, the Inplace-ABN layer, and the output size is The upsampled feature map; then the size of the RGB stream first layer output is The size of the RGB stream feature map and the Noise stream layer 1 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a number of channels of 64, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of

[0036] The size of the output of the 4th decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the 5th decoding layer, and passes through the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, the number of channels of 32, and a step size of 1, the Inplace-ABN layer, and the output is an upsampling feature map of size N*N*3; then the RGB image of size N*N*3 of the first layer of the input RGB stream and the high-frequency noise residual map of size N*N*3 of the first layer of the input Noise stream are added, and connected with the obtained upsampling feature map of size N*N*3 in the channel dimension to obtain a feature map of size N*N*(32+3), which passes through two convolution kernels conv with a width of 3, a height of 3, the number of channels of 16, and a step size of 1 of the feature extraction module conv_block, the Inplace-ABN layer, and the output is a feature map of size N*N*16;

[0037] The feature map of size N*N*16 output by the 5th decoding layer is input into the 6th decoding layer, and passes through the convolution kernel conv with width 1, height 1, number of channels 1, and step size 1 to output a feature map of size N*N*1; then, it passes through the Sigmiod activation function to obtain a single-channel probability mask map of size N*N*1.

[0038] Furthermore, the batch normalization layer norm performs the following processing on the input feature map:

[0039] Calculate the mean of the input feature map data:

[0040]

[0041] Calculate the variance of the input feature map data:

[0042]

[0043] Normalize the feature map data:

[0044]

[0045] Feature restoration after normalizing the feature map data:

[0046]

[0047] Among them, γ and β are the required training parameters.

[0048] The Inplace-ABN layer combines the batch normalization norm layer and the activation function layer.

[0049] The setting of batch normalization layer can not only speed up the convergence of network training, but also alleviate the problem of gradient diffusion in deep networks to a certain extent, making the training of deep network models easier and more stable.

[0050] An image tampering detection method using a dual-stream U-Net image tampering detection network system comprises the following steps:

[0051] S1: Obtain reference image data and tampered image data to be detected, wherein the reference image data includes a reference tampered image and its corresponding tampered annotated image, divide the reference image data into reference image training data and reference image verification data in a ratio of 9:1, and pre-process the reference image training data, the reference image verification data and the tampered image data to be detected respectively;

[0052] S2: Initialize the dual-stream U-Net image tampering detection network system parameters;

[0053] S3: Use the preprocessed benchmark image training data and verification data to train the two-stream U-Net image tampering detection network system, confirm the optimal two-stream U-Net image tampering detection network system, input the tampered image data to be tested into the optimal two-stream U-Net image tampering detection network system, and output a single-channel probability mask map as the tampering detection result.

[0054] The present invention provides a dual-stream U-Net image tampering detection network system. An image tampering detection method using the network can simultaneously detect three types of tampering: splicing, copying and moving, and removal. The method has certain robustness and can still maintain a good detection effect in response to JPEG compression attacks with different compression factors.

[0055] Furthermore, in step S1, the preprocessing of the reference image training data includes the following steps:

[0056] S11-1: simultaneously reading the reference tampered image and its corresponding tampered annotated image in the reference image training data, and cropping the image at a random area cropping rate between 90% and 100%;

[0057] S12-1: The image obtained in step S11-1 is first randomly flipped horizontally with a probability of 50%, and then randomly flipped vertically with a probability of 50%;

[0058] S13-1: The image obtained in step S12-1 is scaled by bicubic interpolation to unify the image size and rotated by a fixed angle, wherein the fixed angle is randomly selected from 0°, 90°, 180° or 270°;

[0059] S14-1: Perform normalization on the reference tampered image obtained in step S13-1. The normalization formula is:

[0060]

[0061] , I (u,v) is the pixel value at coordinate (u, v) in the reference tampered image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,c) is the pixel value at coordinate (u, v) in the normalized reference tampered image;

[0062] The corresponding tampered annotated image is normalized, and the normalization formula is:

[0063]

[0064] , F (u,v) To tamper with the pixel value at coordinate (u,c) in the annotated image, F' (u,v) The pixel value at the coordinate (u, v) in the tampered labeled image after normalization;

[0065] S15-1: Connect the reference tampered image and its corresponding tampered annotated image obtained in step S14-1 in the batch dimension respectively;

[0066] S16-1: Shuffle the reference tampered images and their corresponding tampered annotated images within the same batch size in step S15-1 respectively.

[0067] The preprocessing of the reference image verification data comprises the following steps:

[0068] S11-2: Simultaneously reading the reference tampered image and its corresponding tampered annotated image in the reference image verification data;

[0069] S12-2: The reference tampered image obtained in step S11-2 and its corresponding tampered annotation image are scaled by bicubic interpolation. After unifying the image sizes, the reference tampered image is subjected to random horizontal flip transformation, random vertical flip transformation, and rotation transformation at a fixed angle randomly selected from 0°, 90°, 180°, and 270°, generating a total of 16 transformation conditions, corresponding to 16 groups of reference tampered images, and similarly, 16 groups of tampered annotation images are obtained;

[0070] S13-2: Perform normalization on the 16 groups of reference tampered images obtained in step S12-2.

[0071] The formula is:

[0072]

[0073] , I (u,v) is the pixel value at coordinate (u, v) in the reference tampered image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the normalized reference tampered image;

[0074] The corresponding 16 groups of tampered annotated images are normalized, and the normalization formula is:

[0075]

[0076] , G (u,v) To tamper with the pixel value at coordinate (u,v) in the annotated image, G' (u,v) It is the pixel value at the coordinate (u, v) in the normalized tampered annotation image.

[0077] The preprocessing of the image data to be detected for tampering comprises the following steps:

[0078] S11-3: reading the tampered image in the tampered image data to be tested, scaling the tampered image by using bicubic interpolation, unifying the image size, and then performing random horizontal flip transformation, random vertical flip transformation, and randomly selecting a fixed angle from 0°, 90°, 180°, and 270° for rotation transformation on the tampered image in sequence, generating a total of 16 transformation situations, corresponding to 16 groups of tampered images;

[0079] S12-3: Perform normalization operation on the 16 groups of tampered images obtained in step S11-3, and the normalization formula is:

[0080]

[0081] , I (u,v) To tamper with the pixel value at the coordinate (u, v) in the image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the tampered image after normalization;

[0082] Furthermore, in step S2, the dual-stream U-Net image tampering detection network system parameters are initialized by using the weights of the ImageNet open source data set to respectively initialize the network parameters of the RGB stream and the Noise stream, and prepare an N*N*3 sized RGB image and an N*N*3 sized high-frequency noise residual map obtained by passing through an SRM high-pass filter.

[0083] Furthermore, in step S3, the training of the dual-stream U-Net image tampering detection network system using the preprocessed reference image training data and verification data includes the following steps:

[0084] S31: Input the reference tampered image in the preprocessed reference image training data into the dual-stream U-Net image tampering detection network system to output a single-channel probability mask map, and use the loss function to calculate the loss of the obtained single-channel probability mask map and the tampered annotation image corresponding to the original reference tampered image, and then back-propagate the gradient according to the calculation result to update the dual-stream U-Net image tampering detection network system parameters, where the loss function is Dice loss + Focal loss, and the loss function expression is as follows:

[0085] L Total (M g ,M o )=L Dice (M g ,M o )+L Focal (M g ,M o )

[0086] Among them, (M g ,M o ) is the ground-truth and predicted probability map within a batch, Dice loss and Focalloss can be specifically expressed as:

[0087]

[0088] Among them, M g (i,j) and M o (i,j) represents the pixel value located at (i,j);

[0089] S32: Freeze the parameters of the dual-stream U-Net image tampering detection network system obtained in step S31, input the reference tampered image in the preprocessed reference image verification data into the dual-stream U-Net image tampering detection network system, output 16 groups of single-channel probability mask images, perform inverse transformation on 15 groups of single-channel probability mask images, and then average the 16 groups of single-channel probability mask images, and use the average value as the final output single-channel probability mask image, and use the loss function to calculate the loss of the tampered annotated image corresponding to the original reference tampered image;

[0090] S33: Steps S31 and S32 are continuously repeated. When the loss of the loss function tends to be stable, it is considered that the loss has converged and the training is stopped. At this time, the parameters of the dual-stream U-Net image tampering detection network system are optimal.

[0091] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0092] The network of the present invention adopts a U-type codec network structure as a whole. The encoder includes an RGB stream and a Noise stream. The RGB stream is used to extract high-level tampering features in the image, that is, the contrast and distortion effects in the image, and low-level tampering features, that is, the local dependencies between neighboring pixels, so as to obtain a rough positioning effect; the Noise stream is used as a supplementary stream to reveal the local noise inconsistency in the image; the two streams are combined in a lightweight hierarchical manner, so that the network of the present invention can perceive tampering targets with large differences in shape and size at multiple different scales; after the encoder, the network of the present invention uses a decoder to simultaneously By fusing coding features with jump features, rich low-level tampering traces and spatial positioning information are obtained to meet the needs of accurately locating the tampered target area; at the decoder end, the detailed information of the tampered target and the spatial dimension of the image are gradually restored in the global coding features, and finally a high-resolution prediction probability map is generated to reveal the original area and the tampered area. The image tampering detection method using the dual-stream U-Net image tampering detection network system can simultaneously detect three types of tampering: splicing, copying and moving, and removal. This method has a certain degree of robustness and can still maintain a good detection effect when dealing with JPEG compression attacks with different compression factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is a structural diagram of a two-stream U-Net image tampering detection network system;

[0094] Figure 2 A flowchart of an image tampering detection method using a dual-stream U-Net image tampering detection network system;

[0095] Figure 3 This is the result graph of preprocessing the benchmark image training data;

[0096] Figure 4 The result diagram of preprocessing the benchmark image verification data;

[0097] Figure 5 This is a subjective visual comparison result diagram of the method of the present invention and similar tampering detection algorithms on different data;

[0098] Figure 6 The tampered images, ground-truth images, and probabilistic truncated output images of DS-U-Net under five different data tampering types;

[0099] Figure 7There are three columns of images, wherein the first column of images are three types of tampered images, namely, splicing, copying and moving, and removal; the second column is the Ground True image that reveals the tampered area; and the third column is the binary mask output image obtained by the present invention with a threshold of 0.5. DETAILED DESCRIPTION

[0100] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0101] It is understandable to those skilled in the art that some well-known descriptions may be omitted in the drawings.

[0102] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0103] The embodiment of the present invention uses about 180,000 artificially copied and pasted synthetic image data as reference image data.

[0104] The tampered data to be tested used in Example 1 is the spliced ​​Columbia data.

[0105] A dual-stream U-Net image tampering detection network system used in this embodiment, such as Figure 1 As shown, the specific parameters of its internal structure and the data transmission process include:

[0106] The data of the first data stream layer of the input RGB stream is an image of size N*N*3. After the first data stream layer of the RGB stream, the convolution kernel conv with a width of 7, a height of 7, a number of channels of 64, and a step size of 2; the batch normalization data stream layer norm; the activation function Relu, the final output size is The RGB stream feature map; the data input to the first data stream layer of the Noise stream is a high-frequency noise residual map of size N*N*3, which passes through the convolution kernel conv with a width of 7, a height of 7, a number of channels of 64, and a step size of 2 in the first data stream layer of the Noise stream; the batch normalization data stream layer norm; the activation function Relu, and the final output size is Noise flow feature map;

[0107] The data of the second data stream layer of the input RGB stream is the size of the first data stream layer of the RGB stream. The RGB stream feature map is passed through the RGB stream second data stream layer with a window width of 3, a height of 3, and a maximum pooling data stream layer with a step size of 2; the dense link module denseblock, and the final output size is RGB stream feature map; the data input to the second data stream layer of the Noise stream is the size of the first data stream layer of the Noise stream The Noise stream feature map is passed through the Noise stream second data stream layer with a window width of 3, a height of 3, and a maximum pooling data stream layer with a step size of 2; the dense link module denseblock, and the final output size is Noise flow feature map;

[0108] The data of the third data stream layer of the input RGB stream is the size of the second data stream layer of the RGB stream. The RGB stream feature map is passed through the 3rd data stream layer of the RGB stream, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense block module has a final output size of RGB stream feature map; the data input to the third data stream layer of the Noise stream is the size of the second data stream layer from the Noise stream The Noise stream feature map is passed through the Noise stream third data stream layer, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense block module has a final output size of Noise flow feature map;

[0109] The data of the 4th data stream layer of the input RGB stream is the data from the 3rd data stream layer of the RGB stream. The RGB stream feature map is passed through the 4th data stream layer of the RGB stream, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense link module denseblock, and the final output size is RGB stream feature map; the data input to the 4th data stream layer of the Noise stream is the size of the 3rd data stream layer of the Noise stream. The Noise stream feature map is passed through the Noise stream 4th data stream layer, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense block module has a final output size of Noise flow feature map;

[0110] The data of the 5th data stream layer of the input RGB stream is the size of the 4th data stream layer of the RGB stream. The RGB stream feature map is transformed through the 5th data stream layer of the RGB stream, which reduces the number of feature channels by half and the spatial dimension of the feature map by half. The final output size is RGB stream feature map; the data input to the 5th data stream layer of the Noise stream is the size of the 4th data stream layer of the Noise stream The Noise stream feature map is transformed through the Noise stream 5th data stream layer, which reduces the number of feature channels by half and the spatial dimension of the feature map by half. The final output size is Noise flow feature map.

[0111] The output size is The RGB stream feature map and the Noise stream feature map are added in the channel and input into the upsampling feature extraction module up_conv_block of the first decoding layer. After the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 256, and a step size of 1, the Inplace-ABN layer, the output size is The upsampled feature map; then the size of the RGB stream 4th layer output is The size of the RGB stream feature map and the Noise stream 4th layer output is The Noise stream feature maps are added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a channel number of 1024, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of

[0112] The size of the output of the first decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the second decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 512, and a step size of 1, the Inplace-ABN layer, and the output size is The upsampled feature map; then the size of the RGB stream layer 3 output is The size of the RGB stream feature map and the Noise stream layer 3 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the feature extraction module conv_block with two convolution kernels of width 3, height 3, number of channels 512, and step size 1, and the Inplace-ABN layer. The output size is The feature map of

[0113] The size of the second decoding layer output is The feature map is input into the upsampling feature extraction module up_conv_block of the 3rd decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block is passed through the convolution kernel conv with a width of 3, a height of 3, a number of channels of 256, and a step size of 1. The output size is The upsampled feature map; then the size of the RGB stream second layer output is The size of the RGB stream feature map and the Noise stream layer 2 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a channel number of 256, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of

[0114] The size of the output of the third decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the 4th decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 64, and a step size of 1, the Inplace-ABN layer, and the output size is The upsampled feature map; then the size of the RGB stream first layer output is The size of the RGB stream feature map and the Noise stream layer 1 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a number of channels of 64, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of

[0115] The size of the output of the 4th decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the 5th decoding layer, and passes through the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, the number of channels of 32, and a step size of 1, the Inplace-ABN layer, and the output is an upsampling feature map of size N*N*3; then the RGB image of size N*N*3 of the first layer of the input RGB stream and the high-frequency noise residual map of size N*N*3 of the first layer of the input Noise stream are added, and connected with the obtained upsampling feature map of size N*N*3 in the channel dimension to obtain a feature map of size N*N*(32+3), which passes through two convolution kernels conv with a width of 3, a height of 3, the number of channels of 16, and a step size of 1 of the feature extraction module conv_block, the Inplace-ABN layer, and the output is a feature map of size N*N*16;

[0116] The feature map of size N*N*16 output by the 5th decoding layer is input into the 6th decoding layer, and passes through the convolution kernel conv with width 1, height 1, number of channels 1, and step size 1 to output a feature map of size N*N*1; then, it passes through the Sigmiod activation function to obtain a single-channel probability mask map of size N*N*1.

[0117] The batch normalization layer norm performs the following processing on the input feature map:

[0118] Calculate the mean of the input feature map data:

[0119]

[0120] Calculate the variance of the input feature map data:

[0121]

[0122] Normalize the feature map data:

[0123]

[0124] Feature restoration after normalizing the feature map data:

[0125]

[0126] Among them, γ and β are the required training parameters.

[0127] The Inplace-ABN layer combines the batch normalization norm layer and the activation function layer.

[0128] The above is a dual-stream U-Net image tampering detection network system, its internal structure specific parameters and data transmission process.

[0129] A dual-stream U-Net image tampering detection network system image tampering detection method, such as Figure 2 As shown, the following steps are included:

[0130] S1: Obtain reference image data and tampered image data to be tested, wherein the reference image data includes a reference tampered image and its corresponding tampered annotated image, divide the reference image data into reference image training data and reference image verification data in a ratio of 9:1, and respectively perform the reference image training data, such as Figure 3 As shown, benchmark image verification data, such as Figure 4 As shown, the tampered image data to be detected is preprocessed;

[0131] S11-1: simultaneously reading the reference tampered image and its corresponding tampered annotated image in the reference image training data, and cropping the image at a random area cropping rate between 90% and 100%;

[0132] S12-1: The image obtained in step S11-1 is first randomly flipped horizontally with a probability of 50%, and then randomly flipped vertically with a probability of 50%;

[0133] S13-1: The image obtained in step S12-1 is scaled by bicubic interpolation to unify the image size and rotated by a fixed angle, wherein the fixed angle is randomly selected from 0°, 90°, 180° or 270°;

[0134] S14-1: Perform normalization on the reference tampered image obtained in step S13-1. The normalization formula is:

[0135]

[0136] , I (u,v) is the pixel value at coordinate (u, v) in the reference tampered image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the normalized reference tampered image;

[0137] The corresponding tampered annotated image is normalized, and the normalization formula is:

[0138]

[0139] , G (u,v) To tamper with the pixel value at coordinate (u,v) in the annotated image, G' (u,v) The pixel value at the coordinate (u, v) in the tampered labeled image after normalization;

[0140] S15-1: Connect the reference tampered image and its corresponding tampered annotated image obtained in step S14-1 in the batch dimension respectively;

[0141] S16-1: Shuffle the reference tampered images and their corresponding tampered annotated images within the same batch size in step S15-1 respectively.

[0142] The results of preprocessing the benchmark image training data are as follows: Figure 3 shown.

[0143] The preprocessing of the reference image verification data comprises the following steps:

[0144] S11-2: Simultaneously reading the reference tampered image and its corresponding tampered annotated image in the reference image verification data;

[0145] S12-2: The reference tampered image obtained in step S11-2 and its corresponding tampered annotation image are scaled by bicubic interpolation. After unifying the image sizes, the reference tampered image is subjected to random horizontal flip transformation, random vertical flip transformation, and rotation transformation at a fixed angle randomly selected from 0°, 90°, 180°, and 270°, generating a total of 16 transformation conditions, corresponding to 16 groups of reference tampered images, and similarly, 16 groups of tampered annotation images are obtained;

[0146] S13-2: Perform normalization on the 16 groups of reference tampered images obtained in step S12-2.

[0147] The formula is:

[0148]

[0149] , I (u,v) is the pixel value at coordinate (u, v) in the reference tampered image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the normalized reference tampered image;

[0150] The corresponding 16 groups of tampered annotated images are normalized, and the normalization formula is:

[0151]

[0152] , G (u,v) To tamper with the pixel value at coordinate (u,v) in the annotated image, G' (u,v) It is the pixel value at the coordinate (u, v) in the normalized tampered annotation image.

[0153] The result of preprocessing the benchmark image verification data is as follows: Figure 4 shown.

[0154] The preprocessing of the image data to be detected for tampering comprises the following steps:

[0155] S11-3: reading the tampered image in the tampered image data to be tested, scaling the tampered image by using bicubic interpolation, unifying the image size, and then performing random horizontal flip transformation, random vertical flip transformation, and randomly selecting a fixed angle from 0°, 90°, 180°, and 270° for rotation transformation on the tampered image in sequence, generating a total of 16 transformation situations, corresponding to 16 groups of tampered images;

[0156] S12-3: Perform normalization operation on the 16 groups of tampered images obtained in step S11-3, and the normalization formula is:

[0157]

[0158] , I (u,v) To tamper with the pixel value at the coordinate (u, v) in the image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the tampered image after normalization;

[0159] S2: Initialize the parameters of the dual-stream U-Net image tampering detection network system, that is, use the weights of the ImageNet open source dataset to initialize the network parameters of the RGB stream and the Noise stream respectively, and prepare an N*N*3 RGB image and an N*N*3 high-frequency noise residual map obtained by the SRM high-pass filter;

[0160] S3: Use the preprocessed benchmark image training data and verification data to train the two-stream U-Net image tampering detection network system, confirm the optimal two-stream U-Net image tampering detection network system, input the tampered image data to be tested into the optimal two-stream U-Net image tampering detection network system, and output a single-channel probability mask map as the tampering detection result.

[0161] S31: Input the reference tampered image in the preprocessed reference image training data into the dual-stream U-Net image tampering detection network system to output a single-channel probability mask map, and use the loss function to calculate the loss of the obtained single-channel probability mask map and the tampered annotation image corresponding to the original reference tampered image, and then back-propagate the gradient according to the calculation result to update the dual-stream U-Net image tampering detection network system parameters, where the loss function is Dice loss + Focal loss, and the loss function expression is as follows:

[0162] L Total (M g ,M o )=L Dice (M g ,M o )+L Focal (M g ,M o )

[0163] Among them, (M g ,M o ) is the ground-truth and predicted probability map within a batch, Dice loss and Focalloss can be specifically expressed as:

[0164]

[0165] Among them, M g (i,j) and M o (i,j) represents the pixel value located at (i,j);

[0166] S32: Freeze the parameters of the dual-stream U-Net image tampering detection network system obtained in step S31, input the reference tampered image in the preprocessed reference image verification data into the dual-stream U-Net image tampering detection network system, output 16 groups of single-channel probability mask images, perform inverse transformation on 15 groups of single-channel probability mask images, and then average the 16 groups of single-channel probability mask images, and use the average value as the final output single-channel probability mask image, and use the loss function to calculate the loss of the tampered annotated image corresponding to the original reference tampered image;

[0167] S33: Steps S31 and S32 are continuously repeated. When the loss of the loss function tends to be stable, it is considered that the loss has converged and the training is stopped. At this time, the parameters of the dual-stream U-Net image tampering detection network system are optimal.

[0168] The tampered data to be detected used in Example 2 is spliced ​​DSO-1 data, and the dual-stream U-Net image tampering detection network system and image tampering detection method steps adopted are the same as those in Example 1.

[0169] The tampered data to be detected used in Example 3 is Coverage data of the copy-move type, and the dual-stream U-Net image tampering detection network system and the image tampering detection method steps adopted are the same as those in Example 1.

[0170] The tampered data to be tested used in Example 4 is CASIA data including splicing and copy-move types, and the dual-stream U-Net image tampering detection network system and image tampering detection method steps adopted are the same as those in Example 1.

[0171] The tampered data to be tested used in Example 5 is NIST16 data including splicing, copy movement and removal types. The dual-stream U-Net image tampering detection network system and image tampering detection method steps adopted are the same as those in Example 1.

[0172] The following is a comparative analysis of the method of the present invention with the existing methods: ELA, CFA1, NOI1, J-LSTM, RGB-N, and H-LSTM.

[0173] The evaluation indicators used in the comparative analysis are as follows:

[0174] cIOU represents the intersection-over-union ratio of each category, AUC is the area under the receiver operating characteristic curve, F1 is the balance between precision and recall, ACC is the accuracy, IOU is the intersection-over-union ratio, and MCC is the Matthews correlation coefficient; all evaluation indicators are the higher the better, and the closer to 1 the better.

[0175] The comparison of only splicing type data is shown in Table 1 and Table 2. It can be observed from Table 1 that the network performance of the present invention is better. The reasons are summarized as follows: ① The quality of pre-trained data is low (not visually realistic or perfect), which is mainly reflected in high-level tampering traces (color contrast and brightness contrast), which will cause the model to be more inclined to capture local high-contrast areas and ignore low-level local pixel inconsistencies; ② Some deep learning methods are pre-trained on a large-scale, diverse data, or a real hand-made data; ③ Some methods use ground-truth to assist in obtaining the optimal threshold, and obtain higher evaluation indicators under the optimal threshold. In reality, it is difficult to obtain the corresponding ground-truth, so this solution is not realistic. The present invention only uses 0.5 as a fixed threshold for probability truncation. The present invention can effectively fit the tampering features in splicing type tampering, such as Figure 5 shown.

[0176] The same phenomenon can be observed in Table 2. Compared with the traditional unsupervised method (only focusing on specific tampering traces), the present invention can automatically and simultaneously extract high-level and low-level tampering features in a network learning preference manner. The present invention surpasses the second place by 16% and 12% in the two data respectively.

[0177] Table 1 shows the FI, ACC, AUC, and MCC evaluation index values ​​of Columbia data and DSO-1 data under different tampering detection methods

[0178] Table 1

[0179]

[0180] Table 2 shows the cIOU evaluation index values ​​of Columbia data and DSO-1 data under different tampering detection methods

[0181] Table 2

[0182]

[0183] The comparison results on the data of only copy-move type are shown in Table 3. It can be seen that the present invention ranks first in ACC and F1, and leads the second by 6% and 11.9% respectively. The AUC index of the present invention ranks third, but it should be noted that the SPAN method pre-trains the backbone network on 385 types of forensic detection data, and RGB-N only uses the target detection framework to provide a very rough bounding-box level positioning effect. This result means that the present invention is effective in detecting copy-move type tampering.

[0184] Table 3 shows the ACC, AUC and F1 evaluation index values ​​of Coverage data under different tampering detection methods.

[0185] Table 3

[0186]

[0187] The comparison results on data with various types of tampering are shown in Tables 4 and 5. On CASIA data, the present invention exceeds the second place by an average of 7.05% in F1 and MCC indicators, which means the high adaptability of the present invention. The present invention achieved the best results in indicators ACC, F1, MCC and IOU, which are 97.2%, 77.2%, 77.0% and 62.9% respectively. This effectiveness can be attributed to the diversified pre-training of Example 1 and the SRM high-pass filter adopted by the present invention (effectiveness in revealing local noise inconsistencies, while the removal of traces is mainly inconsistencies in noise).

[0188] Table 4 shows the ACC, MCC and F1 evaluation index values ​​of CASIA data under different tampering detection methods

[0189] Table 4

[0190]

[0191] Table 5 shows the ACC, AUC, MCC, F1 and IOU evaluation index values ​​of NIST16 data under different tampering detection methods

[0192] Table 5

[0193]

[0194] Comparison of subjective vision with different methods Figure 5As shown in the fourth column, it can be seen that the positioning area of ​​the present invention is more complete and the positioning edge is more accurate. The present invention provides output effects on five data, such as Figure 6 As shown, observe Figure 7 It can be concluded that the positioning effect of the present invention is very close to the ground-truth, which means that the present invention is effective in dealing with various types of tampering.

[0195] Table 6 is the robustness test of DS-UNet and the comparison results of different JPEG compression factors.

[0196] Table 6

[0197]

[0198] The robustness test is shown in Table 6. JPEG compression can well cover up high-frequency tampering traces and make tampering more realistic. JPEG compression factors of 95, 75, and 50 are used here. It can be observed from the table that the AUC changes from 86.7% to 97.7% on the Columbia data, and from 82.1% to 82.8% on the NIST16 data. It can be seen that the Columbia data changes more sensitively. This is because the Columbia data is closer to the network input size of the present invention, so most of the high-frequency tampering traces can be retained, making it very sensitive to attacks; while the NIST16 data has a very high resolution, and drastic downsampling will cause the tampering traces to be destroyed to a large extent, so it is not sensitive to JPEG compression.

[0199] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A dual-stream U-Net image tampering detection network system, characterized in that: The network consists of two parts: encoder and decoder. The encoder includes two data transmission paths: RGB stream and Noise stream, each of which includes the 1st to 5th data stream layers; the decoder includes the 1st to 6th decoding layers; A high-pass filter is provided between the first data stream layer of the RGB stream and the first data stream layer of the Noise stream, and an input addition jump channel is provided, leading to the sixth decoding layer. The outputs of the first data stream layer of the RGB stream and the first data stream layer of the Noise stream are provided with an addition jump channel, leading to the fifth decoding layer. The outputs of the second layer of the RGB stream and the second layer of the Noise stream are provided with an addition jump channel, leading to the fourth decoding layer. The outputs of the third layer of the RGB stream and the third layer of the Noise stream are provided with an addition jump channel, leading to the third decoding layer. The outputs of the fourth layer of the RGB stream and the fourth layer of the Noise stream are provided with an addition jump channel, leading to the second decoding layer. The outputs of the fifth layer of the RGB stream and the fifth layer of the Noise stream are added and input to the first decoding layer. The internal coding structure of the RGB stream and the Noise stream includes: The first data stream layer of the RGB stream includes the convolution kernel conv; the batch normalization data stream layer norm; the activation function Relu; the first data stream layer of the Noise stream includes the convolution kernel conv; the batch normalization data stream layer norm; the activation function Relu; The second data stream layer of the RGB stream includes a maximum pooling data stream layer pool; a dense connection module denseblock; the second data stream layer of the Noise stream includes a maximum pooling data stream layer pool; a dense connection module denseblock; The third data stream layer of the RGB stream includes a transposition module transition; a dense link module denseblock; the third data stream layer of the Noise stream includes a transposition module transition; a dense link module denseblock; The 4th data stream layer of the RGB stream includes a transposition module transition; a dense link module denseblock; the 4th data stream layer of the Noise stream includes a transposition module transition; a dense link module denseblock; The fifth data stream layer of the RGB stream includes a transposition module transition; the fifth data stream layer of the Noise stream includes a transposition module transition; The specific parameters of the internal coding structure of the RGB stream and the Noise stream and the data transmission process include: The data of the first data stream layer of the input RGB stream is an image of size N*N*3. After the first data stream layer of the RGB stream, the convolution kernel conv with a width of 7, a height of 7, a number of channels of 64, and a step size of 2; the batch normalization data stream layer norm; the activation function Relu, the final output size is The RGB stream feature map; the data input to the first data stream layer of the Noise stream is a high-frequency noise residual map of size N*N*3, which passes through the convolution kernel conv with a width of 7, a height of 7, a number of channels of 64, and a step size of 2 in the first data stream layer of the Noise stream; the batch normalization data stream layer norm; the activation function Relu, and the final output size is Noise flow feature map; The data of the second data stream layer of the input RGB stream is the size of the first data stream layer of the RGB stream. The RGB stream feature map is passed through the RGB stream second data stream layer with a window width of 3, a height of 3, and a maximum pooling data stream layer with a step size of 2; the dense link module denseblock, and the final output size is RGB stream feature map; the data input to the second data stream layer of the Noise stream is the size of the first data stream layer of the Noise stream The Noise stream feature map is passed through the Noise stream second data stream layer with a window width of 3, a height of 3, and a maximum pooling data stream layer with a step size of 2; the dense link module denseblock, and the final output size is Noise flow feature map; The data of the third data stream layer of the input RGB stream is the size of the second data stream layer of the RGB stream. The RGB stream feature map is passed through the 3rd data stream layer of the RGB stream, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense block module has a final output size of RGB stream feature map; the data input to the third data stream layer of the Noise stream is the size of the second data stream layer from the Noise stream The Noise stream feature map is passed through the Noise stream third data stream layer, which reduces the number of feature channels by half and the spatial dimension of the feature map by half; the dense block module has a final output size of Noise flow feature map; The data of the 4th data stream layer of the input RGB stream is the data from the 3rd data stream layer of the RGB stream. The RGB stream feature map is passed through the 4th data stream layer of the RGB stream, which reduces the number of feature channels by one half and the spatial dimension of the feature map by one half; the dense link module denseblock, and the final output size is RGB stream feature map; the data input to the 4th data stream layer of the Noise stream is the size of the 3rd data stream layer of the Noise stream. The Noise stream feature map is passed through the Noise stream 4th data stream layer, which reduces the number of feature channels by half and the spatial dimension of the feature map by half; the dense block module has a final output size of Noise flow feature map; The data of the 5th data stream layer of the input RGB stream is the size of the 4th data stream layer of the RGB stream. The RGB stream feature map is transformed through the 5th data stream layer of the RGB stream, which reduces the number of feature channels by half and the spatial dimension of the feature map by half. The final output size is 512 RGB stream feature map; the data input to the 5th data stream layer of the Noise stream is the size of the 4th data stream layer of the Noise stream The Noise stream feature map is transformed through the Noise stream 5th data stream layer, which reduces the number of feature channels by half and the spatial dimension of the feature map by half. The final output size is Noise flow feature map.

2. According to claim 1, a dual-stream U-Net image tampering detection network system is characterized in that: The internal decoding structure of the first to sixth decoding layers includes: The first decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer, and the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer; The second decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer; The third decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer; The 4th decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer; The 5th decoding layer includes an upsampling feature extraction module up_conv_block and a feature extraction module conv_block, wherein the upsampling feature extraction module up_conv_block includes a bilinear upsampling layer, a convolution kernel conv and an Inplace-ABN layer; the feature extraction module conv_block includes two convolution kernels conv and an Inplace-ABN layer; The sixth decoding layer includes the convolution kernel conv and the Sigmiod activation function layer.

3. According to claim 2, a dual-stream U-Net image tampering detection network system is characterized in that: The specific parameters of the internal decoding structure and the data transmission process of the first to sixth decoding layers include: The output size is The RGB stream feature map and the Noise stream feature map are added in the channel and input into the upsampling feature extraction module up_conv_block of the first decoding layer. After the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 256, and a step size of 1, the Inplace-ABN layer, the output size is The upsampled feature map; then the size of the RGB stream 4th layer output is The size of the RGB stream feature map and the Noise stream 4th layer output is The Noise stream feature maps are added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a channel number of 1024, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of The size of the output of the first decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the second decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 512, and a step size of 1, the Inplace-ABN layer, and the output size is The upsampled feature map; then the size of the RGB stream layer 3 output is The size of the RGB stream feature map and the Noise stream layer 3 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the feature extraction module conv_block with two convolution kernels of width 3, height 3, number of channels 512, and step size 1, and the Inplace-ABN layer. The output size is The feature map of The size of the second decoding layer output is The feature map is input into the upsampling feature extraction module up_conv_block of the 3rd decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block is passed through the convolution kernel conv with a width of 3, a height of 3, a number of channels of 256, and a step size of 1. The output size is The upsampled feature map; then the size of the RGB stream second layer output is The size of the RGB stream feature map and the Noise stream layer 2 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a channel number of 256, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of The size of the output of the third decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the 4th decoding layer, and the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, a number of channels of 64, and a step size of 1, the Inplace-ABN layer, and the output size is The upsampled feature map; then the size of the RGB stream first layer output is The size of the RGB stream feature map and the Noise stream layer 1 output is The Noise stream feature map is added across the channels, and the resulting size is The upsampled feature maps are connected in the channel dimension to obtain a size of The feature map of is passed through the convolution kernel conv with a width of 3, a height of 3, a number of channels of 64, and a step size of 1 in the feature extraction module conv_block, and the Inplace-ABN layer. The output size is The feature map of The size of the output of the 4th decoding layer is The feature map is input into the upsampling feature extraction module up_conv_block of the 5th decoding layer, and passes through the bilinear upsampling layer of the upsampling feature extraction module up_conv_block, the convolution kernel conv with a width of 3, a height of 3, the number of channels of 32, and a step size of 1, the Inplace-ABN layer, and the output is an upsampling feature map of size N*N*3; then the RGB image of size N*N*3 of the first layer of the input RGB stream and the high-frequency noise residual map of size N*N*3 of the first layer of the input Noise stream are added, and connected with the obtained upsampling feature map of size N*N*3 in the channel dimension to obtain a feature map of size N*N*(32+3), which passes through two convolution kernels conv with a width of 3, a height of 3, the number of channels of 16, and a step size of 1 of the feature extraction module conv_block, the Inplace-ABN layer, and the output is a feature map of size N*N*16; The feature map of size N*N*16 output by the 5th decoding layer is input into the 6th decoding layer, and passes through the convolution kernel conv with width 1, height 1, number of channels 1, and step size 1 to output a feature map of size N*N*1; then it passes through the Sigmiod activation function to obtain a single-channel probability binary map of size N*N*1.

4. An image tampering detection method using the dual-stream U-Net image tampering detection network system according to any one of claims 1 to 3, characterized in that: The following steps are involved: S1: Obtain reference image data and tampered image data to be detected, wherein the reference image data includes a reference tampered image and its corresponding tampered annotated image, divide the reference image data into reference image training data and reference image verification data, and pre-process the reference image training data, the reference image verification data and the tampered image data to be detected respectively; S2: Initialize the dual-stream U-Net image tampering detection network system parameters; S3: Use the preprocessed benchmark image training data and verification data to train the two-stream U-Net image tampering detection network system, confirm the optimal two-stream U-Net image tampering detection network system, input the tampered image data to be tested into the optimal two-stream U-Net image tampering detection network system, and output a single-channel probability mask map as the tampering detection result.

5. The image tampering detection method according to claim 4, characterized in that: In step S1, the reference image data is divided into reference image training data and reference image verification data in a ratio of 9:

1. The preprocessing of the reference image training data includes the following steps: S11-1: simultaneously reading the reference tampered image and its corresponding tampered annotated image in the reference image training data, and cropping the image at a random area cropping rate between 90% and 100%; S12-1: The image obtained in step S11-1 is first randomly flipped horizontally with a probability of 50%, and then randomly flipped vertically with a probability of 50%; S13-1: The image obtained in step S12-1 is scaled by bicubic interpolation to unify the image size and rotated by a fixed angle, wherein the fixed angle is randomly selected from 0°, 90°, 180° or 270°; S14-1: Perform normalization on the reference tampered image obtained in step S13-1. The normalization formula is: I (u,v) is the pixel value at coordinate (u, v) in the reference tampered image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the normalized reference tampered image; The corresponding tampered annotated image is normalized, and the normalization formula is: G (u,v) To tamper with the pixel value at coordinate (u,v) in the annotated image, G' (u,v) The pixel value at the coordinate (u, v) in the tampered labeled image after normalization; S15-1: Connect the reference tampered image and its corresponding tampered annotated image obtained in step S14-1 in the batch dimension respectively; S16-1: Shuffling the reference tampered images and their corresponding tampered annotated images within the same batch size in step S15-1 respectively; The preprocessing of the reference image verification data comprises the following steps: S11-2: Simultaneously reading the reference tampered image and its corresponding tampered annotated image in the reference image verification data; S12-2: The reference tampered image obtained in step S11-2 and its corresponding tampered annotated image are scaled by bicubic interpolation. After unifying the image sizes, the reference tampered image is subjected to random horizontal flip transformation, random vertical flip transformation, and rotation transformation at a fixed angle randomly selected from 0°, 90°, 180°, and 270°, generating a total of 16 transformation conditions, corresponding to 16 groups of reference tampered images, and similarly, 16 groups of tampered annotated images are obtained; S13-2: Perform normalization operation on the 16 groups of reference tampered images obtained in step S12-2. The normalization formula is: I (u,v) is the pixel value at coordinate (u, v) in the reference tampered image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the normalized reference tampered image; The corresponding 16 groups of tampered annotated images are normalized, and the normalization formula is: G (u,v) To tamper with the pixel value at coordinate (u,v) in the annotated image, G' (u,v) The pixel value at the coordinate (u, v) in the tampered labeled image after normalization; The preprocessing of the image data to be detected for tampering comprises the following steps: S11-3: reading the tampered image in the tampered image data to be tested, scaling the tampered image by using bicubic interpolation, unifying the image size, and then performing random horizontal flip transformation, random vertical flip transformation, and randomly selecting a fixed angle from 0°, 90°, 180°, and 270° for rotation transformation on the tampered image in sequence, generating a total of 16 transformation situations, corresponding to 16 groups of tampered images; S12-3: Perform normalization operation on the 16 groups of tampered images obtained in step S11-3, and the normalization formula is: I (u,v) To tamper with the pixel value at the coordinate (u, v) in the image, I mean Represents the mean pixel value of the current channel, I std Represents the standard deviation of the current channel pixel, I' (u,v) is the pixel value at coordinate (u, v) in the tampered image after normalization.

6. The image tampering detection method according to claim 4, characterized in that: In the step S2, the dual-stream U-Net image tampering detection network system parameters are initialized by using the weights of the ImageNet open source data set to respectively initialize the network parameters of the RGB stream and the Noise stream, and prepare an N*N*3 sized RGB image and an N*N*3 sized high-frequency noise residual map obtained by passing through an SRM high-pass filter.

7. The image tampering detection method according to claim 6, characterized in that: In step S3, the training of the dual-stream U-Net image tampering detection network system using the preprocessed reference image training data and verification data includes the following steps: S31: Input the reference tampered image in the preprocessed reference image training data into the dual-stream U-Net image tampering detection network system to output a single-channel probability mask map, and use the loss function to calculate the loss of the obtained single-channel probability mask map and the tampered annotation image corresponding to the original reference tampered image, and then back-propagate the gradient according to the calculation result to update the dual-stream U-Net image tampering detection network system parameters, where the loss function is Dice loss and Focal loss, and the loss function expression is as follows: L Total (M g ,M o )=L Dice (M g ,M o )+L Focal (M g ,M o ) Among them, (M g ,M o ) is the ground-truth and predicted probability map within a batch, and Dice loss and Focal loss are specifically expressed as: Among them, M g (i,j) and M o (i,j) represents the pixel value located at (i,j); S32: Freeze the parameters of the dual-stream U-Net image tampering detection network system obtained in step S31, input the reference tampered image in the preprocessed reference image verification data into the dual-stream U-Net image tampering detection network system, output 16 groups of single-channel probability mask images, perform inverse transformation on 15 groups of single-channel probability mask images, and then average the 16 groups of single-channel probability mask images, and use the average value as the final output single-channel probability mask image, and use the loss function to calculate the loss of the tampered annotated image corresponding to the original reference tampered image; S33: Steps S31 and S32 are continuously repeated. When the loss of the loss function tends to be stable, it is considered that the loss has converged and the training is stopped. At this time, the parameters of the dual-stream U-Net image tampering detection network system are optimal.

Images