A deep multi-scale fusion image forensics method based on PRNU

Abstract

The present invention discloses a deep multi-scale fusion image forensics method based on PRNU. First, the noise residual and the corresponding camera model of the image to be detected are calculated, and the obtained camera PRNU is preprocessed to suppress non-unique noise; secondly, the tampering probability map of different scales is calculated through the correlation between the camera PRNU and the PRNU of the image to be detected; then, a fully connected neural network is constructed and trained to fuse the tampering probability maps of different scales; preferably, the connected areas with too small areas are deleted and morphological expansion is used to compensate for the erosion of the boundaries. The present invention can make it possible to detect various types of tampering in a certain area of ​​the image after the PRNU of the area is tampered, and it has improvements in pixel level, image level and robustness experiments.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia forensics, and in particular relates to a PRNU-based deep multi-scale fusion image forensics method. Background Art

[0002] As more and more software for tampering with images is produced, people can easily tamper with images without leaving obvious traces. This phenomenon has aroused great concern about image security in society. Therefore, it is urgent to determine whether an image has been tampered with. Among them, photo response non-uniformity (PRNU) is a more promising direction. The reason for the formation of PRNU is the defects of the silicon crystal of the camera sensor during the manufacturing process. PRNU is usually also called camera fingerprint, which can be used for image tampering detection and positioning.

[0003] The PRNU-based tampering localization algorithm is roughly divided into the following steps. First, the block in the PRNU of the image to be detected is compared with the block corresponding to the camera PRNU, the normalized correlation between the two is calculated, and then compared with a threshold. Finally, if the correlation between the two is lower than the threshold, it is determined that the area where the block is located has been tampered with.

[0004] According to the above PRNU positioning method, its essence is to determine whether PRNU exists or not. Therefore, PRNU-based technology can be used to detect a variety of different types of tampering, such as copy and paste, splicing, repair, etc. Of course, if the PRNU corresponding to the tampered position has not changed, this tampering is not suitable for detection using PRNU, such as changing the color of the car on the road in the image. PRNU is also limited. It is a weak multiplicative noise that is unchanged in time and independent of the scene. In dark, saturated, and textured areas, PRNU is often so weak that the correlation is very low, so even in areas that have not been tampered with, it may be misjudged as a tampered area. When using PRNU for tampering positioning, it is difficult to detect small tampering. This is because PRNU uses sliding window analysis when performing tampering positioning, so the size of the sliding window affects the resolution of tampering positioning.

[0005] In order to make up for the defect that PRNU is a weak noise, many researchers have studied PRNU extraction. Extracting the PRNU of the source camera can be divided into the following three stages, namely, filtering stage, joint stage, and enhancement stage. In the filtering stage, the noise residual can be obtained by the difference between the images before and after filtering, mainly using wavelet filtering, BM3D filtering, content adaptive filtering, etc. In the joint stage, the noise residuals in multiple images taken by the same camera are combined to obtain PRNU noise, which can mainly be combined using averaging method, maximum likelihood estimation method, weighted average method, etc. In the enhancement stage, the common noise in the image that cannot be used for forensics will be further removed, thereby further enhancing the PRNU noise. The methods used include Fourier transform domain preprocessing, spectral equalization method, etc. Some researchers have proposed the conclusion that PRNU is very weak in dark, saturated, and textured areas. At the same time, in order to reduce the impact of this defect, it is proposed to use a detector to predict correlation. However, most algorithms use a 128*128 sliding window for all processing, which makes these algorithms unsuitable for detecting small tampering.

[0006] In order to detect minor tampering, some algorithms propose to first segment the image and then turn to an object-oriented process to statistically correlate. The disadvantage of this method is that the segmentation in the paper uses a manual-assisted method, not a fully automatic method. Secondly, this method is not very effective for other types of tampering without obvious boundaries. Some researchers have also proposed a method of multi-scale fusion of tampered images, which locates tampering by fusing the results of small sliding windows and large sliding windows. The advantage of this is that it can use the advantages of small windows to detect minor tampering, and use large windows to fully obtain detection statistics related information, but this method has a high false alarm rate and sets complex parameters. In addition to fusing tampering probability maps of different sizes, there are also methods that fuse two features, but they are only useful for a single type of tampering and have no effect on other types of tampering. It can be seen that the existing tampering location methods all have some problems. Summary of the invention

[0007] Purpose of the invention: In view of the problems existing in the prior art, the purpose of the present invention is to provide a PRNU multi-scale fusion tampering positioning method based on a neural network, which can improve the accuracy of tampering positioning in the prior art, reduce the false alarm rate, and at the same time be robust to some common post-processing operations.

[0008] Technical solution: The deep multi-scale fusion image forensics method based on PRNU described in the present invention comprises the following steps:

[0009] (1) Calculate the noise residual of the image to be detected and the corresponding camera model, and preprocess the obtained camera PRNU to suppress non-unique noise;

[0010] (2) Calculate the tampering probability map of different scales through the correlation between the camera PRNU and the PRNU of the image to be detected;

[0011] (3) Construct and train a fully connected neural network to fuse tampering probability maps of different scales;

[0012] (4) Delete connected regions that are too small and use morphological dilation to compensate for boundary erosion.

[0013] Furthermore, the step (1) is implemented by the following formula:

[0014] Y=(1+K)X+θ

[0015] Where Y represents the camera output image containing PRNU noise and other random noises, X represents the pure image without any noise, K represents the multiplicative factor of PRNU, and θ represents the sum of all random noises except PRNU noise;

[0016] Use the residual image W to suppress the main signal X:

[0017]

[0018] in, It represents the denoising form of Y obtained by using a denoising filter. The denoising filters commonly used are wavelet filtering or BM3D, and η represents the denoising error The sum of the small difference terms (XY)K and θ;

[0019] For the image to be detected, the noise residual W is directly used to approximate the PRNU of the image. For the reference PRNU of the target camera, the noise residual W1,…W corresponding to the M images taken by the camera is first calculated. M , and then the PRNU factor of the camera is obtained based on the maximum likelihood estimation of these noise residuals

[0020]

[0021] Where M represents the number of images, W i represents the noise residual block centered at pixel i, Y i represents the image patch centered at pixel i.

[0022] Furthermore, the step (2) is implemented by the following formula:

[0023]

[0024] Among them, q iis the normalized correlation coefficient between the PRNU block of the image and the PRNU block of the camera, r w is an image PRNU block of size w*w centered on pixel i. Similarly, is the PRNU block of the camera, Represents r w The mean of Indicates z w The mean of

[0025] After calculating the parameters of each camera model, six windows of size 32, 48, 64, 96, 128, and 192 are used to calculate the tampering probability map C of the image to be detected. 1 ,C 2 ,C 3 ,C 4 ,C 5 ,C 6 :

[0026]

[0027] Among them, σ0 and σ1 represent the variance of the test statistics of the hypothesis that PRNU does not exist and the hypothesis that PRNU exists, respectively; Represents the correlation predicted by the predictor.

[0028] Furthermore, the fully connected neural network in step (3) is composed of an input layer, three hidden layers and an output layer; wherein the input layer is 6 neural units, the first hidden layer is 64 units, the second hidden layer is 32 units, the third hidden layer is 16 units, and the final output layer is one unit; a dynamically weighted binary cross entropy is used as the loss function, and the Adam optimization algorithm is used to adjust the model weights.

[0029] Furthermore, the step (3) is implemented by the following formula:

[0030] Y=f1(f2(w3*f2(w2*f2(w1*X+b1)+b2)+b3))

[0031] Among them, X represents a vector composed of 6 input units, Y represents a binary value, and w i represents the corresponding weight, b i represents the corresponding bias, f1 and f2 are activation functions, which are defined as:

[0032]

[0033] f2(x)=relu(x)=max(0,x).

[0034] Furthermore, the implementation process of step (4) is as follows:

[0035] The area of ​​each connected tampered region in the decision graph is counted, and the regions with an area less than or equal to 80 pixels are removed; a disk-shaped structural unit with a radius of 4 is used to compensate for the erosion of the detection boundary.

[0036] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are: the present invention realizes the location of image tampering. Since the neural network has a powerful data learning ability compared with the random field model, the present invention can better integrate multi-scale information, achieve more accurate decision-making, and reduce the false alarm rate; at the same time, the present invention has improved image-level accuracy and has good robustness for some common post-processing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a strategy diagram for generating training samples of the present invention;

[0038] Figure 2 It is a neural network framework diagram of the vehicle picking up method of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to the accompanying drawings.

[0040] The present invention proposes a PRNU multi-scale fusion tampering positioning method based on a neural network, which includes two parts: calculating a tampering probability map and fusion. The specific implementation process is as follows:

[0041] Step 1: Calculate the noise residual of the image to be detected and the corresponding camera model, and preprocess the obtained camera PRNU to suppress non-unique noise.

[0042] The exact camera output model for acquiring an image from a digital camera is quite complex, so for convenience, a simplified image acquisition model is given:

[0043] Y=(1+K)X+θ

[0044] Where Y represents the camera output image containing PRNU noise and other random noises, X represents the pure image without any noise, K represents the multiplicative factor of PRNU, and θ represents the sum of all random noises except PRNU noise.

[0045] PRNU noise, KX is very weak. In order to better improve the signal-to-noise ratio, the residual image W is used to suppress the main signal X:

[0046]

[0047] in, It represents the denoising form of Y obtained by using a denoising filter. The denoising filters commonly used are wavelet filtering or BM3D, and η represents the denoising error The sum of the small difference terms (XY)K and θ.

[0048] For the image to be detected, since there is only one image, the noise residual W can be directly used to approximate the PRNU of the image. For the reference PRNU of the target camera, first calculate the noise residual W1,…W corresponding to the M images taken by the camera. M , and then the PRNU factor of the camera is obtained based on the maximum likelihood estimation of these noise residuals

[0049]

[0050] Where M represents the number of images, W i represents the noise residual block centered at pixel i, Y i represents the image patch centered at pixel i.

[0051] The estimated PRNU factor contains system information that is present in every image. These noises are not unique to the sensor. Cameras of the same brand or imaging sensors with the same design structure share these noises, so the PRNU estimated by two different cameras may be slightly correlated, which will increase the error recognition rate. Therefore, before using PRNU, the estimated camera PRNU needs to be preprocessed to suppress these noises. The following two methods are mainly used for preprocessing. The first method is the zero mean operation ZM(K), so that the mean of each row and column of the processed PRNU factor is 0. The second method is to convert ZM(K) to the Fourier domain, and then pass it through a Wiener filter to retain only the noise component.

[0052] Step 2: Calculate the tampering probability map of different scales through the correlation between the camera PRNU and the PRNU of the image to be detected.

[0053] After obtaining the PRNU of the camera and the image to be detected and preprocessing, the detection problem can be expressed as a binary hypothesis test: H0 represents the hypothesis that PRNU does not exist, and H1 represents the hypothesis that PRNU exists.

[0054]

[0055] Among them, σ0 and σ1 represent the variance of the H0 and H1 test statistics respectively, q i represents the normalized correlation coefficient, Represents the correlation predicted by the predictor.

[0056] Then, according to the Neyman-Person method, the threshold γ1 is selected to obtain the desired error acceptance rate. Since PRNU noise is less in dark, saturated, and textured areas, it is easy to generate false positives in these areas. Therefore, a predictive correlator is used to predict its correlation to alleviate this defect. The predictor mainly analyzes the characteristics of each block and sets the threshold γ2 to separate the reliable area from the problem area. Then, the correlation q can be calculated. i Correlation with the prediction of the predictor Compare with the threshold to determine whether the block has been tampered with. During the detection process, for each sliding window, the normalized correlation coefficient q between the PRNU block of the image and the PRNU block of the camera is calculated. i :

[0057]

[0058] Among them, r w is an image PRNU block of size w*w centered on pixel i. Similarly, is the PRNU block of the camera, Represents r w The mean of Indicates z w The mean of .

[0059] After calculating the parameters of each camera model, six windows of size 32, 48, 64, 96, 128, and 192 are used to calculate the tampering probability map C of the image to be detected. 1 ,C 2 ,C 3 ,C 4 ,C 5 ,C 6 :

[0060]

[0061] Where σ0 and σ1 represent the variance of the test statistic for the hypothesis that PRNU does not exist and the hypothesis that PRNU exists, respectively. i represents the normalized correlation coefficient, Represents the correlation predicted by the predictor.

[0062] Although the application of deep learning to multi-scale fusion is a conventional technology, the use of deep learning technology in the multi-scale tampering localization algorithm based on PRNU is still an innovative method. The biggest possible reason for this is the lack of data. For the tampering localization algorithm based on PRNU, the data set used is not only the tampered image, but also the tampered image of the source camera PRNU. It is precisely because of this prerequisite that the number of tampering data sets is far from meeting the requirements of deep learning training. In order to solve this problem, such as Figure 1 As shown in FIG. 1 , the present invention produces some tampered images with known source cameras and real tampering locations, and obtains tampering probability maps C of 6 different scales. 1 ,C 2 ,C 3 ,C 4 ,C 5 ,C 6 . Then, the tampering probability map and the real tampering location corresponding to each tampered image are divided into blocks using 2*2 non-overlapping sliders, and the corresponding small blocks are taken as the data, among which the 6 average values ​​corresponding to the 6 tampering probability maps are used as 6 data avg1, avg2, avg3, avg4, avg5, avg6. The average value corresponding to the real tampering location is compared with 255 / 2, and the small blocks larger than 255 / 2 are classified as tampered blocks avg gt =1, classify the small blocks less than or equal to 255 / 2 into the original blocks avg gt = 0. Finally, 6 data and one label are used as a training sample, namely (avg1, avg2, avg3, avg4, avg5, avg6, avg gt ).

[0063] Step 3: Build and train a fully connected neural network to fuse tampering probability maps of different scales.

[0064] The fully connected neural network model is composed of an input layer, three hidden layers and an output layer, wherein the input layer is 6 neural units, the first hidden layer is 64 units, the second hidden layer is 32 units, the third hidden layer is 16 units, and the final output layer is one unit; the algorithm of the present invention uses the dynamic weighted binary cross entropy as the loss function, uses the Adam optimization algorithm to adjust the model weights, and sets the initial learning rate to 0.001; it is set that after every 3 rounds, if the model performance cannot be improved, the learning rate will be reduced by 50%. This operation of reducing the learning rate is used in conjunction with early stopping; the practice is that when the accuracy of the validation set is not improved in 10 consecutive rounds, it is considered that the accuracy is no longer improved, and early stopping is performed; when using the trained network for testing, the output probability value needs to be compared with the threshold value of 0.75, if it is greater than 0.75, it is considered to be a tampered area, and if it is less than 0.75, it is considered to be a real area.

[0065] like Figure 2 As shown, the present invention uses a fully connected neural network module to fuse the six tampering probability values. The learned weights are used to calculate the corresponding output values, where 0 represents a real block and 1 represents a tampered block. The present invention uses the following fusion formula to predict the final output value:

[0066] Y=f1(f2(w3*f2(w2*f2(w1*X+b1)+b2)+b3))

[0067] Among them, X represents a vector composed of 6 input units, Y represents a binary value, and w i represents the corresponding weight, b i represents the corresponding bias. f1 and f2 are activation functions, which are defined as:

[0068]

[0069] f2(x)=relu(x)=max(0,x)

[0070] After the fully connected neural network is trained, it needs to be used to fuse the tampering probability image. The process of using a neural network for fusion is similar to the production of a training data set. For each tampering probability map, a 2*2 window is used to slide, and the 6 means corresponding to the 6 tampering probability maps are calculated and used as the input of the neural network. Then the output is assigned to the corresponding position in the fused decision map. If it is a tampered image, the corresponding 2*2 block in the decision map is assigned a pixel value of 255. If it is a real image, the corresponding 2*2 block in the decision map is assigned a pixel value of 0. After the 2*2 slider traverses the image, a tampering decision map will be obtained.

[0071] Step 4: Delete the connected regions that are too small and use morphological dilation to compensate for the erosion of the boundary.

[0072] Since it is meaningless to make too small tampering in the image, some unconnected scattered small areas can be removed. The present invention first counts the area of ​​each connected tampering area in the decision graph, and then removes the area with an area less than or equal to 80 pixels. Further, a disc-shaped structural unit with a radius of 4 is used to compensate for the erosion of the detection boundary. The above operation is performed on the image to be detected to complete the tampering location of the image.

[0073] The present invention is end-to-end. Compared with the random field model, the neural network has a stronger data learning ability, which enables better fusion of multi-scale information to achieve more accurate decision-making, that is, reduce the false alarm rate. In addition, the image-level accuracy has been improved, and it also has good robustness for some common post-processing operations.

Claims

1. A deep multi-scale fusion image forensics method based on PRNU, characterized in that: The following steps are involved: (1) Calculate the noise residual of the image to be detected and the corresponding camera model, and preprocess the obtained camera PRNU to suppress non-unique noise; (2) Calculate the tampering probability map of different scales through the correlation between the camera PRNU and the PRNU of the image to be detected; (3) Construct and train a fully connected neural network to fuse tampering probability maps of different scales; (4) Delete the connected areas that are too small and use morphological dilation to compensate for the erosion of the boundary; The fully connected neural network in step (3) is composed of an input layer, three hidden layers and an output layer; wherein the input layer is 6 neural units, the first hidden layer is 64 units, the second hidden layer is 32 units, the third hidden layer is 16 units, and the final output layer is one unit; a dynamically weighted binary cross entropy is used as the loss function, and the Adam optimization algorithm is used to adjust the model weights; The step (3) is implemented by the following formula: B=f1(f2(w3*f2(w2*f2(w1*A+b1)+b2)+b3)) Among them, A represents a vector composed of 6 input units, B represents a binary value, w1, w2, w3 represent the corresponding weights, b1, b2, b3 represent the corresponding biases, f1 and f2 are activation functions, which are defined as: f2(x)=relu(x)=max(0,x).

2. The PRNU-based deep multi-scale fusion image forensics method according to claim 1, characterized in that: The step (1) is implemented by the following formula: Y=(1+K)X+θ Where Y represents the camera output image containing PRNU noise and other random noises, X represents the pure image without any noise, K represents the multiplicative factor of PRNU, and θ represents the sum of all random noises except PRNU noise; Use the residual image W to suppress X: in, represents the denoised form of Y obtained using a denoising filter. The denoising filter used is wavelet filtering or BM3D, and η represents the denoising error The sum of the small difference terms (XY)K and θ; For the image to be detected, the noise residual W is directly used to approximate the PRNU of the image. For the reference PRNU of the target camera, the noise residual W1,…W corresponding to the M images taken by the camera is first calculated. M , and then the PRNU factor of the camera is obtained based on the maximum likelihood estimation of these noise residuals Where M represents the number of images, W i represents the noise residual block centered at pixel i, Y i represents the image patch centered at pixel i.

3. The deep multi-scale fusion image forensics method based on PRNU according to claim 1 is characterized in that: The step (2) is implemented by the following formula: Among them, q i is the normalized correlation coefficient between the PRNU block of the image and the PRNU block of the camera, r w is an image PRNU block of size w*w centered on pixel i. Similarly, is the PRNU block of the camera, Represents r w The mean of Indicates z w The mean of After calculating the parameters of each camera model, six windows of size 32, 48, 64, 96, 128, and 192 are used to calculate the tampering probability map C of the image to be detected. 1 ,C 2 ,C 3 ,C 4 ,C 5 ,C 6 : Among them, σ0 and σ1 represent the variance of the test statistics of the hypothesis that PRNU does not exist and the hypothesis that PRNU exists, respectively; Represents the correlation predicted by the predictor.

4. The PRNU-based deep multi-scale fusion image forensics method according to claim 1, characterized in that: The implementation process of step (4) is as follows: The area of ​​each connected tampered region in the decision graph is counted, and the regions with an area less than or equal to 80 pixels are removed; a disk-shaped structural unit with a radius of 4 is used to compensate for the erosion of the detection boundary.

Images