An Image Tampering Location Method, Device and Terminal Based on Anomaly Detection Technology

Abstract

The present invention discloses an image tampering localization method, device and terminal based on anomaly detection technology. Among them, the above method includes: obtaining an unknown image; controlling the extraction of noise from the unknown image to obtain a noise image, and splitting the noise image into a number of mutually related noise blocks; sequentially inputting the noise blocks into a pre-constructed anomaly detection network for training; when the obtained loss difference is less than a preset loss difference, completing the training and freezing the detection parameters of the current anomaly detection network; sequentially inputting the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, and calculating the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block; generating a heat map based on the anomaly scores, extracting the pixel points in the heat map that are higher than a preset threshold, generating a predicted tampering localization map, and outputting it. It realizes image tampering localization that does not rely on data, has wide applicability and high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly detection, and particularly to an image tampering localization method, device and terminal based on anomaly detection technology. Background Art

[0002] Nowadays, people come into contact with a large amount of text, pictures and videos every day, and people can obtain the information they want to know from them. However, it is impossible to judge whether the information carried by these carriers is true only by human eyes. With the progress of image editing technology, a large number of tampering software have emerged continuously. These tampering software can help people modify a certain pattern of an image or change an important field in the image conveniently and quickly. The development of these new technologies brings convenience to life while also bringing potential security risks. For example, criminals in court use tampering software to modify photos to forge evidence to help themselves defend, and speculators use tampered photos of competitors' scandals to achieve a favorable competitive position for themselves.

[0003] In the prior art, the detection and localization methods for unknown images are mainly divided into supervised schemes and unsupervised schemes. Among them, the supervised scheme relies more on neural networks and a large amount of training data, and there are also certain requirements for the predicted images; while the unsupervised scheme generally has low accuracy and poor localization effect. It can be seen that there is still a lack of unknown image detection and localization schemes with a wide application range, small demand for data volume and high accuracy in the prior art.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the Invention

[0005] Aiming at the problems in the prior art that the supervised image tampering localization scheme is too dependent on the data set and has a small application range, and the unsupervised image tampering localization scheme has low accuracy. There is a lack of an image tampering localization scheme that does not depend on the data set, has a wider applicability and has a higher accuracy. The present invention provides an image tampering localization method based on anomaly detection technology. The unknown image to be predicted is preprocessed by noise extraction and splitting to obtain several images that can be used for training, and then input into a pre-constructed anomaly detection network to obtain an anomaly detection network trained by the unknown image. Then, the noise blocks obtained by splitting the unknown image are input into the anomaly detection network to obtain an anomaly score for evaluating whether the corresponding position of the image is tampered, and then a heat map and an image tampering localization map are generated. This method trains and tests a single image, greatly reducing the dependence on the data set and expanding the application range. At the same time, compared with some deep networks, the network of this method is more lightweight, occupies less hardware resources, is easier to be deployed into daily life, and also provides good image tampering localization accuracy.

[0006] To achieve the above technical effects, a first aspect of the present invention provides an image tampering localization method based on anomaly detection technology, wherein the method includes:

[0007] Obtain an unknown image for which image tampering localization is required;

[0008] Control the extraction of noise from the unknown image to obtain a noise image, and split the noise image into a number of mutually related noise blocks;

[0009] Input the noise blocks into a pre-constructed anomaly detection network for training in sequence;

[0010] When the obtained loss difference is less than a preset loss difference, complete the training and freeze the detection parameters of the current anomaly detection network;

[0011] Input the noise blocks into the trained anomaly detection network in sequence to obtain reconstructed blocks, and calculate the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block;

[0012] Generate a heat map based on the anomaly scores, extract the pixel points in the heat map that are higher than the preset threshold, generate a predicted tampering localization map, and output it.

[0013] Optionally, before the step of obtaining an unknown image for which image tampering localization is required, it includes:

[0014] Construct an anomaly detection network, including an encoder, a fully connected layer, a fitting Gaussian distribution layer, and a decoder.

[0015] Optionally, before the step of obtaining an unknown image for which image tampering localization is required, it further includes:

[0016] Set a preset loss difference for determining whether to stop network training;

[0017] Set a preset threshold.

[0018] Optionally, the step of controlling the extraction of noise from the unknown image to obtain a noise image, and splitting the noise image into a number of mutually related noise blocks includes:

[0019] Extract the noise features of the unknown image through Noiseprint to obtain a noise image;

[0020] Split the noise image, and control to obtain a noise block with a size of 48*48 pixels every 16 pixel units.

[0021] Optionally, the step of when the obtained loss difference is less than a preset loss difference, completing the training and freezing the detection parameters of the current anomaly detection network includes:

[0022] Calculate the loss difference during the training of the anomaly detection network through the following formula:

[0023] Total loss: L = L1 + λ * L2,

[0024] Among them, the image reconstruction loss:

[0025] KL divergence loss:

[0026] Among them, n represents the total number of noise blocks in a training batch, X i represents the original image block corresponding to the current i, X' i represents the block reconstructed after the original image block passes through the network, J represents the number of Gaussian models fitted in the middle layer of the network, u j and σ j respectively represent the parameter mean and standard deviation of the jth Gaussian model, and λ is a hyperparameter with a value between 0 and 1;

[0027] When it is detected that the loss difference L during the training process is less than the preset loss difference, stop the training and freeze the detection parameters of the current anomaly detection network.

[0028] Optionally, the step of sequentially inputting the noise blocks into the trained anomaly detection network to obtain reconstructed blocks and calculating the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block includes:

[0029] Re-input the noise blocks into the anomaly detection network with frozen parameters to obtain reconstructed blocks of the same size;

[0030] Calculate the sum of the squared differences of all pixel points between the noise blocks and the corresponding reconstructed blocks, and use it as the anomaly score of the current noise block.

[0031] Optionally, the step of generating a heat map based on the anomaly scores, extracting the region in the heat map that is higher than the preset threshold to generate a predicted tampering localization map, and outputting includes:

[0032] Control to normalize all anomaly scores to the range of 0 - 1 to obtain normalized anomaly scores;

[0033] Integrate the normalized anomaly scores to automatically generate a heat map with the same size as the unknown image, where the overlapping area of the noise blocks takes the average value of each normalized anomaly score;

[0034] Control to extract the pixel points in the heat map that are higher than the preset threshold to generate a predicted tampering localization map, and output.

[0035] The second aspect of the present invention provides an image tampering localization device based on anomaly detection technology. The device includes:

[0036] An unknown image acquisition module, configured to acquire an unknown image for which image tampering localization is to be performed;

[0037] A noise block acquisition module, configured to control the extraction of noise from the unknown image to obtain a noise image, and split the noise image into a plurality of interrelated noise blocks;

[0038] An anomaly detection network training module, configured to sequentially input the noise blocks into a pre-constructed anomaly detection network for training. When the obtained loss difference is less than a preset loss difference, the training is completed and the detection parameters of the current anomaly detection network are frozen;

[0039] An image tampering localization module, configured to sequentially input the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, calculate the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block, generate a heat map based on the anomaly scores, extract the pixel points in the heat map that are higher than a preset threshold, generate a predicted tampering localization map, and output it.

[0040] The third aspect of the present invention provides an intelligent terminal. The intelligent terminal includes a memory, a processor, and an image tampering localization program based on anomaly detection technology stored on the memory and executable on the processor. When the image tampering localization program based on anomaly detection technology is executed by the processor, the steps of the image tampering localization method based on anomaly detection technology as described in any one of the above are implemented.

[0041] The fourth aspect of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image tampering localization method based on anomaly detection technology as described in any one of the above are implemented.

[0042] As can be seen above, when the solution of the present invention is implemented, first, noise extraction and splitting are performed on the unknown image for which image tampering localization is to be performed, and the split noise blocks are input into a pre-set anomaly detection network for single-image training. After the training is completed, the noise blocks are re-input into the trained anomaly detection network to obtain reconstructed noise blocks. Based on the noise blocks and the reconstructed blocks, the reconstruction error of each noise block, that is, the anomaly score, is calculated. Based on the anomaly scores of each noise block, a heat map with the same size as the unknown image is stitched together. Finally, the part in the heat map that is higher than the preset threshold is extracted to generate a tampering localization map for predicting the tampering position. It can be seen that since the same unknown image is used in both the prediction and training stages, this method does not depend on a data set, has a wide range of applications, and can also achieve good prediction accuracy. Brief Description of the Drawings

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a schematic flowchart of a method for image tampering localization based on anomaly detection technology provided by an embodiment of the present invention;

[0045] Figure 2 It is a schematic diagram of the anomaly detection network structure provided by an embodiment of the present invention;

[0046] Figure 3 It is an embodiment of the present invention Figure 1 The specific flowchart of step S200;

[0047] Figure 4 It is a schematic flowchart of implementing a method for image tampering localization based on anomaly detection technology with a certain landscape picture as an example provided by an embodiment of the present invention;

[0048] Figure 5 It is an embodiment of the present invention Figure 1 The specific flowchart of step S400;

[0049] Figure 6 It is an embodiment of the present invention Figure 1 The specific flowchart of step S500;

[0050] Figure 7 It is an embodiment of the present invention Figure 1 The specific flowchart of step S600;

[0051] Figure 8 It is a schematic diagram of performing morphological erosion and dilation operations on the tampering localization map and extracting the number of connected regions provided by an embodiment of the present invention;

[0052] Figure 9 It is a comparison result diagram of splicing tampering detection between the method of the present invention and the Splicebuster method;

[0053] Figure 10 It is a comparison result diagram of text erasure tampering detection between the method of the present invention and the Splicebuster method;

[0054] Figure 11 It is a comparison result diagram of text addition tampering detection between the method of the present invention and the Splicebuster method;

[0055] Figure 12 It is a schematic structural diagram of an image forgery localization device provided by an embodiment of the present invention based on anomaly detection technology;

[0056] Figure 13 It is a block diagram of the internal structural principle of an intelligent terminal provided by an embodiment of the present invention. Specific embodiments

[0057] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0058] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0059] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0060] Next, in conjunction with the accompanying drawings of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0061] In the following description, many specific details are set forth in order to fully understand the present invention, but the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0062] In recent years, the emergence of image editing software has enabled users without professional image knowledge to easily process image data. Through these convenient image editing software, people can easily retouch selfies to make the people in the photos look better, and can also easily complete image splicing, splicing things that were not originally on the scene into the pictures. Therefore, while providing convenience to people, image editing software has also greatly reduced the cost of image forgery. Especially when people with malicious intentions spread false pictures, it is difficult for people to distinguish the authenticity of the pictures with the naked eye, which can easily cause serious consequences.

[0063] In the prior art, for the problem of identifying whether the above image is tampered with and the specific location where the tampering occurs, there are supervised and unsupervised image tampering localization schemes. Specifically, in the supervised scheme, for example, a neural network is pre-trained with a large amount of training data, and when in use, the image to be predicted is input into the neural network to predict the image tampering location. However, on the one hand, this method requires a large amount of training data and has certain limitations for the predicted images, with a small applicable range; the unsupervised scheme has a low prediction accuracy.

[0064] To solve the problems existing in the prior art, the present invention provides an image tampering localization method based on anomaly detection technology. In an embodiment of the present invention, an unknown image to be subjected to image tampering localization is obtained; controlling the extraction of noise from the unknown image to obtain a noise image, and splitting the noise image into a plurality of mutually related noise blocks; sequentially inputting the noise blocks into a pre-constructed anomaly detection network for training; when the obtained loss difference is less than a preset loss difference, the training is completed and the detection parameters of the current anomaly detection network are frozen; sequentially inputting the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, and calculating the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block; generating a heat map based on the anomaly scores, extracting the pixel points in the heat map that are higher than a preset threshold, generating a predicted tampering localization map, and outputting. By the method of single-image training and single-image testing, the dependence on the data set is greatly reduced, and the applicable range is expanded. At the same time, compared with some deep networks, the network of this method is more lightweight, occupies less hardware resources, is easier to be deployed in daily life, and also provides better image tampering localization accuracy.

[0065] Exemplary method

[0066] As Figure 1 shown, an embodiment of the present invention provides an image tampering localization method based on anomaly detection technology, and the above method includes the following steps:

[0067] Step S100, obtaining an unknown image to be subjected to image tampering localization.

[0068] In this embodiment, first, an unknown image to be subjected to image tampering localization is obtained. When the image tampering localization method based on anomaly detection technology is applied to software, the software obtains the picture input by the user into the software as the unknown image.

[0069] Step S200: Control the extraction of noise from the unknown image to obtain a noise image, and split the noise image into a number of mutually related noise blocks.

[0070] In this embodiment, the software controls the extraction of noise from the unknown image, extracts the noise features in the image, obtains a noise image corresponding to the unknown image, and further splits the noise image into mutually related noise blocks. Specifically, set the size of each split noise block, such as 48*48 pixels, and set to offset a number of sizes in the four directions of up, down, left, and right of the noise block to obtain another noise block associated with the noise block. It should be noted that the offset distance needs to be less than the side length of the noise block, that is, there needs to be an overlapping part between two adjacent noise blocks, so that adjacent noise blocks are mutually related.

[0071] Furthermore, set that when splitting to the edge of the unknown image and the remaining image is not enough to split into a 48*48 pixel noise block, control to reduce the offset distance so that one side of the noise block coincides with the edge of the unknown image. Make all the split noise blocks of the same size, and all pixels of the unknown image are collected, providing a more complete training set for single-image training.

[0072] It can be seen that in this step, different from conventional neural network training, only one image is used. By splitting one image into multiple image data available for training, it provides a basis for single-image training.

[0073] Step S300: Sequentially input the noise blocks into a pre-constructed anomaly detection network for training.

[0074] In this embodiment, the software sequentially inputs the obtained several noise blocks into the pre-constructed anomaly detection network for training, realizing network training through one unknown image.

[0075] Step S400: When the obtained loss difference is less than the preset loss difference, complete the training and freeze the detection parameters of the current anomaly detection network.

[0076] In this embodiment, during the training process of the anomaly detection network, the current loss difference is calculated and obtained in real time according to the pre-set formula. And compare the loss difference with the pre-set preset loss difference. When the loss difference is less than the preset loss difference, stop the training and freeze the detection parameters of the current anomaly detection network.

[0077] Step S500: Sequentially input the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, and calculate the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block.

[0078] In this embodiment, the noise blocks are sequentially input into the anomaly detection network with frozen parameters to obtain reconstructed blocks having the same size as the input noise blocks. Calculate the reconstruction error between the noise blocks and the corresponding output reconstructed blocks, and finally obtain the anomaly scores for evaluating whether there are tampering traces at the corresponding positions of the image.

[0079] Specifically, the reconstruction error reflects the fitting ability of the anomaly detection network for different noise blocks. The fitting at the tampered position is poor and the reconstruction error is large; the fitting at the untampered position is good and the reconstruction error is small. And the anomaly score represents the probability of a pixel being tampered. The higher the score, the greater the probability of the pixel being tampered, and vice versa.

[0080] Step S600: Generate a heat map based on the anomaly scores, extract the pixel points in the heat map that are higher than a preset threshold, generate a predicted tampering localization map, and output it.

[0081] In this embodiment, according to the anomaly scores obtained in the previous step, the noise blocks are re - stitched, and the anomaly scores of the overlapping parts are averaged to automatically generate a heat map. And according to a preset threshold set in advance, the pixel points in the heat map that exceed the preset threshold are extracted to generate a tampering localization map for predicting the tampering position of the unknown image, and output it.

[0082] As can be seen from the above, an image tampering localization method based on anomaly detection technology provided by an embodiment of the present invention obtains an unknown image that needs to be subjected to image tampering localization; controls the extraction of noise from the unknown image to obtain a noise image, and splits the noise image into several mutually - related noise blocks; sequentially inputs the noise blocks into a pre - constructed anomaly detection network for training; when the obtained loss difference is less than a preset loss difference, the training is completed and the detection parameters of the current anomaly detection network are frozen; sequentially input the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, and calculate the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block; generate a heat map based on the anomaly scores, extract the pixel points in the heat map that are higher than a preset threshold, generate a predicted tampering localization map, and output it. It can achieve high - accuracy prediction of tampering positions only through one unknown image and has a wider scope of application.

[0083] In a further embodiment, a program for image tampering localization based on anomaly detection technology is taken as an example for specific description.

[0084] In an application scenario, an anomaly detection network for detecting the location of image tampering is pre-constructed, which includes an encoder, a fully connected layer, a Gaussian distribution fitting layer, and a decoder.

[0085] For example, please refer to Figure 2 , and this anomaly detection network is built by the Pytorch framework. The encoding layer includes a two-dimensional convolution module, a max pooling layer module, a batch normalization module, and a Sigmoid activation function module. Among them, the size of the two-dimensional convolution module is 3*3, and the stride is set to 1; the size of the max pooling layer module is 3*3, and this module is used to capture the maximum value in a certain area; the batch normalization module is used to standardize these input values and reduce the differences between different samples to the same range. On the one hand, it can accelerate the training speed of the model, and it can also make each layer process input values with the same feature distribution, alleviating the problem of gradient disappearance in training. The Sigmoid activation function maps the feature value x to the interval from 0 to 1, reducing the difficulty of network training. The calculation formula is as follows:

[0086]

[0087] The fully connected layer can play a role in adjusting the feature dimension and reducing network redundancy in the network. At the same time, because it has separate parameters for each individual feature, overall it considers the influence of the results of each pixel value. Its output is the result of the weighted sum of each feature.

[0088] In the Gaussian distribution fitting layer, after the noise block passes through the encoder and the fully connected layer, it is encoded into a 20-dimensional vector with a low dimension. At this time, it is divided into two parts, which are called x1 and x2 respectively, each part has 10 dimensions, and then they are recombined to fit into 10 different Gaussian distributions. At this time, x1 represents the mean value, and x2 represents the variance. The Gaussian distribution G is expressed as follows:

[0089] G i = x1 i + x2 i *randn#(2)

[0090] Where i represents the i-th Gaussian distribution, and the value ranges from 1 to 10. randn represents a Gaussian random number with a mean of 1 and a variance of 0 in the same dimension as x1 and x2.

[0091] The decoder has a symmetric structure with the encoder. The decoder and the linear layer restore the 10 Gaussian distributions coming out of the Gaussian distribution fitting layer into a noise block of the same size as the input.

[0092] In an application scenario, the program obtains an unknown image that needs to be located for image tampering.

[0093] For example, a user downloads a landscape photo from the Internet and drags it into the image tampering localization program. At the same time, the program detects and obtains the landscape photo input by the user, that is, the unknown image to be subjected to image tampering localization.

[0094] In an application scenario, the program controls the extraction of noise from the unknown image to obtain a noise image, and splits the noise image into a number of interrelated noise blocks.

[0095] Specifically, as Figure 3 shown, the above step S200 includes:

[0096] Step S201: Extract the noise features of the unknown image through Noiseprint to obtain a noise image;

[0097] Step S202: Split the noise image, and control to obtain a noise block with a size of 48*48 pixels every 16 pixel units.

[0098] For example, please refer to Figure 4 . The picture input by the user contains rocks, sky, birds and sunflowers. The program extracts the noise features with the same length and width as the original picture and a channel number of 1 through Noiseprint to obtain a gray noise image. This network is trained using contrastive loss, which strengthens the noise traces generated by tampering and suppresses the image content, making the extracted noise easier to detect. It can be seen that there are obvious different noise features at the position of the sunflower in the noise image, that is, the noise image obtained through Noiseprint has certain tampering position information.

[0099] Further, the program controls the splitting of the noise image. It is set to obtain a noise block with a size of 48*48 pixels every 16 pixel units, that is, there will be an overlapping part of two-thirds between every two adjacent noise blocks. Through this step, the noise image is split into a number of related noise blocks to obtain a training set available for training.

[0100] In an application scenario, the program sequentially inputs the noise blocks into a pre-constructed anomaly detection network for training.

[0101] For example, all the noise blocks are sent into the anomaly detection network for training. The training parameters are set as batch_size = total number of training patches / 20, learning rate Learning_rate = 0.001, and the optimizer selects the Adam optimizer.

[0102] In an application scenario, when the obtained loss difference is less than a preset loss difference, the program control finishes the training and freezes the detection parameters of the current anomaly detection network.

[0103] Specifically, as Figure 5 shown, the above step S400 includes:

[0104] Step S401: Calculate the loss difference during the training of the anomaly detection network through the following formula:

[0105] Total loss: L = L1 + λ * L2,

[0106] where the image reconstruction loss:

[0107] KL divergence loss:

[0108] where n represents the total number of noise blocks in a training batch, X i represents the original image block corresponding to the current i, X' i represents the block reconstructed from the original image block after passing through the network, J represents the number of Gaussian models fitted in the middle layer of the network, u j and σ j respectively represent the parameter mean and standard deviation of the j-th Gaussian model, and λ is a hyperparameter with a value between 0 and 1;

[0109] Step S402: When it is detected that the loss difference L during the training process is less than the preset loss difference, stop the training and freeze the detection parameters of the current anomaly detection network.

[0110] Among them, before the step of obtaining the unknown image to be subjected to image tampering localization, it further includes:

[0111] Set a preset loss difference for judging whether the network training needs to stop.

[0112] For example, please continue to refer to Figure 4 , before performing image tampering localization on the unknown image, a preset loss difference of 0.01 is set in advance to judge whether the training completion index has been reached during the training process of the anomaly detection network.

[0113] During the training process of the anomaly detection network, calculate the loss difference of the anomaly detection network through the following formula: L = L1 + λ * L2#(3)

[0114] where the image reconstruction loss:

[0115] KL divergence loss:

[0116] Among them, n represents the total number of noise blocks in a training batch, and X i represents the original image block corresponding to the current i, and X' i represents the block reconstructed from the original image block after passing through the network. J represents the number of Gaussian models fitted in the intermediate layer of the network. u j and σ j represent the parameter mean and standard deviation of the j-th Gaussian model. λ is a hyperparameter with a value between 0 and 1, which is used to balance the relationship between L1 and L2.

[0117] For example, during the training process, when it is detected that the loss difference L = 0.025, it is judged to be greater than the preset loss difference of 0.01, so the training continues; when it is detected that the loss difference L = 0.007, it is judged to be less than the preset loss difference of 0.01, and the training is controlled to stop and the detection parameters of the current anomaly detection network are frozen. The training of the anomaly detection network is realized by using an image to be detected.

[0118] In an application scenario, the program sequentially inputs the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, and calculates the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block.

[0119] Specifically, as Figure 6 shown, the above step S500 includes:

[0120] Step S501: Re-input the noise block into the anomaly detection network with frozen parameters to obtain a reconstructed block of the same size;

[0121] Step S502: Calculate the sum of the squared differences of all pixel points between the noise block and the corresponding reconstructed block, and use it as the anomaly score of the current noise block.

[0122] For example, the program sequentially inputs the noise blocks into the anomaly detection network with frozen parameters, that is, the trained network, and will obtain a reconstructed block with a size of 48*48 pixels corresponding to the input noise block. Calculate the reconstruction error between all noise blocks and their corresponding reconstructed blocks, the sum of the squared differences of all pixel points, and use it as the anomaly score S i :

[0123] S i =(X' i -X i ) 2 #(6)

[0124] Among them, X i represents the original image block corresponding to the current i, and X' i represents the block reconstructed from the original image block after passing through the network.

[0125] In an application scenario, the program generates a heat map based on the anomaly scores, extracts the pixel points in the heat map that are higher than a preset threshold, generates a predicted tampering localization map, and outputs it.

[0126] Specifically, as Figure 7 described above, step S600 includes:

[0127] Step S601: Control to normalize all anomaly scores to the range of 0-1 to obtain normalized anomaly scores;

[0128] Step S602: Integrate the normalized anomaly scores to automatically generate a heat map with the same size as the unknown image. Among them, for the overlapping area of noise blocks, take the average value of each normalized anomaly score;

[0129] Step S603: Control to extract the pixel points in the heat map that are higher than a preset threshold, generate a predicted tampering localization map, and output it.

[0130] Among them, before the step of obtaining the unknown image to be subjected to image tampering localization, it further includes:

[0131] Set a preset threshold.

[0132] For example, a preset threshold with a parameter of 0.5 is set in advance to assist in generating the tampering localization map.

[0133] Furthermore, the program normalizes all anomaly scores S i to the range of 0-1 to obtain normalized anomaly scores

[0134]

[0135] where S max represents the maximum value in the anomaly score set S. Similarly, S min represents the minimum value in the anomaly score set S.

[0136] Please continue to refer to Figure 4 , splice all the noise blocks marked with normalized anomaly scores together, take the average value for the overlapping area of adjacent noise blocks, and form a heat map with the normalized anomaly scores. To obtain a binary classification judgment of pixel points, use a preset threshold, set the value of pixel points greater than 0.5 in the heat map to 1, and the rest to 0. Among them, 1 indicates a predicted tampered pixel, and 0 indicates a predicted untampered pixel. In this way, a pixel-level prediction map can be obtained.

[0137] Finally, the control outputs the generated tampering localization map. It can be seen that the prediction result is basically the same as the real tampering area, accurately predicting the tampering position in the unknown image. The image tampering localization scheme that realizes single-image training and single-image testing has a wider scope of application.

[0138] In an application scenario, the program is further improved.

[0139] When the compression quality of the JPEG image is too low, for example, the compression quality factor QF (quality factor), when QF < 80, part of the tampering traces in the image are erased, and the remaining tampering traces are difficult to extract through the Noiseprint network. Therefore, we directly send the test image with QF < 80 into the proposed anomaly detection network for training to obtain the result.

[0140] When the anomaly detection network is overfitting, the reconstruction effect of all blocks is very good, resulting in very small reconstruction errors and it is impossible to distinguish which area has been tampered with. Considering introducing a memory module into the network, the memory module records the features of a limited number of normal blocks, so not all noise blocks will be reconstructed well.

[0141] In an application scenario, this scheme can not only be used to locate known tampered images, but also detect images whose tampering status is unknown, and determine whether a certain image has been tampered with.

[0142] Specifically, the control performs morphological erosion and dilation operations on the tampering localization map, and extracts the number of connected components in the image after the dilation operation;

[0143] When the number of connected components exceeds the preset number, it is determined that the unknown image has not been tampered with.

[0144] For example, since common tampering operations are to tamper with a region, or one or several regions of a picture, such as tampering with signatures, titles, etc., there will be no situation where multiple discrete points in a picture are tampered with. When there is no tampered area in the image, the noise values at all positions tend to be the same, and there is no abnormal noise texture or noise value. Therefore, the anomaly detection algorithm cannot capture a specific area, but will randomly capture many discrete areas, manifested as many discrete points in the prediction result.

[0145] In specific implementation, the program control first performs morphological erosion on the tampering localization map and then performs dilation operation, and sets the size of the kernel to 48*48. This makes the boundary of the prediction area smooth and disconnects narrow connections. After erosion and dilation, the number of connected components of the result is extracted.

[0146] With Figure 8For example, for the unaltered image, the number of connected regions finally extracted by the above method is 97, while for the image with the forgery erasure signature, the number of connected regions finally extracted is 1.

[0147] If it is set that when the number of connected components exceeds a preset number, for example, exceeds 15, it is determined that the unknown image has not been tampered with, and the judgment result of "not tampered with" is output. Then the previous image will correspondingly output the judgment result of "not tampered with"; while the latter image will output the judgment result of "tampered with", and at the same time prompt the tampering position.

[0148] Based on the above-mentioned image forgery localization method based on anomaly detection technology, further test and compare the prediction accuracy of this method:

[0149] First, compare the accuracy for individual types of forgery. As Figure 9 、 10 and as shown in 11, it is a comparison of the accuracy of individual types of forgery between the method of the present invention and the unsupervised scheme Splicebuster. Among them, Figure 9 is the comparison result of splicing forgery detection, Figure 10 is the comparison result of text erasure forgery detection, Figure 11 is the comparison result of text addition forgery detection. It can be seen that the accuracy rate of the method of the present invention is significantly better than that of the unsupervised scheme Splicebuster, and can well locate the forged content.

[0150] Second, compare the forgery performance for large data sets with multiple types of forgery. In the test, the commonly used evaluation indicators at the image pixel level, such as the AUC, F1 index, MCC index, and IoU index, were used.

[0151] AUC = ∫TPR(FPR -1 (x))dx#(8)

[0152]

[0153]

[0154]

[0155] Among them, TPR and FPR respectively represent the true positive rate and the false positive rate, TP and FN respectively represent the number of correctly classified and misclassified forged pixels, and TN and FP respectively represent the number of correctly classified and misclassified original pixels. In the experiment, the fixed threshold was set to 0.5.

[0156] The methods involved in the comparison include unsupervised schemes: ADQ1, a method for detecting alignment in double JPEG compression by utilizing the DCT coefficient distribution; the DCT-based method, a method for detecting inconsistencies in the JPEG DCT coefficient histogram; NADQ, a method for detecting misaligned double JPEG compression by analyzing 8×8 DCT blocks.

[0157] And supervised schemes: forensic similarity, which reveals the tampered areas by comparing the similarity of image patches with a CNN-based network in a sliding window manner; a multi-task fully convolutional network MFCN for splicing localization, which combines the edges and interiors of the tampered areas to achieve prediction; LSTM-EnDec, which fuses the resampled features represented by LSTM with the spatial features extracted by CNN for tampering localization; Mantra-net, which first learns the tampering traces from multiple image operations and then performs tampering localization through local anomaly detection; Dense-FCN, which achieves excellent image tampering localization performance under the condition of a larger receptive field by designing a deep neural network containing dilated convolutions.

[0158] For the fairness of comparison, all supervised schemes are trained on the dataset generated by the automatic cover tampering script and then tested on three major datasets, while the unsupervised schemes are directly tested. The three major datasets include the public dataset NIST16 containing 564 tampered JPEG images, the artificial PS dataset with boundary post-processing (PS-boundary) containing 1000 images, and the artificial PS dataset with arbitrary post-processing (PS-arbitrary) containing 1000 images.

[0159] The performance comparison table of the nine methods is as follows. The numbers in parentheses in the table represent the single-index rankings among all the schemes. Finally, the average of all the index rankings is calculated as the comprehensive evaluation index in this dataset.

[0160] Among them, Table 1 is the network performance comparison on the NIST16 dataset, Table 2 is the network performance comparison on the PS-boundary dataset, and Table 3 is the network performance comparison on the PS-arbitrary dataset.

[0161] Table 1

[0162]

[0163] Table 2

[0164]

[0165] Table 3

[0166]

[0167] From the result analysis on the above three datasets, it can be seen that the accuracy of the proposed solution of the present invention is relatively good, achieving the first, third, and second excellent results on the NIST16, PS-boundary, and PS-arbitrary datasets respectively, and the performance of this method is better than all unsupervised methods. Since the supervised solution trains a large amount of data, while the proposed solution of the present invention only trains a single test data, the performance is slightly weaker than the supervised solution. However, it can be seen from the results that in this case, the proposed solution of the present invention still achieves good results, and in most cases, the performance approaches or even exceeds that of the supervised solution.

[0168] It can be seen that the proposed solution of the present invention is more applicable to the situation where there is no large amount of data available for training in daily life, has a wider application range, and higher accuracy.

[0169] Exemplary device

[0170] As Figure 12 shown, corresponding to the above image tampering localization method based on anomaly detection technology, an embodiment of the present invention further provides an image tampering localization device based on anomaly detection technology. The above device includes:

[0171] An unknown image acquisition module 1210, configured to acquire an unknown image to be subjected to image tampering localization;

[0172] A noise block acquisition module 1220, configured to control the extraction of noise from the unknown image to obtain a noise image, and split the noise image into a plurality of interrelated noise blocks;

[0173] An anomaly detection network training module 1230, configured to sequentially input the noise blocks into a pre-constructed anomaly detection network for training. When the loss difference obtained by training is less than a preset loss difference, the training is completed and the detection parameters of the current anomaly detection network are frozen;

[0174] An image tampering localization module 1240, configured to sequentially input the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, calculate the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block, generate a heat map based on the anomaly scores, extract the pixel points in the heat map that are higher than a preset threshold, generate a predicted tampering localization map, and output it.

[0175] The present invention provides an intelligent terminal, and its principle block diagram can be as Figure 13As shown in the figure. The above intelligent terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements the steps of any one of the above image tampering localization methods based on anomaly detection technology. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0176] Those skilled in the art can understand that Figure 13 the principle block diagram shown in the figure is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the intelligent terminal to which the solution of the present invention is applied. The specific intelligent terminal may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0177] In one embodiment, an intelligent terminal is provided, including a memory, a processor, and a program stored on the above memory and executable on the above processor. When the above program is executed by the above processor, the following operation instructions are performed:

[0178] Obtain an unknown image to be subjected to image tampering localization;

[0179] Control the extraction of noise from the unknown image to obtain a noise image, and split the noise image into a number of interrelated noise blocks;

[0180] Input the noise blocks into a pre-constructed anomaly detection network in sequence for training;

[0181] When the obtained loss difference is less than a preset loss difference, complete the training and freeze the detection parameters of the current anomaly detection network;

[0182] Input the noise blocks into the trained anomaly detection network in sequence to obtain reconstructed blocks, and calculate the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block;

[0183] Generate a heat map based on the anomaly scores, extract the pixel points in the heat map that are higher than a preset threshold, generate a predicted tampering localization map, and output it.

[0184] It should be understood that the sequence numbers of the above steps in the embodiments do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules according to needs, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present invention. The specific working processes of the units and modules in the above-mentioned system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0186] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. For the quantities with the same representation form in each formula, they represent the same quantity without special instructions, and the various formulas can be referred to each other.

[0187] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0188] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the above-mentioned division of modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0189] When the above integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The above computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the above computer program includes computer program code, and the above computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The above computer-readable medium can include: any entity or device capable of carrying the above computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the above computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction.

[0190] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An image tampering localization method based on anomaly detection technology, characterized in that, the method includes: Obtain an unknown image to be subjected to image tampering localization; Control the extraction of noise from the unknown image to obtain a noise image, and split the noise image into a number of mutually related noise blocks; Input the noise blocks into a pre-constructed anomaly detection network in sequence for training; When the obtained loss difference is less than a preset loss difference, complete the training and freeze the detection parameters of the current anomaly detection network; Input the noise blocks into the trained anomaly detection network in sequence to obtain reconstructed blocks, and calculate the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block; Generate a heat map based on the anomaly scores, extract the pixel points in the heat map that are higher than the preset threshold, generate a predicted tampering localization map, and output it.

2. The image tampering localization method based on anomaly detection technology according to claim 1, characterized in that, before the step of obtaining an unknown image to be subjected to image tampering localization includes: Construct an anomaly detection network, including an encoder, a fully connected layer, a fitting Gaussian distribution layer, and a decoder.

3. The image tampering localization method based on anomaly detection technology according to claim 1, characterized in that, before the step of obtaining an unknown image to be subjected to image tampering localization also includes: Set a preset loss difference for judging whether to stop network training; Set a preset threshold.

4. The image tampering localization method based on anomaly detection technology according to claim 1, characterized in that, the step of controlling the extraction of noise from the unknown image to obtain a noise image, and splitting the noise image into a number of mutually related noise blocks includes: Extract the noise features of the unknown image through Noiseprint to obtain a noise image; Split the noise image, and control to obtain a noise block with a size of 48*48 pixels every 16 pixel units.

5. The image tampering localization method based on anomaly detection technology according to claim 1, characterized in that, the step of when the obtained loss difference is less than a preset loss difference, complete the training and freeze the detection parameters of the current anomaly detection network includes: Calculate the loss difference during the training of the anomaly detection network through the following formula: Total loss: L = L1 + λ * L2, Among them, the image reconstruction loss: KL divergence loss: where n represents the total number of noise blocks in a training batch, X i represents the original image block corresponding to the current i, X' i represents the block reconstructed from the original image block after passing through the network, J represents the number of Gaussian models fitted in the intermediate layer of the network, u j and σ j respectively represent the parameter mean and standard deviation of the j-th Gaussian model, and λ is a hyperparameter with a value between 0 and 1; When it is detected that the loss difference L during the training process is less than the preset loss difference, stop the training and freeze the detection parameters of the current anomaly detection network.

6. The image tampering localization method based on anomaly detection technology according to claim 1, characterized in that, the step of inputting the noise blocks into the trained anomaly detection network in sequence to obtain reconstructed blocks, and calculating the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block includes: Re-input the noise blocks into the anomaly detection network with frozen parameters to obtain reconstructed blocks of the same size; Calculate the sum of the squared differences of all pixel points between the noise block and the corresponding reconstructed block, and use it as the anomaly score of the current noise block.

7. A method for image tampering localization based on anomaly detection technology according to claim 1, characterized in that, the steps of generating a heat map based on the anomaly scores, extracting regions in the heat map that are higher than a preset threshold, generating a predicted tampering localization map, and outputting include: controlling to normalize all anomaly scores to the range of 0-1 to obtain normalized anomaly scores; integrating the normalized anomaly scores to automatically generate a heat map with the same size as the unknown image, wherein the overlapping regions of noise blocks take the average value of each normalized anomaly score; controlling to extract pixel points in the heat map that are higher than the preset threshold, generating a predicted tampering localization map, and outputting.

8. An image tampering localization device based on anomaly detection technology, characterized in that, the device includes: an unknown image acquisition module for acquiring an unknown image to be subjected to image tampering localization; a noise block acquisition module for controlling to extract noise from the unknown image to obtain a noise image, and splitting the noise image into a plurality of mutually related noise blocks; an anomaly detection network training module for sequentially inputting the noise blocks into a pre-constructed anomaly detection network for training, and when the obtained loss difference is less than a preset loss difference, completing the training and freezing the detection parameters of the current anomaly detection network; an image tampering localization module for sequentially inputting the noise blocks into the trained anomaly detection network to obtain reconstructed blocks, calculating the reconstruction error between the noise blocks and the corresponding reconstructed blocks to obtain the anomaly scores of each noise block, generating a heat map based on the anomaly scores, extracting pixel points in the heat map that are higher than the preset threshold, generating a predicted tampering localization map, and outputting.

9. An intelligent terminal, characterized in that, the intelligent terminal includes a memory, a processor, and an image tampering localization program based on anomaly detection technology stored on the memory and executable on the processor. When the image tampering localization program based on anomaly detection technology is executed by the processor, the steps of the method for image tampering localization based on anomaly detection technology according to any one of claims 1-7 are implemented.

10. A non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for image tampering localization based on anomaly detection technology according to any one of claims 1-7 are implemented.

Images