A method and device for detecting and checking duplicates of a counterfeit image

Abstract

The application discloses a method for detecting and checking duplicates of a fake image. A first machine learning model determines whether an input image is tampered with; if the input image is determined to be untampered, the whole method is exited; if the input image is determined to be a tampered fake image, the method continues. A second machine learning model locates a tampered region from the fake image. Similarities are calculated between the fake image after removing the tampered region and all images in a database after removing corresponding regions. If the similarity value is greater than or equal to a threshold value, it is considered that the image being calculated in the database is a duplicate image of the fake image, indicating that the fake image is secondly confirmed to be tampered with, and the duplicate image of the fake image is output. If the similarity value is less than the threshold value, it is considered that the fake image does not find a duplicate image in the database; the image is added to the database, and the image is labeled in the database according to the artificial review result. The application improves the accuracy and interpretability of the fake image judgment.

Description

Technical Field

[0001] This invention relates to a method for detecting forged images. Background Technology

[0002] With the widespread use of electronic devices and image processing software, people can more easily acquire and edit images, which has also facilitated image forgery. Some criminals have even used altered and synthesized images to commit economic fraud, causing harm to individuals and society.

[0003] Existing methods for determining whether an image has been forged mainly fall into two categories. The first is simple image classification, which determines whether the input image is forged. The second is simple tamper detection and localization, which determines whether there are tampered areas in the input image and, if so, locates those areas. Because this requires processing all images, it is relatively inefficient. Summary of the Invention

[0004] The technical problem to be solved by this invention is to propose a method to improve the accuracy of forged image detection, and at the same time, to explain the reason for judging it as a forged image by providing the original image with high similarity obtained by deduplication.

[0005] To address the aforementioned technical problems, this invention discloses a method for detecting and deduplicating forged images, comprising the following steps: Step S1: A first machine learning model determines whether the input image has been tampered with; if the input image is determined to be untampered, the entire method exits; if the input image is determined to be a tampered forged image, then proceed to Step S2. Step S2: A second machine learning model locates the tampered region in the forged image. Step S3: After removing the tampered region from the forged image, the similarity is calculated between the forged image and all images in the database after removing the corresponding regions. If the similarity value is greater than or equal to a threshold, the image in the database being calculated is considered a duplicate of the forged image, indicating that the forged image has been confirmed as tampered for the second time, and the duplicate image of the forged image is output. If the similarity value is less than the threshold, it is considered that no duplicate image of the forged image is found in the database; the image is added to the database, and the image is manually reviewed, and labeled in the database according to the manual review results.

[0006] Furthermore, the first machine learning model is a binary classification model used to divide the input image into two categories: the original image that has not been tampered with, and the forged image that has been tampered with.

[0007] Preferably, the first machine learning model employs a ResNet-18 convolutional neural network.

[0008] Furthermore, the training method of the first machine learning model is as follows: (1) Create training data; collect original images, and perform tampering operations on the original images to form fake images; use the original images and fake images as the whole as training data; (2) send the training data into the first machine learning model, and have the first machine learning model classify each image so that the classification results output by the first machine learning model are as consistent as possible with the known categories of each image.

[0009] Furthermore, the training method of the second machine learning model is as follows: (1) Create training data; collect original images, perform tampering operations on the original images to form fake images, and mark the tampered areas on each fake image; use the fake images and the marked tampered areas on the fake images as training data; (2) send the training data into the second machine learning model, and the second machine learning model locates the tampered areas of each fake image, so that the tampered areas output by the second machine learning model are as consistent as possible with the marked tampered areas of each image.

[0010] Preferably, when the first machine learning model and the second machine learning model generate training data, the same content is used only once and repeatedly.

[0011] Furthermore, in step S3, the similarity calculation adopts any one of the perceptual hashing algorithm pHash, the mean hashing algorithm aHash, and the difference hashing algorithm dHash.

[0012] Furthermore, in step S3, if the image in the database is the same size as the forged image, then the corresponding area refers to the area where the tampered area of ​​the forged image is located in the same position as the image in the database; if the image in the database is different sizes from the forged image, then the corresponding area refers to the area where the tampered area of ​​the forged image is located in the same position as the image in the database after the forged image has been scaled to the size of the image in the database.

[0013] Furthermore, in step S3, the similarity threshold is obtained through image prior calculation.

[0014] This invention also discloses a device for detecting and deduplicating forged images, including a tampering detection and classification unit, a tampering region localization unit, and a forged image deduplication unit. The tampering detection and classification unit uses a first machine learning model to determine whether an input image has been tampered with; if the input image is determined to be untampered, the entire processing flow is exited; if the input image is determined to be a tampered forged image, the forged image is passed to the tampering region localization unit. The tampering region localization unit uses a second machine learning model to locate the tampered region from the forged image, and the forged image and its tampered region are passed to the forged image deduplication unit. The forged image deduplication unit calculates the similarity between the forged image after removing the tampered region and all images in the database after removing the corresponding regions; if the similarity value is greater than or equal to a threshold, the image in the database being calculated is considered a duplicate of the forged image, indicating that the forged image has been confirmed as tampered a second time, and the duplicate image is output; if the similarity value is less than the threshold, no duplicate image is found in the database; the image is added to the database, and manually reviewed, and labeled in the database based on the manual review results.

[0015] The technical effects achieved by this invention are as follows: (1) Existing image deduplication techniques are often relatively simple. When comparing tampered images with images in the database, it is often difficult to match them due to the influence of the tampered area. This invention first locates the tampered area and then removes it, which can improve the accuracy of deduplication. (2) This invention first performs tamper detection on the image, and then locates the tampered area in the image with tampering, which reduces the number of images that need to be processed to locate the tampered area; then it finds duplicate images of the forged image, compares the forged image with the duplicate images obtained from the deduplication, which improves the accuracy and interpretability of the forged image judgment. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the method for detecting and deduplicating forged images proposed in this invention.

[0017] Figure 2 This is a schematic diagram of the structure of the forged image detection and deduplication device proposed in this invention.

[0018] The attached figures are labeled as follows: 1 is the tampering detection classification unit, 2 is the tampering area location unit, and 3 is the forged image plagiarism detection unit. Detailed Implementation

[0019] Please see Figure 1 The method for detecting and deduplicating forged images proposed in this invention includes the following steps.

[0020] Step S1: The first machine learning model determines whether the input image has been tampered with. The first machine learning model is a binary classification model used to divide the input image into two categories: the original, untampered image and the tampered, forged image. Preferably, the first machine learning model uses a ResNet-18 convolutional neural network. If the input image is determined to be untampered, the entire method exits. If the input image is determined to be tampered with, proceed to step S2.

[0021] The training method of the first machine learning model is as follows: (1) Create training data. Collect original images, such as personal ID cards, certificates, natural scenes, etc., which are at risk of being forged. Modify the original images to form forged images. The modification operation includes one or more of the following: copying a certain area of ​​the original image and moving it to another area to paste; erasing a certain area of ​​the original image; erasing a certain area of ​​the original image and adding new content (such as text); adding new content (such as text) to the original image; copying a certain area of ​​one image and moving it to another image to paste (called splicing two images). Use the original image and the forged image as the whole as training data. At this time, the category (original image, forged image) of each image is known. (2) Feed the training data into the first machine learning model. The first machine learning model classifies each image so that the classification result output by the first machine learning model is as consistent as possible with the known category of each image. The trained first machine learning model can then be used to classify the input images into two categories: original images and forged images.

[0022] Step S2: The second machine learning model locates the tampered area in the fake image.

[0023] The training method for the second machine learning model is as follows: (1) Create training data. Collect original images, modify them to form fake images, and mark the modified areas on each fake image. Use the fake images and the marked modified areas on the fake images as training data. (2) Feed the training data into the second machine learning model, which locates the modified areas in each fake image, making the modified areas output by the second machine learning model as consistent as possible with the marked modified areas in each image. The trained second machine learning model can then be used to locate the modified areas in the fake images.

[0024] Preferably, when the first and second machine learning models generate training data, the same content is only processed once. This minimizes repetitive work and improves processing efficiency. For example, the process involves only one step: collecting the original image, modifying it to create a fake image, and marking the modified areas on each fake image. The original image and the fake image together serve as the training data for the first machine learning model. The fake image and the marked modified areas together serve as the training data for the second machine learning model.

[0025] Step S3: After removing the tampered areas from the forged image, calculate its similarity with all images in the database after removing the corresponding areas. Similarity calculation can employ algorithms such as perceptual hashing (pHash), average hashing (aHash), and difference hashing (dHash). This step preferably uses perceptual hashing, which removes high-frequency information from the image through Discrete Cosine Transform (DCT), retaining only low-frequency information, thus exhibiting better robustness compared to average hashing and difference hashing.

[0026] If the image in the database is the same size (dimensions, resolution) as the forged image, then the corresponding area refers to the region in the database image where the altered area of ​​the forged image is located. If the image in the database is a different size from the forged image, then the corresponding area refers to the region in the database image where the altered area of ​​the forged image, after being scaled to the size of the image in the database, is located in the same position.

[0027] If the similarity score is greater than or equal to the threshold, it is considered that a duplicate image has been found in the database. This duplicate image could be the original forged image itself, the original image corresponding to the forged image, or another forged image where the original image has been modified in the same area. This indicates that the forged image has been confirmed as tampered with. In this case, the duplicate image of the forged image is output, thus providing evidence that the image has been determined to be a forged image.

[0028] If the similarity score is less than the threshold, the forged image is considered to have no duplicate image found in the database. In this case, the image is added to the database and manually reviewed. Based on the manual review results (whether it is a forged image, and if so, locating the tampered area), the image is marked in the database. If this image is subsequently used for tampering, it can be identified through comparison.

[0029] The similarity threshold is calculated using image priors. For example, it is obtained by calculating the hash value of multiple existing forged images after removing the tampered areas and the corresponding original images after removing the corresponding areas, ensuring a detection rate (e.g., >99%). The existing forged images and original images are, for example, training data generated by a first machine learning model and / or a second machine learning model.

[0030] Please see Figure 2 The image detection and deduplication device proposed in this invention includes a tampering detection and classification unit 1, a tampering area positioning unit 2, and an image deduplication unit 3. Figure 2 The device shown corresponds to Figure 1 The method shown.

[0031] The tampering detection and classification unit 1 is used to determine whether the input image has been tampered with using a first machine learning model. If the input image is determined to be untampered, the entire processing flow is terminated. If the input image is determined to be tampered with, the image is passed to the tampered area location unit 2.

[0032] The tampered area location unit 2 is used to locate the tampered area from the forged image using a second machine learning model, and the forged image and its tampered area are transmitted to the forged image deduplication unit 3.

[0033] The forged image deduplication unit 3 is used to calculate the similarity between a forged image (after removing the tampered areas) and all images in the database (after removing the corresponding areas). If the similarity value is greater than or equal to a threshold, it is considered that a duplicate image has been found in the database. This indicates that the forged image has been confirmed as tampered, and the duplicate image is output. If the similarity value is less than the threshold, it is considered that no duplicate image has been found in the database. In this case, the image is added to the database and manually reviewed. Based on the manual review results, the image is labeled in the database.

[0034] Compared with existing technologies, the method for detecting and deduplicating forged images proposed in this invention has the following beneficial effects.

[0035] First, this invention performs tampering detection classification on all images to be inspected (step S1), and then locates the tampered area only for images determined to have been tampered with (step S2). The pre-classification judgment model reduces the range of images for which tampered areas need to be located, significantly improving the processing speed and efficiency of the tampering detection system.

[0036] Second, the present invention only checks for duplicate images in the database that are determined to have been tampered with (step S3), which reduces the range of images that need to be checked and greatly improves the speed and efficiency of image deduplication.

[0037] Third, this invention calculates the similarity between the forged image after removing the tampered areas and other samples in the database after removing the corresponding tampered areas. Based on the similarity check results, the forged image can be confirmed a second time that it has indeed been tampered with, thereby improving the accuracy of forged image detection and providing favorable evidence for the determination of forged images, thus enhancing interpretability.

[0038] Fourth, this invention improves the detection rate of duplicate images by calculating the similarity threshold through image priors.

[0039] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting and checking a counterfeit image, characterized in that, Comprising the following steps: Step S1: The first machine learning model judges whether the input picture is tampered with; if the input picture is determined to be unaltered, the whole method is exited; if the input picture is determined to be a fake picture that is tampered with, step S2 is continued; Step S2: The second machine learning model locates the tampered region from the fake picture; Step S3: The similarity is calculated between the fake picture after removing the tampered region and all pictures in the database after removing the corresponding region; If the similarity value is greater than or equal to the threshold value, it is considered that the picture being calculated in the database is a duplicate picture of the fake picture, indicating that the fake picture is twice confirmed to be tampered with, and the duplicate picture of the fake picture is output at this time; If the similarity value is less than the threshold value, it is considered that no duplicate picture of the fake picture is found in the database; the picture is added to the database, and the picture is manually reviewed according to the manual review result, and the picture is labeled in the database according to the manual review result.

2. The method of claim 1, wherein the method further comprises: The first machine learning model is a binary classification model, which is used to divide the input picture into two categories: original picture without tampering and fake picture with tampering.

3. The method of claim 1, wherein the method further comprises: The first machine learning model adopts ResNet-18 convolutional neural network.

4. The method of claim 1, wherein the method further comprises: The training method of the first machine learning model is as follows: (1) making training data; collecting original pictures, performing tampering operation on the original pictures to form fake pictures; taking the whole of the original picture and the fake picture as training data; (2) sending the training data into the first machine learning model, and classifying each picture by the first machine learning model, so that the classification result output by the first machine learning model is consistent with the known category of each picture as much as possible.

5. The method of claim 1, wherein the method further comprises: The training method of the second machine learning model is as follows: (1) making training data; collecting original pictures, performing tampering operation on the original pictures to form fake pictures, and labeling the tampered region on each fake picture; taking the whole of the fake picture and the labeled tampered region on the fake picture as training data; (2) sending the training data into the second machine learning model, and locating the tampered region of each fake picture by the second machine learning model, so that the tampered region output by the second machine learning model is consistent with the labeled tampered region of each picture as much as possible.

6. The method of claim 4 or 5, wherein, When making training data, the same content is only performed once for the first machine learning model and the second machine learning model.

7. The method of claim 1, wherein the method further comprises: In step S3, the similarity calculation adopts any one of perceptual hash algorithm pHash, mean hash algorithm aHash and difference value hash algorithm dHash.

8. The method of claim 1, wherein the method further comprises: In step S3, if the pictures in the database and the fake picture are of the same size, the corresponding region refers to the region of the same position of the tampered region of the fake picture in the picture in the database; if the pictures in the database and the fake picture are of different sizes, the corresponding region refers to the region of the same position of the tampered region of the fake picture after scaling the fake picture to the size of the picture in the database after scaling the fake picture to the size of the picture in the database.

9. The method of claim 1, wherein the method further comprises: In step S3, the similarity threshold value is calculated by priori of picture.

10. A device for detecting and checking a counterfeit image, characterized by, It comprises a tampering detection and classification unit, a tampered region locating unit and a fake picture duplicate checking unit. The tamper detection classification unit is configured to determine whether the input picture is tampered by using a first machine learning model; if the input picture is determined to be not tampered, the whole processing flow is exited; If the input picture is determined to be a fake picture that is tampered, the fake picture is transmitted to a tamper region positioning unit; The tamper region positioning unit is configured to locate a tamper region from the fake picture by using a second machine learning model, and the fake picture and the tamper region thereof are transmitted to a fake picture duplicate checking unit; The fake picture duplicate checking unit is configured to calculate a similarity between the fake picture after removing the tamper region and all pictures in a database after removing corresponding regions; if a similarity value is greater than or equal to a threshold value, it is considered that a picture being calculated in the database is a duplicate picture of the fake picture, indicating that the fake picture is twice confirmed to be tampered, and the duplicate picture of the fake picture is output; If the similarity value is less than the threshold value, it is considered that no duplicate picture of the fake picture is found in the database; the picture is added to the database, and the picture is manually reviewed, and the picture is labeled in the database according to the manual review result.

Images