A robust tampered image localization method based on multi-task and contrastive learning

Abstract

The application discloses a robust tampered image positioning method based on multi-task and contrast learning, and comprises the following steps: constructing a tampered positioning neural network based on multi-task learning and contrast learning in advance; generating a plurality of tampered training samples close to a real scene on line through a data enhancement module of the tampered positioning neural network based on multi-task learning and contrast learning constructed in advance; inputting the tampered training samples into the constructed tampered positioning neural network based on multi-task learning and contrast learning to train the tampered positioning neural network, training the whole tampered positioning network, and obtaining a trained tampered positioning neural network; inputting an image to be tested into the trained tampered positioning neural network to position the image to be tested, and outputting a positioning result. The application can accurately position a tampered area in a tampered image.

Description

Technical Field

[0001] This invention relates to the field of multimedia information security and forensics technology, and in particular to a robust method, apparatus, smart terminal and storage medium for locating tampered images based on multi-task and contrastive learning. Background Technology

[0002] With the development of technologies such as image processing and artificial intelligence, an increasing number of tampered images are being created and used for illegal purposes. Once these forged images are widely disseminated, they will inevitably cause serious social impact. Therefore, there is an urgent need to develop tampering location methods capable of identifying tampered images and pinpointing the tampered areas.

[0003] Several tampering localization methods have been proposed in the prior art. Although these methods have achieved good performance in laboratory environments, their localization performance deteriorates significantly when tested on post-processed datasets (JPEG compression, scaling, blurring, and social media transmission, etc.). This is because these methods rarely consider real-world scenarios during model design and training, resulting in extracted features lacking sufficient robustness and generalization.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] The technical problem to be solved by this invention is that the features extracted by existing technologies for tamper location lack sufficient robustness and generalization, resulting in poor accuracy in tamper location. This invention provides a robust tamper image location method, device, smart terminal, and storage medium based on multi-task and contrastive learning. The robust tamper image location method provided by this invention based on multi-task and contrastive learning can accurately locate the tampered areas in a tampered image.

[0006] The technical solution adopted by this invention to solve the problem is as follows:

[0007] This invention provides a robust image tampering localization method based on multi-task and contrastive learning, wherein the method includes:

[0008] A tamper localization neural network based on multi-task learning and contrastive learning is pre-constructed;

[0009] By pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tamper localization neural network, multiple tamper training samples that closely resemble real-world scenarios are generated online.

[0010] The modified training samples are input into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, and the entire tampering localization network is trained to obtain the trained tampering localization neural network.

[0011] The image to be tested is input into the trained tamper localization neural network, which performs tamper localization on the image and outputs the localization result.

[0012] The robust tamper-proof image localization method based on multi-task and contrastive learning, wherein the step of pre-constructing a tamper-proof localization neural network based on multi-task learning and contrastive learning includes:

[0013] A tamper localization neural network based on multi-task learning and contrastive learning is pre-constructed;

[0014] The tamper localization neural network based on multi-task learning and contrastive learning includes: a data augmentation module, a feature extraction module, a contrastive learning module, a localization decoder, and a restoration decoder;

[0015] The data augmentation module is used to simulate possible tampering and post-processing operations in real-world scenarios.

[0016] The feature extraction module is connected to the data augmentation module. The feature extraction module is used to receive the image output by the data augmentation module and extract features related to the two tasks of image restoration and tamper location from the input image.

[0017] The contrastive learning module is connected to the feature extraction module. The contrastive learning module is used to constrain the learned features at the feature level using contrastive learning, so that the similar features of the tampered region and the original region are concentrated, while the dissimilar features of the tampered region and the original region are far apart.

[0018] The positioning decoder is connected to the feature extraction module, and the positioning decoder is used to generate the result of tampered positioning.

[0019] The restoration decoder is connected to the feature extraction module, and the restoration decoder is used to generate the restored image.

[0020] The robust image tampering localization method based on multi-task and contrastive learning, wherein the step of inputting the tampering training sample into the constructed tampering localization neural network based on multi-task and contrastive learning for training, and training the entire tampering localization network to obtain the trained tampering localization neural network includes:

[0021] During the training phase, data augmentation modules are used to simulate possible tampering and post-processing operations in real-world scenarios.

[0022] The feature extraction module receives the image output by the data augmentation module and extracts features related to the two tasks of image restoration and tamper localization from the input image.

[0023] By using the contrastive learning module, the features learned by the feature extraction module are constrained, so that similar features of the tampered region and the original region are concentrated, while dissimilar features of the tampered region and the original region are far apart.

[0024] The restoration decoder module receives features constrained by the contrastive learning module, trains it using restoration loss, and outputs the restored image. Simultaneously, the restored image serves as input to the feature extraction module, providing another version of the image for training the feature extraction module, resulting in the trained tamper localization neural network.

[0025] The robust image tampering localization method based on multi-task and contrastive learning, wherein the step of generating multiple near-real-world tampering training samples online by pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tampering localization neural network includes:

[0026] By pre-constructing a data augmentation module based on a tamper location neural network using multi-task learning and contrastive learning, more tamper training samples are generated by simulating tampering operations online.

[0027] Meanwhile, various post-processing chains that may occur in reality are simulated online to further process the tampered images and generate multiple tampering training samples that closely resemble real-world scenarios online.

[0028] The robust image tampering localization method based on multi-task and contrastive learning, wherein the step of inputting the tampering training sample into the constructed tampering localization neural network based on multi-task and contrastive learning for training, and training the entire tampering localization network to obtain the trained tampering localization neural network includes:

[0029] The modified training samples are input into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, thereby training the entire tampering localization network.

[0030] The tamper location network is a network with pixel-level tamper location capability obtained through supervised learning using training samples; it is called a trained tamper location neural network.

[0031] The robust tampered image localization method based on multi-task and contrastive learning, wherein the feature extraction module includes multiple sequentially connected sub-feature extraction units, each sub-feature extraction unit including an overlapping encoder, an efficient self-attention mechanism unit, a hybrid feedforward network block, and a feature layer, which are sequentially connected.

[0032] The overlapping encoder is used to extract overlapping features from different regions of the features output by the image or the previous transformer block.

[0033] The efficient attention mechanism unit is used to improve the traditional attention mechanism;

[0034] The hybrid feedforward network block is used to introduce position information into the transformer block by replacing the traditional position encoder with a convolutional layer;

[0035] The feature layer is only for the purpose of showing readers the intermediate processes involved.

[0036] The robust image tampering localization method based on multi-task and contrastive learning, wherein the data augmentation module that pre-constructs a tampering localization neural network based on multi-task and contrastive learning includes: an online tampering module and a post-processing operation chain simulation module connected in sequence;

[0037] The online tampering module is used to further tamper with the training images through three tampering operations: splicing, copying and pasting, and removing, to obtain a variety of tampered images.

[0038] The post-processing operation module is used to simulate various post-processing operation chains that may exist in real life by randomly using rotation, flipping, JPEG compression, mean filtering, scaling, median filtering, Gaussian filtering, and Gaussian noise operations, and to perform post-processing operations on diverse tampered images.

[0039] A robust image tampering localization device based on multi-task and contrastive learning, wherein the device comprises:

[0040] Pre-built modules are used to pre-build tamper localization neural networks based on multi-task learning and contrastive learning;

[0041] The tampering training sample generation control module is used to generate multiple tampering training samples that closely resemble real-world scenarios online by pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tampering localization neural network.

[0042] The training control module is used to input the tampered training samples into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, train the entire tampering localization network, and obtain the trained tampering localization neural network.

[0043] The positioning control module is used to input the image to be tested into the trained tamper positioning neural network, perform tamper positioning on the image to be tested, and output the positioning result.

[0044] A smart terminal includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to implement the steps of any one of the methods when executed by one or more processors.

[0045] A non-transitory computer-readable storage medium, wherein, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described above.

[0046] The beneficial effects of this invention are as follows: This invention provides a robust image tampering localization method based on multi-task and contrastive learning. This invention recognizes that tampering inevitably leaves traces in an image, and a trained localization network can capture these traces. This invention pre-constructs a tampering localization neural network based on multi-task learning and contrastive learning, which can detect the presence of tampering traces in a tampered image with high accuracy and robustness, thereby achieving tampering localization. This method can play an important role in various practical applications involving image security. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the robust tampered image localization method based on multi-task and contrastive learning provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of a deep neural network that is pre-constructed to restore the image before tampering localization, based on multi-task and contrastive learning, as provided in the embodiments of the present invention.

[0050] Figure 3 This is a functional principle block diagram of the data augmentation module of the robust tampered image localization method based on multi-task and contrastive learning provided in the embodiments of the present invention.

[0051] Figure 4 This is a table showing the tampering localization performance data of different model variants of the robust tampering image localization method based on multi-task and contrastive learning provided by this invention.

[0052] Figure 5 This is a graph showing the localization performance of various tampering localization methods based on multi-task and contrastive learning, as provided in the embodiments of the present invention, on the CASIAv1, NIST16, Columbia, DSO, and IMD2020 datasets.

[0053] Figure 6This is a diagram illustrating the robustness of various tampering localization methods based on multi-task and contrastive learning for social media transmission processing, as described in this invention.

[0054] Figure 7 This is a block diagram illustrating the principle of a robust tampered image localization device based on multi-task and contrastive learning, as provided in an embodiment of the present invention.

[0055] Figure 8 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0058] Several methods for tampering with localization have been proposed in the existing technology, but the features extracted by tampering with localization in the existing technology lack sufficient robustness and generalization, resulting in poor accuracy of tampering with localization.

[0059] To address the problems of existing technologies, this invention discloses a robust tampering image localization method based on multi-task and contrastive learning. This invention proposes a tampering localization neural network based on multi-task learning and contrastive learning. Through the multi-task learning strategy, the shared encoder, needing to optimize two tasks simultaneously, can effectively reduce overfitting of extracted features. Simultaneously, the introduction of a contrastive learning module makes features within the same category more similar, while keeping features from different categories as far apart as possible, further improving the robustness of the entire framework.

[0060] Exemplary methods

[0061] like Figure 1 As shown in the illustration, this embodiment of the invention provides a robust image tampering localization method based on multi-task and contrastive learning, which can be applied to smart terminals. In this embodiment, the method includes the following steps:

[0062] Step S100: Pre-construct a tamper localization neural network based on multi-task learning and contrastive learning; the tamper localization neural network based on multi-task learning and contrastive learning includes: a data augmentation module, a feature extraction module, a contrastive learning module, a localization decoder, and a restoration decoder;

[0063] The data augmentation module is used to simulate possible tampering and post-processing operations in real-world scenarios, thereby increasing the diversity of training data and reducing the differences from the displayed scenario data.

[0064] The feature extraction module is connected to the data augmentation module. The feature extraction module is mainly used to extract features related to the two tasks of image restoration and tamper location.

[0065] The contrastive learning module is connected to the feature extraction module. The contrastive learning module is mainly used to constrain the learned features at the feature level using contrastive learning, so that the features of the same type (tampered regions and tampered regions, original regions and original regions) are more concentrated, while the features of different types (tampered regions and original regions) are kept as far apart as possible, thereby improving the robustness and generalization of the entire model.

[0066] The positioning decoder is connected to the feature extraction module, and the positioning decoder is mainly used to generate the result of tampering with the positioning.

[0067] The restoration decoder is connected to the feature extraction module, and the restoration decoder is mainly used to generate the restored image.

[0068] Specifically, such as Figure 3 As shown, Figure 3 This is a functional principle block diagram of the data augmentation module of the robust tampered image localization method based on multi-task and contrastive learning provided in the embodiments of the present invention. The data augmentation module of the present invention, which is pre-constructed based on multi-task learning and contrastive learning tampering localization neural network, includes: an online tampering module and a post-processing operation chain simulation module connected in sequence.

[0069] The online tampering module uses three common tampering operations (splicing, copying and pasting, and removing) to further tamper with the training images, resulting in diverse tampered images and thus increasing the diversity of the training samples.

[0070] Post-processing module: By randomly using operations such as rotation, flipping, JPEG compression, mean filtering, scaling, median filtering, Gaussian filtering, and Gaussian noise, it simulates various post-processing operation chains that may exist in real life, and performs post-processing operations on diverse tampered images, thereby reducing the difference between the distribution of training data and actual test data, and further improving the generalization of the tampering localization model.

[0071] Step S200. By pre-constructing a data augmentation module for a tamper location neural network based on multi-task learning and contrastive learning, more tamper training samples that closely resemble real-world scenarios are generated online for training the constructed tamper location neural network based on multi-task learning and contrastive learning.

[0072] Step S300. Input the tampered training sample into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, train the entire tampering localization network, and obtain the trained tampering localization neural network;

[0073] In this embodiment of the invention, during the training phase, the data augmentation module simulates possible tampering and post-processing operations in a real-world scenario, thereby increasing the diversity of training data and reducing the difference from the displayed scenario data. The feature extraction module receives the image output by the data augmentation module and extracts features related to the two tasks of image restoration and tamper localization from the input image. The contrastive learning module uses contrastive learning to constrain the features learned by the feature extraction module, making features of the same type (tampered regions and tampered regions, original regions and original regions) more concentrated, while features of different types (tampered regions and original regions) are kept as far apart as possible, thereby improving the robustness and generalization of the entire model. The restoration decoder module receives the features constrained by the contrastive learning module, trains using a restoration loss, and outputs the restored image. Simultaneously, the restored image can serve as input to the feature extraction module, providing another version of the image for training the feature extraction module. The localization decoder receives the features constrained by the contrastive learning module, trains using a localization loss, and outputs the localization result.

[0074] Step S400. Input the image to be tested into the trained tamper localization neural network, perform tamper localization on the image to be tested, and output the localization result.

[0075] In this embodiment of the invention, during the testing or application phase, the test image is used as the input of the feature extraction module. After the feature extraction module extracts features from the image, it is sent to the positioning decoder to directly obtain the tampering positioning result of the test image.

[0076] In this embodiment of the invention, the data augmentation module and the contrastive learning module are used only during the training phase and not during the testing phase.

[0077] The principle of this invention is that when image editing software tampers with an image, it leaves traces of the tampering in the tampered area. The robust image tampering localization method proposed in this invention, based on multi-task and contrastive learning, learns robust features from training data to distinguish between tampered and original areas, thereby achieving the function of locating the tampered area in the tampered image.

[0078] In a specific embodiment of the present invention:

[0079] (1) A multi-task network is constructed by sharing the same encoder and using specific decoders for different tasks. Specifically, joint optimization is performed on both the restoration and localization tasks. Since both tasks aim to learn semantically independent features, the shared encoder can learn more robust features when optimizing these two tasks together.

[0080] (2) This invention introduces a contrastive learning module to constrain the features extracted by the feature encoder, making the features extracted by the same category more concentrated, while the features extracted by different categories are kept as far apart as possible, thereby further improving the robustness of the extracted features.

[0081] In step S200 of this embodiment of the invention, a data augmentation module is used to generate more tampered samples that closely resemble real-world scenarios online for network training. Specifically, more tampered training samples are generated by simulating tampering operations online for training. Simultaneously, various post-processing chains that may occur in practice are simulated online to further process the tampered images, generating multiple tampered training samples that closely resemble real-world scenarios online. This reduces the distributional discrepancy between training data and actual test data, thereby improving the robustness of the entire localization model.

[0082] In step S300, specifically: the tampered training samples are input into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, and the entire tampering localization network is trained; the tampering localization network is a network with pixel-level tampering localization capability obtained by supervised learning through training samples, and is called the trained tampering localization neural network.

[0083] In step S400, the trained network is used to locate the tampering in the image to be tested and output the location result.

[0084] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0085] In this embodiment of the invention, the specific steps for constructing the training sample dataset and the test dataset for the pre-built tamper localization neural network based on multi-task learning and contrastive learning are as follows:

[0086] This invention can use the publicly available CASIAv2 dataset as training data for its method, and CASIAv1, NIST16, Columbia, DSO, IMD2020, and their social transmission versions as final test data. CASIAv2 is a dataset of manipulated images, and this invention selects 5123 manipulated images to train the tamper localization neural network constructed based on multi-task learning and contrastive learning.

[0087] Implement a tamper localization method based on multi-task and contrastive learning:

[0088] like Figure 1 As shown, the tamper localization neural network constructed in this invention based on multi-task learning and contrastive learning includes: a data augmentation module, a feature extraction module, a contrastive learning module, a localization decoder, and a restoration decoder. The feature extraction module comprises multiple sequentially connected sub-feature extraction units. Each sub-feature extraction unit includes a sequentially connected overlapping encoder, an efficient self-attention mechanism unit, a hybrid feedforward network block, and a feature layer. The overlapping encoder is used to extract overlapping features from different regions of the image or the features output by the previous transformer block.

[0089] The efficient attention mechanism unit reduces the computational complexity of attention by improving upon traditional attention mechanisms.

[0090] The hybrid feedforward network block introduces position information into the transformer block by replacing the traditional position encoder with a convolutional layer.

[0091] The feature layer is only for the purpose of showing the intermediate processes to the reader.

[0092] The tamper localization neural network based on multi-task learning and contrastive learning in this embodiment of the invention mainly consists of a shared encoder, two task-specific decoders, and a data augmentation module.

[0093] Training the location-tampering network:

[0094] This invention implements the proposed method using PyTorch 1.9.0 and conducts all experiments on an NVIDIA A100 GPU (a type of graphics card). PyTorch is an open-source Python machine learning library.

[0095] like Figure 2As shown, this invention uses a SegFormer (semantic analysis) feature extraction module based on a MiT-B5 encoder as the backbone network of the proposed method, and initializes it with ImageNet-1K (a dataset) pre-trained weights. During the training and testing phases, the image is first scaled to 512x512, and then input into a tamper localization neural network based on multi-task learning and contrastive learning. The batch size is set to 12, and the learning rate is set to 10. -4 The number of epochs during the training phase was set to 40. After each epoch, the learning rate was reduced by a factor of 0.9.

[0096] The principle of this invention is as follows: When image editing software tampers with an image, it leaves tampering traces in the tampered area, while the untampered area does not have these traces. The tampering localization neural network proposed in this invention, based on multi-task learning and contrastive learning, is a tampering localization framework that can learn more robust features and better capture tampering traces, thereby achieving the function of locating tampered images.

[0097] The following are some specific embodiments and experimental results of the method of the present invention:

[0098] Example 1

[0099] To verify the effectiveness of the proposed framework, this invention compared the performance of the tampered localization network on several public datasets under different conditions. The test results are as follows: Figure 4 As shown in the table, Figure 4 The table shows the tamper localization performance data of different model variants in the embodiments of the present invention. As can be seen from the table, when the data augmentation module, the contrastive learning module, and the multi-task learning module are used simultaneously, the proposed tamper localization network achieves the highest average performance on the CASIAv1, NIST16, Columbia, and IMD2020 datasets. The average F1 score is improved from 0.504 to 0.607 compared to the baseline model.

[0100] Example 2

[0101] To verify that the method proposed in this invention has better performance than current tamper location methods, this invention compares the proposed method with current tamper location methods, and the results are as follows: Figure 5 As shown, Figure 5 This table shows the localization performance of various tampering localization methods of the present invention on the CASIAv1, NIST16, Columbia, DSO, and IMD2020 datasets. From... Figure 5As can be seen, the method proposed in this invention achieves better performance on most test sets and the best average performance compared to other tamper location methods.

[0102] Example 3

[0103] In real-world scenarios, tampered images may be transmitted through various social media platforms, and current tamper detection algorithms show a significant decrease in performance under such conditions. Therefore, this invention verifies the robustness of the proposed method under different social media platforms. The test results are shown in Table 3. Figure 6 This is a diagram illustrating the robustness of various tampering location methods in embodiments of the present invention for handling social media transmissions. From... Figure 6 As shown in the figures, the method proposed in this invention achieves the best performance when faced with various social media transmission scenarios, including Facebook, WhatsApp, Weibo, and WeChat. This further demonstrates the robustness of the algorithm proposed in this invention for social media post-processing.

[0104] Exemplary device

[0105] like Figure 7 As shown in the figure, an embodiment of the present invention provides a robust tamper-proof image localization device based on multi-task and contrastive learning, the device comprising:

[0106] Pre-built module 510 is used to pre-build a tamper localization neural network based on multi-task learning and contrastive learning;

[0107] The tampering training sample generation control module 520 is used to generate multiple tampering training samples that closely resemble real-world scenarios online by pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tampering localization neural network.

[0108] The training control module 530 is used to input the tampered training sample into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, train the entire tampering localization network, and obtain the trained tampering localization neural network.

[0109] The positioning control module 540 is used to input the image to be tested into the trained tampering positioning neural network, perform tampering positioning on the image to be tested, and output the positioning result, as described above.

[0110] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 8As shown, the smart terminal includes a processor, memory, network interface, display screen, and camera connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a robust tamper-proof image localization method based on multi-task and contrastive learning. The display screen can be an LCD screen or an e-ink screen, and the camera is pre-installed within the smart terminal.

[0111] Those skilled in the art will understand that Figure 8 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0112] In one embodiment, a smart terminal is provided, including a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0113] A tamper localization neural network based on multi-task learning and contrastive learning is pre-constructed;

[0114] By pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tamper localization neural network, multiple tamper training samples that closely resemble real-world scenarios are generated online.

[0115] The modified training samples are input into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, and the entire tampering localization network is trained to obtain the trained tampering localization neural network.

[0116] The image to be tested is input into the trained tamper localization neural network, which performs tamper localization on the image and outputs the localization result, as described above.

[0117] The step of pre-constructing the tamper localization neural network based on multi-task learning and contrastive learning includes:

[0118] A tamper localization neural network based on multi-task learning and contrastive learning is pre-constructed;

[0119] The tamper localization neural network based on multi-task learning and contrastive learning includes: a data augmentation module, a feature extraction module, a contrastive learning module, a localization decoder, and a restoration decoder;

[0120] The data augmentation module is used to simulate possible tampering and post-processing operations in real-world scenarios.

[0121] The feature extraction module is connected to the data augmentation module. The feature extraction module is used to receive the image output by the data augmentation module and extract features related to the two tasks of image restoration and tamper location from the input image.

[0122] The contrastive learning module is connected to the feature extraction module. The contrastive learning module is used to constrain the learned features at the feature level using contrastive learning, so that the similar features of the tampered region and the original region are concentrated, while the dissimilar features of the tampered region and the original region are far apart.

[0123] The positioning decoder is connected to the feature extraction module, and the positioning decoder is used to generate the result of tampered positioning.

[0124] The restoration decoder is connected to the feature extraction module, and the restoration decoder is used to generate the restored image.

[0125] The robust image tampering localization method based on multi-task and contrastive learning, wherein the step of inputting the tampering training sample into the constructed tampering localization neural network based on multi-task and contrastive learning for training, and training the entire tampering localization network to obtain the trained tampering localization neural network includes:

[0126] During the training phase, data augmentation modules are used to simulate possible tampering and post-processing operations in real-world scenarios.

[0127] The feature extraction module receives the image output by the data augmentation module and extracts features related to the two tasks of image restoration and tamper localization from the input image.

[0128] By using the contrastive learning module, the features learned by the feature extraction module are constrained, so that similar features of the tampered region and the original region are concentrated, while dissimilar features of the tampered region and the original region are far apart.

[0129] The restoration decoder module receives features constrained by the contrastive learning module, trains it using restoration loss, and outputs the restored image. Simultaneously, the restored image serves as input to the feature extraction module, providing another version of the image for training the feature extraction module, resulting in the trained tamper localization neural network.

[0130] The step of generating multiple near-real-world tamper training samples online by pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tamper localization neural network includes:

[0131] By pre-constructing a data augmentation module based on a tamper localization neural network using multi-task learning and contrastive learning, more tamper training samples are generated by simulating tampering operations online.

[0132] Meanwhile, various post-processing chains that may occur in reality are simulated online to further process the tampered images and generate multiple tampering training samples that closely resemble real-world scenarios online.

[0133] The step of inputting the tampered training samples into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, and training the entire tampering localization network to obtain the trained tampering localization neural network includes:

[0134] The modified training samples are input into the constructed tamper location neural network based on multi-task learning and contrastive learning for training, thereby training the entire tamper location network.

[0135] The tamper location network is a network with pixel-level tamper location capability obtained through supervised learning using training samples; it is called a trained tamper location neural network.

[0136] The feature extraction module includes multiple sub-feature extraction units connected in sequence. Each sub-feature extraction unit includes an overlapping encoder, an efficient self-attention mechanism unit, a hybrid feedforward network block, and a feature layer connected in sequence.

[0137] The overlapping encoder is used to extract overlapping features from different regions of the features output by the image or the previous transformer block.

[0138] The efficient attention mechanism unit is used to improve the traditional attention mechanism;

[0139] The hybrid feedforward network block is used to introduce position information into the transformer block by replacing the traditional position encoder with a convolutional layer;

[0140] The feature layer is only for the purpose of showing the intermediate processes to the reader.

[0141] The robust image tampering localization method based on multi-task and contrastive learning, wherein the data augmentation module that pre-constructs a tampering localization neural network based on multi-task and contrastive learning includes: an online tampering module and a post-processing operation chain simulation module connected in sequence;

[0142] The online tampering module is used to further tamper with the training images through three tampering operations: splicing, copying and pasting, and removal, resulting in a variety of tampered images.

[0143] The post-processing operation module is used to simulate various post-processing operation chains that may exist in real life by randomly using rotation, flipping, JPEG compression, mean filtering, scaling, median filtering, Gaussian filtering, and Gaussian noise operations, and to perform post-processing operations on diverse tampered images, as described above.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0145] In summary, this invention discloses a robust method, apparatus, smart terminal, and storage medium for tampering image localization based on multi-task and contrastive learning. This invention leverages the fact that tampering inevitably leaves traces in images, and a trained localization network can capture these traces. This invention pre-constructs a tampering localization neural network based on multi-task learning and contrastive learning, which can detect the presence of tampering traces in tampered images with high accuracy and robustness, thereby achieving tampering localization. This method can play an important role in various practical applications involving image security.

[0146] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A robust method for tampering image localization based on multi-task and contrastive learning, characterized in that, The method includes: A tamper localization neural network based on multi-task learning and contrastive learning is pre-constructed; By pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tamper localization neural network, multiple tamper training samples that closely resemble real-world scenarios are generated online. The step of generating multiple near-real-world tamper training samples online by pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tamper localization neural network includes: By pre-constructing a data augmentation module based on a tamper localization neural network using multi-task learning and contrastive learning, more tamper training samples are generated by simulating tampering operations online. Meanwhile, various post-processing chains that may occur in reality are simulated online to further process the tampered images and generate multiple tampered training samples that closely resemble real-world scenarios online. The steps of pre-constructing a tamper localization neural network based on multi-task learning and contrastive learning include: A tamper localization neural network based on multi-task learning and contrastive learning is pre-constructed; The data augmentation module is used to simulate possible tampering and post-processing operations in real-world scenarios. The tamper localization neural network based on multi-task learning and contrastive learning includes: a data augmentation module, a feature extraction module, a contrastive learning module, a localization decoder, and a restoration decoder; The feature extraction module is connected to the data augmentation module. The feature extraction module is used to receive the image output by the data augmentation module and extract features related to the two tasks of image restoration and tamper location from the input image. The contrastive learning module is connected to the feature extraction module. The contrastive learning module is used to constrain the learned features at the feature level using contrastive learning, so that the similar features of the tampered region and the original region are concentrated, while the dissimilar features of the tampered region and the original region are far apart. The positioning decoder is connected to the feature extraction module, and the positioning decoder is used to generate the result of tampered positioning. The restoration decoder is connected to the feature extraction module, and the restoration decoder is used to generate the restored image; The feature extraction module includes multiple sub-feature extraction units connected in sequence. Each sub-feature extraction unit includes an overlapping encoder, an efficient self-attention mechanism unit, a hybrid feedforward network block, and a feature layer connected in sequence. The overlapping encoder is used to extract overlapping features from different regions of the features output by the image or the previous transformer block. The efficient self-attention mechanism unit is used to improve the traditional attention mechanism; The hybrid feedforward network block is used to introduce position information into the transformer block by replacing the traditional position encoder with a convolutional layer; The feature layer is only for the purpose of showing the intermediate process to the reader; The pre-built data augmentation module based on multi-task learning and contrastive learning tamper localization neural network includes: The online tampering module and the post-processing operation chain simulation module are connected in sequence; The online tampering module is used to further tamper with the training images through three tampering operations: splicing, copying and pasting, and removal, resulting in a variety of tampered images. The post-processing operation chain simulation module is used to simulate various post-processing operation chains that may exist in real life by randomly using rotation, flipping, JPEG compression, mean filtering, scaling, median filtering, Gaussian filtering, and Gaussian noise operations, and to perform post-processing operations on diverse tampered images. Data augmentation modules can increase the diversity of training data and reduce the discrepancies with the data in the display scenario; The data augmentation module generates multiple tampered training samples that closely resemble real-world scenarios online, thereby reducing the distributional discrepancy between training data and actual test data and improving the robustness of the entire localization model. The modified training samples are input into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, and the entire tampering localization network is trained to obtain the trained tampering localization neural network. The image to be tested is input into the trained tamper localization neural network, which performs tamper localization on the image and outputs the localization result.

2. The robust image tampering localization method based on multi-task and contrastive learning according to claim 1, characterized in that, The step of inputting the modified training samples into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, and training the entire tampering localization network to obtain the trained tampering localization neural network includes: During the training phase, data augmentation modules are used to simulate possible tampering and post-processing operations in real-world scenarios. The feature extraction module receives the image output by the data augmentation module and extracts features related to the two tasks of image restoration and tamper localization from the input image. By using the contrastive learning module, the features learned by the feature extraction module are constrained, so that similar features of the tampered region and the original region are concentrated, while dissimilar features of the tampered region and the original region are far apart. The restoration decoder module receives features constrained by the contrastive learning module, trains it using restoration loss, and outputs the restored image. Simultaneously, the restored image serves as input to the feature extraction module, providing another version of the image for training the feature extraction module, resulting in the trained tamper localization neural network.

3. The robust image tampering localization method based on multi-task and contrastive learning according to claim 1, characterized in that, The step of inputting the modified training samples into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, and training the entire tampering localization network to obtain the trained tampering localization neural network includes: The modified training samples are input into the constructed tamper location neural network based on multi-task learning and contrastive learning for training, thereby training the entire tamper location network. The tamper location network is a network with pixel-level tamper location capability obtained through supervised learning using training samples; it is called a trained tamper location neural network.

4. A robust image tampering localization device based on multi-task and contrastive learning, characterized in that, The apparatus is applied to the robust tamper-proof image localization method based on multi-task and contrastive learning as described in any one of claims 1-3, and the apparatus comprises: Pre-built modules are used to pre-build tamper localization neural networks based on multi-task learning and contrastive learning; The tampering training sample generation control module is used to generate multiple tampering training samples that closely resemble real-world scenarios online by pre-constructing a data augmentation module based on multi-task learning and contrastive learning-based tampering localization neural network. The training control module is used to input the tampered training samples into the constructed tampering localization neural network based on multi-task learning and contrastive learning for training, train the entire tampering localization network, and obtain the trained tampering localization neural network. The positioning control module is used to input the image to be tested into the trained tamper positioning neural network, perform tamper positioning on the image to be tested, and output the positioning result.

5. A smart terminal, characterized in that, The method includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors to implement the steps of the method as described in any one of claims 1-3.

6. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-3.

Images