Deep image inpainting tampering detection method based on adaptive tampering trace learning

Abstract

The application discloses a deep image inpainting tampering detection method based on adaptive tampering trace learning, and belongs to the technical field of digital image forensics. The method first processes the input image by using an adaptive differential convolution module to suppress image content and enhance tampering traces; then, multi-scale and fine-grained features are extracted in parallel by a multi-scale hollow convolution module and a dense connection network; next, a neural network structure search module is used to automatically optimize the feature extraction path to adapt to diversified tampering types; then, a global and local double-branch attention enhancement module is used to fuse multi-level features and simultaneously improve the internal consistency and boundary accuracy of the tampering area; finally, a decoder module is used to output a pixel-level tampering area mask. The application can adaptively learn tampering features and maintain high precision and strong robustness under post-processing conditions such as JPEG compression, scaling, noise addition and the like, and is suitable for fields such as digital forensics and media content security.

Description

Technical Field

[0001] This invention relates to the fields of digital image processing and information security technology, and in particular to an image forensics method based on deep learning, specifically a deep image restoration and tamper detection method based on adaptive tamper trace learning. Background Technology

[0002] In recent years, with the rapid development of deep learning technologies such as Generative Adversarial Networks (GANs), deep image inpainting technology has made significant progress. This type of technology can seamlessly remove specific objects, text, or watermarks from images and generate alternative content that is visually highly consistent with the original scene.

[0003] Existing methods for detecting deep image restoration tampering mostly rely on convolutional neural networks (CNNs) to extract subtle traces of tampering. These methods typically employ fixed preprocessing operators at the network input, such as high-pass filters or constrained convolutions, to suppress strong content signals in the original image, thereby highlighting potential tampering traces. However, these fixed filters lack adaptability and cannot dynamically adjust according to different image content, diverse restoration algorithms, and varying degrees of post-processing (such as JPEG compression, noise addition, and image scaling), resulting in insufficient detection performance and robustness in complex scenes.

[0004] Furthermore, most current mainstream detection network structures are manually designed fixed architectures. This "one-size-fits-all" design is difficult to adapt to varying data distributions and tampering patterns, resulting in decreased generalization ability and significantly reduced detection accuracy when facing cross-datasets or unknown tampering types. Simultaneously, many methods fail to effectively combine global contextual information with local boundary details, easily leading to blurring, breaks, or missed detections at region edges when locating tampered areas, thus affecting the accuracy of localization.

[0005] Therefore, there is an urgent need in this field for a novel deep image restoration tampering detection technology that can adaptively learn and enhance tampering trace features, be robust to various restoration algorithms and post-processing, and accurately locate the boundaries of the tampered area to improve the accuracy and reliability of digital image forensics. Summary of the Invention

[0006] This invention provides a deep image restoration tampering detection method based on adaptive tampering trace learning. The main purpose is to adaptively learn and enhance tampering trace features, be robust to various restoration algorithms and post-processing, and accurately locate the boundaries of tampered areas, thereby improving the accuracy and reliability of digital image forensics.

[0007] In a first aspect, the present invention provides a deep image restoration tampering detection method based on adaptive tampering trace learning, comprising the following steps:

[0008] S1. Acquire the digital image to be detected and input the digital image into the adaptive differential convolution module. The adaptive differential convolution module processes the digital image through a convolution kernel with updatable weights to suppress image content and enhance tampering traces, thereby obtaining the first tampering feature.

[0009] S2. The first tampering feature is input in parallel to the front end of the dense connection network of the multi-scale dilated convolution module and the dense connection network module; the multi-scale dilated convolution module extracts features under different receptive fields to generate multi-scale tampering features; the front end of the dense connection network extracts fine-grained features to generate front end depth features;

[0010] S3. Input the multi-scale tampering features and the front-end deep features into the neural network structure search module, and perform search and optimization by the preset operation combination in the neural network structure search module to generate high-level tampering features;

[0011] S4. Input the front-end deep features into the back-end of the dense connection network of the dense connection network module to extract deep features and generate back-end deep features;

[0012] S5. The high-level tampering features and the back-end deep features are input into the attention enhancement module for feature fusion and enhancement. The global attention branch in the attention enhancement module enhances global semantic consistency, and the local attention branch optimizes the boundary features of the tampered region, thereby obtaining the enhanced fused features.

[0013] S6. The enhanced fusion features are input to the decoder module, which performs upsampling and feature reconstruction to output a pixel-level tampering region mask corresponding to the size of the digital image.

[0014] As an optional implementation of the first aspect of this application, in step S1, the adaptive differential convolution module includes at least one convolutional layer, the convolutional kernel size of the convolutional layer is 3×3, the stride is 1, and the padding method is SAME; the weights of the convolutional kernel are iteratively updated during training using the gradient descent algorithm.

[0015] As an optional implementation of the first aspect of this application, in step S2, the multi-scale dilated convolution module includes at least four sets of parallel convolutional units, the dilation rates of the four sets of convolutional units being 1, 2, 4 and 8 respectively; the kernel size of each set of convolutional units is 3×3, and the activation function is the modified linear unit ReLU; the multi-scale dilated convolution module concatenates the output features of the four sets of convolutional units in the channel dimension, and performs dimensionality reduction and fusion through a 1×1 convolutional layer to generate the multi-scale tampering features.

[0016] As an optional implementation of the first aspect of this application, in steps S2 and S4, the front end and the back end of the dense connection network are composed of multiple dense blocks and transition layers stacked alternately; wherein, each dense block includes multiple convolutional units, and the input of each convolutional unit is formed by concatenating the input features of the dense block with the output features of all previous convolutional units in the dense block; each transition layer includes a 1×1 convolutional layer and an average pooling layer.

[0017] As an optional implementation of the first aspect of this application, in step S3, the neural network structure search module uses multiple micro-network modules to search for differentiable architectures; its preset operation combination search space includes at least standard convolution, dilated convolution, depthwise separable convolution, max pooling, average pooling, and identity mapping.

[0018] As an optional implementation of the first aspect of this application, in step S5, the global attention branch generates a weight matrix by calculating the similarity between any two features at any position on the feature map, and weights the features based on a multi-head attention mechanism; the local attention branch calculates the correlation of local features within a sliding window of a preset size, wherein the size of the sliding window is 3×3, 5×5, or 7×7.

[0019] As an optional implementation of the first aspect of this application, in step S6, the decoder module adopts a U-shaped network structure, including multiple deconvolution layers, and fuses with features from the encoding stage through skip connections; the final output layer of the decoder module adopts a Sigmoid activation function to generate a probabilistic tampering region mask; the method further includes a step of thresholding the tampering region mask to obtain a binarized mask.

[0020] As an optional implementation of the first aspect of this application, the method further includes a model training step, wherein the model training step utilizes a training dataset containing labeled tampered masks to perform end-to-end supervised training on the adaptive differential convolution module, multi-scale dilated convolution module, dense connection network module, neural network structure search module, attention enhancement module, and decoder module; the loss function used in the training is a weighted combination of binary cross-entropy loss and focus loss.

[0021] In a second aspect, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0022] Thirdly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. Adaptive Feature Enhancement: Adaptive differential convolution modules are used to replace traditional fixed filters. The convolution kernel weights can be dynamically adjusted according to the input data, thereby more effectively suppressing image content and highlighting diverse tampering artifacts, significantly improving the targeting and effectiveness of trace extraction.

[0025] 2. Deep fusion of multi-path features: Through the collaborative work of multi-scale dilated convolution, densely connected networks and neural network structure search modules, feature extraction and fusion from different scales, depths and structural paths are achieved, effectively taking into account both global structural information and local fine-grained texture, and enhancing the model's ability to express complex tampering patterns.

[0026] 3. Dynamic optimization of network structure: The introduction of a neural network structure search module enables the network to automatically learn and optimize the topology of its feature extraction, thus overcoming the limitations of manually designing a fixed architecture and greatly improving the model's generalization ability and adaptability to different repair algorithms, different datasets and unknown scenarios.

[0027] 4. Balancing boundary accuracy and global consistency: By combining global and local dual attention mechanisms, it can ensure the consistency of features within the tampered region from a global perspective, while also finely depicting the region boundary from a local perspective, effectively solving the problems of boundary ambiguity and missed detection in traditional methods.

[0028] 5. High robustness: The design of this invention enables it to maintain high detection accuracy and robustness when faced with common post-processing operations such as JPEG compression, noise addition, and image scaling, making it more suitable for complex real-world application scenarios. Attached Figure Description

[0029] Figure 1 This is a flowchart of a deep image restoration and tamper detection method based on adaptive tamper trace learning, according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the network model structure corresponding to a deep image restoration and tamper detection method based on adaptive tamper trace learning according to an embodiment of the present invention.

[0031] Figure 3 This is a network structure diagram of the adaptive differential convolution module in an embodiment of the present invention;

[0032] Figure 4 This is a network structure diagram of the densely connected network module in an embodiment of the present invention.

[0033] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0036] Example 1

[0037] The present invention proposes a deep image inpainting tamper detection method based on adaptive tamper trace learning, the overall process of which is as follows: Figure 1 As shown, its corresponding network model structure is as follows: Figure 2 As shown. The method specifically includes the following steps:

[0038] S1: Acquire the digital image to be detected and input the digital image into the adaptive differential convolution module. The adaptive differential convolution module processes the digital image through a convolution kernel with updatable weights to suppress image content and enhance tampering traces, thereby obtaining the first tampering feature.

[0039] The network's initial module, the Adaptive Differential Convolutional Module, takes a digital image to be detected and feeds it as input. The core idea of ​​this module is to learn a convolutional kernel capable of predicting pixel neighborhood information. By differencing the original pixel values ​​with the predicted values, it amplifies anomalous signals, i.e., tampering traces. Unlike a fixed high-pass filter, the kernel weights in this module are adaptively learned and updated during model training using a gradient descent algorithm.

[0040] Specifically, this module uses iteratively updated weights to predict the input feature map and highlights forgery traces by differencing it from the original feature map. It also incorporates multi-scale feature representations to enhance the model's robustness, ultimately outputting a stable and clear forgery trace feature map. This provides high-quality input for subsequent neural architecture search and attention modules, enabling accurate pixel-level forgery region localization. Specifically, for the input feature map f, the adaptive differential convolution module predicts the resulting feature map f. o The formula is:

[0041]

[0042] Where x represents the current position of the pixel, x n Let x represent the pixel positions around x, S represent the receptive field, b be the bias term, ω be the weight coefficients of the convolution kernel, and θ be a weight parameter used to control the contribution of the input feature map f(x), where θ∈[0,1]. In this way, the adaptive differential convolution module can adaptively capture forgery traces in the image, providing important feature information for subsequent image tampering detection.

[0043] In this embodiment, the adaptive differential convolution module, as follows: Figure 3 As shown, its convolutional layer uses a 3×3 convolutional kernel (the weights of the convolutional kernel are iteratively updated during training using the gradient descent algorithm), with a stride of 1 and a padding method of SAME to keep the feature map size unchanged. After processing, the output feature map (i.e., the first tampered feature) suppresses most of the original image content through adaptive differencing, while enhancing artifacts such as boundary discontinuities and texture inconsistencies introduced by the restoration operation.

[0044] S2: The first tampering feature is input in parallel to the multi-scale dilated convolution module and the front end of the dense connection network; the multi-scale dilated convolution module extracts features under different receptive fields to generate multi-scale tampering features; the front end of the dense connection network extracts fine-grained features to generate front end depth features.

[0045] The first tampering feature output from step S1 is processed in parallel via two paths:

[0046] The input goes through a multi-scale dilated convolutional module. This module contains four sets of parallel convolutional units with dilation rates of 1, 2, 4, and 8, respectively. This allows the network to obtain receptive fields of different sizes at the same layer, thereby simultaneously capturing local details and broader contextual information of the tampered region. The output features of each set of convolutional units are concatenated along the channel dimension, and then fused and reduced in dimensionality by a 1×1 convolution to finally generate multi-scale tampering features.

[0047] Another input goes to the front end of the densely connected network, such as... Figure 2 and Figure 4 As shown, the DenseNet module consists of a front-end and a back-end. Through its unique connection mechanism (each layer's input contains the outputs of all preceding layers), DenseNet achieves extreme feature reuse and efficient gradient propagation. This helps extract very fine-grained tampering traces and effectively suppresses noise artifacts introduced by post-processing such as JPEG compression. The front-end typically contains a first Dense Block and subsequent Transition Layers. Its output represents the depth features of the front-end.

[0048] S3: Input the multi-scale tampering features and the front-end deep features into the neural network structure search module, and perform search and optimization by the preset operation combination in the neural network structure search module to generate high-level tampering features.

[0049] The multi-scale tampering features generated in step S2 are fused with the front-end depth features and fed as input into the neural network structure search module. This embodiment employs three micro-network modules to perform Differentiable Architecture Search (DARTS), automatically finding the optimal computational unit (Cell) structure within a predefined search space. This search space includes various candidate operations, such as 3x3 standard convolution, 5x5 dilated convolution, 3x3 depthwise separable convolution, max pooling, average pooling, and identity mapping (skip connections). By learning the weights of these operations during training, the network can automatically discover the most suitable feature extraction path combinations to distinguish tampered and untampered regions, thereby generating more expressive high-level tampering features.

[0050] S4: Input the front-end deep features into the back-end of the densely connected network to extract deep features and generate back-end deep features.

[0051] The front-end depth features output from the densely connected network in step S2 are then fed into the rear-end of the densely connected network, as follows: Figure 2 and Figure 4 As shown, the back end of a densely connected network typically contains multiple (e.g., three) dense blocks and transition layers between them. By using deeper network layers, tampered features can be further extracted and abstracted to obtain more discriminative deep representations, i.e., back-end deep features.

[0052] S5: Input the high-level tampering features and the back-end deep features into the attention enhancement module for feature fusion and enhancement. The global attention branch in the attention enhancement module enhances global semantic consistency, and the local attention branch optimizes the boundary features of the tampered region, thereby obtaining the enhanced fused features.

[0053] The high-level tampering features output from step S3 are fused with the back-end deep features output from step S4, and then fed into the attention enhancement module. This module contains two parallel branches:

[0054] Global Attention Branch: This branch calculates the correlation between any two pixel locations on the feature map, generating a global attention map. In this way, long-distance dependencies between features can be explicitly modeled, allowing semantically similar regions (both tampered or both real) to mutually enhance each other, thereby improving feature consistency within the tampered region.

[0055] Local attention branch: This branch computes the correlation of local features within a fixed-size sliding window (e.g., 3×3, 5×5, or 7×7). It focuses on information interactions within the neighborhood, helps maintain the local continuity of features, and can particularly enhance the feature responses at the edges of tampered regions, thus making the final localization boundary clearer and more accurate.

[0056] The outputs of the two branches are fused to obtain the final enhanced fused feature.

[0057] S6: The enhanced fusion features are input to the decoder module, which performs upsampling and feature reconstruction to output a pixel-level tampering region mask corresponding to the size of the digital image.

[0058] The enhanced fused features output from step S5 are input to the decoder module. The decoder employs a classic U-shaped network structure, progressively upsampling through a series of deconvolutional layers (or transposed convolutional layers) to restore the spatial resolution of the feature maps. Simultaneously, skip connections are used to fuse the upsampled features from each layer of the decoder with the corresponding feature maps from the encoder (i.e., the part consisting of S1-S5). This design helps recover spatial detail lost during downsampling. The final layer of the decoder uses a sigmoid activation function, outputting a probability mask with the same size as the original input image and pixel values ​​between [0, 1]. Finally, by setting a threshold (e.g., 0.5), the probability mask is binarized; areas with a pixel value of 1 are the detected tampered areas, and areas with a pixel value of 0 are the real areas, thus achieving precise tamper location.

[0059] In practical implementation, the network model involved in the method described in this invention needs to be trained through supervised learning. The training dataset may include publicly available image tampering detection datasets (such as CASIA, COVERAGE, etc.) and artificially synthesized restored tampered images. When artificially synthesizing data, various object masks can be automatically generated using an instance segmentation model, and then various advanced image restoration algorithms can be used to tamper with the images to enhance the model's generalization ability. During training, the loss function used is a weighted combination of binary cross-entropy loss and focal loss. Focal loss can effectively solve the problem of imbalanced positive and negative samples and allow the model to focus more on learning samples that are difficult to classify (such as small tampered regions or blurred boundaries), thereby further improving detection accuracy.

[0060] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a deep image restoration tamper detection method based on adaptive tamper trace learning, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0061] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a deep image restoration tamper detection method based on adaptive tamper trace learning, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0062] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0063] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0064] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0065] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A deep image restoration and tamper detection method based on adaptive tamper trace learning, characterized in that, Includes the following steps: S1. Acquire the digital image to be detected and input the digital image into the adaptive differential convolution module. The adaptive differential convolution module processes the digital image through a convolution kernel with updatable weights to suppress image content and enhance tampering traces, thereby obtaining the first tampering feature. S2. The first tampering feature is input in parallel to the front end of the dense connection network of the multi-scale dilated convolution module and the dense connection network module; the multi-scale dilated convolution module extracts features under different receptive fields to generate multi-scale tampering features; the front end of the dense connection network extracts fine-grained features to generate front end depth features; S3. The multi-scale tampering features and the front-end deep features are input into the neural network structure search module, and the preset operation combination in the neural network structure search module is used to search and optimize to generate high-level tampering features; the neural network structure search module uses multiple micro-network modules to search for differentiable architectures; Its preset operation combination search space includes at least standard convolution, dilated convolution, depthwise separable convolution, max pooling, average pooling, and identity mapping; S4. Input the front-end deep features into the back-end of the dense connection network of the dense connection network module to extract deep features and generate back-end deep features; S5. The high-level tampering features and the back-end deep features are input into the attention enhancement module for feature fusion and enhancement. The global attention branch in the attention enhancement module enhances global semantic consistency, and the local attention branch optimizes the boundary features of the tampered region, thereby obtaining the enhanced fused features. The global attention branch generates a weight matrix by calculating the similarity between any two features on the feature map, and weights the features based on a multi-head attention mechanism. The local attention branch calculates the correlation of local features within a sliding window of a preset size, which is 3×3, 5×5, or 7×7. S6. The enhanced fusion features are input to the decoder module, which performs upsampling and feature reconstruction to output a pixel-level tampering region mask corresponding to the size of the digital image.

2. The deep image restoration and tamper detection method based on adaptive tamper trace learning according to claim 1, characterized in that, In step S1, the adaptive differential convolution module includes at least one convolutional layer with a kernel size of 3×3, a stride of 1, and a padding method of SAME; the weights of the convolutional kernel are iteratively updated during training using a gradient descent algorithm.

3. The deep image restoration and tamper detection method based on adaptive tamper trace learning according to claim 1, characterized in that, In step S2, the multi-scale dilated convolution module includes at least four sets of parallel convolutional units, with dilation rates of 1, 2, 4, and 8 for the four sets of convolutional units, respectively; the kernel size of each set of convolutional units is 3×3, and the activation function is the modified linear unit ReLU; the multi-scale dilated convolution module concatenates the output features of the four sets of convolutional units along the channel dimension, and performs dimensionality reduction and fusion through a 1×1 convolutional layer to generate the multi-scale tampering features.

4. The deep image restoration and tamper detection method based on adaptive tamper trace learning according to claim 1 or 2, characterized in that, In steps S2 and S4, the front and back segments of the dense connection network are composed of multiple dense blocks and transition layers stacked alternately; wherein, each dense block includes multiple convolutional units, and the input of each convolutional unit is formed by concatenating the input features of the dense block with the output features of all previous convolutional units in the dense block; each transition layer includes a 1×1 convolutional layer and an average pooling layer.

5. The deep image restoration and tamper detection method based on adaptive tamper trace learning according to claim 1, characterized in that, In step S6, the decoder module adopts a U-shaped network structure, including multiple deconvolutional layers, and fuses with features from the encoding stage through skip connections; the final output layer of the decoder module adopts a Sigmoid activation function to generate a probabilistic tampering region mask; the method further includes a step of thresholding the tampering region mask to obtain a binarized mask.

6. The deep image restoration and tamper detection method based on adaptive tamper trace learning according to claim 1, characterized in that, The method further includes a model training step, which uses a training dataset containing labeled tampered masks to perform end-to-end supervised training on the adaptive differential convolution module, multi-scale dilated convolution module, dense connection network module, neural network structure search module, attention enhancement module, and decoder module; the loss function used in the training is a weighted combination of binary cross-entropy loss and focus loss.

7. An electronic device, characterized in that, The method includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the deep image restoration tamper detection method based on adaptive tamper trace learning as described in any one of claims 1-6.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the deep image restoration tamper detection method based on adaptive tamper trace learning as described in any one of claims 1-6.

Images