An image inpainting forensics method based on multi-feature dynamic fusion

Abstract

The application discloses an image restoration forensics method based on multi-feature dynamic fusion, and belongs to the technical field of image forensics. Firstly, trace enhancement is performed on an input image by using spatial domain high-pass filtering and frequency domain high-pass filtering, and single input is expanded into multi-input features. Then, efficient feature learning is realized by combining RGB information and enhanced trace features through a multi-feature dynamic fusion encoder. Next, a multi-scale feature enhancement module is introduced to expand the multi-scale perception ability of the network. Finally, pixel-level detection and positioning are performed on the tampered area by using a decoder with a skip connection. The application fully utilizes the complementary advantages of various trace enhancement methods, combines dynamic convolution and multi-scale context information, improves the robustness and generalization performance of the forensics model, and can realize high-precision identification and positioning of the image restoration area in different forensics scenes.

Description

Technical Field

[0001] This invention relates to the field of image forensics technology in computer vision, specifically to an image restoration forensics method based on multi-feature dynamic fusion. Background Technology

[0002] In the digital media era, the emergence of image restoration technology has posed an extremely serious challenge to digital information security. Due to the ease of use of image restoration algorithms, editing and tampering with images has become convenient and widespread, and the altered images are difficult to detect. Given the extremely low cost of maliciously altering images, forged images containing false information may be used in news media, scientific research, and criminal evidence. This will greatly threaten network information security and have serious negative impacts on society.

[0003] Image inpainting refers to the technique of restoring missing areas of an image after removing some information from the original image. Deep learning-based image inpainting offers superior restoration results, posing a greater challenge to image forensics. Image inpainting forensics involves identifying and locating tampered areas in an image. However, research on deep learning-based image inpainting forensics is still limited, and inaccurate location of tampered areas remains a problem. Therefore, there is a need to propose an image forensics method oriented towards deep image inpainting that can achieve more accurate location of tampered areas. Summary of the Invention

[0004] There is limited research on forensic methods for deep image restoration, and existing methods often suffer from inaccurate identification and localization of tampered areas. This invention proposes a multi-feature dynamic fusion-based image restoration forensics method, which achieves superior performance compared to existing forensics techniques across various deep image restoration approaches.

[0005] This invention provides an image restoration and forensics method based on multi-feature dynamic fusion, comprising the following steps: (1) Input expansion: The RGB image to be examined is subjected to two trace enhancement methods, namely spatial domain high-pass filtering and frequency domain high-pass filtering, to generate two residual feature maps, which together with the original RGB image constitute three sets of input features; (2) Multi-feature dynamic fusion encoding: After mapping the three sets of input features to the same number of channels, the channels are stacked, and then the encoder composed of dynamic convolutional blocks and downsampling operations is used to realize the dynamic fusion of multiple features and extract and repair tampered features. (3) Multi-scale feature enhancement: The feature map output by the encoder is input into the multi-scale feature enhancement module. Through the global feature extraction branch and multiple pyramid convolution branches with different kernel sizes, multi-scale context information is extracted and features are aggregated. (4) Decoding and localization: The enhanced feature map is upsampled and stacked by a decoder with skip connections, and finally the classifier outputs the pixel-level repair region detection and localization results.

[0006] In some instances, the spatial domain high-pass filtering in step (1) maps the RGB image to a 3-channel residual feature map by using three high-pass filter kernels with initialized weights as convolution kernels.

[0007] In some instances, the frequency domain high-pass filtering in step (1) specifically involves dividing the RGB image into three single-channel images, performing discrete cosine transform (DCT) on each, removing low-frequency information, inversely transforming back to the spatial domain, and then stacking the channel dimensions to obtain the residual feature map.

[0008] In some instances, low-frequency information is removed by removing the first 1 / 18 of the frequency points in the frequency domain image in zig-zag order, multiplying the frequency domain image by the low-frequency mask, where the low-frequency regions in the low-frequency mask are set to 0, and the remaining regions are set to 1.

[0009] In some instances, when mapping the three sets of input features to the same number of channels in step (2), each input is assigned a convolutional block consisting of two consecutive convolutional units, with each convolutional unit being a 3×3 convolutional layer, a BN layer, and a ReLU activation function in sequence.

[0010] In some instances, the dynamic convolutional block in step (2) sets the number of parallel convolutional kernels to 3, the kernel size to 3×3, and aggregates multiple parallel convolutional kernels after assigning weights using the channel attention mechanism; the downsampling operation uses max pooling with a stride of 2.

[0011] In some instances, step (2) involves performing three dynamic convolutions and two downsampling operations on the feature map after channel stacking, resulting in 32, 64, and 128 channel numbers for the output feature map at input image scales of 1, 1 / 2, and 1 / 4, respectively.

[0012] In some instances, the multi-scale feature enhancement module in step (3) includes: Branch 1: Adaptive average pooling is used to reduce the size of the input feature map space to 11×11. After extracting features with 11×11 convolution kernels, the original size is restored by bilinear interpolation to obtain the global feature map. Other branches: consist of four pyramidal convolutional layers with different kernel sizes, namely 9×9, 7×7, 5×5 and 3×3, with corresponding feature group numbers of 8, 4, 2 and 1, and output feature map channels of 1 / 2, 1 / 4, 1 / 8 and 1 / 8 of the input channel number, respectively. After stacking all branch output feature maps through channels, a 1×1 convolutional layer is used to aggregate multi-scale contextual information.

[0013] In some instances, the decoder in step (4) consists of an upsampling module, a convolutional block, a 1×1 convolutional layer, and a Softmax layer; the upsampling module uses a transposed convolution with a stride of 2; the number of output feature map channels at the 1 / 2 and 1 input image scales are set to 64 and 32, respectively; finally, the 32 channels are mapped to 2 channels through a 1×1 convolution, and the evidence collection result is output by the Softmax classifier.

[0014] In some instances, the method uses the cross-entropy loss function as the network's loss function to train the network model.

[0015] In some instances, the method is applied to tampered images generated by various deep learning-based image restoration algorithms, enabling pixel-level detection and localization of the restoration area.

[0016] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. This invention forms an end-to-end image restoration and forensics method through the organic combination of four steps: input expansion, multi-feature dynamic fusion encoding, multi-scale feature enhancement, and decoding localization. It can achieve pixel-level detection and localization of tampered areas for various depth image restoration schemes, overcoming the problem of inaccurate localization of deep restoration images in existing forensics methods.

[0017] 2. This invention uses three high-pass filter kernels with initialized weights to perform spatial domain high-pass filtering, which can effectively enhance the subtle edge and texture anomalies introduced by the restoration operation in the image, providing richer tampering clues for subsequent feature learning.

[0018] 3. This invention employs Discrete Cosine Transform (DCT) and selectively removes low-frequency information, which can highlight the restoration artifacts in the frequency domain; by quantitatively removing the first 1 / 18 of the frequency components in a zig-zag sequence, it not only preserves the effective frequency domain tampering characteristics but also reduces computational redundancy and enhances the model's sensitivity to frequency domain anomalies.

[0019] 4. This invention uses a convolutional block consisting of two consecutive convolutional units (3×3 convolution + BN + ReLU) to perform feature mapping on three sets of inputs, which ensures the non-linear expressive power of feature extraction. At the same time, the BN layer accelerates model convergence and improves training stability.

[0020] 5. This invention proposes a multi-feature dynamic fusion encoder based on dynamic convolution. By adaptively allocating weights to three parallel convolutional kernels through a channel attention mechanism, it can dynamically fuse RGB information and two trace enhancement features according to the input content, significantly improving the network's generalization ability and robustness to diverse tampering methods. At three scales (original image, 1 / 2, 1 / 4), the number of channels is set to 32, 64, and 128 respectively, achieving a reasonable balance between feature dimension and spatial resolution.

[0021] 6. The multi-scale feature enhancement module proposed in this invention integrates global feature extraction and pyramid convolution multi-branch structure. The global branch provides overall contextual information, and the four branches with different convolutional kernel sizes (9×9 to 3×3) capture restoration artifact features from macro to micro. This effectively solves the problem of insufficient contextual information caused by a single receptive field and significantly improves the classification accuracy and edge localization accuracy of the tampered region.

[0022] 7. This invention employs a decoder structure with skip connections. By upsampling through transposed convolution and stacking it with the skip connection features of the corresponding layer of the encoder, spatial detail information is effectively recovered. Finally, pixel-level binary classification results are output through 1×1 convolution and a Softmax classifier, achieving high-precision localization of the repaired area.

[0023] 8. This invention uses the cross-entropy loss function for network training. This loss function is suitable for pixel-level classification tasks and can effectively measure the difference between the predicted probability distribution and the true label, ensuring that the model converges to the optimal evidence collection performance.

[0024] In summary, compared with existing forensic techniques, this invention achieves significant improvements in quantitative indicators: the F1 score is 16.03% higher and the IoU is 8.59% higher. This indicates that this invention maintains a high recall rate while possessing extremely high precision, enabling more accurate location of the repaired tampered areas in images. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the method provided in an embodiment of the present invention; Figure 2 This is the spatial high-pass filter kernel provided in the embodiments of the present invention; Figure 3This is a schematic diagram of frequency domain removal of low-frequency information provided in an embodiment of the present invention, wherein (a) is a frequency domain image, (b) is a low-frequency part mask, and (c) is the filtered image; Figure 4 This is a schematic diagram of the multi-scale feature enhancement module provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] In the following description, specific embodiments of the invention will be illustrated with reference to steps and symbols performed by one or more computers, unless otherwise stated. Therefore, these steps and operations will be referred to several times as being performed by a computer, and computer execution as referred to herein includes operations by a computer processing unit representing electronic signals of data in a structured format. This operation transforms the data or maintains it at a location in the computer's memory system, which can be reconfigured or otherwise alter the operation of the computer in a manner well known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of the invention described above are not intended to be limiting, and those skilled in the art will understand that many of the following steps and operations can also be implemented in hardware.

[0029] The terms "module" or "unit" as used herein can be considered as software objects executing on the computing system. Different components, modules, engines, and services described herein can be considered as implementations on the computing system. The apparatus and methods described herein are preferably implemented in software, but can also be implemented in hardware, both of which are within the scope of this invention.

[0030] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0031] This invention provides an image restoration and forensics method based on multi-feature dynamic fusion, comprising the following four main steps: (1) By employing spatial domain high-pass filtering and frequency domain high-pass filtering to extend single-input features to multi-input features; (2) The proposed multi-feature dynamic fusion encoder combines RGB information and subtle trace information to achieve effective feature learning; (3) After encoder feature learning, a multi-scale feature enhancement module is added to expand the network’s multi-scale perspective; (4) Based on the above features, pixel-level detection and localization of the tampered area being repaired are achieved by using a decoder with skip connections.

[0032] Furthermore, this invention expands a single input through two trace enhancement methods, extracts image forensics features through a multi-feature dynamic fusion method of the encoder and a multi-scale enhancement module, and finally achieves identification and forensics of different restoration areas through a decoder with skip connections, thereby providing an accurate and efficient image restoration forensics scheme. The structural diagram is shown below. Figure 1 As shown, it mainly consists of four parts: input expansion, multi-feature dynamic fusion encoder, multi-scale feature enhancement module, and decoder. (1) Input expansion: First, the RGB image is mapped to a 3-channel residual feature map by using three high-pass filter kernels with initialized weights as convolution kernels. The high-pass filter kernels are as follows: Figure 2 As shown. Secondly, the RGB image is treated as three independent single-channel images, each transformed to the frequency domain using Discrete Cosine Transform (DCT). After removing low-frequency information, it is inversely transformed back to the spatial domain, and then the residual feature map is obtained by stacking the channel dimensions. For example... Figure 3 As shown, the low-frequency components of the frequency domain image after DCT transformation are located in the upper left corner. Therefore, the first 1 / 18 of the frequency points in zig-zag order are removed, i.e., the frequency domain image ( Figure 3(a) and low-frequency mask ( Figure 3 (b) The dot product yields the filtered image. Figure 3 (c)). Figure 3 (b) The black area corresponds to the low frequency part and is set to 0, while the remaining white area is set to 1.

[0033] (2) Multi-feature dynamic fusion encoder: Two 3-channel feature maps obtained from RGB images and extended inputs are used as three sets of inputs, and each is mapped to a 16-channel feature map using two convolutional blocks. The convolutional block consists of two consecutive convolutional units. The convolutional unit structure is set as follows: 3×3 convolutional layer, BN layer and ReLU activation function. The three sets of channels are stacked to obtain 48-channel features. Then, the encoder composed of dynamic convolutional blocks and downsampling operations is used to realize the dynamic fusion of multiple features. The dynamic convolutional block sets the number of parallel convolutional kernels to 3 and the kernel size to 3×3. The dynamic convolution uses the channel attention mechanism to assign weights to multiple parallel convolutional kernels and then aggregates them. The downsampling operation is a max pooling operation with a stride of 2. The encoder of this method performs three dynamic convolutions and two downsampling operations on the stacked 48-channel feature map to effectively extract and repair tampered features. The number of channels of the output feature map at the input image scales of 1, 1 / 2 and 1 / 4 are 32, 64 and 128, respectively.

[0034] (3) Multi-scale feature enhancement module: such as Figure 4 As shown, branch one of the multi-scale feature enhancement module employs adaptive average pooling while maintaining reasonable spatial resolution, reducing the spatial size of the input feature map to 11×11. Features are extracted through convolution operations with an 11×11 kernel and a stride of 1, and then bilinear interpolation is used to map them back to the initial size to obtain the global feature map. Other branches consist of pyramid convolutions composed of four convolutional layers with kernels of different sizes, from top to bottom: 9×9, 7×7, 5×5, and 3×3. The corresponding feature group numbers are 8, 4, 2, and 1, respectively, and the number of output feature map channels are 1 / 2, 1 / 4, 1 / 8, and 1 / 8 of the number of input feature map channels, respectively. The output feature maps from all branches are then stacked by channel dimension, and finally, a 1×1 convolutional layer is used to aggregate multi-scale contextual information.

[0035] (4) Decoder: such as Figure 1As shown, the decoder consists of an upsampling module, a convolutional block, a 1×1 convolutional layer, and a Softmax layer. The upsampling module uses a transposed convolution with a stride of 2 to progressively restore the feature map to the original image size. First, the decoder performs two upsampling operations on the feature map output by the multi-scale feature enhancement module. After each upsampling, the feature map of the skip-connected parts is stacked, and a convolutional block operation is inserted. The number of channels of the output feature map at the 1 / 2 and 1 input image scales are set to 64 and 32, respectively. Then, a convolutional layer with a kernel size of 1×1 maps the 32-channel feature map to 2 channels. Finally, a Softmax classifier is used to normalize it to obtain the output, which is the forensic result.

[0036] This invention, through the aforementioned technical solutions, comprehensively utilizes multiple trace enhancement methods to highlight abnormal features. Employing a multi-feature dynamic fusion encoder, it exhibits better forensic performance and generalization capabilities, and can adaptively handle different forensic scenarios. A multi-scale enhancement module is proposed to strengthen multi-scale feature representation, thereby improving global classification accuracy. The method provided by this invention not only directly utilizes RGB image information for detecting visual restoration traces but also simultaneously uses spatial domain high-pass filtering and frequency domain high-pass filtering for trace enhancement, revealing abnormal features of the tampered image and achieving mutual complementarity of abnormal features. A multi-feature dynamic fusion method based on dynamic convolution is proposed, which can fully combine the advantages of multiple effective restoration trace features, greatly improving the network's robustness and forensic performance. A multi-scale enhancement module is proposed to expand the multi-scale field of view, solving the problem of insufficient contextual information caused by the network's local receptive field.

[0037] In another embodiment of the present invention, an image restoration and forensics method based on multi-feature dynamic fusion is proposed. First, spatial domain high-pass filtering and frequency domain high-pass filtering are used to enhance the restoration of tampering traces. Second, a multi-feature dynamic fusion encoder is proposed to extract effective image forensics features. Third, a proposed multi-scale feature enhancement module expands the multi-scale perspective. Finally, a decoder with skip connections is used to locate the tampered area to generate the final forensics result. The specific implementation steps are as follows: Step 1: Obtain the image restoration dataset. Select several representative deep restoration algorithms to construct the image dataset. The image dataset contains the image after being tampered with by the restoration algorithm and its corresponding restoration location label. Step 2: Use the cross-entropy loss function as the network's loss function to train the network model and obtain the trained network model. Step 3: For the image to be used for evidence collection, the RGB image is subjected to spatial domain high-pass filtering and frequency domain high-pass filtering to obtain the feature map after trace enhancement, which is then expanded into multiple inputs; Step 4: The proposed multi-feature dynamic fusion encoder is used to extract and fuse forensic features from the image and the enhanced feature map; Step 5: Use the proposed multi-scale enhancement module to further expand the multi-scale perspective of the extracted and fused features, and enhance the multi-scale feature representation; Step Six: Based on the decoder structure with skip connections, identify and locate the repaired area to obtain the final evidence result.

[0038] The method proposed in this embodiment of the invention is implemented based on the PyTorch framework, and the model is trained and tested in an Ubuntu environment. The forensic performance of the model is evaluated using the obtained image restoration dataset, and different forensic schemes are tested on the test set to obtain the forensic metrics F1 and IOU, as shown in Table 1.

[0039] Table 1 compares the performance of the proposed method with existing methods in terms of F1 (%) and IoU (%).

[0040] Metric Explanation: IOU (Intersection over Union) is one of the most important evaluation metrics for object detection and semantic segmentation. In forensic work, it refers to the ratio of the intersection and union of detected tampered regions and real tampered regions. F1 score is a comprehensive metric for measuring the accuracy of binary classification models, taking into account both precision and recall. Both are standard metrics in the field of forensics, and higher values ​​for both indicate better forensic performance.

[0041] Table 1 shows the quantitative comparison results of different forensic techniques. The data comparison shows that, compared to existing forensic techniques, the method provided in this embodiment of the invention achieves an F1 score 16.03% higher and an IOU 8.59% higher than the suboptimal method. Therefore, this invention has superior forensic performance compared to other forensic methods.

[0042] This invention may be implemented in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered illustrative rather than limiting in all respects, for example: 1) multimedia file types are not limited to images; 2) image restoration schemes are not limited to those used in the embodiments; 3) the selection of network configuration parameters is not limited to the configurations in the embodiments, such as the number of downsampling or upsampling operations, the number of feature channels at different sizes, and the size of the convolutional kernel. The scope of this invention is indicated by the appended claims rather than the foregoing description. All variations falling within the meaning and scope of equivalent technical solutions of the claims are included within their scope.

[0043] The above provides a detailed description of an image restoration and forensics method based on multi-feature dynamic fusion provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for image restoration and forensics based on multi-feature dynamic fusion, characterized in that, include: The RGB image to be examined is subjected to two trace enhancement methods, namely spatial domain high-pass filtering and frequency domain high-pass filtering, to generate two residual feature maps, which together with the original RGB image constitute three sets of input features. After mapping the three sets of input features to the same number of channels, the channels are stacked, and then an encoder consisting of dynamic convolutional blocks and downsampling operations is used to achieve dynamic fusion of multiple features and extract and repair tampered features. The feature map output by the encoder is input into the multi-scale feature enhancement module. Through the global feature extraction branch and multiple pyramid convolution branches with different kernel sizes, multi-scale contextual information is extracted and features are aggregated. The enhanced feature map is upsampled and stacked by a decoder with skip connections, and finally the classifier outputs pixel-level repair region detection and localization results.

2. The method according to claim 1, characterized in that, Spatial domain high-pass filtering uses three high-pass filter kernels with initialized weights as convolution kernels to map an RGB image to a 3-channel residual feature map.

3. The method according to claim 2, characterized in that, The frequency domain high-pass filtering is specifically as follows: the RGB image is divided into three single-channel images, and discrete cosine transform (DCT) is performed on each of them. After removing low-frequency information, the image is inversely transformed back to the spatial domain, and then the residual feature map is obtained by stacking the channel dimensions.

4. The method according to claim 3, characterized in that, The method for removing low-frequency information is as follows: remove the first 1 / 18 of the frequency points of the frequency domain image in zig-zag order, and multiply the frequency domain image with the low-frequency mask. In the low-frequency mask, the low-frequency corresponding area is set to 0, and the remaining areas are set to 1.

5. The method according to claim 4, characterized in that, When mapping the three sets of input features to the same number of channels, each input is used as a convolutional block consisting of two consecutive convolutional units. Each convolutional unit is a 3×3 convolutional layer, a BN layer, and a ReLU activation function in sequence.

6. The method according to claim 5, characterized in that, The dynamic convolutional block sets the number of parallel convolutional kernels to 3, with a kernel size of 3×3. It uses a channel attention mechanism to assign weights to multiple parallel convolutional kernels before aggregation. The downsampling operation uses max pooling with a stride of 2.

7. The method according to claim 6, characterized in that, The feature maps after channel stacking are subjected to three dynamic convolutions and two downsampling operations in sequence. The number of channels in the output feature maps at the input image scales of 1, 1 / 2 and 1 / 4 are 32, 64 and 128, respectively.

8. The method according to claim 7, characterized in that, The multi-scale feature enhancement module includes: Branch 1: Adaptive average pooling is used to reduce the size of the input feature map space to 11×11. After extracting features with 11×11 convolution kernels, the original size is restored by bilinear interpolation to obtain the global feature map. Other branches: consist of four pyramidal convolutional layers with different kernel sizes, namely 9×9, 7×7, 5×5 and 3×3, with corresponding feature group numbers of 8, 4, 2 and 1, and output feature map channels of 1 / 2, 1 / 4, 1 / 8 and 1 / 8 of the input channel number, respectively. After stacking all branch output feature maps through channels, a 1×1 convolutional layer is used to aggregate multi-scale contextual information.

9. The method according to claim 8, characterized in that, The decoder consists of an upsampling module, a convolutional block, a 1×1 convolutional layer, and a Softmax layer. The upsampling module uses a transposed convolution with a stride of 2. The number of output feature map channels is set to 64 and 32 at input image scales of 1 / 2 and 1, respectively. Finally, the 32 channels are mapped to 2 channels through a 1×1 convolution, and the evidence collection result is output by the Softmax classifier.

10. The method according to claim 1, characterized in that, The method uses the cross-entropy loss function as the network loss function to train the network model.

Images