Image restoration forensics method, electronic device and storage medium based on comprehensive similarity deformer and central triplet loss

By using the encoder-decoder architecture, a comprehensive similarity deformer and a central triple loss function in the image repair forensics method, the problem of insufficient global feature extraction capabilities and blurred boundaries in the image repair forensics is solved, and the detection accuracy and freedom are improved. It is awesome and achieves end-to-end image repair and forensics.

CN118552448BActive Publication Date: 2024-11-19TIANJIN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410686413.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-30
Publication Date
2024-11-19
Estimated Expiration
2044-05-30

AI Technical Summary

Technical Problem

The existing image repair forensics methods have problems with insufficient global feature extraction capabilities and blurred boundaries when detecting the repaired area of ​​the image, and have poor robust performance, making it difficult to detect post-processed repaired images.

Method used

Using an encoder-decoder-based infrastructure, the integrated similar deformation and central triple loss function are introduced to build a densely connected dual-stream network, combined with Resnet-50 and SwinTransformer-Tiny, enhance boundary feature representation, and pass periodic cosine annealing. Optimizer and data augmentation technology improve network robustness.

Benefits of technology

More accurate image repair area detection and boundary recognition are achieved, which improves the robustness of the network for post-processed images and can achieve good results on different data sets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118552448B_ABST
    Figure CN118552448B_ABST
Patent Text Reader

Abstract

The present invention relates to an image restoration forensics method based on a comprehensive similarity deformer and a central triplet loss, and belongs to the field of multimedia forensics technology. The prior art has limited ability to extract features of the restoration area, and it is often difficult to accurately identify the boundaries of the restoration area, and has poor robustness when facing post-processing operations. The method disclosed in the present invention builds a forensics network based on the infrastructure of an encoder-decoder. The encoder is composed of a densely connected two-stream backbone network. The dense fusion of local features and global features produces a more effective feature representation. A comprehensive similarity deformer is introduced in the decoder to enhance the tampering boundary features. A central triplet loss function is designed to supervise the image restoration forensics network and enhance the robustness of the model. The method disclosed in the present invention effectively improves the problem of the network's difficulty in accurately identifying the boundaries of the restoration area and poor robustness while improving the accuracy of network restoration forensics.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and pattern recognition, and in particular to an image restoration and forensics method based on a comprehensive similarity deformer and central triplet loss, an electronic device and a storage medium. Background Art

[0002] Image inpainting is a common and effective image editing technique, which aims to fill in and repair missing areas in a visually reasonable way by using the existing information of the image. Due to its powerful image editing capabilities, this technology is widely used in various fields, such as cultural heritage protection, crime evidence analysis, and personal photo restoration. However, image inpainting can also be used maliciously to achieve some unethical or even illegal purposes. With the increasing popularity of deep learning-based inpainting methods, the malicious use of image inpainting is also increasing. For example: malicious users can use image inpainting technology to forge images to deceive the media to report fake news; or to create false evidence to defame others and mislead legal institutions; and malicious users may use image inpainting technology to restore blurred images to violate the privacy of others, etc. In recent years, the issue of multimedia information security has attracted more and more attention. Therefore, it is crucial to accurately detect and locate the inpainted area of ​​the image, and it is worth conducting in-depth research on this.

[0003] Traditional restoration forensics methods mainly determine which area of ​​the image has been tampered with by detecting the similarity between image blocks. Capturing local tampering traces based on convolutional neural networks is a recognized method in the field of restoration forensics. Training on large-scale data sets has achieved significantly better results than traditional methods while maintaining the generalization of the network. However, convolutional neural networks pay more attention to local information and lack the ability to extract global features in restoration forensics tasks, which limits the performance of restoration forensics. In addition, there are few modules in the existing technology that extract features for the boundaries of the restoration area, resulting in blurred boundaries of the restoration area in the detection results. More importantly, malicious image tamperers will use post-processing operations to process the restoration image to cover up the restoration traces. The robust performance of existing technologies is poor, and it is often difficult to detect restoration images that have been post-processed. Summary of the invention

[0004] In view of the above problems, the present invention provides an image restoration forensics method, an electronic device and a storage medium based on a comprehensive similarity deformer and a center triplet loss, in order to solve at least one of the above problems.

[0005] According to a first aspect of the present invention, there is provided an image restoration forensics method based on a comprehensive similarity deformer and a center triplet loss function, comprising:

[0006] An image restoration forensics model is built based on the encoder-decoder infrastructure, and a comprehensive similarity deformer is introduced into the decoder of the image restoration forensics model. The encoder of the image restoration forensics model adopts a densely connected backbone network.

[0007] Obtain public image dataset samples, and use a variety of image restoration methods to process the public image dataset samples, and use the processed image restoration forensic dataset samples to train the image restoration forensic model;

[0008] The pre-designed triple loss function is combined with the cross entropy loss function as the total loss function, and the image restoration forensics model is trained using the total loss function and the image restoration forensics dataset samples to obtain the trained image restoration forensics model;

[0009] The trained image restoration forensics model is used to detect the target restoration image, identify the restored area in the target restoration image, and obtain the restoration forensics result.

[0010] According to an embodiment of the present invention, the image restoration forensics model is built based on the encoder-decoder infrastructure, and the comprehensive similarity deformer is introduced into the decoder of the image restoration forensics model, including:

[0011] A spatial latent enrichment model is added before the Resnet-50 backbone network, and the Resnet-50 with the filter added and the SwinTransformer-Tiny are densely connected to obtain a densely connected two-stream backbone network as an encoder.

[0012] Add a synthetic similarity transformer as a decoder to the original decoder consisting of convolutional layers and upsampling layers;

[0013] The encoder and decoder are spliced ​​together to obtain an image restoration forensics model, and the encoder is used to downsample the target restoration image.

[0014] The decoder is used to upsample the output of the encoder to obtain a restoration forensics result with the same resolution as the target restoration image.

[0015] According to an embodiment of the present invention, the above-mentioned dual-stream backbone network includes a CNN stream composed of Resnet-50 and a Transformer stream composed of SwinTransformer-Tiny.

[0016] According to an embodiment of the present invention, after adjusting the size and number of channels of the output feature map of each stage in the CNN stream, it is spliced ​​into the input of the same-level stage and all upper-level stages in the Transformer stream according to the channel dimension.

[0017] According to an embodiment of the present invention, in the above decoder, the convolutional layers and upsampling layers are repeatedly stacked multiple times, and then a comprehensive similarity transformer is added to enhance the boundary feature representation.

[0018] According to an embodiment of the present invention, the above-mentioned comprehensive similarity deformer includes multiple network flows, each network flow is constructed by inserting a similarity perception layer between multiple convolutional layers, and the multiple network flows can process feature maps of different resolutions;

[0019] Among them, the similarity perception layer is used to detect pixel-level inconsistencies for fine-grained feature modeling. After adjusting the size and number of channels, the feature maps output by multiple network flows are spliced ​​with the output of the first convolution block in the last network flow according to the channel dimension, and the image restoration forensics result is obtained after passing through another convolution layer.

[0020] According to an embodiment of the present invention, a pre-designed triple loss function is combined with a cross entropy loss function as a total loss function, and an image restoration forensics model is trained using the total loss function and a sample of an image restoration forensics dataset, and the trained image restoration forensics model includes:

[0021] The central triplet loss function is designed, and the cross entropy loss function is combined as the total loss function to perform data enhancement on the image restoration forensics dataset samples, obtain enhanced image restoration forensics samples, and pre-train the Resnet-50 and Swin Transformer-Tiny in the image restoration forensics model;

[0022] Based on the periodic cosine annealing method, the image restoration forensics model is trained using the SGD optimizer, enhanced image restoration forensics samples, pre-trained Resnet-50, pre-trained Swin Transformer-Tiny and the total loss function to obtain the trained image restoration forensics model.

[0023] According to an embodiment of the present invention, the image restoration forensics dataset samples are enhanced to obtain enhanced image restoration forensics samples, including:

[0024] The samples of the image restoration forensic dataset are randomly flipped, randomly cropped, randomly noised, and randomly transformed in color space to obtain enhanced image restoration forensic samples.

[0025] According to a second aspect of the present invention, there is provided an electronic device, comprising:

[0026] one or more processors;

[0027] a storage device for storing one or more programs,

[0028] When one or more programs are executed by one or more processors, the one or more processors execute an image restoration forensics method based on a comprehensive similarity deformer and a central triplet loss.

[0029] According to a third aspect of the present invention, there is provided a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute an image restoration forensics method based on a comprehensive similarity deformer and a center triplet loss.

[0030] Compared with the prior art, the image restoration forensics method based on comprehensive similarity deformer and center triplet loss provided by the present invention makes full use of the restoration features extracted from each stream in a densely connected two-stream network, providing richer restoration features for the decoder part; the decoder with the addition of a comprehensive similarity deformer can alleviate the problem of blurred forensic boundaries and enhance the boundary detection effect; the center triplet loss can allow the network to still have good forensic performance when facing post-processed images, thereby improving the robustness of the network; end-to-end image restoration forensics can be achieved, and good results can be achieved on different data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0032] Figure 1 is a flowchart of an image restoration forensics method based on a comprehensive similarity deformer and a center triplet loss function according to an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of the architecture of an image restoration forensics model according to an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of the structure of a comprehensive similarity deformer according to an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of test results and scheme comparison according to an embodiment of the present invention;

[0036] Figure 5 A block diagram of an electronic device suitable for implementing an image restoration forensics method based on a comprehensive similarity deformer and a center triplet loss function according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0038] Image restoration is to fill in missing areas with existing information in the image. However, image restoration can also be used for illegal purposes, and malicious image restoration can lead to many serious consequences. Therefore, image restoration forensics methods are needed to accurately detect and locate the restored areas of the image.

[0039] In response to the possible abuse problems in image restoration, technicians in this field have provided many image restoration forensics methods. For example, the detection method based on zero-connectivity features and fuzzy membership. The suspicious area is screened by zero-connectivity features, and then the restoration blocks in the suspicious area are identified by calculating the fuzzy membership, and the final restoration tampering area is located by the cut set; for example, the central pixel mapping is used to search for suspicious block pairs, which accelerates the search for suspicious blocks in terms of loading factor and search range, and the maximum zero-connectivity area marking and fragment splicing detection technology are used to locate the tampering area. In recent years, in order to improve the detection efficiency and enhance the robustness of the algorithm against geometric transformations, the image inpainting detection algorithm based on deep learning has also been continuously developed. For example, the encoder-decoder network based on CNN automatically extracts tampering traces through neural networks, realizes image pixel-level prediction and locates the restoration area; the image restoration tampering forensics method based on the Faster R-CNN network realizes the bounding box positioning of the restoration area. However, this method can only obtain the bounding box marked with the confidence score, and cannot obtain the real area of ​​the restoration tampering area, and the positioning accuracy needs to be further improved. In order to improve the detection performance of existing algorithms for invisible image restoration methods, relevant technicians have provided a new end-to-end image restoration detection network IID-Net, which uses a neural architecture search algorithm to search for an appropriate network architecture to further extract potential restoration features. Other technicians have introduced the attention mechanism into the FCN fully connected network to further focus on the parts of the features extracted by the neural network related to the restoration traces. Although traditional image restoration forensics methods have achieved many technical results, there are still technical problems such as poor versatility, limited application scenarios, and slow processing speed.

[0040] Capturing local tampering traces based on convolutional neural networks is a recognized method in the field of restoration forensics. Training on large-scale data sets has achieved significantly better results than traditional methods while maintaining the generalization of the network. However, convolutional neural networks pay more attention to local information, and their ability to extract global features in restoration forensics tasks is insufficient, which limits the performance of restoration forensics. In addition, there are few modules in the existing technology that extract features for the boundaries of the restoration area, resulting in blurred boundaries of the restoration area in the detection results. More importantly, malicious image tamperers will use post-processing operations to process the restoration image to cover up the restoration traces. The robust performance of the existing technology is poor, and it is often difficult to detect the restoration image after post-processing.

[0041] In view of the defects in the prior art, the purpose of the present invention is to propose an image restoration forensics method, design a dual-stream encoder to improve the network's ability to extract restoration features. In view of the problem of blurred detection boundaries in restoration forensics tasks, a module is designed to supplement the edges of the restoration area. And a loss function is designed to improve the robustness of the network in the face of post-processing operations.

[0042] Figure 1 is a flowchart of an image restoration forensics method based on a comprehensive similarity deformer and a center triplet loss function according to an embodiment of the present invention.

[0043] like Figure 1 As shown, the above-mentioned image restoration forensics method based on the comprehensive similarity deformer and the central triplet loss function includes operations S110 to S140.

[0044] In operation S110, an image restoration forensics model is built based on an encoder-decoder infrastructure, and a comprehensive similarity deformer is introduced into a decoder of the image restoration forensics model, wherein the encoder of the image restoration forensics model adopts a densely connected stream backbone network.

[0045] In operation S120, a public image dataset sample is obtained, and the public image dataset sample is processed using a variety of image restoration methods, and the image restoration forensic dataset sample obtained after the processing is used to train an image restoration forensic model.

[0046] Optionally, for public data sets, randomly remove areas of different shapes, sizes, and locations in the image to produce image data to be repaired. Select a variety of typical image restoration methods, repair the image data to be restored, and construct multiple image restoration forensic data sets. Typical image restoration methods include traditional image restoration algorithms (algorithms based on non-deep learning) and image restoration algorithms based on deep learning. This ensures the richness and diversity of the constructed image restoration forensic data set, which facilitates the reasonable evaluation of subsequent image deep restoration forensic methods.

[0047] In the preferred embodiment of the present invention, a public image dataset Place365 is selected, from which 20,250 images of size 256×256×3 are selected, and some areas are randomly removed. The shapes of the randomly removed areas include rectangles, circles, and irregular shapes, and the area sizes include 1%, 5%, and 10%. Then five image restoration methods are used to process these 20,250 images. These five image restoration methods include a diffusion-based image restoration method, a sample-based image restoration method, and three deep learning-based image restoration methods. A total of 20,250×5 images are generated. After division, there are 18,000×5 images as training sets, 1,350×5 images as verification sets, and 900×5 images as test sets.

[0048] In operation S130, the pre-designed triple loss function is combined with the cross entropy loss function as the total loss function, and the image restoration forensics model is trained using the total loss function and the image restoration forensics dataset samples to obtain a trained image restoration forensics model.

[0049] In operation S140, the trained image restoration forensics model is used to detect the target restoration image, identify the restored area in the target restoration image, and obtain a restoration forensics result.

[0050] Compared with the prior art, the image restoration forensics method based on comprehensive similarity deformer and center triplet loss provided by the present invention makes full use of the restoration features extracted from each stream in a densely connected two-stream network, providing richer restoration features for the decoder part; the decoder with the addition of a comprehensive similarity deformer can alleviate the problem of blurred forensic boundaries and enhance the boundary detection effect; the center triplet loss can allow the network to still have good forensic performance when facing post-processed images, thereby improving the robustness of the network; end-to-end image restoration forensics can be achieved, and good results can be achieved on different data sets.

[0051] According to an embodiment of the present invention, the above-mentioned image restoration forensics model is built based on the encoder-decoder infrastructure, and the comprehensive similarity deformer is introduced into the decoder of the image restoration forensics model, including: adding a spatial domain latent rich model in front of the Resnet-50 backbone network, and densely connecting the Resnet-50 with the filter and the Swin Transformer-Tiny to obtain a densely connected two-stream backbone network as an encoder; adding a comprehensive similarity deformer as a decoder in the original decoder composed of a convolutional layer and an upsampling layer; splicing the encoder and the decoder to obtain an image restoration forensics model, and using the encoder to downsample the target restoration image; using the decoder to upsample the output result of the encoder to obtain a restoration forensics result with the same resolution as the target restoration image.

[0052] According to an embodiment of the present invention, the above-mentioned dual-stream backbone network includes a CNN stream composed of Resnet-50 and a Transformer stream composed of SwinTransformer-Tiny.

[0053] According to an embodiment of the present invention, after adjusting the size and number of channels of the output feature map of each stage in the CNN stream, it is spliced ​​into the input of the same-level stage and all upper-level stages in the Transformer stream according to the channel dimension.

[0054] According to an embodiment of the present invention, in the above decoder, the convolutional layers and upsampling layers are repeatedly stacked multiple times, and then a comprehensive similarity transformer is added to enhance the boundary feature representation.

[0055] According to an embodiment of the present invention, the above-mentioned comprehensive similarity deformer includes multiple network flows, each network flow is constructed by inserting a similarity perception layer between multiple convolutional layers, and the multiple network flows can process feature maps of different resolutions; wherein the similarity perception layer is used to detect pixel-level inconsistencies for fine-grained feature modeling, and the feature maps output by the multiple network flows are spliced ​​with the output of the first convolution block in the last network flow according to the channel dimension after adjusting the size and the number of channels, and the image restoration forensics result is obtained after passing through another convolutional layer.

[0056] The following is a specific embodiment and combined with the attached Figure 2 and 3 The construction process and specific architecture of the above-mentioned image restoration forensics model provided by the present invention are further described in detail.

[0057] Figure 2 Schematic diagram of the architecture of an image restoration forensics model according to an embodiment of the present invention.

[0058] Figure 3 is a schematic diagram of the structure of a comprehensive similarity deformer according to an embodiment of the present invention.

[0059] First, a spatial domain hidden rich model (Steganalysis Rich Model, SRM) is added before the Resnet-50 backbone network, and the Resnet-50 with added filters and the Swin Transformer-Tiny are densely connected to form a densely connected two-stream backbone network as the encoder part of the forensic network.

[0060] Secondly, in the traditional decoder composed of convolution layer and upsampling layer, a comprehensive similarity transformer is added as the decoder part of the forensic network; optionally, in the embodiment of the present invention, the convolution layer is composed of 3×3 convolution, batch normalization BN and activation function Relu. The upsampling layer is composed of a single upsampling function.

[0061] Finally, the encoder and decoder are spliced ​​together to build an image restoration forensics network (i.e., image restoration forensics model). The encoder downsamples the restored image, and the decoder upsamples the encoder output to obtain a forensics result with the same resolution as the original.

[0062] Alternatively, if Figure 2As shown in the figure, the encoder consists of a densely connected dual-stream backbone network. The dual streams refer to the CNN stream composed of Resnet-50 and the Transformer stream composed of Swin Transformer-Tiny. The dense connection means that the output feature map of each stage in the Transformer stream is resized and the number of channels is adjusted, and then spliced ​​to the input of the same stage and all the upper stages in the CNN stream according to the channel dimension, and then processed by the convolution layer to restore the original number of channels. The output feature map of each stage of the Transformer stream is recorded as The output feature map of each stage of the CNN stream is For the first stage, input The output feature map of the Transformer flow First, the output feature map of the Transforme stream is resized to the same size as Same. represents the composite operation in the convolution layer, represents the upsampling layer, and the adjusted feature map is , as shown in formula (1):

[0063] (1);

[0064] Furthermore, the densely connected form of this stage can be expressed by formula (2):

[0065] (2),

[0066] in, Represents the first Composite operations in a stage. This dense connection can supplement the global features extracted from each stage of the Transformer stream to the CNN stream, thus reducing feature waste.

[0067] Optionally, the feature map sizes output by the four stages of the CNN stream and the Transformer stream are 64×64×96, 32×32×192, 16×16×384, and 8×8×768, respectively.

[0068] Optionally, the convolutional layers and upsampling layers in the decoder are stacked three times repeatedly, and then a comprehensive similarity transformer is added to enhance the boundary feature representation. The comprehensive similarity transformer consists of three network flows, such as Figure 3As shown in the figure, each network stream is constructed by inserting a similarity-aware layer between two convolutional layers. The difference between the three network streams is that they accept and output input feature maps of different resolutions, which are 1 / 4, 1 / 2, and 1 times the input image resolution respectively. The similarity-aware layer plays a key role in the comprehensive similarity deformer, which is used to detect pixel-level inconsistencies for fine-grained feature modeling. After the output feature maps of the three network streams are adjusted in size and number of channels, they are concatenated with the output of the first convolutional block in the last network stream according to the channel dimension, and then pass through a convolutional layer to obtain the final forensic result. The similarity-aware layer converts the repair-related semantic features into the similarity space, extracts the boundary shape-related information, and then uses the improved channel attention to learn the importance of different channels of the feature map. The similarity-aware layer first calculates the feature similarity between each pixel and the surrounding pixels, and records them. Input feature map for similarity perception layer The middle position is The feature of a pixel point is recorded In a B×B local window, the position is The features of the adjacent pixels are calculated as shown in formula (3):

[0069] (3),

[0070] Among them, Represents pixel With adjacent pixels The similarity of features, Represents the cosine similarity calculation function. After calculating the feature similarity between each pixel and its adjacent pixels, a vector of size M×N×B×B is obtained. After changing the shape of the vector and removing the similarity calculation between each pixel and itself, a similarity feature map is obtained. In order to distinguish The importance of each channel is further improved by using the improved channel attention mechanism. Specifically, the channel importance weight is introduced , the weight can be updated iteratively during the training process, and the improved channel attention mechanism can be expressed as formula (4):

[0071] (4),

[0072] in, Represents element-wise multiplication. represents the output of the improved channel attention. Furthermore, the calculation process of the comprehensive similarity transformer module can be expressed by formula (5):

[0073] (5),

[0074] in, represents the output of the comprehensive similarity deformer, represents the input of the similarity perception layer in the top stream of the comprehensive similarity transformer, , , They represent the outputs of the three similarity-aware layers from top to bottom in the comprehensive similarity deformer.

[0075] The similarity perception layers of different network flows can extract boundary features of repaired areas of different sizes, which can effectively improve the problem of blurred detection boundaries encountered in image repair forensics.

[0076] Optionally, the input feature map sizes of the three similarity perception layers from bottom to top in the comprehensive similarity deformer are 64×64×96, 128×128×48, and 256×256×24, respectively, and the selected local window size is 3×3, so the corresponding output feature map sizes are 64×64×8, 128×128×8, and 256×256×8, respectively.

[0077] According to an embodiment of the present invention, a pre-designed triplet loss function is combined with a cross entropy loss function as a total loss function, and the image restoration forensic model is trained using the total loss function and a sample of an image restoration forensic dataset to obtain a trained image restoration forensic model, including: designing a central triplet loss function, and combining the cross entropy loss function as a total loss function, performing data enhancement on the sample of an image restoration forensic dataset to obtain enhanced image restoration forensic samples, and pre-training Resnet-50 and Swin Transformer-Tiny in the image restoration forensic model; based on periodic cosine annealing, the image restoration forensic model is trained using an SGD optimizer, enhanced image restoration forensic samples, pre-trained Resnet-50, pre-trained Swin Transformer-Tiny, and a total loss function to obtain a trained image restoration forensic model.

[0078] According to an embodiment of the present invention, the above-mentioned data enhancement of the image restoration forensic data set samples to obtain the enhanced image restoration forensic samples includes: randomly flipping, randomly cropping, randomly adding noise, and randomly transforming the color space of the image restoration forensic data set samples to obtain the enhanced image restoration forensic samples.

[0079] The process of training the image restoration forensics model based on the total loss function of the central triplet and the cross entropy provided by the present invention is further described in detail below through a specific embodiment.

[0080] Firstly, the center triplet loss is designed and combined with the cross entropy loss as the total loss of the network for network training. Secondly, the image restoration forensics dataset is augmented and expanded using random flipping and random JPEG compression. Optionally, the compression factor of random JPEG compression is randomly selected from 70 to 100. Finally, the pre-trained weights of Resnet-50 and SwinTransformer-Tiny are loaded to train the restoration forensics network.

[0081] Optionally, the center triplet loss replaces the anchor sample in the triplet with a learnable vector based on the triplet loss. The learnable vector is randomly initialized and can be iteratively updated during the training process to represent the feature center of the repaired area. The center triplet loss can be expressed by formula (6):

[0082] (6),

[0083] in Represents the average distance from the pixel features of the repaired area to the learnable vector

[0084] Represents the average distance from the pixel features of the non-repaired area to the learnable vector, is the margin in the triplet loss. The feature vector representing the pixel point, represents a learnable vector, then and The calculation formulas are shown in (7) and (8):

[0085] (7),

[0086] (8),

[0087] in, Indicates the number of elements in a collection. represents the L2 norm, Represents the set of pixels in the repaired area, Represents the set of pixels in the non-painted area.

[0088] Under the constraint of the central triplet loss, the features of the repaired area can be made more concentrated, which improves the intra-class compactness and inter-class separability of the network, ensures that the network still has good forensic effects when facing difficult samples that have undergone post-processing operations such as JPEG compression, and improves the robustness of the network.

[0089] Preferably, the total number of training cycles is 100, the repair forensics network sums the central triplet loss and the cross entropy loss, calculates the total loss of the predicted value and the label, and then updates the network parameters and the learnable vector in the central triplet loss by back propagation until the training loss reaches the minimum and the IOU indicators of the training set and the validation set are close.

[0090] In an optional embodiment of the present invention, an SGD optimizer is used as a training optimizer, the initial value of the learning rate is set to 1e-3, the learning rate is decayed by periodic cosine annealing, and the number of batch samples is set to 24.

[0091] Optionally, the trained restoration forensics network is deployed on a test system, and images processed by different restoration methods are used as input to obtain forensic results.

[0092] In order to better illustrate the advantages and effectiveness of the above method provided by the present invention, the present invention conducts specific experiments and combines the attached Figure 4 The above method provided by the present invention is tested and verified.

[0093] Figure 4 1 is a schematic diagram of test results and scheme comparison according to an embodiment of the present invention.

[0094] In an optional embodiment of the present invention, the trained model is deployed on the test system, and the generalization result of the model on the substation equipment target detection is obtained using the test set image data. The software configuration of the test system is python3.8, pytorch1.9.0, cuda11.2, pycharm2023.3; the hardware configuration is intel core i7-11700K 8 core 16 thread cpu, RTX-3090 graphics card (24G video memory) × 2, 16G memory.

[0095] For the image restoration forensics model obtained in the preferred embodiment of the present invention, 3×900 test set image data (corresponding to test sets 1-3) in the image restoration forensics dataset are used for testing, and the IoU and F1 indicators of the detection results are recorded as shown in Table 1. The IoU and F1 indicators of the detection results are recorded as shown in Table 2. In Tables 1 and 2, Comparison Scheme 1 is a “deep learning method for patch-based image restoration forensics”, and Comparison Scheme 2 is an “image restoration detection network based on neural structure search and attention”.

[0096] Some test effect pictures are as follows Figure 4 As shown in the test result diagram, it can be seen that the image restoration forensics method of this embodiment is tested on the image restoration dataset and can accurately identify the restoration area of ​​the image in the test set.

[0097] In Table 1 and Table 2, it can be seen from Table 1 that the repair forensic network of this embodiment is tested on the test sets generated by five repair methods, and the IoU is 93.41%, 94.71%, and 92.23%, respectively, and the F1 scores are 96.45%, 97.21%, and 95.84%, respectively. Compared with the comparative schemes 1 and 2, they all have obvious advantages. Especially on the test set 3, the IoU of the detection results of this embodiment is 42.54% and 37.81% higher than that of the comparative schemes 1 and 2, respectively. It can be seen from Table 2 that when the pictures of the test set 1 are compressed by JPEG (compression factor is 90) or Gaussian white noise is added (signal-to-noise ratio is 50), this embodiment can still obtain good forensic results, with IoU of 91.13% and 93.54%, and F1 scores of 94.96% and 96.51%. It is far higher than the forensic performance of comparative schemes 1 and 2. Combining Table 1 and Table 2, it can be shown that the present invention has excellent forensic performance and good robustness.

[0098] Depend on Figure 4 It can be seen that when the repair area is a regular shape (circle, rectangle), the present invention can basically detect the shape of the repair area completely. When the repair area is an irregular shape, the present invention can still roughly detect the repair area, especially in the boundary part, and the detection effect is far better than the comparison scheme. And for the repair area with a very small area, the present invention still has good detection results.

[0099] Table 1. Test results of the model on the test set generated by the three repair methods and comparison of the schemes

[0100]

[0101] Table 2. Test results of the model on the test set generated after compression and noise addition and comparison of the schemes

[0102]

[0103] The beneficial effects of the present invention are that, compared with the prior art, the densely connected two-stream network makes full use of the restoration features extracted from each stream, providing the decoder part with richer restoration features; the decoder with the addition of a comprehensive similarity deformer can alleviate the problem of blurred forensic boundaries and enhance the boundary detection effect; the center triplet loss can allow the network to still have good forensic performance when facing post-processed images, thereby improving the network robustness; end-to-end image restoration forensics can be achieved, and good results can be achieved on different data sets.

[0104] The present invention may be implemented in other specific forms without departing from its spirit or essential characteristics. The described embodiments are considered to be illustrative in all respects and not restrictive. For example, the network structure proposed in the present invention can also be used for video stream restoration and evidence collection, not limited to image restoration and evidence collection; the types of restoration operations are not limited to the four types mentioned in the embodiments; the network training method and the parameter configuration of the learning rate optimization strategy are non-restrictive and preferred choices, and those skilled in the art can select the network training method and parameter configuration according to various indicators such as detection accuracy and efficiency.

[0105] Figure 5 A block diagram of an electronic device suitable for implementing an image restoration forensics method based on a comprehensive similarity deformer and a center triplet loss function according to an embodiment of the present invention is schematically shown.

[0106] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0107] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0108] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0109] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0110] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image restoration forensics method based on comprehensive similarity deformer and central triplet loss function, It is characterized in that include: An image restoration forensics model is built based on an encoder-decoder infrastructure, and a comprehensive similarity deformer is introduced into the decoder of the image restoration forensics model, wherein the encoder of the image restoration forensics model adopts a densely connected two-stream backbone network; Obtaining a public image dataset sample, and processing the public image dataset sample using a variety of image restoration methods, and using the processed image restoration forensic dataset sample to train the image restoration forensic model; The pre-designed triple loss function is combined with the cross entropy loss function as a total loss function, and the image restoration forensics model is trained using the total loss function and the image restoration forensics dataset sample to obtain a trained image restoration forensics model; Using the trained image restoration forensics model to detect the target restoration image, identify the restored area in the target restoration image, and obtain the restoration forensics result; Among them, building an image restoration forensics model based on the basic architecture of encoder-decoder, and introducing a comprehensive similarity deformer into the decoder of the image restoration forensics model includes: A spatial domain latent enrichment model is added before the Resnet-50 backbone network, and the Resnet-50 with the filter added and the SwinTransformer-Tiny are densely connected to obtain the densely connected two-stream backbone network as an encoder; Adding the comprehensive similarity transformer as a decoder in an original decoder consisting of a convolutional layer and an upsampling layer; The encoder and the decoder are spliced ​​to obtain the image restoration forensics model, and the encoder is used to downsample the target restoration image; Upsampling the output result of the encoder by using the decoder to obtain a restoration forensics result having the same resolution as the target restoration image; The convolutional layers and upsampling layers in the decoder are repeatedly stacked multiple times, and then the comprehensive similarity deformer is added to enhance the boundary feature representation; The comprehensive similarity deformer includes multiple network flows, each of which is constructed by inserting a similarity perception layer between multiple convolutional layers, and the multiple network flows can process feature maps of different resolutions; The similarity perception layer is used to detect pixel-level inconsistencies for fine-grained feature modeling. After adjusting the size and number of channels, the feature maps output by the multiple network flows are spliced ​​with the output of the first convolution block in the last network flow according to the channel dimension, and the image restoration forensics result is obtained after passing through another convolution layer. The similarity perception layer calculates the feature similarity between each pixel and its adjacent pixels using the following formula: , in Represents pixel With adjacent pixels The similarity of features, Represents the cosine similarity calculation function; The center triplet loss can be expressed by the following formula: , in, Represents the average distance from the pixel features of the repaired area to the learnable vector Represents the average distance from the pixel features of the non-repaired area to the learnable vector, is the margin in the triplet loss.

2. The method according to claim 1, It is characterized in that The dual-stream backbone network includes a CNN stream consisting of Resnet-50 and a Transformer stream consisting of Swin Transformer-Tiny.

3. The method according to claim 2, It is characterized in that After adjusting the size and number of channels of the output feature map of each stage in the CNN stream, it is spliced ​​into the input of the same stage and all upper stages in the Transformer stream according to the channel dimension.

4. The method according to claim 1, It is characterized in that The pre-designed triple loss function is combined with the cross entropy loss function as the total loss function, and the image restoration forensics model is trained using the total loss function and the image restoration forensics dataset sample, and the trained image restoration forensics model includes: Design a central triplet loss function, and combine the cross entropy loss function as the total loss function, perform data enhancement on the image restoration forensics dataset samples to obtain enhanced image restoration forensics samples, and pre-train Resnet-50 and Swin Transformer-Tiny in the image restoration forensics model; Based on the periodic cosine annealing method, the image restoration forensics model is trained using the SGD optimizer, the enhanced image restoration forensics sample, the pre-trained Resnet-50, the pre-trained Swin Transformer-Tiny and the total loss function to obtain a trained image restoration forensics model.

5. The method according to claim 4, It is characterized in that Data enhancement is performed on the image restoration forensics dataset samples, and the enhanced image restoration forensics samples include: The enhanced image restoration forensic samples are obtained by randomly flipping, randomly cropping, randomly adding noise, and randomly transforming the color space of the image restoration forensic data set samples.

6. An electronic device, include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 5.