An image provenance tracing method based on residual fusion mechanism

Abstract

The present invention relates to an image tracing and forensics method based on a residual fusion mechanism, and belongs to the field of information security. The present invention first pre-processes and selects the input image; then the input image enters the image tracing model architecture of the residual fusion mechanism for detection, and the architecture is divided into two branches, the upper branch uses a specific method to extract noise residual information, and the lower branch uses a deep learning network to automatically extract noise residual information, and then the feature information extracted by the upper and lower branches is fused and refined by residual connection; then the refined residual features are sent to the classification module for mapping and tracing the category of the shooting device; finally, the detection results are intuitively displayed through a visualization page. The present invention fuses the residual information extracted by the specific method and the deep learning method, combining the advantages of the two, so that the extracted residual fingerprint features are deeper and more delicate, and the tracing process is fast and robust.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and in particular relates to an image provenance tracing method based on a residual fusion mechanism. Background Art

[0002] With the advancement of technology, digital cameras and mobile phones have become accessible to everyone due to their intelligent designs and affordable prices. This has made image acquisition, transmission, and storage more convenient, making images a crucial medium for information transmission and storage. However, various image editing tools have made image editing and manipulation increasingly easy, making image security issues increasingly prominent. Image forensics methods can be used to effectively determine image copyright and authenticity issues at critical moments, providing strong support for combating crime and maintaining social harmony and stability. Summary of the Invention

[0003] (1) Technical issues to be solved

[0004] Existing image tracing and forensic technology either relies on traditional methods to manually extract residual features, or relies on deep learning methods to automatically learn camera fingerprint features. The feature information obtained is relatively simple and cannot extract potential information that is beneficial for tracing the origin of image devices. It is less effective when tracing the origin of images to cameras and mobile phone devices.

[0005] (2) Technical solution

[0006] In order to solve the above technical problems, the present invention provides an image provenance tracing method based on a residual fusion mechanism, comprising the following steps:

[0007] Step 1: Classify and preprocess the input image

[0008] If the input image is a full-size image, no processing is performed; if the input image is a user-defined size image, the image is cropped according to the user-provided custom size, discarding the edge parts that do not meet the defined size, and retaining multiple image blocks that meet the defined size;

[0009] In step 1, the cropping rule used when cropping the image is as follows: if the user enters a custom size of M×M, the image is divided into multiple M×M sizes from left to right and from top to bottom, and the excess size on the right and left sides of the image is deleted;

[0010] Step 2: Residual feature extraction, fusion and classification

[0011] This step uses a residual information fusion method of a specific method and a deep learning method to extract the fingerprint information contained in the image. Among them, the upper and lower branches are used to extract the fingerprint residual information of the preprocessed image. The upper branch uses a specific method to extract residual features, and the lower branch uses a deep learning network to extract residual features. The residual features extracted by the upper branch are down-sampled at different levels and then fused and refined with the residual features extracted by the lower branch in a residual connection manner to obtain a feature map. Finally, according to the information in the feature map, the classification module is used to map the features to the recognition target to match the target device model as the classification result. The features are fingerprint features.

[0012] Step 3: Visualization of test results

[0013] For full-size images, the detection result is the full-size image and its corresponding shooting device category;

[0014] For image blocks of size M×M, the detection result is the shooting device category corresponding to each image block and the positioning map of the tampered image blocks after the image blocks are spliced ​​into a complete image.

[0015] The present invention also provides a system for implementing the method, characterized in that it includes a specific residual information extractor, a DPCE ConvNeXt network, a residual information fusion module and a classification module.

[0016] (3) Beneficial effects

[0017] This paper discloses an image provenance tracing method based on residual fusion mechanism, which has the following advantages:

[0018] (1) The present invention supports dual feature fusion. Existing image tracing and forensics technologies either rely on traditional methods to manually extract residual features or rely on deep learning methods to automatically learn camera fingerprint features. The feature information obtained is relatively simple and cannot extract potential fingerprint information that is beneficial for image tracing. When tracing images to cameras and mobile devices, the effectiveness is low. The present invention fuses the features extracted by specific methods and deep learning methods to explore deeper camera fingerprints, thereby effectively realizing image tracing and forensics.

[0019] (2) The present invention supports the tracing of images taken by digital cameras and mobile phones. Most current image tracing and forensic technologies only support the detection of images taken by cameras or mobile phones, and their application scope is relatively small. The present invention uses images taken by digital cameras and mobile smartphones as training data. The trained model can be used to trace the origin of digital images to mobile phones and cameras, which has wider application and better meets the needs of real society.

[0020] (3) The present invention is applicable to images of any size. Since the detection network is built using a specific method and fully convolutional layers, it can extract and calculate features for images of any size (including images with different resolutions captured by different devices or images that have been cropped). In particular, the detection of small-sized images can also locate tampered areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of digital image detection of the present invention;

[0022] Figure 2 This is a model architecture diagram of the present invention;

[0023] Figure 3 Detection examples and tampered area positioning map, where (a) is the original image, (b) is the tampered image, and (c) is the detection result. DETAILED DESCRIPTION

[0024] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0025] In order to solve the problem of image traceability and forensics at the device level, the present invention proposes an image traceability and forensics method based on a residual fusion mechanism. This method combines Lukas's light response non-uniformity (PRNU) residual information extraction method with a deep learning residual information extraction method to obtain more comprehensive fingerprint information. This method is based on a complete set of digital image feature extraction model architectures. From input images to output results, it has simple operations and intuitive result presentations. The architecture includes: image preprocessing, image residual feature extraction fusion and classification, and detection result presentation modules. The image preprocessing module can choose whether to perform cropping detection on the image; the image residual feature extraction module extracts fingerprint features from the image to be detected, including a residual information extraction branch based on a specific method and a camera fingerprint feature extraction branch based on deep learning. After being processed by a specific method, the residual information extraction branch is sequentially down-sampled at different levels and then subjected to feature fusion with the deep learning branch. The image feature classification module is responsible for mapping the classification results of the fused learning features. The result display area displays the detected image and results. For cropped detected images, the classification of each cropped area can also be visualized. If any area has been tampered with, the tampered area can be located intuitively.

[0026] In the method of the present invention, the input image is first pre-processed and selected. The full-size image can be used for detection, or the full-size image can be cropped into an image block of a custom size for detection; then the input image enters the image tracing model architecture of the residual fusion mechanism for detection, which is divided into two branches, the upper branch uses a specific method to extract noise residual information, and the lower branch uses a deep learning network to automatically extract noise residual information. After that, the feature information extracted by the upper and lower branches is fused and refined by residual connection; the refined residual features are then sent to the classification module for mapping and tracing the category of the shooting equipment; finally, the detection results are intuitively displayed through a visualization page. The present invention fuses the residual information extracted by the specific method and the deep learning method, combining the advantages of the two, so that the extracted residual fingerprint features are deeper and more delicate, the tracing process is fast and robust, and plays an important role in public security organs, judicial organs and other related fields. It can be used to verify the authenticity of images, determine copyright ownership, narrow the scope of suspicious persons, and even track and identify criminal suspects.

[0027] In order to make the purpose, content and advantages of the present invention more clearly explained, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0028] Reference Figure 1 As shown, the present invention provides an image tracing and forensics method based on the residual fusion mechanism, which mainly includes three core steps: image preprocessing, residual feature extraction, fusion and classification, and detection result visualization.

[0029] Step 1: Preprocess the input image

[0030] The method proposed in this paper can detect full images or cropped images of any size captured by different cameras and mobile smartphones. In the image preprocessing selection module, users can select preprocessing methods, including full size and custom size. That is, the input image is full size or user-defined size. The specific preprocessing process is as follows:

[0031] (1) Data Receiving Service

[0032] The numbers to be tested are uploaded to the testing system and await pre-processing selection.

[0033] (2) Image preprocessing selection settings

[0034] Both full-size and custom-size detection are available. Full-size images are not processed; custom-size images are cropped based on the user-provided size. During cropping, edges that do not conform to the defined size are discarded, while image blocks that conform to the defined size are retained.

[0035] The specific cropping rules used when cropping an image are as follows: if the user enters a custom size of M×M, the image is divided into multiple M×M sizes from left to right and from top to bottom, and the redundant sizes on the right and left sides of the image are deleted.

[0036] Step 2: Residual feature extraction, fusion and classification

[0037] This step uses a residual information fusion method of a specific method and a deep learning method to extract the fingerprint information contained in the image. Specifically, it is divided into two branches, the upper and lower branches, to extract the fingerprint residual information of the preprocessed image. The upper branch uses a specific method to extract residual features, and the lower branch uses a deep learning network to extract residual features. The residual features extracted by the upper branch are down-sampled at different levels and then fused and refined with the residual features extracted by the lower branch in a residual connection manner to obtain a feature map. Finally, it is mapped to the recognition target through the classification module to match the target device model (classification result). In the classification module, the global average pooling layer integrates the global spatial information of the feature map, and the linear layer maps the extracted features to the classification result. The features are fingerprint features.

[0038] (2) Classification category mapping

[0039] The classification module maps to the target device model through a linear layer.

[0040] The specific method for extracting residual features uses Lukas' light-corresponding non-uniformity fingerprint noise extraction method. Lukas uses wavelet filters to extract noise residuals. The wavelet denoising filter is mainly completed in two stages and four steps. The first stage completes the local variance estimation of the image. The second stage uses the Wiener filter in the wavelet domain to obtain the final denoised image estimate. The specific steps are as follows:

[0041] S11. Perform a fourth-order wavelet decomposition on each color channel of the color image, and denote the vertical subband, horizontal subband, and diagonal subband as h(i, j), v(i, j), and d(i, j), respectively, where (i, j) traverses the index set Y of the decomposition level;

[0042] S12. In each subband, the local variance of the original noise-free image corresponding to each wavelet coefficient is estimated using the maximum a posteriori probability estimation method on four types of square neighborhoods N with sizes of W×W (3×3, 5×5, 7×7, 9×9), as shown in formula (1):

[0043]

[0044] Among them, h(x,y) is the horizontal subband value, W is the sliding window size, N is the number of windows, and Y is the wavelet subband value. is the variance of the N center pixels in each square neighborhood;

[0045] S13. From the sliding windows of different sizes, the minimum value is selected as the final local variance estimation result. The specific process is shown in formula (2):

[0046]

[0047] S14. Obtain the wavelet coefficients by the Wiener filter, as shown in formula (3):

[0048]

[0049] S15. It should be noted that the above steps (1) to (4) need to be repeated for each color channel. Finally, the clean image F′(i, j) is obtained by inverse wavelet transform. The residual estimate is obtained by subtracting the original image F(i, j) from the clean image F′(i, j).

[0050] R(i,j)=F(i,j)-F′(i,j),(i,j)∈Y(4)

[0051] In step 2, the deep learning network uses the DPCE ConvNeXt network. The steps of extracting residual features by the DPCE ConvNeXt network are as follows: feature extraction is implemented through downsampling, layer normalization and four groups of DPCE ConvNeXt modules; the DPCE ConvNeXt module has two branches, each branch passes through a 1×1 convolution layer, a 7×7 large window convolution layer and a 1×1 convolution layer in sequence. After that, the results of the two branches are added together and aggregated and strengthened through channel attention.

[0052] The specific steps of the residual feature fusion module are as follows: The residual feature fusion module consists of a parallel connection between a specific method branch and a deep learning (DPCE ConvNeXt) branch. The residual features extracted by the specific method branch are downsampled through four convolutions and then converged into the deep learning network branch using residual connections. The four DPCE ConvNeXt modules perform feature fusion and refinement.

[0053] Step 3: Visualization of test results

[0054] This step can directly output the detected full-size image and its classification results, or it can stitch the cropped image blocks together to display the areas that may have been tampered with.

[0055] (1) Visualization of full-size image recognition results

[0056] The recognition results of the full-size image are displayed as the input full-size image and the matched target device model.

[0057] (2) Visualization of cropped image recognition results

[0058] The recognition results of the cropped small image blocks are positioned and matched to the original image. The matched target device model is displayed in the image frame, and the masked original image is displayed to easily locate the tampered image area.

[0059] In step 3, the visualization of the test results is as follows:

[0060] S31, input a full-size image, and output the full-size image and its corresponding shooting device category;

[0061] S32 , inputting image blocks of size M×M, and outputting the categories of shooting devices corresponding to the image blocks and a positioning map of image blocks that may be tampered with after the image blocks are aggregated into a complete image.

[0062] Reference Figure 2 As shown in the figure, the residual fusion image tracing forensics model architecture provided by the present invention consists of a specific residual information extractor, a DPCE ConvNeXt network architecture, a residual information fusion module and a classification module.

[0063] (1) Residual Information Extractor

[0064] The residual information extractor of the specific method branch uses Lukas' residual information extraction scheme. It primarily removes low-frequency interference through wavelet filtering. The wavelet denoising filter is divided into two stages: the first stage estimates the local variance of the image, and the second stage uses the Wiener filter in the wavelet domain to obtain the final denoised image estimate.

[0065] (2) DPCE ConvNeXt network architecture

[0066] The deep learning method branch uses the DPCE ConvNeXt network architecture to extract learnable features. This part consists of downsampling, layer normalization (LN), and four groups of DPCE ConvNeXt modules. The feature extraction module starts with a 4×4 convolutional layer with a stride of 4. During this process, the size of the feature map will be reduced, and the network will discard some redundant information while retaining high-level abstract features. Layer normalization LN is then used for layer normalization. After that, four groups of DPCE modules with the same design are used to further extract features. Each DPCE module contains three DPCE ConvNeXt modules in series. After normalization, downsampling is performed using a 2×2 convolutional layer with a stride of 2.

[0067] (3) Residual information fusion module

[0068] To effectively perform information fusion, the parameters of the algorithm branches of specific methods are fixed and excluded from network updates. Fusion is then performed using a 4×4 convolutional block with a stride of 4 and three feature refinement modules. The feature refinement module consists of a 3×3 convolutional layer, a normalization layer, and a 2×2 convolutional block with a stride of 2 for downsampling and feature extraction. The dimensions of the feature maps obtained after each feature refinement module are consistent with those of the model extraction branch. This allows for feature fusion with the DPCE ConvNeXt module through a residual connection after downsampling in each residual extraction branch.

[0069] (4) Classification module

[0070] In the classification module, the global average pooling layer integrates the global spatial information of the feature map, and the linear layer maps the extracted features to the classification results.

[0071] As can be seen, the image provenance forensics method proposed in this paper, based on a residual fusion mechanism, introduces a specific image residual information extraction technique. After downsampling at different levels, it is residually connected with the deep learning network branch. Through this dual information fusion approach, "fingerprint information" that facilitates image provenance is extracted, thereby enabling the source tracking of digital images captured by digital cameras or mobile smartphones. Furthermore, the traceability of small-sized image blocks can also locate the location of image tampering.

[0072] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image provenance tracing method based on residual fusion mechanism, characterized by: The following steps are involved: Step 1: Classify and preprocess the input image If the input image is a full-size image, no processing is performed; if the input image is a user-defined size image, the image is cropped according to the user-provided custom size, discarding the edge parts that do not meet the defined size, and retaining multiple image blocks that meet the defined size; In step 1, the cropping rule used when cropping the image is as follows: if the user enters a custom size of M×M, the image is divided into multiple M×M sizes from left to right and from top to bottom, and the excess size on the right and left sides of the image is deleted; Step 2: Residual feature extraction, fusion and classification This step uses a residual information fusion method of a specific method and a deep learning method to extract the fingerprint information contained in the image. Among them, the upper and lower branches are used to extract the fingerprint residual information of the preprocessed image. The upper branch uses a specific method to extract residual features, and the lower branch uses a deep learning network to extract residual features. The residual features extracted by the upper branch are down-sampled at different levels and then fused and refined with the residual features extracted by the lower branch in a residual connection manner to obtain a feature map. Finally, according to the information in the feature map, the classification module is used to map the features to the recognition target to match the target device model as the classification result. The features are fingerprint features. Step 3: Visualization of test results For full-size images, the detection result is the full-size image and its corresponding shooting device category; For an M×M image block, the detection result is the camera category corresponding to each image block and the location map of the tampered image block after each image block is spliced ​​into a complete image; In step 2, the specific method for extracting residual features is to adopt Lukas' light response non-uniformity fingerprint noise extraction method, wherein Lukas uses a wavelet filter to extract noise residuals.

2. The method according to claim 1, wherein In step 2, the classification module uses a global average pooling layer to integrate the global spatial information of the feature map, and then uses a linear layer to map the extracted features to the classification results.

3. The method according to claim 1, wherein In step 2, a specific method is used to extract residual features, which is divided into two stages and four steps. In the first stage, the local variance of the image is estimated. In the second stage, the Wiener filter is used in the wavelet domain to obtain the final denoised image estimate. The specific steps are as follows: S11. Perform a fourth-order wavelet decomposition on each color channel of the color image, and denote the vertical subband, horizontal subband, and diagonal subband as h(i, j), v(i, j), and d(i, j), respectively, where (i, j) traverses the index set Y of the decomposition level; S12. In each subband, the local variance of the original noise-free image corresponding to each wavelet coefficient is estimated using the maximum a posteriori probability estimation method on four types of square neighborhoods N of size W×W, as shown in formula (1): Among them, h(x,y) is the horizontal subband value, W is the sliding window size, N is the number of windows, and Y is the wavelet subband value. is the variance of the N center pixels in each square neighborhood; S13. From the sliding windows of different sizes, the minimum value is selected as the final local variance estimation result. The specific process is as shown in formula (2): S14. Obtain the wavelet coefficients by the Wiener filter, as shown in formula (3): S15. Repeat steps S11 to S14 for each color channel. Finally, obtain the clean image F′(i, j) by inverse wavelet transform. Subtract the original image F(i, j) from the clean image F′(i, j) to obtain the residual estimate: R(i,j)=F(i,j)-F′(i,j),(i,j)∈Y (4).

4. The method according to claim 1, wherein In step 2, the deep learning network is a DPCE ConvNeXt network. The steps of extracting residual features by the DPCE ConvNeXt network are as follows: feature extraction is achieved through downsampling, layer normalization and four groups of DPCEConvNeXt modules; the DPCE ConvNeXt module has two branches, each branch passes through a 1×1 convolution layer, a 7×7 large window convolution layer and a 1×1 convolution layer in sequence, and then the results of the two branches are added together and aggregated and strengthened through channel attention.

5. The method according to claim 1, wherein In step 2, the residual features extracted by the upper branch are downsampled by four convolutions and then converged to the lower branch in the form of residual connections, and feature fusion and refinement are performed by four DPCE ConvNeXt modules.

6. The method according to claim 4, wherein Feature extraction starts with a 4×4 convolutional layer with a stride of 4. During this process, the size of the feature map is reduced, redundant information is discarded, and high-level abstract features are retained. Then layer normalization LN is used for layer normalization; after that, four groups of identical DPCE modules are used to further extract features. Each DPCE module contains three DPCE ConvNeXt modules in series. After normalization, a 2×2 convolutional layer with a stride of 2 is used for downsampling.

7. The method according to claim 1, wherein In step 2, the residual features extracted by the upper branch are downsampled at different levels and then fused and refined with the residual features extracted by the lower branch in the form of residual connection. The fusion is performed through a 4×4 convolution block with a step size of 4 and 3 feature refinement modules. The feature refinement module is composed of a 3×3 convolution layer, a normalization layer and a 2×2 convolution block with a step size of 2 for downsampling and feature extraction.

8. The method according to any one of claims 1 to 7, characterized in that The method is applied in the field of information security technology.

9. A system for implementing the method according to any one of claims 1 to 8, characterized in that: It includes a specific residual information extractor, a DPCE ConvNeXt network, a residual information fusion module and a classification module.

Images