Image traceability evidence obtaining method based on residual fusion mechanism

Abstract

The invention relates to an image traceability evidence obtaining method based on a residual fusion mechanism, and belongs to the field of information security. The method comprises the following steps: firstly, preprocessing and selecting an input image; then the input image enters an image traceability model architecture of a residual fusion mechanism for detection, the architecture is divided into an upper branch and a lower branch, the upper branch uses a specific method to extract noise residual information, and the lower branch uses a deep learning network to automatically extract noise residual information; fusing and refining the feature information extracted from the upper branch and the lower branch in a residual connection mode; the refined residual features are sent to a classification module for mapping traceability of shooting equipment categories; finally, the detection result is visually displayed through a visual page. According to the invention, the residual information extracted by a specific method and a deep learning method is fused, and the advantages of the two methods are combined, so that the extracted residual fingerprint features are deeper and finer, and the traceability process is rapid and good in robustness.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and in particular relates to an image source tracing and evidence collection method based on a residual fusion mechanism. Background Art

[0002] With the development of science and technology, digital cameras and mobile phones have become accessible to everyone due to their intelligent design and low prices. Image acquisition, transmission and storage have also become convenient, and images have become an important medium for information transmission and storage. However, various image editing tools make it easier to edit and tamper with images, and image security issues have become increasingly prominent. Image forensics methods can make effective judgments on image copyright and authenticity issues at critical moments, and provide strong support for combating crimes and maintaining social harmony and stability. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] Existing image tracing forensics technologies either rely on traditional methods to manually extract residual features, or rely on deep learning methods to automatically learn camera fingerprint features. The feature information obtained is relatively simple and cannot extract potential information that is helpful for tracing the image device. When tracing the image to the camera and mobile phone device, the effectiveness is low.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention provides an image source tracing and forensics method based on residual fusion mechanism, comprising the following steps:

[0007] Step 1: Classify and preprocess the input image

[0008] If the input image is a full-size image, no processing is performed; if the input image is a user-defined size image, the image is cropped according to the user-defined size provided, and the edge parts that do not meet the defined size are discarded during cropping, and multiple image blocks that meet the defined size are retained;

[0009] In step 1, the cropping rule used when cropping the image is as follows: if the user inputs a custom size of M×M, the image is divided into multiple M×M sizes from left to right and from top to bottom, and the redundant sizes on the right and left sides of the image are deleted;

[0010] Step 2: Residual feature extraction, fusion and classification

[0011] This step uses a residual information fusion method of a specific method and a deep learning method to extract the fingerprint information contained in the image, wherein the upper and lower branches are used to extract the fingerprint residual information of the preprocessed image, the upper branch uses a specific method to extract residual features, and the lower branch uses a deep learning network to extract residual features. The residual features extracted by the upper branch are downsampled at different levels and then fused and refined with the residual features extracted by the lower branch in a residual connection manner to obtain a feature map. Finally, according to the information of the feature map, a classification module is used to map the features to the recognition target to match the target device model as the classification result. The features are fingerprint features.

[0012] Step 3: Visualization of test results

[0013] For full-size images, the detection result is the full-size image and its corresponding shooting device category;

[0014] For an image block of size M×M, the detection result is the shooting device category corresponding to each image block and a positioning map of the tampered image blocks after each image block is spliced ​​into a complete image.

[0015] The present invention also provides a system for implementing the method, characterized in that it includes a specific residual information extractor, a DPCE ConvNeXt network, a residual information fusion module and a classification module.

[0016] (III) Beneficial effects

[0017] The present invention discloses an image tracing and forensics method based on residual fusion mechanism, which has the following three main advantages:

[0018] (1) The present invention supports dual feature fusion. Existing image tracing forensics technologies either rely on traditional methods to manually extract residual features, or rely on deep learning methods to automatically learn camera fingerprint features. The feature information obtained is relatively simple and cannot extract potential fingerprint information that is beneficial to image tracing. When tracing the image to the camera and mobile phone device, the effectiveness is low. The present invention fuses the features extracted by the specific method and the deep learning method to explore deeper camera fingerprints, thereby effectively realizing image tracing forensics.

[0019] (2) The present invention supports the tracing of images taken by digital cameras and mobile phones. Most of the current image tracing and forensic technologies only support the detection of images taken by cameras or mobile phones, and their application scope is relatively small. The present invention uses images taken by digital cameras and mobile smart phones as training data. The trained model can be used to trace the origin of digital images to mobile phones and cameras, which has a wider application and better meets the needs of real society.

[0020] (3) The present invention is applicable to images to be inspected of any size. Since the detection network is built using a specific method and a full convolutional layer, the detection network can perform feature extraction and calculation for images to be inspected of any size (including images with different resolutions taken by different devices, or images that have been cropped). In particular, the detection of small-sized images can also locate the tampered area. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a digital image detection flow chart of the present invention;

[0022] Figure 2 This is a model architecture diagram of the present invention;

[0023] Figure 3 Detection examples and tampered area positioning map, where (a) is the original image, (b) is the tampered image, and (c) is the detection result. DETAILED DESCRIPTION

[0024] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0025] In order to solve the problem of image tracing forensics at the device level, the present invention proposes an image tracing forensics method based on a residual fusion mechanism, which combines Lukas's light response non-uniformity (PRNU) residual information extraction method with a deep learning residual information extraction method to obtain more comprehensive fingerprint information. The method is implemented based on a complete digital image feature extraction model architecture. From the input image to the output result, it has simple operation and intuitive result display. The architecture includes: image preprocessing, image residual feature extraction fusion and classification, and detection result display module. The image preprocessing module can choose whether to perform cropping detection on the image; the image residual feature extraction module extracts fingerprint features from the image to be detected, including a residual information extraction branch based on a specific method and a camera fingerprint feature extraction branch based on deep learning. After the residual information extraction branch is processed by a specific method, it is down-sampled at different levels and then fused with the deep learning branch for features. The image feature classification module is responsible for mapping the classification results of the features after fusion learning. The result display area will display the detection image and results. For the cropped detection image, the classification of each cropped area can also be visualized. If any area is tampered with, the tampered area can be located intuitively.

[0026] In the method of the present invention, the input image is first preprocessed and selected, and the full-size image can be used for detection, or the full-size image can be cropped into an image block of a custom size for detection; then the input image enters the image tracing model architecture of the residual fusion mechanism for detection, and the architecture is divided into two branches, the upper branch uses a specific method to extract noise residual information, and the lower branch uses a deep learning network to automatically extract noise residual information, and then the feature information extracted by the upper and lower branches is fused and refined by residual connection; then the refined residual features are sent to the classification module for mapping and tracing of the shooting equipment category; finally, the detection results are intuitively displayed through a visualization page. The present invention fuses the residual information extracted by the specific method and the deep learning method, combines the advantages of the two, makes the extracted residual fingerprint features deeper and more delicate, and the tracing process is fast and robust. It plays an important role in public security organs, judicial organs and other related fields, and can be used to verify the authenticity of images, determine copyright ownership, narrow the scope of suspicious persons, and even track and determine criminal suspects.

[0027] In order to make the purpose, content and advantages of the present invention more clearly explained, the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings and examples.

[0028] Reference Figure 1 As shown, the present invention provides an image tracing and forensics method based on residual fusion mechanism, which mainly includes three core steps: image preprocessing, residual feature extraction, fusion and classification, and detection result visualization.

[0029] Step 1: Preprocess the input image

[0030] The method proposed in the present invention can detect complete images taken by different cameras and mobile smart phones or images of any size after cropping. In the image preprocessing selection module, users can select the preprocessing method, including full size and custom size, that is, the input image is full size or user-defined size; the specific preprocessing process is as follows:

[0031] (1) Data Receiving Service

[0032] The numbers to be tested are uploaded to the testing system and await preprocessing selection.

[0033] (2) Image preprocessing selection settings

[0034] Provides full-size detection and custom-size detection. Full-size images are not processed in any way; for custom-size images, they are cropped according to the custom size provided by the user. When cropping, the edge parts that do not meet the defined size are discarded, and the image blocks that meet the defined size are retained.

[0035] The specific cropping rules used when cropping an image are as follows: if the user inputs a custom size of M×M, the image is divided into multiple M×M sizes from left to right and from top to bottom, and the redundant sizes on the right and left sides of the image are deleted.

[0036] Step 2: Residual feature extraction, fusion and classification

[0037] This step uses a residual information fusion method of a specific method and a deep learning method to extract the fingerprint information contained in the image. Specifically, it is divided into two branches, upper and lower, to extract fingerprint residual information from the preprocessed image. The upper branch uses a specific method to extract residual features, and the lower branch uses a deep learning network to extract residual features. The residual features extracted by the upper branch are fused and refined with the residual features extracted by the lower branch in a residual connection manner after downsampling at different levels to obtain a feature map, and finally mapped to the recognition target through a classification module to match the target device model (classification result). In the classification module, after the global average pooling layer integrates the global spatial information of the feature map, the linear layer maps the extracted features to the classification result, and the features are fingerprint features;

[0038] (2) Classification category mapping

[0039] The classification module is mapped to the target device model through a linear layer.

[0040] The specific method for extracting residual features adopts Lukas' light response non-uniform fingerprint noise extraction method, and Lukas uses wavelet filters to extract noise residuals. The wavelet denoising filter is mainly divided into two stages and a total of four steps. In the first stage, the local variance estimation of the image is completed. In the second stage, the Wiener filter is used in the wavelet domain to obtain the final denoised image estimation. The specific steps are as follows:

[0041] S11, performing a fourth-order wavelet decomposition on each color channel of the color image, and representing the vertical subband, horizontal subband, and diagonal subband as: h(i, j), v(i, j), and d(i, j), respectively, where (i, j) traverses the index set Y of the decomposition level;

[0042] S12. In each subband, the local variance of the original noise-free image corresponding to each wavelet coefficient is estimated using the maximum a posteriori probability estimation method on four types of square neighborhoods N with sizes of W×W (3×3, 5×5, 7×7, 9×9), as shown in formula (1):

[0043]

[0044] Among them, h(x,y) is the horizontal subband value, W is the sliding window size, N is the number of windows, and Y is the wavelet subband value. is the variance of the N center pixels in each square neighborhood;

[0045] S13. From the sliding windows of different sizes, the minimum value is selected as the final local variance estimation result. The specific process is shown in formula (2):

[0046]

[0047] S14. The wavelet coefficients are obtained by the Wiener filter, as shown in formula (3):

[0048]

[0049] S15. It should be noted that the above steps (1) to (4) need to be repeated for each color channel. Finally, the clean image F′(i, j) is obtained by inverse wavelet transform, and the residual estimate is obtained by subtracting the original image F(i, j) from the clean image F′(i, j).

[0050] R(i,j)=F(i,j)-F′(i,j),(i,j)∈Y(4)

[0051] In step 2, the deep learning network uses the DPCE ConvNeXt network. The steps of extracting residual features of the DPCE ConvNeXt network are as follows: feature extraction is implemented through downsampling, layer normalization and four groups of DPCE ConvNeXt modules; the DPCE ConvNeXt module has two branches, each branch passes through a 1×1 convolution layer, a 7×7 large window convolution layer and a 1×1 convolution layer in sequence, and then the results of the two branches are added together and aggregated and strengthened through channel attention.

[0052] The specific steps of the residual feature fusion module are as follows: The residual feature fusion module is composed of a specific method branch and a deep learning (DPCE ConvNeXt) branch in parallel. The residual features extracted by the specific method branch are downsampled by four convolutions and then converged to the deep learning network branch in the form of residual connections, and the four DPCE ConvNeXt modules perform feature fusion and refinement.

[0053] Step 3: Visualization of test results

[0054] This step can directly output the detected full-size image and its classification results, or it can stitch the cropped image blocks to display the areas that may be tampered with.

[0055] (1) Visualization of full-size image recognition results

[0056] The recognition results of the full-size image are displayed as the input full-size image and the matched target device model.

[0057] (2) Visualization of cropped image recognition results

[0058] The recognition results of the cropped small image blocks are positioned and matched to the original image. The matched target device model is displayed in the image frame, and the positioned and matched original image is displayed with a mask, so that the tampered image area can be easily located.

[0059] In step 3, the visualization of the detection results is as follows:

[0060] S31, input a full-size image, and output the full-size image and its corresponding shooting device category;

[0061] S32, inputting image blocks of size M×M, and outputting the categories of photographing devices corresponding to the image blocks and a positioning map of image blocks that may be tampered with after the image blocks are aggregated into a complete image.

[0062] Reference Figure 2 As shown, the residual fusion image tracing forensics model architecture provided by the present invention consists of a specific residual information extractor, a DPCE ConvNeXt network architecture, a residual information fusion module and a classification module.

[0063] (1) Residual Information Extractor

[0064] The residual information extractor of the specific method branch adopts Lukas' residual information extraction scheme. Low-frequency interference is mainly removed by wavelet filter. The wavelet denoising filter is mainly divided into two stages. In the first stage, the local variance estimation of the image is completed. In the second stage, the Wiener filter is used in the wavelet domain to obtain the final denoised image estimation.

[0065] (2) DPCE ConvNeXt network architecture

[0066] The deep learning method branch uses the DPCE ConvNeXt network architecture to extract learnable features. This part consists of downsampling, layer normalization (LN), and four groups of DPCE ConvNeXt modules. The feature extraction module starts with a 4×4 convolutional layer with a stride of 4. During this process, the size of the feature map will be reduced, and the network will discard some redundant information while retaining high-level abstract features. Then layer normalization LN is used for layer normalization. After that, four groups of DPCE modules with the same design are used to further extract features. Each DPCE module contains three DPCE ConvNeXt modules in series. After normalization, downsampling is performed using a 2×2 convolutional layer with a stride of 2.

[0067] (3) Residual information fusion module

[0068] In order to effectively perform information fusion, the parameters of the branch algorithm of a specific method are fixed and do not participate in the network update. Then, they are fused through a 4×4 convolution block with a step size of 4 and three feature refinement modules. In the feature refinement module, a 3×3 convolution layer, a normalization layer, and a 2×2 convolution block with a step size of 2 are used for downsampling and feature extraction. The dimensions of the feature maps obtained after each feature refinement module are consistent with the model extraction branch. In this way, feature fusion can be performed with the DPCE ConvNeXt module through residual connection after downsampling of each residual extraction branch.

[0069] (4) Classification module

[0070] In the classification module, after the global average pooling layer integrates the global spatial information of the feature map, the linear layer maps the extracted features to the classification results.

[0071] It can be seen that the image tracing forensics method based on residual fusion mechanism proposed in the present invention introduces a specific image residual information extraction technology, and after downsampling at different levels, residual connection is performed with the deep learning network branch. Through this dual information fusion method, "fingerprint information" that is conducive to image tracing can be extracted, thereby realizing the source tracking of digital images taken by digital cameras or mobile smartphones. Furthermore, the tracing of the source of small-size image blocks can also locate the tampered position of the image.

[0072] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image tracing and forensics method based on residual fusion mechanism, characterized in that: The following steps are involved: Step 1: Classify and preprocess the input image If the input image is a full-size image, no processing is performed; if the input image is a user-defined size image, the image is cropped according to the user-defined size provided, and the edge parts that do not meet the defined size are discarded during cropping, and multiple image blocks that meet the defined size are retained; In step 1, the cropping rule used when cropping the image is as follows: if the user inputs a custom size of M×M, the image is divided into multiple M×M sizes from left to right and from top to bottom, and the redundant sizes on the right and left sides of the image are deleted; Step 2: Residual feature extraction, fusion and classification This step uses a residual information fusion method of a specific method and a deep learning method to extract the fingerprint information contained in the image, wherein the upper and lower branches are used to extract the fingerprint residual information of the preprocessed image, the upper branch uses a specific method to extract residual features, and the lower branch uses a deep learning network to extract residual features. The residual features extracted by the upper branch are downsampled at different levels and then fused and refined with the residual features extracted by the lower branch in a residual connection manner to obtain a feature map. Finally, according to the information of the feature map, a classification module is used to map the features to the recognition target to match the target device model as the classification result. The features are fingerprint features. Step 3: Visualization of test results For full-size images, the detection result is the full-size image and its corresponding shooting device category; For an image block of size M×M, the detection result is the shooting device category corresponding to each image block and a positioning map of the tampered image blocks after each image block is spliced ​​into a complete image.

2. The method according to claim 1, characterized in that In step 2, the classification module uses a global average pooling layer to integrate the global spatial information of the feature map, and then uses a linear layer to map the extracted features to the classification results.

3. The method according to claim 1, characterized in that In step 2, the specific method for extracting residual features is to use Lukas' light response non-uniformity fingerprint noise extraction method, wherein Lukas uses a wavelet filter to extract noise residuals.

4. The method according to claim 1, characterized in that In step 2, a specific method is used to extract residual features, which is divided into two stages and four steps. In the first stage, the local variance estimation of the image is completed. In the second stage, the Wiener filter is used in the wavelet domain to obtain the final denoised image estimation. The specific steps are as follows: S11, performing a fourth-order wavelet decomposition on each color channel of the color image, and representing the vertical subband, horizontal subband, and diagonal subband as: h(i, j), v(i, j), and d(i, j), respectively, where (i, j) traverses the index set Y of the decomposition level; S12. In each subband, the local variance of the original noise-free image corresponding to each wavelet coefficient is estimated using the maximum a posteriori probability estimation method on four types of square neighborhoods N of size W×W, as shown in formula (1): Among them, h(x,y) is the horizontal subband value, W is the sliding window size, N is the number of windows, and Y is the wavelet subband value. is the variance of the N center pixels in each square neighborhood; S13. From the sliding windows of different sizes, the minimum value is selected as the final local variance estimation result. The specific process is as shown in formula (2): S14. Obtain wavelet coefficients by Wiener filter, as shown in formula (3): S15. Repeat steps S11 to S14 for each color channel. Finally, obtain the clean image F′(i, j) by inverse wavelet transform, and make a difference between the original image F(i, j) and the clean image F′(i, j) to obtain the residual estimate: R(i,j)=F(i,j)-F′(i,j),(i,j)∈Y (4).

5. The method according to claim 1, characterized in that In step 2, the deep learning network is a DPCE ConvNeXt network. The steps of extracting residual features of the DPCE ConvNeXt network are as follows: feature extraction is achieved through downsampling, layer normalization and four groups of DPCEConvNeXt modules; the DPCE ConvNeXt module has two branches, each branch passes through a 1×1 convolution layer, a 7×7 large window convolution layer and a 1×1 convolution layer in sequence, and then the results of the two branches are added together and aggregated and strengthened through channel attention.

6. The method according to claim 1, characterized in that In step 2, the residual features extracted from the upper branch are downsampled by four convolutions and then converged to the lower branch in the form of residual connections, and feature fusion and refinement are performed by four DPCE ConvNeXt modules.

7. The method according to claim 5, characterized in that Feature extraction starts with a 4×4 convolutional layer with a stride of 4. During the process, the size of the feature map is reduced, and redundant information is discarded while retaining high-level abstract features; Then layer normalization LN is used for layer normalization; after that, four groups of identical DPCE modules are used to further extract features. Each DPCE module contains three DPCE ConvNeXt modules connected in series. After normalization, a 2×2 convolutional layer with a stride of 2 is used for downsampling.

8. The method according to claim 1, characterized in that In step 2, the residual features extracted by the upper branch are downsampled at different levels and then fused and refined with the residual features extracted by the lower branch in the form of residual connection. They are fused through a 4×4 convolution block with a step size of 4 and 3 feature refinement modules. The feature refinement module consists of a 3×3 convolution layer, a normalization layer and a 2×2 convolution block with a step size of 2 for downsampling and feature extraction.

9. The method according to any one of claims 1 to 8, characterized in that The method is applied in the field of information security technology.

10. A system for implementing the method according to any one of claims 1 to 9, characterized in that: It includes a specific residual information extractor, a DPCE ConvNeXt network, a residual information fusion module and a classification module.

Images