Multi-feature fusion prnu extraction method and device based on double-layer mixed model

Abstract

The application belongs to the field of PRNU extraction, and provides a multi-feature fusion PRNU extraction method and device based on a double-layer hybrid model. The method comprises the following steps: based on a to-be-traced image, a double-layer hybrid model is used to obtain a noise feature map; the correlation between the PRNU of the noise feature map and the reference PRNU of a camera is calculated to determine whether the to-be-traced image is shot by the camera; wherein the process of using the double-layer hybrid model comprises the following steps: based on the to-be-traced image, a first feature extraction network and a second feature extraction network are used respectively to obtain a first feature map and a second feature map; the first feature map and the second feature map are fused to obtain a first fusion feature map; after the first fusion feature map is enhanced and compressed, the to-be-traced image is fused to obtain a second fusion feature map; after the second fusion feature map is enhanced and compressed, a noise feature map is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of PRNU extraction, and particularly relates to a multi-feature fusion PRNU extraction method and device based on a double-layer hybrid model. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] In recent years, source camera identification, as one of the most important tasks in digital image forensics, has received more and more attention. It requires to reliably link a digital image to the camera that took it. The method based on photo response non-uniformity (PRNU) noise can effectively solve this problem. PRNU is mainly caused by the defects of the imaging sensor in the manufacturing process of the silicon wafer. At the same time, PRNU does not change with the environment for a long time, and it is unique and stable like human fingerprints. Therefore, PRNU can be used as the fingerprint of the imaging sensor to identify the digital imaging device, and then complete the image source identification task. We first use a PRNU extraction algorithm to extract PRNU from the image, and then calculate the correlation between the image PRNU and the reference PRNU of the camera. If the correlation exceeds the set threshold, it is considered that the image is taken by the camera. Therefore, we can regard it as the "camera fingerprint" of identifying the source of the digital image, which is used to identify the source of a digital image.

[0004] In order to extract reliable PRNU fingerprints, researchers have proposed many effective PRNU extraction algorithms. For example, using wavelet filtering denoising method to extract PRNU fingerprint, using convolutional neural network to capture PRNU noise by learning high-level features, using dense connection hierarchical denoising network to extract PRNU fingerprint, etc. However, the PRNU related components in the noise extracted by the above methods are less, and the extraction accuracy of PRNU fingerprint needs to be improved.

[0005] Moreover, most of the existing deep learning PRNU estimation schemes take additive white Gaussian noise (AWGN) as the target for output. And use MSE loss function to train, that is, force the network output to be close to the target, however, the original camera identification task based on PRNU focuses on the proportional correlation of two PRNUs, rather than the numerical correlation, and the MSE loss function is more inclined to the numerical correlation, which will lead to poor quality of the PRNU extracted by the filter. SUMMARY

[0006] In order to solve the technical problems in the background art, the application provides a multi-feature fusion PRNU extraction method and device based on a double-layer hybrid model, which can retain more PRNU related components in the extracted noise and improve the extraction accuracy of PRNU.

[0007] In order to achieve the above purpose, the application adopts the following technical solutions:

[0008] The first aspect of the application provides a multi-feature fusion PRNU extraction method based on a double-layer hybrid model.

[0009] A multi-feature fusion PRNU extraction method based on a double-layer hybrid model comprises:

[0010] Obtaining a to-be-traced image;

[0011] Based on the to-be-traced image, a double-layer hybrid model is used to obtain a noise feature map;

[0012] The correlation between the PRNU of the noise feature map and the reference PRNU of the camera is calculated to determine whether the to-be-traced image is taken by the camera;

[0013] The process of using the double-layer hybrid model comprises: based on the to-be-traced image, using a first feature extraction network and a second feature extraction network respectively to obtain a first feature map and a second feature map; fusing the first feature map and the second feature map to obtain a first fusion feature map; after enhancing and compressing the first fusion feature map, fusing it with the to-be-traced image to obtain a second fusion feature map; after enhancing and compressing the second fusion feature map, obtaining a noise feature map; at least one convolution layer of the first feature extraction network uses an adjacent attention mechanism with an expansion factor of at least 2.

[0014] Further, the first convolution layer of the first feature extraction network uses a deformable convolution network, and the fifth layer, the ninth layer and the twelfth layer use an adjacent attention mechanism with an expansion factor of at least 2.

[0015] Further, the adjacent attention mechanism is obtained by positioning the self-attention operation to the adjacent pixels within the window.

[0016] Further, the second feature extraction network uses a hybrid network of neural network and Transformers attention mechanism.

[0017] Further, before using the double-layer hybrid model, it further comprises: training the double-layer hybrid model using a loss function to adjust the hyperparameters of the double-layer hybrid model.

[0018] Further, the loss function is

[0019] L = 1 - p(W, K)

[0020] wherein p(,) represents a p loss function, W is a provenance image PRNU estimated by a network model, and K is a reference PRNU of a camera.

[0021] Further, each of the enhanced processes comprises fusion of feature maps, batch normalization BN, and ReLU algorithm.

[0022] The second aspect of the present application provides a multi-feature fusion PRNU extraction device based on a double-layer hybrid model.

[0023] A multi-feature fusion PRNU extraction system based on a double-layer hybrid model, comprising:

[0024] A data acquisition module configured to acquire a to-be-provenance image;

[0025] A feature extraction module configured to obtain a noise feature map based on the to-be-provenance image by using a double-layer hybrid model;

[0026] A provenance module configured to calculate the correlation between the PRNU of the noise feature map and the reference PRNU of a camera to determine whether the to-be-provenance image is taken by the camera;

[0027] A double-layer hybrid model construction module configured to obtain a first feature map and a second feature map by using a first feature extraction network and a second feature extraction network respectively based on the to-be-provenance image; fuse the first feature map and the second feature map to obtain a first fused feature map; fuse the first fused feature map after enhancement and compression with the to-be-provenance image to obtain a second fused feature map; obtain the noise feature map after enhancement and compression of the second fused feature map; at least one convolution layer of the first feature extraction network uses an adjacent attention mechanism with a dilation factor of at least 2.

[0028] The third aspect of the present application provides a computer readable storage medium.

[0029] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the multi-feature fusion PRNU extraction method based on a double-layer hybrid model according to the first aspect described above.

[0030] The fourth aspect of the present application provides a computer device.

[0031] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multi-feature fusion PRNU extraction method based on a double-layer hybrid model according to the first aspect described above.

[0032] Compared with the prior art, the present application has the beneficial effects that:

[0033] The present application uses image-PRNU pairs and a proportional correlation-based p loss function to train the network model, which can further enhance the extraction performance of the model for PRNU. The multi-feature fusion PRNU extraction network based on a double-layer hybrid model can extract different features and enhance the generalization ability of the extraction network, such as PRNU noise in different cameras. For complex noise images, the fusion of global and local features can better extract the PRNU details hidden in the image and enhance the expression ability of the PRNU noise extraction network. BRIEF DESCRIPTION OF DRAWINGS

[0034] The drawings accompanying the specification of the present application form a part thereof and serve to provide further understanding of the present application, the illustrative embodiments of the present application and its description serve to explain the present application and do not constitute an improper limitation of the present application.

[0035] Figure 1 is a structure diagram of the multi-feature fusion PRNU extraction network based on a double-layer hybrid model shown by the present application;

[0036] Figure 2 is a calculation process diagram of the adjacent attention mechanism shown by the present application;

[0037] Figure 3 is a diagram showing the attention range of a single pixel in the adjacent attention mechanism (NA) and the extended adjacent attention mechanism (DiNA) shown by the present application;

[0038] Figure 4 is a structure diagram of the DiNAT Block network shown by the present application;

[0039] Figure 5 is a flowchart of the multi-feature fusion PRNU extraction method based on a double-layer hybrid model shown by the present application;

[0040] Figure 6 is a performance comparison diagram of different PRNU estimation methods based on ROC curves under different cropping sizes in the Daxing dataset shown by the present application; Figure 6 (a) in (a) represents a performance comparison diagram of different PRNU estimation methods based on ROC curves under the size of 64x64 in the Daxing dataset; Figure 6 (b) in (a) represents a performance comparison diagram of different PRNU estimation methods based on ROC curves under the size of 96x96 in the Daxing dataset; Figure 6 (c) in (a) represents a performance comparison diagram of different PRNU estimation methods based on ROC curves under the size of 128x128 in the Daxing dataset; Figure 6Fig. 4 (d) shows the performance comparison of different PRNU estimation methods based on ROC curve in the Daxing dataset at 256x256 size.

[0041] Figure 7 Fig. 4 (d) shows the performance comparison of different PRNU estimation methods based on ROC curve in the Daxing dataset at 256x256 size. Figure 7 Fig. 4 (a) shows the performance comparison of different PRNU estimation methods based on ROC curve in the Dresden dataset at 64x64 size. Figure 7 Fig. 4 (b) shows the performance comparison of different PRNU estimation methods based on ROC curve in the Dresden dataset at 96x96 size. Figure 7 Fig. 4 (c) shows the performance comparison of different PRNU estimation methods based on ROC curve in the Dresden dataset at 128x128 size. Figure 7 Fig. 4 (d) shows the performance comparison of different PRNU estimation methods based on ROC curve in the Dresden dataset at 256x256 size. DETAILED DESCRIPTION

[0042] The present application will be further described with reference to the drawings and examples.

[0043] It should be noted that the following detailed description is merely exemplary and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0044] It is also important to note that the terms used herein are not intended to limit the exemplary embodiments to the specific embodiments which are described herein. Rather, it is contemplated that the exemplary embodiments are intended to encompass all possible embodiments which are within the scope of the present application. As used herein, unless otherwise indicated, the use of the singular includes the plural, and the use of the plural includes the singular. Also, as used herein, unless otherwise indicated, the use of the term "and / or," as well as the use of "or" means "and" or "or". In addition, it should also be noted that, as used herein, the terms "the" and "a" are intended to mean "one or more" or "at least one", and not "only one".

[0045] It should be noted that the flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of the present disclosure. It should be noted that each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which can include one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the drawings. For example, two blocks noted in succession can in fact be executed substantially concurrently or can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts and / or block diagrams and combinations of blocks in the flowcharts and / or block diagrams can be implemented by special-purpose hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0046] Embodiment one

[0047] Explanation of terms: PRNU: the photo response non-uniformity (PRNU) is unique identification for a single shooting device due to the existence of anomalies in the silicon wafer used in the process of image from shooting to forming.

[0048] The embodiment provides a multi-feature fusion PRNU extraction method based on a double-layer hybrid model. The embodiment takes the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server and is realized through interaction of the terminal and the server. The server can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, security services CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. In the embodiment, the method includes the following steps:

[0049] Obtaining a to-be-traced image;

[0050] Based on the to-be-traced image, a double-layer hybrid model is used to obtain a noise feature map;

[0051] The correlation between the PRNU of the noise feature map and the reference PRNU of the camera is calculated to determine whether the to-be-traced image is shot by the camera or not;

[0052] The process of adopting the double-layer hybrid model comprises: based on the image to be traced, first feature extraction network and second feature extraction network are adopted respectively to obtain first feature map and second feature map; the first feature map and the second feature map are fused to obtain a first fusion feature map; after the first fusion feature map is enhanced and compressed, the first fusion feature map is fused with the image to be traced to obtain a second fusion feature map; after the second fusion feature map is enhanced and compressed, a noise feature map is obtained; at least one convolution layer of the first feature extraction network adopts an adjacent attention mechanism with an expansion factor of at least 2.

[0053] The robustness and effectiveness of the PRNU fingerprint usually depend on the denoising filter, so this embodiment focuses on how to introduce an effective image noise extraction method to improve the quality of the PRNU camera fingerprint as much as possible. On the other hand, the preparation of training data and the design of the loss function are key steps in PRNU fingerprint extraction. Most existing deep learning PRNU estimation schemes output with additive white Gaussian noise (AWGN) as the target. And use the MSE loss function to train, that is, force the network output to be close to the target, however, the original camera identification task based on PRNU focuses on the proportional correlation of two PRNUs, rather than the numerical correlation, and the MSE loss function is more inclined to the numerical correlation, which will lead to poor quality of the PRNU extracted by the filter, and the following examples make the difference more obvious. Suppose there is a four-pixel camera, and its reference PRNU is:

[0054]

[0055] where, and are two PRNUs estimated by the CNN filter:

[0056]

[0057] According to the standard of the MSE loss function, relative to the reference is a better PRNU estimate because However, if we check the similarity based on the proportional correlation, is a better PRNU estimate than because Therefore, for PRNU extraction, it is necessary to use the ρ loss function instead of the MSE loss function.

[0058] The multi-feature fusion PRNU extraction network structure based on the double-layer hybrid model proposed in this embodiment is shown in Figure 1 It is composed of three modules: feature extraction block (FEB), enhancement block (EB) and compression block (CB).

[0059] Specifically, the feature extraction block first extracts global and local features from the given image to be traced by a sparse mechanism. Transformers tend to have greater model capacity, but due to the lack of proper inductive bias, their generalization ability may be worse than convolutional networks. In order to effectively combine the advantages of the two architectures of Transformers and convolutional networks, the first network of the feature extraction block uses a hybrid model of the two architectures to implement the sparse mechanism, thereby improving the feature extraction capability of the network. Then, in order to gradually enhance the residual information, the enhancement block fuses global and local features, and provides supplementary information for the following network through a two-stage mechanism. Next, the compression block is stacked to extract the obtained residual image and reduce the number of local parameters, and the compression network is used to reduce the computational cost. The specific introduction of each module is as follows:

[0060] (1) Feature extraction block (FEB): In order to extract accurate features, we use a multi-feature fusion method to enhance the representation ability of the network. That is, FEB includes a first network FEBnet1 and a second network FEBnet2.

[0061] For the first network FEBnet1, the sparse mechanism used is an important component. Specifically, the first layer adopts a deformable convolution network structure. Ordinary convolution operations may change the original noise distribution in the image, which may increase the training difficulty of image PRNU extraction. The deformable convolution that utilizes the relationship between surrounding pixels can effectively solve this problem. The deformable convolution has good performance in obtaining more context information. The 5th layer, the 9th layer and the 12th layer can capture rich context information through a series of adjacent attention mechanism layers (DiNAT) with a dilation factor of 2 + batch normalization BN + ReLU operation. The adjacent attention mechanism is an effective and scalable sliding window attention mechanism, as shown in Figure 2 The adjacent attention mechanism with a dilation factor of 1 and 2 respectively shows the attention range of a single pixel in the adjacent attention mechanism, and the adjacent attention mechanism with a dilation factor of 2 can capture more global context and longer distance interdependence, and expand the receptive field by a factor without additional cost, thereby significantly improving the performance of PRNU extraction. The structure of the DiNAT module is as follows: Figure 3 Figure 4 ​As shown, the DiNAT module is similar to most Transformers: attention computation is followed by an MLP with Layer Norm and skip connections in between. The first network performs ordinary convolution + batch normalization BN + ReLU operation on other layers except the 1st, 5th, 9th and 12th layers, and the extracted features are relatively less than the adjacent attention mechanism layer with a dilation factor of 2, and the combination of adjacent attention mechanism and ordinary convolution makes good use of sparsity and can better extract potential PRNU noise details.

[0062] The second network adopts the form of ordinary convolution + ReLU. The outputs of the two are fused later, and different networks can provide complementary information from multiple views. The filter size of all convolution layers is set to 3x3, and the adjacent window size of DiNAT is set to 7x7. Convolutional neural networks often have better generalization ability and faster convergence speed, which benefits from their strong advantage in inductive bias, while Transformers attention mechanism has higher model capacity and can benefit from larger data sets. The hybrid model based on the above design combining convolutional neural networks and Transformers attention mechanism can achieve better generalization ability and model capacity.

[0063] (2) Enhancement Block (EB): The enhancement block collects and fuses global and local features to provide supplementary information for the subsequent. EB includes EB1 and EB2. EB1 integrates the outputs of the two sub-networks, and through the splicing operation, two different types of features from different networks are integrated together. It is known that the features obtained by the dilated convolution in the first network are different from those obtained in the second network, which leads to a great difference in the feature distribution obtained in EB1. Therefore, batch normalization BN is used to eliminate the adverse effects. Finally, the linear features obtained are converted into nonlinear features using the ReLU algorithm. Multi-feature fusion can enhance the robustness of the obtained features. EB2 fuses global features and original image information in the same way to improve the expression ability of the network.

[0064] (3) Compression Block (CB): The compression block refines the extracted information and reduces the number of local parameters to reduce the computational cost. CB consists of CB1, CB2 and CB3, all of which are implemented using 1x1 convolution. Using 1x1 convolution reduces the dimension of the features, thereby compressing the data and improving the overall efficiency of the network.

[0065] In summary, the multi-feature fusion PRNU extraction network based on the double-layer hybrid model can extract different features and enhance the generalization ability of the extraction network, such as PRNU noise in different cameras. For complex noise images, the fusion of global and local features can better extract noise details hidden in the image and enhance the expression ability of the noise extraction network.

[0066] The overall processing flowchart of the embodiment is as shown in Figure 5 The specific process is as follows:

[0067] We first use the noise extraction network to extract noise residuals from multiple images of the same camera, then estimate a camera reference PRNU closest to the photoresponse non-uniformity PRNU from a large number of noise residuals by maximum likelihood estimation, and at the same time use the noise extraction network to extract the PRNU noise residuals of the image from the trace image. Finally, the similarity between the trace image and the fingerprint of the imaging device is measured by the peak correlation energy ratio (PCE). The process of calculating PCE is shown in the following formula:

[0068]

[0069] wherein, Y=W,X and Y have a dimension of m*n, I is the trace image, is the camera reference PRNU estimated by maximum likelihood estimation, W is the PRNU noise residual extracted from the trace image, p(s;X,Y) is the correlation function between X and Y, p(s peak :X,Y) is the maximum value of the correlation function, sign(p(s peak :X,Y)) represents the sign of the function when the maximum value is taken, and N is a small block region centered on s peak

[0070] Most existing deep learning PRNU estimation schemes take additive white Gaussian noise (AWGN) as the target for output, and our scheme takes the reference PRNU of the camera as the output target of the network model, and takes the image-PRNU pair as the form of training data. We randomly selected 10 cameras from the VISION dataset, randomly selected 40 images for each camera, and cut 238720 image-PRNU pairs of 40*40 pixels from these images and their corresponding reference PRNU for network model training.

[0071] ​The test dataset is selected from 40 cameras of 11 models in the Dresden dataset, and 100 images are randomly selected from each camera. In order to prepare the data sample, four images of the same size are cropped from the center of each original image, and PRNU extraction experiments are performed on the four cropping sizes mentioned above, that is, 256x256, 128x128, 96x96 and 64x64. Once a camera is fixed as the reference camera, the images captured by the camera (in the corresponding position) are used as positive samples, and the images from the other three cameras are used as negative samples. Similarly, 100 images are randomly selected from each camera in the Daxing dataset, which contains 40 cameras of 6 models. Similarly, four images of the same size are cropped from the center of each original image, and PRNU extraction experiments are performed on the four cropping sizes mentioned above. Once a camera is fixed as the reference camera, the images captured by the camera (in the corresponding position) are used as positive samples, and the images from the other three cameras are used as negative samples.

[0072] In order to enable the model to be effectively trained and improve the ability of the model to extract PRNU, we use the correlation-based ρ loss function to train our network model. Since |ρ|≤1, and the loss function is expected to be smaller during the training process, we define the ρ loss function as:

[0073] L = 1 - ρ(W, K)

[0074] where W is the PRNU of the trace image estimated by the network model, and K is the reference PRNU of the camera. The back propagation function of the ρ loss function can be defined as:

[0075]

[0076] The ROC curve is selected as an index for evaluating the performance of the PRNU camera fingerprint extraction algorithm in this embodiment. For the ROC curve, the true positive rate (TPR) and the false positive rate (FPR) are calculated by adding the true positives and false positives of each imaging device at the detection threshold. After the PCE value is obtained, the true positive rate and the false positive rate can be calculated by setting different thresholds, and the corresponding ROC curve graph can be drawn. The true positive rate, also known as the recall rate or the probability of detection, calculates the ratio of the true positive samples detected as true in the entire actual true sample set. The false positive rate, also known as the false alarm rate, calculates the ratio of the true positive samples detected as false in the entire actual false sample set.

[0077] The training of the network model is performed by using an image-PRNU pair and a proportional correlation-based p loss function, and the extraction performance of the model for the PRNU can be further enhanced. The multi-feature fusion PRNU extraction network based on the double-layer mixed model can extract different features and enhance the generalization ability of the extraction network, such as PRNU noise in different cameras. For complex noise images, the global and local features are fused, the PRNU details hidden in the image can be better extracted, and the expression ability of the PRNU noise extraction network is enhanced.

[0078] The performance of the algorithm can be intuitively and effectively displayed by using the ROC curve. We compared the method in the document "Effective PRNU extraction via densely connected hierarchical network" with the method of the present application in the test set, Figure 6 、 Figure 7 is a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Daxing and Dresden data sets under different cropping sizes shown in the present application; Figure 6 (a) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Daxing data set under the size of 64x64; Figure 6 (b) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Daxing data set under the size of 96x96; Figure 6 (c) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Daxing data set under the size of 128x128;

[0079] Figure 6 (d) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Daxing data set under the size of 256x256; Figure 7 (a) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Dresden data set under the size of 64x64; Figure 7 (b) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Dresden data set under the size of 96x96;

[0080] Figure 7 (c) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Dresden data set under the size of 128x128; Figure 7 (d) in the figure represents a performance comparison diagram of different PRNU estimation methods based on the ROC curve in the Dresden data set under the size of 256x256. Figure 6 andFigure 7 Both of them show that the proposed PRNU extraction algorithm achieves better performance than other algorithms in the Effective PRNU extraction via densely connected hierarchical network.

[0081] Embodiment Two

[0082] The embodiment provides a multi-feature fusion PRNU extraction system based on a double-layer hybrid model.

[0083] A multi-feature fusion PRNU extraction system based on a double-layer hybrid model comprises:

[0084] A data acquisition module configured to acquire a to-be-traced image.

[0085] A feature extraction module configured to obtain a noise feature map based on the to-be-traced image by using a double-layer hybrid model.

[0086] A tracing module configured to calculate the correlation between the PRNU of the noise feature map and the reference PRNU of a camera to determine whether the to-be-traced image is captured by the camera.

[0087] A double-layer hybrid model construction module configured to obtain a first feature map and a second feature map by using a first feature extraction network and a second feature extraction network respectively based on the to-be-traced image; fuse the first feature map and the second feature map to obtain a first fused feature map; fuse the first fused feature map after enhancement and compression with the to-be-traced image to obtain a second fused feature map; and obtain the noise feature map after enhancement and compression of the second fused feature map; at least one convolution layer of the first feature extraction network uses an adjacent attention mechanism with a dilation factor of at least 2.

[0088] It should be noted that the above data acquisition module, feature extraction module, tracing module and double-layer hybrid model construction module have the same examples and application scenarios as the steps in Embodiment One, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0089] Embodiment Three

[0090] The embodiment provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the multi-feature fusion PRNU extraction method based on a double-layer hybrid model as described in Embodiment One.

[0091] Embodiment Four

[0092] The embodiment provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps in the multi-feature fusion PRNU extraction method based on a double-layer mixed model when the program is executed.

[0093] The above merely provides preferred embodiments of the present application but not for limiting the present application. For those skilled in the art, the present application can have various modifications and changes. Any modified, equivalent replaced, improved, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-feature fusion PRNU extraction method based on a two-layer hybrid model, characterized in that, include: Obtain the image to be traced; Based on the image to be traced, a two-layer hybrid model is used to obtain the noise feature map; The correlation between the PRNU of the noise feature map and the reference PRNU of the camera is calculated to determine whether the image to be traced was taken by that camera. The process of employing a two-layer hybrid model includes: based on the image to be traced, a first feature extraction network and a second feature extraction network are used respectively to obtain a first feature map and a second feature map; the first feature map and the second feature map are fused to obtain a first fused feature map; the first fused feature map is enhanced and compressed, and then fused with the image to be traced to obtain a second fused feature map; the second fused feature map is enhanced and compressed to obtain a noise feature map; at least one convolutional layer of the first feature extraction network employs an adjacent attention mechanism with an expansion factor of at least 2. The first convolutional layer of the first feature extraction network adopts a deformable convolutional network, and the fifth, ninth and twelfth layers adopt an adjacent attention mechanism with an expansion factor of at least 2. The adjacent attention mechanism is obtained by locating the self-attention operation to adjacent pixels within the window; The second feature extraction network employs a hybrid network of neural networks and Transformers attention mechanisms.

2. The PRNU extraction method based on a two-layer hybrid model according to claim 1, characterized in that, Before employing the two-layer mixture model, the process includes: training the two-layer mixture model using a loss function and adjusting the hyperparameters of the two-layer mixture model.

3. The multi-feature fusion PRNU extraction method based on a two-layer hybrid model according to claim 1, characterized in that, The loss function is: in, express The loss function is W, which is the source image PRNU estimated by the network model, and K is the reference PRNU of the camera.

4. The PRNU extraction method based on a two-layer hybrid model according to claim 1, characterized in that, Each of the enhancement processes includes feature map fusion, batch normalization (BN), and the ReLU algorithm.

5. A multi-feature fusion PRNU extraction system based on a two-layer hybrid model, characterized in that, include: The data acquisition module is configured to acquire the image to be traced. The feature extraction module is configured to: obtain a noise feature map based on the image to be traced using a two-layer hybrid model; The tracing module is configured to: calculate the correlation between the PRNU of the noise feature map and the reference PRNU of the camera to determine whether the image to be traced was taken by the camera; A two-layer hybrid model construction module is configured to: obtain a first feature map and a second feature map based on the image to be traced, using a first feature extraction network and a second feature extraction network respectively; fuse the first feature map and the second feature map to obtain a first fused feature map; enhance and compress the first fused feature map, and then fuse it with the image to be traced to obtain a second fused feature map; enhance and compress the second fused feature map to obtain a noise feature map; at least one convolutional layer of the first feature extraction network employs an adjacent attention mechanism with a dilation factor of at least 2. The first convolutional layer of the first feature extraction network adopts a deformable convolutional network, and the fifth, ninth and twelfth layers adopt an adjacent attention mechanism with an expansion factor of at least 2. The adjacent attention mechanism is obtained by locating the self-attention operation to adjacent pixels within the window; The second feature extraction network employs a hybrid network of neural networks and Transformers attention mechanisms.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multi-feature fusion PRNU extraction method based on a two-layer hybrid model as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multi-feature fusion PRNU extraction method based on a two-layer hybrid model as described in any one of claims 1-4.

Images