PRNU matching method and system based on structural reparameterization

Abstract

This invention discloses a PRNU matching method and system based on structural reparameterization, belonging to the field of camera traceability technology. The technical problem to be solved is how to reduce the computational burden of the PRNU matching model and improve PRNU matching efficiency when the reference PRNU and the test PRNU are geometrically out of sync. The method comprises the following steps: constructing a sample dataset based on a preprocessed test PRNU, a preprocessed reference PRNU, and the correlation matrix between the test PRNU and the reference PRNU; constructing a PRNU matching model, which is an improved EfficientNet model after structural reparameterization; taking the test PRNU and the reference PRNU as input, and outputting a correlation score between the test PRNU and the reference PRNU using the trained PRNU matching model.

Description

Technical Field

[0001] The present invention relates to the field of camera traceability technology, and in particular to a PRNU matching method and system based on structure reparameterization. Background Art

[0002] In recent years, camera source identification has garnered increasing attention as one of the most important tasks in digital image forensics. This requires reliably linking a digital image to the camera that captured it. Methods based on photo response non-uniformity (PRNU) noise can effectively address this problem. Like a human fingerprint, a PRNU is unique to a single camera and stable across environmental conditions. A PRNU matching algorithm calculates the correlation between the image PRNU and a camera's reference PRNU. If this correlation exceeds a set threshold, the image is considered to have been captured by that camera. Therefore, this method can be considered a "camera fingerprint" for identifying the source of a digital image, effectively identifying the origin of the image.

[0003] The classic solution for PRNU matching is to use the peak correlation energy ratio (PCE). However, this method is computationally intensive and has low matching accuracy. When scanning large databases of "camera fingerprints," the processing time can be prohibitively long. With the widespread application of deep learning technology in image attribution tasks, this problem has been alleviated to some extent. Researchers have proposed using convolutional neural networks (CNNs) for PRNU matching. This two-channel CNN learns a patch-level comparison of camera fingerprints and image noise. After appropriate training, this method is faster and more accurate than the standard PCE method. However, this method only considers the case where the reference PRNU and test PRNU are geometrically synchronized. Furthermore, the model has a high number of parameters and computational complexity, which still consumes unnecessary computing resources and results in low computational efficiency, ultimately affecting the efficiency of PRNU matching.

[0004] How to reduce the computational burden of the PRNU matching model and improve the PRNU matching efficiency when the reference PRNU and the test PRNU are geometrically out of sync is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a PRNU matching method and system based on structural reparameterization to solve the technical problem of how to reduce the computational burden of the PRNU matching model and improve the PRNU matching efficiency when the reference PRNU and the test PRNU are geometrically out of sync.

[0006] In a first aspect, the present invention provides a PRNU matching method based on structural reparameterization, comprising the following steps:

[0007] Randomly select multiple cameras, take multiple images with each camera as test images, extract the PRNU of each test image as the test PRNU, and calculate the reference PRNU of each camera;

[0008] After preprocessing the test PRNU and the reference PRNU, the correlation matrix between the test PRUN and the reference PRNU is calculated, and a sample data set is constructed based on the correlation matrix between the preprocessed test PRNU, the preprocessed reference PRNU, and the test PRUN and the reference PRNU;

[0009] Construct a PRNU matching model, which is an improved EfficientNet model after structural reparameterization, including a convolutional layer module, an MBConv1 module, and a Stage structure. For each stage in the Stage structure, the first module in the stage is an MBConv6 module, and the other modules are PConvB modules. The convolutional layer module, the MBConv1 module, the MBConv6 module, and the PConvB module are all processed by structural reparameterization, and the ECA attention layer is introduced into the MBConv1 module and the MBConv6 module to replace the original SE attention layer. The PRNU matching model is used to take the reference PRNU and the test PRNU as input and predict the correlation score between the reference PRNU and the test PRNU;

[0010] Training the PRNU matching model based on the sample data set to obtain a trained PRNU matching model;

[0011] For the image to be tested, the test PRNU of the image to be tested is extracted, and the reference PRNU of the matching camera is calculated. The test PRNU and the reference PRNU are used as input, and the correlation score between the test PRNU and the reference PRNU is predicted and output by the trained PRNU matching model. If the correlation score meets the threshold, it is determined that the test PRNU is correlated with the reference PRNU, and the matching camera is the source camera of the image to be tested.

[0012] Preferably, the test PRNU and the reference PRNU are preprocessed, including:

[0013] Clip the reference PRNU from the center to Q*Q pixels to obtain the clipped reference PRNU;

[0014] Clip the test PRNU from the center to within P*P pixels to obtain the clipped test PRNU where P is less than Q;

[0015] Normalize the trimmed test PRNU and the trimmed reference PRNU respectively to obtain the normalized test PRNU and the normalized reference PRNU;

[0016] Performing a padding operation on the normalized test PRNU, so that the padded test PRNU has the same size as the reference PRNU;

[0017] Among them, the trimmed test PRNU and the trimmed reference PRNU are the noise map to be processed I, whose size is M×N, and the normalized noise map is

[0018]

[0019] Preferably, the correlation matrix ρ between the test PRUN and the reference PRNU is calculated as follows:

[0020]

[0021] Among them, W and K d Represents W and K respectively d represents the mean of , ||-|| represents the two-norm operation, · represents the dot multiplication operation of the matrix, and the size of the correlation matrix ρ is the same as that of the reference PRNU.

[0022] Preferably, the MBConv1 module includes a k×k convolutional layer with reparameterized structure, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer;

[0023] The MBConv6 module includes a 1×1 convolutional layer, a k×k convolutional layer with structural reparameterization, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer;

[0024] The PConvB module includes a k×k convolutional layer with reparameterized structure, a Swish activation layer, and a 1×1 convolutional layer;

[0025] Taking the MBConv1 module, MBConv6 module, and PConvB module as the target modules, the structure of the reparameterized k×k convolutional layer of the target module is expressed as follows: during training, the k×k convolutional layer is replaced with three or four branches, namely, a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, a 1×1 convolutional layer followed by a BatchNorm layer, and a BatchNorm layer. When the feature map at the input and output sizes of the target module remain unchanged, the branch includes the BatchNorm layer, and when the feature map at the input and output sizes of the target module change, the branch does not include the BatchNorm layer; during inference, the branch is replaced with a k×k convolutional layer;

[0026] The structural reparameterization is calculated by the following formula:

[0027] Reparam(K*K)=BV(Conv(K,K))+BN(Conv(3,3))+BN(Conv(1,1))+BN

[0028] Reparam(K*K) represents the reparameterized convolution, Conv(K, K) represents the k×k convolution, and BN() represents the BatchNorm layer.

[0029] Preferably, for the input feature map, the PConvB module is used to extract features from the first or last continuous channel in the feature map through a convolution operation, while the remaining channels remain unchanged;

[0030] For the output of the PConv module, the feature maps of each channel are integrated through two point-by-point convolution PWConvs, and a standardized convolution layer and a ReLu activation layer are set between the two point-by-point convolution PWConvs.

[0031] Preferably, the ECA attention layer is configured to perform the following:

[0032] Perform a global average pooling operation on the input feature map, change the h and w dimensions in the feature map to 1, and only retain the c dimension to obtain a 1×1×c feature vector;

[0033] A one-dimensional convolution operation with a convolution kernel size of k is used to obtain a feature map of the same size as the feature vector of 1×1×c. The feature map is then normalized and the Sigmoid activation function is applied to obtain the weights of each channel.

[0034] Multiply the weights by the corresponding elements of the input feature map to obtain the final output feature map.

[0035] Preferably, when training the PRNU matching model based on the sample data set, the sample data set is divided into a training set, a validation set and a test set;

[0036] The constructed PRNU matching model is trained, verified, and tested based on the training set, validation set, and test set. During the model training, validation, and testing process, cross entropy is used as the loss function. The cross entropy loss function L for binary classification is expressed as follows:

[0037]

[0038] Among them, y represents the true label, Represents the model's predicted probability for a given sample. The closer to the true label y, the smaller the L value, indicating that the classification error rate of the model is smaller. Conversely, the larger the L value, the greater the classification error rate of the model.

[0039] Based on the loss function, the network parameters of the PRNU matching model are learned using the Adam optimization algorithm and the back-propagation algorithm. During the network parameter learning process, the learning rate and batch size are set, an early stopping training strategy is adopted, and a patience value A and an epoch maximum training value B are set. When the accuracy of correctly matching the noise in the validation set no longer increases within A epochs or reaches B in epochs, model training is stopped and the network parameters that achieve the maximum accuracy of correctly matching the noise in the validation set are saved to obtain the final trained PRNU matching model.

[0040] In a second aspect, the present invention provides a PRNU matching system based on structural reparameterization, configured to perform PRNU matching using a PRNU matching method based on structural reparameterization as described in any one of the first aspects, so as to perform camera traceability on an image to be tested. The system comprises:

[0041] An image acquisition module, wherein the image acquisition module is used to randomly select multiple cameras, capture multiple images as test images through each camera, extract the PRNU of each test image as a test PRNU, and calculate a reference PRNU for each camera;

[0042] a sample processing module configured to preprocess the test PRNU and the reference PRNU, calculate a correlation matrix between the test PRUN and the reference PRNU, and construct a sample data set based on the preprocessed test PRNU, the preprocessed reference PRNU, and the correlation matrix between the test PRUN and the reference PRNU;

[0043] A model construction module, which is used to construct a PRNU matching model. The PRNU model is an improved EfficientNet model after structural reparameterization, including a convolutional layer module, an MBConv1 module, and a Stage structure. For each stage in the Stage structure, the first module in the stage is an MBConv6 module, and the other modules are PConvB modules. The convolutional layer module, the MBConv1 module, the MBConv6 module, and the PConvB module are all processed by structural reparameterization, and the ECA attention layer is introduced into the MBConv1 module and the MBConv6 module to replace the original SE attention layer. The PRNU matching model is used to take the reference PRNU and the test PRNU as input and predict the correlation score between the output reference PRNU and the test PRNU;

[0044] A model training module, wherein the model training module is used to train the PRNU matching model based on a sample data set to obtain a trained PRNU matching model;

[0045] A matching module is used to extract the test PRNU of the image to be tested and calculate the reference PRNU of the matching camera. The matching module takes the test PRNU and the reference PRNU as input and predicts the correlation score between the test PRNU and the reference PRNU through the trained PRNU matching model. If the correlation score meets the threshold, it is determined that the test PRNU is correlated with the reference PRNU, and the matching camera is the source camera of the image to be tested.

[0046] Preferably, the sample processing module is used to perform the following preprocessing operations on the test PRNU and the reference PRNU:

[0047] Clip the reference PRNU from the center to Q*Q pixels to obtain the clipped reference PRNU;

[0048] Clip the test PRNU from the center to within P*P pixels to obtain the clipped test PRNU where P is less than Q;

[0049] Normalize the trimmed test PRNU and the trimmed reference PRNU respectively to obtain the normalized test PRNU and the normalized reference PRNU;

[0050] Performing a padding operation on the normalized test PRNU, so that the padded test PRNU has the same size as the reference PRNU;

[0051] Among them, the trimmed test PRNU and the trimmed reference PRNU are the noise map to be processed I, whose size is M×N, and the normalized noise map is

[0052]

[0053] The correlation matrix ρ of the test PRUN and reference PRNU is calculated as follows:

[0054]

[0055] in, and Represents W and K respectively d The mean of , ||-|| represents the two-norm operation, · represents the matrix point multiplication operation, and the size of the correlation matrix ρ is the same as that of the reference PRNU;

[0056] When the PRNU matching model is trained based on a sample data set, the model training module is used to perform the following:

[0057] Divide the sample data set into training set, validation set and test set;

[0058] The constructed PRNU matching model is trained, verified, and tested based on the training set, validation set, and test set. During the model training, validation, and testing process, cross entropy is used as the loss function. The cross entropy loss function for binary classification is expressed as follows:

[0059]

[0060] Among them, y represents the true label, Represents the model's predicted probability for a given sample. The closer to the true label y, the smaller the L value, indicating that the classification error rate of the model is smaller. Conversely, the larger the L value, the greater the classification error rate of the model.

[0061] Based on the loss function, the network parameters of the PRNU matching model are learned using the Adam optimization algorithm and the back-propagation algorithm. During the network parameter learning process, the learning rate and batch size are set, an early stopping training strategy is adopted, and a patience value A and an epoch maximum training value B are set. When the accuracy of correctly matching the noise in the validation set no longer increases within A epochs or reaches B in epochs, model training is stopped and the network parameters that achieve the maximum accuracy of correctly matching the noise in the validation set are saved to obtain the final trained PRNU matching model.

[0062] Preferably, the MBConv1 module includes a k×k convolutional layer with reparameterized structure, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer;

[0063] The MBConv6 module includes a 1×1 convolutional layer, a k×k convolutional layer with structural reparameterization, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer;

[0064] The PConvB module includes a k×k convolutional layer with reparameterized structure, a Swish activation layer, and a 1×1 convolutional layer;

[0065] Taking the MBConv1 module, MBConv6 module, and PConvB module as the target modules, the structure of the reparameterized k×k convolutional layer of the target module is expressed as follows: during training, the k×k convolutional layer is replaced with three or four branches, namely, a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, a 1×1 convolutional layer followed by a BatchNorm layer, and a BatchNorm layer. When the feature map at the input and output sizes of the target module remain unchanged, the branch includes the BatchNorm layer, and when the feature map at the input and output sizes of the target module change, the branch does not include the BatchNorm layer; during inference, the branch is replaced with a k×k convolutional layer;

[0066] The structural reparameterization is calculated by the following formula:

[0067] Reparam(K*K)=BN(Conv(K,K))+BN(Conv(3,3))+BN(Conv(1,1))+BN

[0068] Where Reparam(K*K) represents the reparameterized convolution, Conv(K, K) represents the k×k convolution, and BN() represents the BatchNorm layer;

[0069] For the input feature map, the PConvB module is used to extract features from the first or last continuous channel in the feature map through convolution operation, while the other channels remain unchanged;

[0070] For the output of the PConv module, the feature maps of each channel are integrated through two point-by-point convolution PWConvs, and a standardized convolution layer and a ReLu activation layer are set between the two point-by-point convolution PWConvs;

[0071] The ECA attention layer is used to perform the following:

[0072] Perform a global average pooling operation on the input feature map, change the h and w dimensions in the feature map to 1, and only retain the c dimension to obtain a 1×1×c feature vector;

[0073] A one-dimensional convolution operation with a convolution kernel size of k is used to obtain a feature map of the same size as the feature vector of 1×1×c. The feature map is then normalized and the Sigmoid activation function is applied to obtain the weights of each channel.

[0074] Multiply the weights by the corresponding elements of the input feature map to obtain the final output feature map.

[0075] The PRNU matching method and system based on structural reparameterization of the present invention have the following advantages:

[0076] 1. Calculate the correlation matrix between the reference PRNU and the test PRNU of the test image and input it into the constructed PRNU matching model for model training, so that the matching model can withstand cropping and shifting;

[0077] 2. For the constructed PRNU matching model, structural reparameterization technology and lightweight partial convolution (PConv) are used to optimize the EfficientNet network. At the same time, the SE attention module in the original model is replaced with the ECA attention mechanism. This can reduce the number of model parameters and computational complexity, reduce the use of computing resources, and improve the PRNU matching efficiency while ensuring the accuracy of the matching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0079] The present invention will be further described below with reference to the accompanying drawings.

[0080] Figure 1 This is a flowchart of a PRNU matching method based on structure reparameterization in Example 1;

[0081] Figure 2 This is a structural block diagram of a PRNU matching model in a PRNU matching method based on structural reparameterization in Example 1. DETAILED DESCRIPTION

[0082] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0083] The embodiments of the present invention provide a PRNU matching method and system based on structural reparameterization to solve the technical problem.

[0084] Example 1:

[0085] The present invention provides a PRNU matching method based on structural reparameterization, comprising the following steps:

[0086] S100, randomly selecting multiple cameras, using each camera to capture multiple images as test images, extracting the PRNU of each test image as a test PRNU, and calculating a reference PRNU for each camera;

[0087] S200, after preprocessing the test PRNU and the reference PRNU, calculating the correlation matrix between the test PRUN and the reference PRNU, and constructing a sample data set based on the preprocessed test PRNU, the preprocessed reference PRNU, and the correlation matrix between the test PRUN and the reference PRNU;

[0088] S300, constructing a PRNU matching model, wherein the PRNU model is an improved EfficientNet model after structural reparameterization, including a convolutional layer module, an MBConv1 module, and a Stage structure. For each stage in the Stage structure, the first module in the stage is an MBConv6 module, and the other modules are PConvB modules. The convolutional layer module, the MBConv1 module, the MBConv6 module, and the PConvB module are all subjected to structural reparameterization processing, and the ECA attention layer is introduced into the MBConv1 module and the MBConv6 module to replace the original SE attention layer. The PRNU matching model is used to take the reference PRNU and the test PRNU as input and predict the correlation score between the reference PRNU and the test PRNU;

[0089] S400, training the PRNU matching model based on a sample data set to obtain a trained PRNU matching model;

[0090] S500: For the image to be tested, extract the test PRNU of the image to be tested, and calculate the reference PRNU of the matching camera. With the test PRNU and the reference PRNU as input, predict and output a correlation score between the test PRNU and the reference PRNU through the trained PRNU matching model. If the correlation score meets a threshold, it is determined that the test PRNU is correlated with the reference PRNU, and the matching camera is the source camera of the image to be tested.

[0091] In this embodiment, step S100 is the preparation of sample data. To improve the applicability of the PRNU matching model, this step randomly selects 36 cameras from two different datasets (Dresden and Vision). Each camera has about 200 images captured to extract the test PRNU.

[0092] In order to effectively train and converge the PRNU matching model, the test PRNU and reference PRNU obtained in step S100 are preprocessed in step S200. The specific implementation of the preprocessing includes the following operations:

[0093] (1) Clip the reference PRNU from the center to Q*Q pixels, where Q is 256, to obtain the clipped reference PRNU;

[0094] The test PRNU is clipped from the center to within P*P pixels to obtain the clipped test PRNU where P is less than Q and P takes values of 128, 140, and 180;

[0095] (2) Normalizing the trimmed test PRNU and the trimmed reference PRNU respectively to obtain the normalized test PRNU and the normalized reference PRNU;

[0096] (3) Perform padding operation on the normalized test PRNU, so that the padded test PRNU has the same size as the reference PRNU.

[0097] Among them, the trimmed test PRNU and the trimmed reference PRNU are the noise map to be processed I, whose size is M×N, and the normalized noise map is

[0098]

[0099] The correlation matrix ρ between the test PRUN and the reference PRNU in step S200 is calculated as follows:

[0100]

[0101] in, and Represents W and K respectively d represents the mean of , ||-|| represents the two-norm operation, · represents the dot multiplication operation of the matrix, and the size of the correlation matrix ρ is the same as that of the reference PRNU.

[0102] During the sample data processing of step S200, the reference PRNU is first cropped from the center to Q×Q pixels to limit the complexity of the network. In order to better compare the matching performance of different models, Q is taken as 256. Then, the test PRNU extracted from each image is cropped from the center to a P×P pixel area, where the value of P should be smaller than Q. P is taken as 128, 140 and 180 to evaluate the matching performance of the model at different cropping sizes.

[0103] Step S300 is to build a PRNU matching model. This model is an improved EfficientNet model after structural reparameterization and is named RepEfficientNet. The structural reparameterization technology is used to reduce the FLOPs and memory usage of the model. The model structure is as follows Figure 2 As shown, the convolutional layer module corresponds to Figure 2 (b) is the standard convolution after structural reparameterization. The MBConv1 module corresponds to Figure 2 (c) includes a k×k convolutional layer with structure reparameterization, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer. The Stage structure is as follows Figure 2As shown in (d), there are six stages in total. In Stage 1 to Stage 6, since MBConv6 can more conveniently change the size and number of channels of the feature map, the first block at the beginning of each stage in this embodiment uses MBConv6 to adjust the size and number of channels of the feature map, and the remaining blocks use PConvB instead of MBConvB6, thereby simply and effectively making full use of the redundancy between different channels of the feature map. Figure 2 (f) and Figure 2 (g) Correspond to the MBConv6 module and PConvB module respectively. The MBConv6 module includes a 1×1 convolution layer, a structurally reparameterized k×k convolution layer, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolution layer; the PConvB module includes a structurally reparameterized k×k convolution layer, a Swish activation layer, and a 1×1 convolution layer.

[0104] Table 1 shows the changes in the number of channels and layers in each stage of the baseline model RepEfficientNet-B0. For the PRNU noise matching model, the proposed structure has lower computational resource consumption and higher matching efficiency.

[0105] Table 1:

[0106]

[0107] The PConvB module (partial convolution module) can reduce the redundant calculations between different channels in the middle layer of the neural network and reduce memory access. Its structure is as follows Figure 2 As shown in (e), for the input feature map, the PConvB module is used to extract features from the first or last continuous channel in the feature map through convolution operation, while the other channels remain unchanged. Assume the following partial ratios:

[0108]

[0109] PConv's FLOPs are only 1 / 16 of standard convolution, and it has smaller memory access:

[0110]

[0111] When the fraction ratio r = 1 / 4, PConv requires only 1 / 4 the memory access of standard convolution. To fully utilize information from all channels, the embodiment uses two point-wise convolutions (PWConv) after PConv to integrate the feature maps of each channel, and places the normalization and ReLu activation layers between the two PWConvs, maintaining feature extraction diversity while reducing computational latency.

[0112] ECA Attention Mechanism: Efficient Channel Attention (ECA) is a lightweight and efficient attention mechanism. The design idea of the ECA attention mechanism is to learn the weight of each channel by using one-dimensional convolution in the channel dimension, thereby achieving an efficient attention mechanism.

[0113] In this embodiment, the ECA attention layer is used to perform the following:

[0114] (1) Perform a global average pooling operation on the input feature map, change the h and w dimensions in the feature map to 1, and only retain the c dimension to obtain a 1×1×c feature vector;

[0115] (2) A one-dimensional convolution operation with a convolution kernel size of k is performed to obtain a feature map of the same size as the feature vector of 1×1×c. The feature map is normalized and the Sigmoid activation function is performed to obtain the weights of each channel.

[0116] (3) Multiply the weights by the corresponding elements of the input feature map to obtain the final output feature map.

[0117] To avoid manually adjusting the parameter k during cross-validation, the ECA module determines the value of k in an adaptive manner.

[0118] For neural networks, multi-branch architecture training is more stable and easier. For example, the residual connection of ResNet can solve the problem of gradient disappearance very well. Although its shortcut does not take up computing power, it doubles the memory usage, which will affect the model inference speed when used in actual application scenarios. The single-branch structure has fast inference speed and saves memory. However, training is more difficult and the training performance is low. In order to solve the above problems, this embodiment uses structural reparameterization to decouple the training and inference architectures. The basic idea is to first construct a series of structures (generally used for training) and convert their parameters into another set of parameters (generally used for inference), thereby converting the multi-branch model into a single-path model. In real scenarios, training resources are generally relatively abundant. This embodiment is more concerned with the cost and performance during inference. Therefore, this embodiment wants the structure during training to be larger and have good properties, such as higher accuracy or other properties. The converted inference structure is smaller in scale and retains these properties. The entire process is based on Figure 2 (c) is an example. The right side is the MBConv1 module after structural reparameterization. From shallow to deep, it includes: k×k convolution layer based on structural reparameterization, Swish activation layer, ECA attention mechanism layer and 1×1 convolution layer. Figure 2(c) As shown on the left, the k×k convolutional layer based on structural reparameterization means that during training, the k×k convolutional layer is replaced with three or four branches, namely a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, and a 1×1 convolutional layer followed by a BatchNorm layer. Since the input and output sizes of the feature map in the MBConvB1 module remain unchanged, an additional BatchNorm layer branch is included. Conversely, if the sizes are different, this BatchNorm layer branch is not included. During inference, these branches are replaced with a k×k convolutional layer. Structural reparameterization can be expressed as follows:

[0119] Reparam(K*K)=BV(Conv(K,K))+BN(Conv(3,3))+BN(Conv(1,1))+BN

[0120] Reparam(K*K) represents the reparameterized convolution, Conv(K, K) represents the k×k convolution, and BN() represents the BatchNorm layer. In this embodiment, K=3.

[0121] In this embodiment, the MBConv1 module, the MBConv6 module, and the PConvB module are used as target modules. The structurally reparameterized k×k convolutional layer of the target module is expressed as follows: during training, the k×k convolutional layer is replaced with three or four branches, namely, a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, a 1×1 convolutional layer followed by a BatchNorm layer, and a BatchNorm layer. When the input and output sizes of the feature map remain unchanged at the target module, the branch includes the BatchNorm layer; when the input and output sizes of the feature map change at the target module, the branch does not include the BatchNorm layer. During inference, the branch is replaced with a k×k convolutional layer.

[0122] Step S400 model training, this step constructs a sample data set based on the preprocessed test PRNU, the preprocessed reference PRNU, and the correlation matrix between the test PRNU and the reference PRNU, and randomly divides the sample data set into three categories: a training set (accounting for 60% of the total), a validation set (20%), and a test set (20%).

[0123] The constructed PRNU matching model is trained, verified, and tested based on the training set, validation set, and test set. During the model training, validation, and testing process, cross entropy is used as the loss function. The cross entropy loss function L for binary classification is expressed as follows:

[0124]

[0125] Among them, y represents the true label, Represents the model's predicted probability for a given sample. The closer to the true label y, the smaller the L value, indicating that the classification error rate of the model is smaller. Conversely, the larger the L value, the greater the classification error rate of the model.

[0126] Based on the loss function, the network parameters of the PRNU matching model are learned using the Adam optimization algorithm and the back-propagation algorithm. During the network parameter learning process, the learning rate is set to 0.001, the batch size is set to 72, the early stopping training strategy is adopted, the patience value is set to 30, and the maximum epoch training value is set to 500. When the accuracy of correctly matching the noise in the validation set no longer improves within 30 epochs or the epoch reaches 500, the model training is stopped and the network parameters with the maximum accuracy of correctly matching the noise in the validation set are saved to obtain the final trained PRNU matching model.

[0127] The application scenario of this model is to identify the source camera from a limited pool of devices. The PRNU of the images in the test set is matched one by one with the reference PRNUs of 36 cameras. The camera with the highest score is the camera associated with the test image. Therefore, the accuracy of correctly tracing the source of each test image is an important evaluation metric. Since our goal is to make the model lighter and improve the matching efficiency of the model, this solution uses two evaluation metrics: accuracy and average comparison time per PRNU pair.

[0128] Tables 2 to 4 show the matching performance of different models at matching sizes of 256-128, 256-140, and 256-180, respectively. As shown in the tables, the RepEfficientNet proposed in this embodiment can achieve the best matching performance at different sizes.

[0129] Table 2. Performance comparison of different models with matching size of 256-128

[0130]

[0131] Table 3. Performance comparison of different models with matching size of 256-140

[0132]

[0133] Table 4. Performance comparison of different models with matching size of 256-180

[0134]

[0135] The method of this embodiment is to calculate the correlation matrix of the reference PRNU and the test PRNU of the test image and then input them into the neural network, so that the matching model can withstand cropping and shifting, and use structural reparameterization technology and lightweight partial convolution (PConv) to optimize the EfficientNet network. At the same time, the SE attention module in the original model is replaced with the ECA module, which can reduce the number of parameters and calculations of the model, reduce the occupancy of computing resources, and improve the matching efficiency of PRNU, while ensuring the accuracy of the matching results.

[0136] Example 2:

[0137] The present invention provides a PRNU matching system based on structural reparameterization, including an image acquisition module, a sample processing module, a model construction module, a model training module and a matching module. The system can execute the method disclosed in Example 1 to perform camera tracing on the image to be tested.

[0138] The image acquisition module is used to randomly select multiple cameras, take multiple images through each camera as test images, extract the PRNU of each test image as the test PRNU, and calculate the reference PRNU of each camera.

[0139] In this embodiment, in order to improve the applicability of the PRNU matching model, the image acquisition module is used to randomly select 36 cameras from two different datasets (Dresden and Vision), and each camera has about 200 images captured to extract the test PRNU.

[0140] The sample processing module is used to preprocess the test PRNU and the reference PRNU, calculate the correlation matrix of the test PRUN and the reference PRNU, and construct a sample data set based on the preprocessed test PRNU, the preprocessed reference PRNU and the correlation matrix of the test PRUN and the reference PRNU.

[0141] In order to effectively train and converge the PRNU matching model, as a specific implementation of preprocessing, the sample processing module is used to perform the following operations:

[0142] (1) Clip the reference PRNU from the center to Q*Q pixels, where Q is 256, to obtain the clipped reference PRNU;

[0143] The test PRNU is clipped from the center to within P*P pixels to obtain the clipped test PRNU where P is less than Q and P takes values of 128, 140, and 180;

[0144] (2) Normalizing the trimmed test PRNU and the trimmed reference PRNU respectively to obtain the normalized test PRNU and the normalized reference PRNU;

[0145] (3) Perform padding operation on the normalized test PRNU, so that the padded test PRNU has the same size as the reference PRNU.

[0146] Among them, the trimmed test PRNU and the trimmed reference PRNU are the noise map to be processed I, whose size is M×N, and the normalized noise map is

[0147]

[0148] The correlation matrix ρ of the test PRUN and reference PRNU is calculated as follows:

[0149]

[0150] in, and Represents W and K respectively d represents the mean of , ||-|| represents the two-norm operation, · represents the dot multiplication operation of the matrix, and the size of the correlation matrix ρ is the same as that of the reference PRNU.

[0151] The model construction module is used to construct a PRNU matching model. The PRNU model is an improved EfficientNet model after structural reparameterization, including a convolutional layer module, an MBConv1 module and a Stage structure. For each stage in the Stage structure, the first module in the stage is an MBConv6 module and the other modules are PConvB modules. Among them, the convolutional layer module, MBConv1 module, MBConv6 module and PConvB module are all processed by structural reparameterization, and the ECA attention layer is introduced into the MBConv1 module and the MBConv6 module to replace the original SE attention layer. The PRNU matching model is used to take the reference PRNU and the test PRNU as input and predict the correlation score between the output reference PRNU and the test PRNU.

[0152] The model constructed in this embodiment is an improved EfficientNet model after structural reparameterization, which uses structural reparameterization technology to reduce the FLOPs and memory usage of the model. Figure 2 As shown, the convolutional layer module corresponds to Figure 2 (b) is the standard convolution after structural reparameterization. The MBConv1 module corresponds to Figure 2 (c) includes a k×k convolutional layer with structure reparameterization, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer. The Stage structure is as follows Figure 2As shown in (d), there are six stages in total. In Stage 1 to Stage 6, since MBConv6 can more conveniently change the size and number of channels of the feature map, the first block at the beginning of each stage in this embodiment uses MBConv6 to adjust the size and number of channels of the feature map, and the remaining blocks use PConvB instead of MBConvB6, thereby simply and effectively making full use of the redundancy between different channels of the feature map. Figure 2 (f) and Figure 2 (g) Correspond to the MBConv6 module and PConvB module respectively. The MBConv6 module includes a 1×1 convolution layer, a structurally reparameterized k×k convolution layer, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolution layer; the PConvB module includes a structurally reparameterized k×k convolution layer, a Swish activation layer, and a 1×1 convolution layer.

[0153] The PConvB module (partial convolution module) can reduce the redundant calculations between different channels in the middle layer of the neural network and reduce memory access. Its structure is as follows Figure 2 As shown in (e), for the input feature map, the PConvB module is used to extract features from the first or last continuous channel in the feature map through convolution operation, while the other channels remain unchanged. Assume the following partial ratios:

[0154]

[0155] PConv's FLOPs are only 1 / 16 of standard convolution, and it has smaller memory access:

[0156]

[0157] When the fraction ratio r = 1 / 4, PConv requires only 1 / 4 the memory access of standard convolution. To fully utilize information from all channels, the embodiment uses two point-wise convolutions (PWConv) after PConv to integrate the feature maps of each channel, and places the normalization and ReLu activation layers between the two PWConvs, maintaining feature extraction diversity while reducing computational latency.

[0158] ECA Attention Mechanism: Efficient Channel Attention (ECA) is a lightweight and efficient attention mechanism. The design idea of the ECA attention mechanism is to learn the weight of each channel by using one-dimensional convolution in the channel dimension, thereby achieving an efficient attention mechanism.

[0159] In this embodiment, the ECA attention layer is used to perform the following:

[0160] (1) Perform a global average pooling operation on the input feature map, change the h and w dimensions in the feature map to 1, and only retain the c dimension to obtain a 1×1×c feature vector;

[0161] (2) A one-dimensional convolution operation with a convolution kernel size of k is performed to obtain a feature map of the same size as the feature vector of 1×1×c. The feature map is normalized and the Sigmoid activation function is performed to obtain the weights of each channel.

[0162] (3) Multiply the weights by the corresponding elements of the input feature map to obtain the final output feature map.

[0163] To avoid manually adjusting the parameter k during cross-validation, the ECA module determines the value of k in an adaptive manner.

[0164] This embodiment uses structural reparameterization to decouple the architecture during training and inference. The basic idea is to first construct a series of structures (generally used for training) and convert their parameters into another set of parameters (generally used for inference), thereby converting the multi-branch model into a single-path model. In real scenarios, training resources are generally relatively abundant. This embodiment is more concerned with the cost and performance during inference. Therefore, this embodiment wants the structure during training to be larger and have good properties, such as higher accuracy or other properties. The converted inference structure is smaller in scale and retains these properties. The entire process is based on Figure 2 (c) is an example. The right side is the MBConv1 module after structural reparameterization. From shallow to deep, it includes: k×k convolution layer based on structural reparameterization, Swish activation layer, ECA attention mechanism layer and 1×1 convolution layer. Figure 2 (c) As shown on the left, the k×k convolutional layer based on structural reparameterization means that during training, the k×k convolutional layer is replaced with three or four branches, namely a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, and a 1×1 convolutional layer followed by a BatchNorm layer. Since the input and output sizes of the feature map in the MBConvB1 module remain unchanged, an additional BatchNorm layer branch is included. Conversely, if the sizes are different, this BatchNorm layer branch is not included. During inference, these branches are replaced with a k×k convolutional layer. Structural reparameterization can be expressed as follows:

[0165] Reparam(K*K)=BN(Conv(K,K))+BN(Conv(3,3))+BN(Conv(1,1))+BN

[0166] Reparam(K*K) represents the reparameterized convolution, Conv(K, K) represents the k×k convolution, and BN() represents the BatchNorm layer. In this embodiment, K=3.

[0167] In this embodiment, the MBConv1 module, the MBConv6 module, and the PConvB module are used as target modules. The structurally reparameterized k×k convolutional layer of the target module is expressed as follows: during training, the k×k convolutional layer is replaced with three or four branches, namely, a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, a 1×1 convolutional layer followed by a BatchNorm layer, and a BatchNorm layer. When the input and output sizes of the feature map remain unchanged at the target module, the branch includes the BatchNorm layer; when the input and output sizes of the feature map change at the target module, the branch does not include the BatchNorm layer. During inference, the branch is replaced with a k×k convolutional layer.

[0168] The model training module is used to train the PRNU matching model based on the sample data set to obtain a trained PRNU matching model.

[0169] As a specific implementation of the model training module, when performing model training, the model constructs a sample data set based on the preprocessed test PRNU, the preprocessed reference PRNU, and the correlation matrix of the test PRUN and the reference PRNU, and randomly divides the sample data set into three categories: training set (60% of the total), validation set (20%), and test set (20%).

[0170] The constructed PRNU matching model is trained, verified, and tested based on the training set, validation set, and test set. During the model training, validation, and testing process, cross entropy is used as the loss function. The cross entropy loss function L for binary classification is expressed as follows:

[0171]

[0172] Among them, y represents the true label, Represents the model's predicted probability for a given sample. The closer to the true label y, the smaller the L value, indicating that the classification error rate of the model is smaller. Conversely, the larger the L value, the greater the classification error rate of the model.

[0173] Based on the loss function, the network parameters of the PRNU matching model are learned using the Adam optimization algorithm and the back-propagation algorithm. During the network parameter learning process, the learning rate is set to 0.001, the batch size is set to 72, the early stopping training strategy is adopted, the patience value is set to 30, and the maximum epoch training value is set to 500. When the accuracy of correctly matching the noise in the validation set no longer improves within 30 epochs or the epoch reaches 500, the model training is stopped and the network parameters with the maximum accuracy of correctly matching the noise in the validation set are saved to obtain the final trained PRNU matching model.

[0174] For the image to be tested, the matching module is used to extract the test PRNU of the image to be tested and calculate the reference PRNU of the matching camera. Taking the test PRNU and reference PRNU as input, the trained PRNU matching model predicts and outputs the correlation score between the test PRNU and the reference PRNU. If the correlation score meets the threshold, it is determined that the test PRNU is correlated with the reference PRNU, and the matching camera is the source camera of the image to be tested.

[0175] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A PRNU matching method based on structural reparameterization, characterized in that: The steps include: Randomly select multiple cameras, take multiple images with each camera as test images, extract the PRNU of each test image as the test PRNU, and calculate the reference PRNU of each camera; After preprocessing the test PRNU and the reference PRNU, the correlation matrix between the test PRUN and the reference PRNU is calculated, and a sample data set is constructed based on the correlation matrix between the preprocessed test PRNU, the preprocessed reference PRNU, and the test PRUN and the reference PRNU; Construct a PRNU matching model, which is an improved EfficientNet model after structural reparameterization, including a convolutional layer module, an MBConv1 module, and a Stage structure. For each stage in the Stage structure, the first module in the stage is an MBConv6 module, and the other modules are PConvB modules. The convolutional layer module, the MBConv1 module, the MBConv6 module, and the PConvB module are all processed by structural reparameterization, and the ECA attention layer is introduced into the MBConv1 module and the MBConv6 module to replace the original SE attention layer. The PRNU matching model is used to take the reference PRNU and the test PRNU as input and predict the correlation score between the reference PRNU and the test PRNU; Training the PRNU matching model based on the sample data set to obtain a trained PRNU matching model; For the image to be tested, the test PRNU of the image to be tested is extracted, and the reference PRNU of the matching camera is calculated. The test PRNU and the reference PRNU are used as input, and the correlation score between the test PRNU and the reference PRNU is predicted and output by the trained PRNU matching model. If the correlation score meets the threshold, the test PRNU is determined to be correlated with the reference PRNU, and the matching camera is the source camera of the image to be tested. The MBConv1 module includes a reparameterized k×k convolutional layer, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer; The MBConv6 module includes a 1×1 convolutional layer, a k×k convolutional layer with structural reparameterization, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer; The PConvB module includes a k×k convolutional layer with reparameterized structure, a Swish activation layer, and a 1×1 convolutional layer; Taking the MBConv1 module, MBConv6 module, and PConvB module as the target modules, the structure of the reparameterized k×k convolutional layer of the target module is expressed as follows: during training, the k×k convolutional layer is replaced with three or four branches, namely, a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, a 1×1 convolutional layer followed by a BatchNorm layer, and a BatchNorm layer. When the feature map at the input and output sizes of the target module remain unchanged, the branch includes the BatchNorm layer, and when the feature map at the input and output sizes of the target module change, the branch does not include the BatchNorm layer; during inference, the branch is replaced with a k×k convolutional layer; The structural reparameterization is calculated by the following formula: Reparam(K*K)=BN(Conv(K,K))+BN(Conv(3,3))+BN(Conv(1,1))+BN Reparam(K*K) represents the reparameterized convolution, Conv(K,K) represents the k×k convolution, and BN() represents the BatchNorm layer.

2. The PRNU matching method based on structural reparameterization according to claim 1, characterized in that: Preprocess the test PRNU and reference PRNU, including: Clip the reference PRNU from the center to Q*Q pixels to obtain the clipped reference PRNU; Clip the test PRNU from the center to within P*P pixels to obtain the clipped test PRNU where P is less than Q; Normalize the trimmed test PRNU and the trimmed reference PRNU respectively to obtain the normalized test PRNU and the normalized reference PRNU; Performing a padding operation on the normalized test PRNU, so that the padded test PRNU has the same size as the reference PRNU; Among them, the trimmed test PRNU and the trimmed reference PRNU are the noise map to be processed I, whose size is M×N, and the normalized noise map is 3. The PRNU matching method based on structural reparameterization according to claim 1, characterized in that: The correlation matrix ρ of the test PRUN and reference PRNU is calculated as follows: in, and Represents W and K respectively d represents the mean of , ||-|| represents the two-norm operation, · represents the dot multiplication operation of the matrix, and the size of the correlation matrix ρ is the same as that of the reference PRNU.

4. The PRNU matching method based on structural reparameterization according to claim 1, characterized in that: For the input feature map, the PConvB module is used to extract features from the first or last continuous channel in the feature map through convolution operation, while the other channels remain unchanged; For the output of the PConv module, the feature maps of each channel are integrated through two point-by-point convolution PWConvs, and a standardized convolution layer and a ReLu activation layer are set between the two point-by-point convolution PWConvs.

5. The PRNU matching method based on structural reparameterization according to claim 1, characterized in that: The ECA attention layer is used to perform the following: Perform a global average pooling operation on the input feature map, change the h and w dimensions in the feature map to 1, and only retain the c dimension to obtain a 1×1×c feature vector; A one-dimensional convolution operation with a convolution kernel size of k is used to obtain a feature map of the same size as the feature vector of 1×1×c. The feature map is then normalized and the Sigmoid activation function is applied to obtain the weights of each channel. Multiply the weights by the corresponding elements of the input feature map to obtain the final output feature map.

6. The PRNU matching method based on structural reparameterization according to claim 1, characterized in that: When training the PRNU matching model based on the sample data set, the sample data set is divided into a training set, a validation set and a test set; The constructed PRNU matching model is trained, verified, and tested based on the training set, validation set, and test set. During the model training, validation, and testing process, cross entropy is used as the loss function. The cross entropy loss function L for binary classification is expressed as follows: Among them, y represents the true label, Represents the model's predicted probability for a given sample. The closer to the true label y, the smaller the L value, indicating that the classification error rate of the model is smaller. Conversely, the larger the L value, the greater the classification error rate of the model. Based on the loss function, the network parameters of the PRNU matching model are learned using the Adam optimization algorithm and the back-propagation algorithm. During the network parameter learning process, the learning rate and batch size are set, an early stopping training strategy is adopted, and a patience value A and an epoch maximum training value B are set. When the accuracy of correctly matching the noise in the validation set no longer increases within A epochs or reaches B in epochs, model training is stopped and the network parameters that achieve the maximum accuracy of correctly matching the noise in the validation set are saved to obtain the final trained PRNU matching model.

7. A PRNU matching system based on structural reparameterization, characterized in that: The system is configured to perform PRNU matching by using a PRNU matching method based on structural reparameterization according to any one of claims 1 to 6, so as to perform camera traceability on an image to be tested, the system comprising: An image acquisition module, wherein the image acquisition module is used to randomly select multiple cameras, capture multiple images as test images through each camera, extract the PRNU of each test image as a test PRNU, and calculate a reference PRNU for each camera; a sample processing module configured to preprocess the test PRNU and the reference PRNU, calculate a correlation matrix between the test PRNU and the reference PRNU, and construct a sample data set based on the preprocessed test PRNU, the preprocessed reference PRNU, and the correlation matrix between the test PRUN and the reference PRNU; A model construction module, which is used to construct a PRNU matching model. The PRNU model is an improved EfficientNet model after structural reparameterization, including a convolutional layer module, an MBConv1 module, and a Stage structure. For each stage in the Stage structure, the first module in the stage is an MBConv6 module, and the other modules are PConvB modules. The convolutional layer module, the MBConv1 module, the MBConv6 module, and the PConvB module are all processed by structural reparameterization, and the ECA attention layer is introduced into the MBConv1 module and the MBConv6 module to replace the original SE attention layer. The PRNU matching model is used to take the reference PRNU and the test PRNU as input and predict the correlation score between the output reference PRNU and the test PRNU; A model training module, wherein the model training module is used to train the PRNU matching model based on a sample data set to obtain a trained PRNU matching model; The matching module is used to extract the test PRNU of the image to be tested and calculate the reference PRNU of the matching camera. The matching module takes the test PRNU and the reference PRNU as input and predicts the correlation score between the test PRNU and the reference PRNU through the trained PRNU matching model. If the correlation score meets the threshold, it is determined that the test PRNU is correlated with the reference PRNU, and the matching camera is the source camera of the image to be tested.

8. The PRNU matching system based on structural reparameterization according to claim 7, characterized in that: The sample processing module is used to perform the following preprocessing operations on the test PRNU and the reference PRNU: Clip the reference PRNU from the center to Q*Q pixels to obtain the clipped reference PRNU; Clip the test PRNU from the center to within P*P pixels to obtain the clipped test PRNU where P is less than Q; Normalize the trimmed test PRNU and the trimmed reference PRNU respectively to obtain the normalized test PRNU and the normalized reference PRNU; Performing a padding operation on the normalized test PRNU, so that the padded test PRNU has the same size as the reference PRNU; Among them, the trimmed test PRNU and the trimmed reference PRNU are the noise map to be processed I, whose size is M×N, and the normalized noise map is The correlation matrix ρ of the test PRUN and reference PRNU is calculated as follows: in, and Represents W and K respectively d The mean of , ||-|| represents the two-norm operation, · represents the matrix point multiplication operation, and the size of the correlation matrix ρ is the same as that of the reference PRNU; When the PRNU matching model is trained based on a sample data set, the model training module is used to perform the following: Divide the sample data set into training set, validation set and test set; The constructed PRNU matching model is trained, verified, and tested based on the training set, validation set, and test set. During the model training, validation, and testing process, cross entropy is used as the loss function. The cross entropy loss function for binary classification is expressed as follows: Among them, y represents the true label, Represents the model's predicted probability for a given sample. The closer to the true label y, the smaller the L value, indicating that the classification error rate of the model is smaller. Conversely, the larger the L value, the greater the classification error rate of the model. Based on the loss function, the network parameters of the PRNU matching model are learned using the Adam optimization algorithm and the back-propagation algorithm. During the network parameter learning process, the learning rate and batch size are set, an early stopping training strategy is adopted, and a patience value A and an epoch maximum training value B are set. When the accuracy of correctly matching the noise in the validation set no longer increases within A epochs or reaches B in epochs, model training is stopped and the network parameters that achieve the maximum accuracy of correctly matching the noise in the validation set are saved to obtain the final trained PRNU matching model.

9. The PRNU matching system based on structural reparameterization according to claim 7, characterized in that: The MBConv1 module includes a reparameterized k×k convolutional layer, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer; The MBConv6 module includes a 1×1 convolutional layer, a k×k convolutional layer with structural reparameterization, a Swish activation layer, an ECA attention mechanism layer, and a 1×1 convolutional layer; The PConvB module includes a k×k convolutional layer with reparameterized structure, a Swish activation layer, and a 1×1 convolutional layer; Taking the MBConv1 module, MBConv6 module, and PConvB module as the target modules, the structure of the reparameterized k×k convolutional layer of the target module is expressed as follows: during training, the k×k convolutional layer is replaced with three or four branches, namely, a k×k convolutional layer followed by a BatchNorm layer, a 3×3 convolutional layer followed by a BatchNorm layer, a 1×1 convolutional layer followed by a BatchNorm layer, and a BatchNorm layer. When the feature map at the input and output sizes of the target module remain unchanged, the branch includes the BatchNorm layer, and when the feature map at the input and output sizes of the target module change, the branch does not include the BatchNorm layer; during inference, the branch is replaced with a k×k convolutional layer; The structural reparameterization is calculated by the following formula: Reparam(K*K)=BN(Conv(K,K))+BN(Conv(3,3))+BN(Conv(1,1))+BN Where Reparam(K*K) represents the reparameterized convolution, Conv(K,K) represents the k×k convolution, and BN() represents the BatchNorm layer; For the input feature map, the PConvB module is used to extract features from the first or last continuous channel in the feature map through convolution operation, while the other channels remain unchanged; For the output of the PConv module, the feature maps of each channel are integrated through two point-by-point convolution PWConvs, and a standardized convolution layer and a ReLu activation layer are set between the two point-by-point convolution PWConvs; The ECA attention layer is used to perform the following: Perform a global average pooling operation on the input feature map, change the h and w dimensions in the feature map to 1, and only retain the c dimension to obtain a 1×1×c feature vector; A one-dimensional convolution operation with a convolution kernel size of k is used to obtain a feature map of the same size as the feature vector of 1×1×c. The feature map is then normalized and the Sigmoid activation function is applied to obtain the weights of each channel. Multiply the weights by the corresponding elements of the input feature map to obtain the final output feature map.

Images