Image noise reduction model compression method based on structure re-parameterization

Abstract

The invention provides an image noise reduction model compression method based on structure re-parameterization, and the method comprises the steps: S1, constructing a dark light image data set, and carrying out the image preprocessing; s2, loading to a multi-branch noise reduction network; s3, training the model by adopting an MAE loss function and an Adam optimizer; s4, performing structure re-parameterization on the trained multi-branch model; and S5, loading the complete image into the model after the re-parameterization. According to the method, through a structure re-parameterization technology, a large model can be constructed for training, and the large model is converted into a small model suitable for edge equipment and an embedded system during reasoning, so that resource limitation and real-time reasoning requirements are met.

Description

Technical Field

[0001] The invention belongs to the technical field of image denoising, and in particular relates to an image denoising model compression method based on structural re-parameterization. Background Art

[0002] In recent years, with the rapid development of computer vision, the applications derived from it have become more and more extensive. In computer vision tasks such as image analysis and understanding, noise is very likely to affect the accuracy of visual algorithms, resulting in deviations in the judgment and understanding of image content. By performing noise reduction processing on images or videos, the image quality can be significantly improved and the interference of noise on visual algorithms can be reduced. For example, in the field of photography, image noise reduction algorithms can be used to reduce noise in photos taken under low-light conditions, improve image quality and detail visibility, and image enhancement technology can also use noise reduction algorithms to improve image clarity and quality. In the field of medical imaging, such as MRI, CT, and ultrasound, image noise reduction algorithms can reduce the interference of noise on medical image quality and diagnosis, which helps improve doctors' accurate interpretation of images and provide more reliable medical diagnoses. In video processing, image noise reduction algorithms can be applied to each frame of a video sequence to reduce noise and artifacts in the video. With the development of technology and the growing demand for applications, image noise reduction algorithms will play a role in more fields, providing clear and accurate image data to support various applications.

[0003] At present, most mainstream image denoising algorithms are based on deep learning technology, or combine traditional denoising algorithms with deep learning technology. Since most of the application scenarios of image denoising are edge devices such as mobile phones and surveillance cameras, and the existing denoising models are large in scale, have many parameters and large in computation, they require high computing power for edge devices. Through model compression technology, large models can be converted into small models suitable for edge devices and embedded systems to meet resource constraints and real-time reasoning requirements.

[0004] Classic image denoising algorithms, such as mean filtering, median filtering and Gaussian filtering, have limited effects on complex noise and detail restoration. Due to the powerful fitting ability of neural networks, significant progress has been made in the field of image denoising. By building a deep neural network model, the mapping relationship between noisy images and noise-free images can be learned to achieve high-quality image denoising effects. Some commonly used deep learning-based image denoising models include autoencoders, convolutional neural networks and generative adversarial networks. The above models can achieve high-quality image denoising effects while expanding the network scale. However, due to the large amount of computation, they cannot be deployed to edge devices and embedded systems with limited computing resources, resulting in limited applications. Model compression technology is usually used to reduce the size of the model, making the model easier to deploy.

[0005] Common model compression techniques include network pruning, knowledge distillation, low-rank decomposition, and quantization. Network pruning compresses the model by iterative pruning and fine-tuning, and reduces the redundancy and complexity of the model by identifying and removing neurons, channels, or layers that contribute less to the model. Knowledge distillation is a method of learning the knowledge of a complex model by training a smaller and simpler model. By passing information such as the predicted distribution, intermediate features, or soft labels of a complex model as an auxiliary target, the complexity of the model is reduced. Low-rank decomposition is a method of reducing the number of parameters and computation of the model by approximately decomposing the weight matrix of the model into a low-rank matrix. Common low-rank decomposition methods include singular value decomposition (SVD) and eigenvalue decomposition. Quantization is a method of reducing the size and computation of the model by reducing the number of bits of parameters and activation values ​​in the model. Common quantization methods include fixed-point quantization and floating-point quantization, which can represent parameters and activation values ​​as fixed-point or floating-point numbers with lower bits, thereby reducing memory usage and computation requirements.

[0006] The neural network-based image denoising model can achieve high-quality denoising effects, but it has the characteristics of large number of parameters and large amount of computation. It cannot be deployed on edge devices and embedded systems with limited computing resources, which limits its application.

[0007] In addition, the commonly used terms in the prior art include:

[0008] Structural reparameterization: First, a series of structures are constructed (generally for training), and their parameters are equivalently converted into another set of parameters (generally for reasoning), thereby converting this series of structures into another series of structures.

[0009] Model compression: Reduce the number of parameters and computational complexity of deep learning models such as neural networks without losing much model accuracy, in order to save memory and increase reasoning speed.

[0010] Black frame: An image captured by a camera in a completely dark environment, containing only noise that is unrelated to the signal.

[0011] Flat-field frame: An image captured by a camera in front of a white paper or wall with uniform lighting.

[0012] RAW domain image: A picture with original information taken and recorded by the camera, without compression or any processing.

[0013] Bayer format: The raw data format obtained using the Bayer array sensor, the general format of which is: odd-numbered scanning line output RGRG...even-numbered scanning line output GBGB...

[0014] U-net: U-net was published in 2015 and is a variant of FCN. Its original intention was to solve problems in biomedical images. Due to its significant effect, it has been widely used in various directions of semantic segmentation.

[0015] Black level correction: Due to the influence of dark current, the actual raw data output by the sensor in a dark environment is not all 0. The process of subtracting the reference dark current signal from the obtained image signal is called black level correction.

[0016] Data enhancement: Data enhancement is an effective method to expand the size of data samples. Image data enhancement methods usually include spatial geometric transformation, pixel color transformation, and multi-sample synthesis. Summary of the invention

[0017] In order to solve the above problems, the purpose of this application is to: Large image denoising models require more storage space and bandwidth for storage and transmission, and usually require huge computing resources for training and reasoning. Deployment on edge devices and embedded systems faces resource constraints, such as limited memory, computing power, and power consumption. Through structural reparameterization technology, large models can be built for training, and converted into small models suitable for edge devices and embedded systems during reasoning to meet resource constraints and real-time reasoning requirements.

[0018] Specifically, the present invention provides an image denoising model compression method based on structural reparameterization, the method comprising the following steps:

[0019] S1. Construct a dark light image dataset and perform image preprocessing, that is, collect high-quality reference images, signal-free black frames, and flat field frames respectively, calibrate noise distribution parameters according to black frames and flat field frames, model different noise sources, and then synthesize the modeled noise with high-quality reference images to obtain simulated noisy images, obtain a "noisy image + reference image" paired dataset, and perform black level correction and channel conversion preprocessing on it;

[0020] S2. First, a U-net-type basic network is designed. Then, a diversified branch module with re-parameterizable structure is designed. By replacing the standard residual module in the basic network with a diversified branch module, a large-scale multi-branch network is obtained for training. The noisy image is loaded into the multi-branch denoising network, and the corresponding noise map is output.

[0021] S3. Use MAE loss function and Adam optimizer to train the model, that is, train a multi-branch network to train the noisy image X after preprocessing and data enhancement input Input into the multi-branch network using the diversified branch module, and the predicted noise map is obtained through forward propagation The MAE function is used to calculate the loss of the predicted image and the reference image, and the network parameters are updated through the Adam optimizer;

[0022] S4. Performing structural reparameterization on the trained multi-branch model, that is, after completing the training of the multi-branch network, converting the diversified branch modules in the network into single-branch modules for reasoning through structural reparameterization;

[0023] S5. Load the complete image into the reparameterized model and output the denoised image, that is, load the preprocessed, uncropped noisy image into the structurally reparameterized network for denoising to obtain the denoised image.

[0024] The step S1 further comprises:

[0025] S1.1 Image acquisition: Collect high signal-to-noise ratio images as clean background images in indoor and outdoor scenes, and control their brightness levels by dividing by different coefficients to simulate dark light environments. In addition, it is necessary to collect black frames in a dark environment and collect flat field frames in front of a white wall or paper with uniform lighting.

[0026] S1.2 Noise calibration and data simulation; In a dark light environment, the noise sources are divided into signal-related noise and signal-independent noise. Signal-related noise is photon shot noise, and signal-independent noise includes readout noise and row noise. Photon shot noise follows Poisson distribution, and the system gain can be calibrated through flat field frames to calculate Poisson distribution. Readout noise follows Tukey-lambda distribution, and row noise follows Gaussian distribution. The parameters required for the two types of distributions can be calibrated through black frames. After obtaining the parameters of different noise source distributions, the noise under different system gains can be modeled, and the modeled noise can be superimposed on a clean background to obtain a simulated image with noise.

[0027] S1.3 Data preprocessing, black level correction and normalization are performed on the RAW image in Bayer format, the single-channel output image is converted into a 4-channel image, and the image spatial resolution is reduced to 1 / 4 of the original; S1.4 Data enhancement, random cropping of the image after data preprocessing, random rotation and random flipping of the cropped image blocks, and the enhanced image blocks are used as the input of the network.

[0028] The step S1.3 further comprises:

[0029] The black level correction and normalization of the Bayer format RAW image is shown in the following formula:

[0030] In formula (1), bit is the number of bits of the RAW image. After black level correction and normalization, the single-channel output image is converted to a 4-channel image, and the image spatial resolution is reduced to 1 / 4 of the original one.

[0031] The step S2 further comprises:

[0032] S2.1, design a basic denoising network in the form of U-Net. Design a lightweight network as the basic denoising model. The model consists of input convolution, encoder, decoder, output convolution and cross-layer connection. The input of the input convolution is a 4-channel noise image. The encoder consists of three consecutive "downsampling + standard residual modules". The decoder consists of three "upsampling + standard residual modules". The output of the output convolution is a 4-channel noise map. The cross-layer connection is used to integrate the detailed information extracted at each stage of downsampling in the upsampling stage.

[0033] S2.2, design diversified branch modules that can perform structural reparameterization to upgrade the basic network to a multi-branch network with stronger performance for training;

[0034] In the encoder part, the first convolution in the standard residual module a is changed to a multi-branch structure with an output channel of C and a 1×1 convolution with an output channel of 2C, where the multi-branch structure consists of a 1×1 convolution, two 3×3 convolutions, two consecutive 1×1 convolutions and 3×3 convolutions, and consecutive 1×1 convolutions and average pooling AVG; the second convolution in the standard residual module a is changed to channel splitting, a multi-branch structure with an output channel of C, channel merging, and channel shuffling, thereby obtaining a diversified branch module a;

[0035] In the decoder part, the first convolution in the standard residual module b is changed to a 1×1 convolution with an output channel of C and a multi-branch structure with an output channel of C; the second convolution in the standard residual module b is changed to channel splitting, a multi-branch structure with an output channel of C / 2, channel merging and channel shuffling, and the final output channel is C, thereby obtaining a diversified branch module b.

[0036] In the step S2.2,

[0037] In the encoder part, for the standard residual module a, assuming that the input channel is C and the output channel is 2C, its parameter quantity is:

[0038] 2C×3×3×C+2C×3×3×2C+2C×1×1×C=56C 2 ;

[0039] The second convolution in the standard residual module a is changed to a multi-branch structure with channel segmentation, output channels C, channel merging and channel shuffling, thereby obtaining a diversified branch module a, whose parameter quantity is:

[0040] (C×C×3×3×4+C×C×1×1×4)×2+2C×C×1×1=82C 2 ;

[0041] In the decoder part, for the standard residual module b, assuming that the input channel is 2C and the output channel is C, the parameter amount is:

[0042] 2C×3×3×C+C×3×3×C+C×1×1×2C=29C 2 ;

[0043] The second convolution in the standard residual module b is changed to a multi-branch structure with channel segmentation, an output channel of C / 2, channel merging and channel shuffling, and the final output channel is C, thereby obtaining a diversified branch module b, whose parameter quantity is:

[0044] C×C×3×3×4+C×C×1×1×4+C×2C×1×1+C×C×3×3+C×C×1×1=52C 2 .

[0045] The step S3 further comprises:

[0046] The MAE function is used to calculate the loss of the predicted image and the reference image, as shown in the following formula:

[0047]

[0048] The Adam optimizer is used to update the network parameters. The cosine annealing learning rate reduction strategy is used during the training process, and the half cycle of the cosine function is set as the maximum number of iterations.

[0049] In the step S4, for the diversified branch module a, the converted parameter amount is, compared with the standard residual module a, the compression ratio is 64.3%; for the diversified branch module b, the converted parameter amount is, and the compression ratio is 55.2%; since the resolution of the feature map within the module has not changed, the compression ratio of the parameter amount is equivalent to the compression ratio of the calculation amount.

[0050] The step S4 further comprises:

[0051] (1), 1×1 convolution is converted to 3×3 convolution: pad the 1×1 convolution kernel with 0 elements around it, i.e., Pad operation, to convert it to 3×3 convolution kernel, and perform Replicate operation with offset, as shown in formula (3) and formula (4), where K 1×1∈R D ×C×1×1 , K′ 1×1 ∈R D×C×3×3 , b 1×1 ∈R D×1×1 , b′ 1×1 ∈R D×H×W , C and D represent the number of input and output channels respectively, H and W represent the width and height of the output feature map;

[0052] K′ 1×1 =Pad(K 1×1 ,0) Formula (3);

[0053] b′ 1×1 =Replicate(b 1×1 ) formula (4);

[0054] (2) The continuous 1×1 convolution and 3×3 convolution are converted to 3×3 convolution: First, the zeroth dimension and the first dimension of the 1×1 convolution are swapped by the transposition operation, and then the 3×3 convolution kernel is used to perform a convolution operation on the transposed result to obtain the converted 3×3 convolution. The conversion process is shown in formula (5) and formula (6); where K 1×1 ∈R D×C×1×1 , K 3×3 ∈R D×D×3×3 , K 1×1→3×3 ∈R D×C×3×3 , b 1×1→3×3 ∈R D×H×W ;

[0055] K 1×1→3×3 =K 3×3 *Transpose(K 1×1 ) formula (5);

[0056] b 1×1→3×3 =Replicate(b 1×1 )*K 3×3 +Replicate(b 3×3 ) formula (6);

[0057] (3), the continuous 1×1 convolution and AVG are converted to 3×3 convolution: the average pooling layer AVG can be equivalent to a fixed weight convolution. When converting AVG to a 3×3 convolution, a fixed weight of 1 / 9 is set for the current input channel, and the others are set to 0. Then, according to (2), the continuous 1×1 convolution and 3×3 convolution are converted to 3×3 convolution. The conversion process is shown in formulas (7), (8), and (9); where K 1×1 ∈R D×C×1×1 , K avg ∈R D×D×3×3 , K 1×1→avg ∈RD×C×3×3 , b 1×1→avg ∈R D×H×W ;

[0058]

[0059] K 1×1→avg =K avg *Transpose(K 1×1 ) formula (8);

[0060] b 1×1→avg =Replicate(b 1×1 )*K 1×1→avg Formula (9);

[0061] (4), the identity mapping is converted to a 3×3 convolution: The identity mapping is equivalent to a 1×1 convolution with the unit matrix as the convolution kernel, and then converted to a 3×3 convolution by padding 0 around it, as shown in formula (10) and formula (11); where K identity ∈R D ×C×3×3 , b identity ∈R D×H×W ;

[0062]

[0063] b identity =Replicate(0) formula (11);

[0064] (5) After each branch is converted to a 3×3 convolution, the weights and biases of all branches are fused to obtain K fuse and b fuse , as shown in formula (12) and formula (13), multiple branches can be converted into a single 3×3 convolution; where K fuse ∈R D×C×3×3 , b fuse ∈R D×H×W ;

[0065] K fuse =K′ 1×1 +K 1×1→3×3 +K 1×1→avg +K identity Formula (12);

[0066] b fuse = b′ 1×1 +b 1×1→3×3 +b 1×1→avg +b identity Formula (13).

[0067] Therefore, the advantages of this application are:

[0068] (1) A high-performance multi-branch model is used during training, and converted into a lightweight single-channel model through structural reparameterization during inference. This makes it easier to deploy on edge devices and embedded systems with limited computing resources to meet real-time inference requirements.

[0069] (2) After the structural reparameterization, the single-path model not only greatly reduces the amount of computation, but also does not have operations with high memory access costs such as group convolution and element-by-element addition in the network, making it more efficient during inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0071] Figure 1 This is a schematic diagram of the application process.

[0072] Figure 2 It is a schematic diagram of the network structure used for training in step S2 of this application.

[0073] Figure 3 In this application, the second convolution in the standard residual module a is changed to a multi-branch structure with channel splitting, output channels of C, channel merging and channel shuffling, thereby obtaining a schematic diagram of a diversified branch module a.

[0074] Figure 4 This application changes the second convolution in the standard residual module b into a multi-branch structure with channel splitting, an output channel of C / 2, channel merging and channel shuffling, and the final output channel is C, thereby obtaining a schematic diagram of the diversified branch module b. DETAILED DESCRIPTION

[0075] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0076] like Figure 1 As shown, this application proposes an image denoising model compression method based on structural reparameterization. The overall process is as follows Figure 1 As shown, the following steps are included:

[0077] Step S1. Construct a dark light image data set and perform image preprocessing, that is, collect high-quality reference images, signal-free black frames, and flat field frames respectively, calibrate noise distribution parameters according to black frames and flat field frames, model different noise sources, and then synthesize the modeled noise with the high-quality reference image to obtain a simulated noisy image, obtain a "noisy image + reference image" paired data set, and perform black level correction and channel conversion preprocessing on it, which specifically includes the following steps:

[0078] Step S1.1 Image acquisition. Collect high signal-to-noise ratio images in indoor and outdoor scenes as clean background images, and control their brightness levels by dividing by different coefficients to simulate a dark light environment. In addition, it is necessary to collect black frames in a dark environment and collect flat field frames in front of a white wall or paper with uniform lighting.

[0079] Step S1.2 Noise calibration and data simulation. In a dark light environment, the noise sources are divided into signal-related noise and signal-independent noise. Signal-related noise is photon shot noise, and signal-independent noise includes readout noise and row noise. Related studies have shown that photon shot noise obeys Poisson distribution, and the system gain can be calibrated through flat field frames to calculate Poisson distribution; readout noise obeys Tukey-lambda distribution, and row noise obeys Gaussian distribution. The parameters required for the two types of distributions can be calibrated through black frames. After obtaining the parameters of different noise source distributions, the noise under different system gains can be modeled, and the modeled noise can be superimposed on a clean background to obtain a simulated image with noise.

[0080] Step S1.3 Data preprocessing: Perform black level correction and normalization on the Bayer format RAW image, as shown in the following formula:

[0081]

[0082] In formula (1), bit is the number of bits of the RAW image. After black level correction and normalization, the single-channel output image is converted to a 4-channel image, and the image spatial resolution is reduced to 1 / 4 of the original. On the one hand, reducing the resolution can reduce the computational overhead of the network, and on the other hand, it can avoid destroying the Bayer format when performing data augmentation operations.

[0083] Step S1.4 Data enhancement. Randomly crop the image after data preprocessing, and randomly rotate and flip the cropped image block, and use the enhanced image block as the input of the network. Step S2. First, design a basic network in the form of U-net, and then design a diversified branch module that can be re-parameterized. By replacing the standard residual module in the basic network with a diversified branch module, a larger multi-branch network is obtained for training; the noisy image is loaded into the multi-branch denoising network, and the corresponding noise map is output. The network structure used for training is as follows: Figure 2 As shown, the specific steps are as follows:

[0084] Step S2.1: Design a basic denoising network in the form of U-Net. The U-Net network is the most classic network structure in the "image to image" task. Referring to the U-Net structure, a lightweight network is designed as the basic denoising model. The model mainly consists of input convolution, encoder, decoder, output convolution and cross-layer connection. The input of the input convolution is a 4-channel noise image, the encoder consists of three consecutive "downsampling + standard residual modules", the decoder consists of three "upsampling + standard residual modules", the output of the output convolution is a 4-channel noise map, and the cross-layer connection is used to integrate the detail information extracted at each stage of downsampling in the upsampling stage, which is conducive to the decoder to restore high-resolution information to the greatest extent;

[0085] Step S2.2 designs a diversified branch module that can perform structural reparameterization, and upgrades the basic network to a multi-branch network with stronger performance for training. The specific structure of the diversified branch module is as follows: Figure 3 As shown. In the encoder part, for the standard residual module a, assuming that the input channel is C and the output channel is 2C, its parameter quantity is:

[0086] 2C×3×3×C+2C×3×3×2C+2C×1×1×C=56C

[0087] The first convolution in the standard residual module a is changed to a multi-branch structure with an output channel of C and a 1×1 convolution with an output channel of 2C, where the multi-branch structure consists of a 1×1 convolution, two 3×3 convolutions, two consecutive 1×1 convolutions and 3×3 convolutions, and consecutive 1×1 convolutions and average pooling (AVG); the second convolution in the standard residual module a is changed to channel splitting, a multi-branch structure with an output channel of C, channel merging, and channel shuffling, thereby obtaining a diversified branch module a, as shown in FIG. Figure 3 As shown, its parameters are:

[0088] (C×C×3×3×4+C×C×1×1×4)×2+2C×C×1×1=82C 2

[0089] In the decoder part, for the standard residual module b, assuming that the input channel is 2C and the output channel is C, the parameter amount is:

[0090] 2C×3×3×C+C×3×3×C+C×1×1×2C=29C 2

[0091] The first convolution in the standard residual module b is changed to a 1×1 convolution with an output channel of C and a multi-branch structure with an output channel of C; the second convolution in the standard residual module b is changed to channel splitting, a multi-branch structure with an output channel of C / 2, channel merging and channel shuffling, and the final output channel is C, thereby obtaining a diversified branch module b, such as Figure 4 As shown, its parameters are:

[0092] C×C×3×3×4+C×C×1×1×4+C×2C×1×1+C×C×3×3+C×C×1×1=52C 2

[0093] Compared with the standard residual module, the diversified branch module has more parameters, and there are differences between the branches, with different feature extraction modes, and has more powerful feature extraction capabilities;

[0094] Step S3. Use the MAE loss function and Adam optimizer to train the model, that is, train a multi-branch network. input Input into the multi-branch network using the diversified branch module, and the predicted noise map is obtained through forward propagation The MAE function is used to calculate the loss of the predicted image and the reference image, as shown in the following formula:

[0095]

[0096] The Adam optimizer is used to update the network parameters. The cosine annealing learning rate reduction strategy is used during the training process, and the half cycle of the cosine function is set as the maximum number of iterations.

[0097] Step S4. Perform structural reparameterization on the trained multi-branch model, that is, perform structural reparameterization on the trained multi-branch network. After the training of the multi-branch network is completed, the diversified branch modules in the network are converted into single-branch modules through structural reparameterization. Its structure is as follows: Figure 3 and Figure 4 As shown. For the diversified branch module a, the converted parameter amount is 20C 2 , compared with the standard residual module a, the compression ratio is 64.3%; for the diversified branch module b, the number of parameters after conversion is 13C 2 , the compression ratio is 55.2%. Since the resolution of the feature map in the module does not change, the compression ratio of the parameter amount is equivalent to the compression ratio of the calculation amount;

[0098] The specific steps for structural reparameterization of multi-branch structures are as follows:

[0099] (1) 1×1 convolution is converted to 3×3 convolution: Pad the 1×1 convolution kernel to convert it to a 3×3 convolution kernel, and perform a replicate operation with the offset, as shown in formulas (3, 4), where K 1×1 ∈R D×C×1×1 , K′ 1×1 ∈R D×C×3×3 , b 1×1 ∈R D×1×1 , b′ 1×1 ∈R D×H×W , C and D represent the number of input and output channels respectively, H and W represent the width and height of the output feature map;

[0100] K′ 1×1 =Pad(K 1×1 ,0) Formula (3);

[0101] b′ 1×1 =Replicate(b 1×1 ) formula (4);

[0102] (2) Continuous 1×1 convolution and 3×3 convolution are converted to 3×3 convolution: First, the zeroth dimension and the first dimension of the 1×1 convolution are swapped by transposing the operation, and then the 3×3 convolution kernel is used to perform a convolution operation on the transposed result to obtain the converted 3×3 convolution. The conversion process is shown in formulas (5, 6); where K 1×1 ∈R D×C×1×1 , K 3×3 ∈R D×D×3×3 , K 1×1→3×3 ∈R D×C×3×3 , b 1×1→3×3 ∈R D×H×W ;

[0103] K 1×1→3×3 =K 3×3 *Transpose(K 1×1 ) formula (5);

[0104] b 1×1→3×3 =Replicate(b 1×1 )*K 3×3 +Replicate(b 3×3 ) formula (6);

[0105] (3) Continuous 1×1 convolution and AVG are converted to 3×3 convolution: The average pooling layer AVG can be equivalent to a fixed-weight convolution. When converting AVG to a 3×3 convolution, a fixed weight of 1 / 9 is set for the current input channel, and the others are set to 0. Then, according to (2), the continuous 1×1 convolution and 3×3 convolution are converted to 3×3 convolution. The conversion process is shown in formulas (7, 8, and 9); where K1×1 ∈R D×C×1×1 , K avg ∈R D×D×3×3 , K 1×1→avg ∈R D×C×3×3 , b 1×1→avg ∈R D×H×W ;

[0106]

[0107] K 1×1→avg =K avg *Transpose(K 1×1 ) formula (8);

[0108] b 1×1→avg =Replicate(b 1×1 )*K 1×1→avg Formula (9);

[0109] (4) Identity mapping is converted to 3×3 convolution: The identity mapping is equivalent to a 1×1 convolution with the unit matrix as the convolution kernel, which is then converted to a 3×3 convolution by padding 0s around it, as shown in formulas (10, 11); where K identity ∈R D×C×3×3 , b identity ∈R D×H×W ;

[0110]

[0111] b identity =Replicate(0) formula (11);

[0112] (5) After each branch is converted to a 3×3 convolution, the weights and biases of all branches are fused to obtain K fuse and b fuse , as shown in formulas (12, 13), multiple branches can be converted into a single 3×3 convolution; where K fuse ∈R D×C×3×3 , b fuse ∈R D ×H×W .

[0113] K fuse =K′ 1×1 +K 1×1→3×3 +K 1×1→avg +K identity Formula (12);

[0114] b fuse = b′ 1×1 +b 1×1→3×3 +b 1×1→avg +b identityFormula (13).

[0115] Step S5. Load the complete image into the reparameterized model, load the preprocessed, uncropped noisy image into the structurally reparameterized network for denoising, and obtain the denoised image.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for compressing an image denoising model based on structural reparameterization, characterized in that: The method comprises the following steps: S1. Construct a dark light image dataset and perform image preprocessing, that is, collect high-quality reference images, signal-free black frames, and flat field frames respectively, calibrate noise distribution parameters according to black frames and flat field frames, model different noise sources, and then synthesize the modeled noise with high-quality reference images to obtain simulated noisy images, obtain a "noisy image + reference image" paired dataset, and perform black level correction and channel conversion preprocessing on it; S2. First, a U-net-type basic network is designed. Then, a diversified branch module with re-parameterizable structure is designed. By replacing the standard residual module in the basic network with a diversified branch module, a large-scale multi-branch network is obtained for training. The noisy image is loaded into the multi-branch denoising network, and the corresponding noise map is output. S3. Use MAE loss function and Adam optimizer to train the model, that is, train a multi-branch network to train the noisy image X after preprocessing and data enhancement input Input into the multi-branch network using the diversified branch module, and the predicted noise map is obtained through forward propagation The MAE function is used to calculate the loss of the predicted image and the reference image, and the network parameters are updated through the Adam optimizer; S4. Performing structural reparameterization on the trained multi-branch model, that is, after completing the training of the multi-branch network, converting the diversified branch modules in the network into single-branch modules for reasoning through structural reparameterization; S5. Load the complete image into the reparameterized model and output the denoised image, that is, load the preprocessed, uncropped noisy image into the structurally reparameterized network for denoising to obtain the denoised image.

2. The image denoising model compression method based on structural reparameterization according to claim 1, characterized in that: The step S1 further comprises: S1.1 Image acquisition: Collect high signal-to-noise ratio images as clean background images in indoor and outdoor scenes, and control their brightness levels by dividing by different coefficients to simulate dark light environments. In addition, it is necessary to collect black frames in a dark environment and collect flat field frames in front of a white wall or paper with uniform lighting. S1.2 Noise calibration and data simulation; In a dark light environment, the noise sources are divided into signal-related noise and signal-independent noise. Signal-related noise is photon shot noise, and signal-independent noise includes readout noise and row noise. Photon shot noise follows Poisson distribution, and the system gain can be calibrated through flat field frames to calculate Poisson distribution. Readout noise follows Tukey-lambda distribution, and row noise follows Gaussian distribution. The parameters required for the two types of distributions can be calibrated through black frames. After obtaining the parameters of different noise source distributions, the noise under different system gains can be modeled, and the modeled noise can be superimposed on a clean background to obtain a simulated image with noise. S1.3 Data preprocessing, black level correction and normalization are performed on the RAW image in Bayer format, the single-channel output image is converted into a 4-channel image, and the image spatial resolution is reduced to 1 / 4 of the original; S1.4 Data enhancement, random cropping of the image after data preprocessing, random rotation and random flipping of the cropped image blocks, and the enhanced image blocks are used as the input of the network.

3. The image denoising model compression method based on structural reparameterization according to claim 2, characterized in that: The step S1.3 further comprises: The black level correction and normalization of the Bayer format RAW image is shown in the following formula: In formula (1), bit is the number of bits of the RAW image. After black level correction and normalization, the single-channel output image is converted to a 4-channel image, and the image spatial resolution is reduced to 1 / 4 of the original one.

4. The image denoising model compression method based on structural reparameterization according to claim 1, characterized in that: The step S2 further comprises: S2.1, design a basic denoising network in the form of U-Net. Design a lightweight network as the basic denoising model. The model consists of input convolution, encoder, decoder, output convolution and cross-layer connection. The input of the input convolution is a 4-channel noise image. The encoder consists of three consecutive "downsampling + standard residual modules", the decoder consists of three "upsampling + standard residual modules", the output of the output convolution is a 4-channel noise map, and the cross-layer connection is used to integrate the detailed information extracted at each stage of downsampling in the upsampling stage; S2.2, design diversified branch modules that can perform structural reparameterization to upgrade the basic network to a multi-branch network with stronger performance for training; In the encoder part, the first convolution in the standard residual module a is changed to a multi-branch structure with an output channel of C and a 1×1 convolution with an output channel of 2C, where the multi-branch structure consists of a 1×1 convolution, two 3×3 convolutions, two consecutive 1×1 convolutions and 3×3 convolutions, and consecutive 1×1 convolutions and average pooling AVG; the second convolution in the standard residual module a is changed to channel splitting, a multi-branch structure with an output channel of C, channel merging, and channel shuffling, thereby obtaining a diversified branch module a; In the decoder part, the first convolution in the standard residual module b is changed to a 1×1 convolution with an output channel of C and a multi-branch structure with an output channel of C; the second convolution in the standard residual module b is changed to channel splitting, a multi-branch structure with an output channel of C / 2, channel merging and channel shuffling, and the final output channel is C, thereby obtaining a diversified branch module b.

5. The image denoising model compression method based on structural reparameterization according to claim 4, characterized in that: In step S2.2, In the encoder part, for the standard residual module a, assuming that the input channel is C and the output channel is 2C, its parameter quantity is: 2C×3×3×C+2C×3×3×2C+2C×1×1×C=56C 2 ; The second convolution in the standard residual module a is changed to a multi-branch structure with channel segmentation, output channels C, channel merging and channel shuffling, thereby obtaining a diversified branch module a, whose parameter quantity is: (C×C×3×3×4+C×C×1×1×4)×2+2C×C×1×1=82C 2 ; In the decoder part, for the standard residual module b, assuming that the input channel is 2C and the output channel is C, the parameter amount is: 2C×3×3×C+C×3×3×C+C×1×1×2C=29C 2 ; The second convolution in the standard residual module b is changed to a multi-branch structure with channel segmentation, an output channel of C / 2, channel merging and channel shuffling, and the final output channel is C, thereby obtaining a diversified branch module b, whose parameter quantity is: C×C×3×3×4+C×C×1×1×4+C×2C×1×1+C×C×3×3+C×C×1×1=52C 2 。 6. The image denoising model compression method based on structural reparameterization according to claim 1, characterized in that: The step S3 further comprises: The MAE function is used to calculate the loss of the predicted image and the reference image, as shown in the following formula: The Adam optimizer is used to update the network parameters. The cosine annealing learning rate reduction strategy is used during the training process, and the half cycle of the cosine function is set as the maximum number of iterations.

7. The image denoising model compression method based on structural reparameterization according to claim 1, characterized in that: In the step S4, for the diversified branch module a, the converted parameter amount is, compared with the standard residual module a, the compression ratio is 64.3%; for the diversified branch module b, the converted parameter amount is, and the compression ratio is 55.2%; since the resolution of the feature map within the module has not changed, the compression ratio of the parameter amount is equivalent to the compression ratio of the calculation amount.

8. The image denoising model compression method based on structural reparameterization according to claim 4, characterized in that: The step S4 further comprises: (1) 1×1 convolution is converted to 3×3 convolution: Pad the 1×1 convolution kernel by filling 0 elements around it, which is the Pad operation, to convert it to a 3×3 convolution kernel. The offset is used to perform the Replicate operation, as shown in formula (3) and formula (4), where K 1×1 ∈R D×C×1×1 , K′ 1×1 ∈R D×C×3×3 , b 1×1 ∈R D×1×1 , b′ 1×1 ∈R D×H×W , C and D represent the number of input and output channels respectively, H and W represent the width and height of the output feature map; K′ 1×1 =Pad(K 1×1 ,0) Formula (3); b′ 1×1 =Replicate(b 1×1 ) formula (4); (2) Continuous 1×1 convolution and 3×3 convolution are converted to 3×3 convolution: First, the zeroth dimension and the first dimension of the 1×1 convolution are swapped by transposing the operation, and then the 3×3 convolution kernel is used to perform convolution operation on the transposed result to obtain the converted 3×3 convolution. The conversion process is shown in formulas (5) and (6); where K 1×1 ∈R D×C×1×1 , K 3×3 ∈R D×D×3×3 , K 1×1→3×3 ∈R D×C×3×3 , b 1×1→3×3 ∈R D×H×W ; K 1×1→3×3 =K 3×3 *Transpose(K 1×1 ) Formula (5); b 1×1→3×3 =Replicate(b 1×1 )*K 3×3 +Replicate(b 3×3 ) formula (6); (3) Continuous 1×1 convolution and AVG are converted to 3×3 convolution: The average pooling layer AVG can be equivalent to a fixed-weight convolution. When converting AVG to a 3×3 convolution, a fixed weight of 1 / 9 is set for the current input channel, and the others are set to 0. Then, according to (2), the continuous 1×1 convolution and 3×3 convolution are converted to 3×3 convolution. The conversion process is shown in formulas (7), (8), and (9); where K 1×1 ∈R D×C×1×1 , K avg ∈R D×D×3×3 , K 1×1→avg ∈R D×C×3×3 , b 1×1→avg ∈R D×H×W ; K 1×1→avg =K avg *Transpose(K 1×1 ) Formula (8); b 1×1→avg =Replicate(b 1×1 )*K 1×1→avg Formula (9); (4) Identity mapping is converted to 3×3 convolution: The identity mapping is equivalent to a 1×1 convolution with the unit matrix as the convolution kernel, which is then converted to a 3×3 convolution by padding 0s around it, as shown in formula (10) and formula (11); where K identity ∈R D×C×3×3 , b identity ∈R D×H×W ; b identity =Replicate(0) Formula(11); (5) After each branch is converted to a 3×3 convolution, the weights and biases of all branches are fused to obtain K fuse and b fuse , as shown in formula (12) and formula (13), multiple branches can be converted into a single 3×3 convolution; where K fuse ∈R D×C×3×3 , b fuse ∈R D×H×W ; K fuse =K ′ 1×1 +K 1×1→3×3 +K 1×1→avg +K identity Formula (12); b fuse =b ′ 1×1 +b 1×1→3×3 +b 1×1→avg +b identity Formula (13).

Images