UNet network-based image noise reduction model training method, computer device and storage medium
Through the image noise reduction model training method based on UNet network, the problem that image noise reduction model in the prior art is difficult to deploy on low-computing equipment, and efficient image noise reduction on low-computing equipment is achieved, which is suitable for use in scenarios where bandwidth and computing power are limited.
Patent Information
- Application Number
- CN202411739019.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-18
AI Technical Summary
The existing deep learning image noise reduction algorithms have problems such as excessive model, high bandwidth requirements, average noise reduction effect, and difficulty in deploying on low-computing devices.
The image noise reduction model training method based on UNet network is adopted. By collecting original image data in different scenarios, the noise-true image pair is generated, and the teacher model and the student model are trained to output the final image noise reduction model. This method includes gain training and knowledge distillation, adapting to low computing power platforms.
The image noise reduction model deployed on low-computing equipment is realized, with good noise reduction effect and improved computing efficiency, and is suitable for use in scenarios where bandwidth and computing power are limited.
Smart Images
Figure CN119648565A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image denoising, and more specifically, to a UNet-based image denoising model training method, a computer device applying the UNet-based image denoising model training method, and a computer-readable storage medium applying the UNet-based image denoising model training method. Background Art
[0002] After decades of research on image denoising, a large number of excellent algorithms have been applied in common devices. When the prior distribution of noise is known, denoising algorithms such as NLM and BM3D can work well, but due to the complexity of the source of noise in reality, it is difficult to accurately model it. When faced with lower illumination scenes and lower-cost equipment, the iteration of traditional algorithms gradually reaches a bottleneck, and it is increasingly difficult to make breakthrough progress. With people's continuous pursuit of image quality, in recent years, researchers have applied deep learning to image and video denoising. More and more research has been conducted, and the image denoising effect achieved has also been greatly improved compared to traditional algorithms. Therefore, applying deep learning to image denoising is a technical direction with great potential.
[0003] When using deep learning for noise reduction, the first two parts to be solved are the noise-truth data pair and the model structure. In practice, it is difficult to collect completely corresponding truth values (noise-free images) and noise (corresponding noisy images). Current research has constructed data pairs by directly sprinkling Gaussian-Poisson-uniform noise on the truth image, but this is different from the actual noise model. The trained model has a general effect on image noise denoising in reality. Furthermore, researchers have established a noise mathematical model by analyzing the causes and processes of noise generation to produce noise that is more in line with reality, but this is still different from reality.
[0004] In addition, existing deep learning denoising algorithms have the disadvantages of being too large in model, having high bandwidth requirements, having average denoising effects, and being difficult to deploy on low-computing devices (1T to 2T FLOPS).
[0005] Therefore, a more optimized image denoising model needs to be considered. Summary of the invention
[0006] The first objective of the present invention is to provide an image denoising model training method based on a UNet network, which can enable the image denoising model to have a better denoising effect and is easy to deploy on a low-computing power device.
[0007] The second object of the present invention is to provide a computer device that can enable the image denoising model to have a better denoising effect and is easy to deploy on low computing power equipment.
[0008] The third object of the present invention is to provide a computer-readable storage medium that can enable the image denoising model to have a better denoising effect and is easy to deploy on low-computing power devices.
[0009] In order to achieve the above-mentioned first purpose, the image denoising model training method based on the UNet network provided by the present invention includes: collecting original image data under different scenes, the original image data including noise-free image sequences and noise image sequences of different intensities; generating noise-true value image pairs according to the original image data, wherein the noise-true value image pairs include noise images and true value images under the same scene; using the UNet network to build a teacher model and a student model; using the noise-true value image pairs to train the teacher model and the student model, and outputting the final image denoising model; wherein the step of using the noise-true value image pairs to train the teacher model and the student model includes: using the teacher model to train the noise-true value image pairs corresponding to a preset number of gain multiples to obtain a sub-teacher model under each gain multiple, and using the preset number of sub-teacher models to perform knowledge distillation to the student model to obtain the final image denoising model.
[0010] It can be seen from the above scheme that the image denoising model training method based on the UNet network of the present invention uses the noise-true value image pairs to train the teacher model and the student model, and uses the teacher model to train the noise-true value image pairs corresponding to the preset number of gain multiples, and obtains the sub-teacher model under each gain multiple, and performs knowledge distillation of the preset number of sub-teacher models to the student model to obtain the final image denoising model. The teacher model adopts the method of divided gain training, which can more accurately learn the noise under the gain and obtain better denoising results.
[0011] In a further scheme, the step of using a preset number of sub-teacher models to perform knowledge distillation on the student model includes: adding the denoising results output by all sub-teacher models with corresponding weights to obtain the final teacher model denoising result; and performing knowledge distillation on the student model using the final teacher model denoising result.
[0012] In a further scheme, the sum of the weights of all sub-teacher models is equal to 1.
[0013] It can be seen that different teacher model weights make the network focus on more rewarding gain scenarios and improve the noise reduction effect.
[0014] In a further scheme, the step of using noise-true value image pairs to train the teacher model and the student model also includes: when training the student model, obtaining a motion offset matrix; combining the motion offset matrix with the L1 loss function to obtain a loss function that emphasizes motion.
[0015] It can be seen that in the training of the student model, in order to better integrate the moving / static areas, a motion estimation module was added and the corresponding loss function was improved. By setting the motion offset matrix, different weights can be assigned to the moving area and the static area, thereby obtaining better fusion and noise reduction results.
[0016] In a further solution, the motion offset matrix is obtained by the following formula: weight=CLIP(abs(img_a-img_b),TH1,TH2), where img_a represents the current frame, img_b represents the previous frame, and TH1 and TH2 represent the threshold range for judging motion.
[0017] It can be seen that the motion offset matrix limits the input value between TH1 and TH2 through the clipping function, that is, the absolute value of the pixel difference between the two frames is clipped within the range determined by TH1 and TH2, thereby obtaining the current input motion offset matrix weight. This motion offset matrix is consistent with the input size and is used to assign different weights to the motion area and the static area in the future to better perform image fusion and noise reduction.
[0018] In a further scheme, both the teacher model and the student model include UNet fused fusion network and UNet denoise denoising network; UNet fused fusion network and UNet denoise denoising network are composed of preset basic residual unit, maximum pooling, deconvolution, Cat and jump connection.
[0019] It can be seen that by setting up the UNet fused fusion network and the UNet denoise denoising network, the input of the current frame and the fusion information of all previous frames, the fusion network outputs implicit fused time domain information, which is directly output as the input of the next frame fusion network. The output of the denoising network and the true value are used to obtain the loss function, so as to train the network until convergence. Through the processing of maximum pooling downsampling and preset basic residual units, a higher receptive field and more advanced image features can be obtained. Combined with the features of the direct output by channel, information of different dimensions can be retained to obtain more effective denoising results. At the same time, jump connection can combine the denoising results with the initial input to denoise while retaining the effective information of the input.
[0020] In a further scheme, the preset basic residual unit includes a first Rep structure block, a second Rep structure block and a ReLU activation function; the input data entering the preset basic residual unit is divided into two paths, one of which is directly output without any operation; the other path extracts input features through the first Rep structure block, then increases nonlinearity through the ReLU activation function, and finally further extracts features through the second Rep structure block to obtain the output of the path; the outputs of the two paths are combined through a point-by-point add operation to obtain the output of the preset basic residual unit; the first Rep structure block and the second Rep structure block both include parallel 3×3 convolution kernels, 1×1 convolution kernels and a jump connection.
[0021] It can be seen that the preset basic residual unit includes the first Rep structure block, the second Rep structure block and the ReLU activation function. By increasing the complexity of the network structure in the training phase, more useful features are learned, and the structure is losslessly transformed in testing / inference without increasing the testing / inference cost. In this way, the improvement of the noise reduction effect is achieved by increasing the cost of the training phase rather than increasing the cost of the testing phase, which is more conducive to deployment.
[0022] In a further scheme, the teacher model uses two preset basic residual units in series in each layer, the student model uses one pre-basic residual unit in each layer, and the number of channels of the student model is half of the number of channels of the teacher model.
[0023] It can be seen that the use of two preset basic residual units in series in each layer of the teacher model allows the teacher model to perform more in-depth feature extraction of the input data. Each basic residual unit is able to learn specific features of the input data, and two connected in series can capture more complex and advanced feature representations. At the same time, it can also improve the model's expressiveness, so that the teacher model can better adapt to complex tasks and data distributions, and improve the accuracy and generalization of the model. The student model uses a single basic residual unit and has half the number of channels compared to the teacher model, making the structure simpler and requiring less computation. Using a single basic residual unit and reducing the number of channels can greatly reduce the number of parameters and computational complexity of the model, thereby improving the model's inference speed and operating efficiency. This is very important for some resource-constrained application scenarios, such as mobile devices and embedded systems.
[0024] In order to achieve the second purpose of the present invention, the present invention provides a computer device including a processor and a memory, the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned image denoising model training method based on the UNet network are implemented.
[0025] In order to achieve the third purpose of the present invention, the computer-readable storage medium provided by the present invention stores a computer program thereon, and when the computer program is executed by a controller, the steps of the above-mentioned UNet network-based image denoising model training method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a flow chart of an embodiment of the image denoising model training method based on the UNet network of the present invention.
[0027] Figure 2 It is a structural schematic diagram of a preset basic residual unit in an embodiment of the image denoising model training method based on a UNet network of the present invention.
[0028] Figure 3 It is a schematic diagram of the UNet network structure in an embodiment of the image denoising model training method based on the UNet network of the present invention.
[0029] Figure 4 It is a flow chart of network training of two consecutive frames of images in an embodiment of the image denoising model training method based on the UNet network of the present invention.
[0030] Figure 5 It is a connection diagram of two jump connections between the UNet fused fusion network and the UNet denoise denoising network in an embodiment of the image denoising model training method based on the UNet network of the present invention.
[0031] Figure 6 It is a flowchart of the process of knowledge distillation from the teacher model to the student model in the embodiment of the image denoising model training method based on the UNet network of the present invention.
[0032] Figure 7 It is an input image that has not been denoised in the embodiment of the image denoising model training method based on the UNet network of the present invention.
[0033] Figure 8 It is the output image after denoising in the embodiment of the image denoising model training method based on the UNet network of the present invention.
[0034] The present invention is further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION
[0035] Example of image denoising model training method based on UNet network:
[0036] The UNet network-based image denoising model training method of the present invention is a computer program applied to a computer device and is used for image denoising model training.
[0037] like Figure 1As shown, in this embodiment, when the image denoising model training method based on the UNet network is working, step S1 is first executed to collect original image data in different scenes. Among them, the original image data includes a noise-free image sequence and a noise image sequence of different intensities. The original image data retains the most original information of the image, and the collected original image data provides the most complete image information required for denoising. When collecting original image data in different scenes, first, for different static and moving scenes, noise-free image sequences and noise image sequences of different intensities are collected as training data, and a part of them is used as a test data set for subjective evaluation. It should be noted that in the process of collecting data, the noise-true value image is obtained by controlling the exposure time and gain. Under the same brightness, the 1x gain image is the noise-free true value image; the image taken after dividing the exposure time by the corresponding multiple and multiplying by the same multiple gain is used as the noise image. At the same time, there is also noise image data taken directly in a low-light environment.
[0038] After collecting the original image data in different scenes, execute step S2 to generate a noise-true value image pair according to the original image data. The noise-true value image pair includes a noise image and a true value image in the same scene. When generating a noise-true value image pair according to the original image data, Gaussian-Poisson-uniform noise is added to the noise-free true value image to construct a noise-true value data pair. This data pair contains static and moving scenes and can be used as a training / test data set. The average of multiple noise image sequences of different intensities in the same static scene is taken as the corresponding true value. The true value image of the moving scene cannot be obtained by averaging. Similarly, the noise-true value image of the static scene obtained can be used as a training / test data set. The true value includes the image obtained by averaging and the collected noise-free image. The true value and the corresponding noise image are divided into training and testing data sets in a ratio of 8:2. For example, a noise-true value image pair refers to a noise image and a corresponding noise-free true value image constituting a pair of data. If the noise image and the true value image are respectively continuous and one-to-one corresponding, they constitute a continuous noise-true value image sequence pair. The ratio of 8:2 means that if there are 100 pairs of noise-true value image pairs, 80 pairs of noise-true value images are used as training, and the remaining 20 pairs are used as testing.
[0039] Generate noise-true value image pairs, and after obtaining the data set, execute step S3 to build a teacher model and a student model using the UNet network. In order to make the denoising model compatible with the deployment of low-computing power devices on the end side, the present invention adopts a two-stage training method based on teachers and students, so different model structures are required, and they are called teacher models and student models. The teacher model has more parameters and a deeper and larger network structure. The student model is homologous to the teacher model, but has reduced parameters and a streamlined structure. Homologous means that the teacher and student models are of the same type of structure, such as the VGG series, UNet series, and ResNet series.
[0040] See also Figure 2 In this embodiment, Figure 2 The preset basic residual unit that constitutes the UNet network structure is shown. The preset basic residual unit includes the first Rep structure block, the second Rep structure block and the ReLU activation function; the input data entering the preset basic residual unit is divided into two paths, one of which is directly output without any operation; the other path extracts input features through the first Rep structure block, then increases nonlinearity through the ReLU activation function, and finally further extracts features through the second Rep structure block to obtain the output of this path; the outputs of the two paths are combined through the point-by-point addition operation to obtain the output of the preset basic residual unit; the first Rep structure block and the second Rep structure block both include parallel 3×3 convolution kernels, 1×1 convolution kernels and a jump connection. It should be noted that the preset basic residual units are different in the two stages of training and testing / inference, where testing and inference both refer to the process of outputting results after the input data passes through the network. During training, the 3×3 convolution kernel in the basic residual unit will be replaced with a Rep structure block. A Rep structure block contains a parallel 3×3 convolution kernel, a 1×1 convolution kernel and a jump connection. During test deployment, a Rep structure will be losslessly fused back to a 3×3 convolution kernel. This operation is to increase the complexity of the network structure in the training phase, learn more useful features, and losslessly transform the structure in the test / inference without increasing the test / inference cost. In this way, the improvement of the noise reduction effect is achieved by increasing the cost of the training phase rather than the cost of the test phase, which is obviously more conducive to deployment.
[0041] Figure 3 The UNet network structure is shown. In this embodiment, the UNet network is composed of a preset basic residual unit, maximum pooling, deconvolution, Cat and jump connection. Figure 3 The UNet network structure shown is a three-layer UNet structure (divided by resolution, Figure 3The UNet network structure shown involves 3 image resolutions). The initial input data is divided into two paths after passing through a preset basic residual unit. One path is directly output to the end of UNet, and the other path is further processed after passing through maximum pooling downsampling and basic residual units. The part that continues to be processed is repeatedly divided into two paths, one is directly output, and the other is processed by the basic residual unit. Through downsampling and processing by basic residual units, a higher receptive field and more advanced image features can be obtained. By combining the features of the direct output by channel, information of different dimensions can be retained to obtain more effective denoising results. The use of jump connections is mainly based on the consideration of improving the effect. By combining the denoising results and the initial input data, denoising can be achieved while retaining the effective information of the input data.
[0042] In this embodiment, each layer of the teacher model uses two preset basic residual units in series, each layer of the student model uses a pre-basic residual unit, and the number of channels of the student model is half the number of channels of the teacher model. The use of two preset basic residual units in series in each layer of the teacher model allows the teacher model to perform more in-depth feature extraction on the input data. Each basic residual unit can learn the specific features of the input data, and two connected in series can capture more complex and advanced feature representations. At the same time, the model expression ability can also be improved, so that the teacher model can better adapt to complex tasks and data distribution, and improve the accuracy and generalization ability of the model. The student model uses a single basic residual unit and the number of channels is halved compared to the teacher model, making the structure simpler and the amount of calculation smaller. Using a single basic residual unit and reducing the number of channels can greatly reduce the number of parameters and computational complexity of the model, thereby improving the reasoning speed and operating efficiency of the model. This is very important for some resource-constrained application scenarios, such as mobile devices and embedded systems.
[0043] See also Figure 4 In this embodiment, the teacher model and the student model both include a UNet fused network and a UNet denoise network. Figure 4The network training process of two consecutive frames of images is shown. The input of the network comes from two sources: the input of the current frame and the fusion information of all previous frames; the output of the fusion network is implicit fusion time domain information, which is directly output as the input of the next frame fusion network; the output of the denoising network is compared with the true value to obtain the loss function, so as to train the network until convergence. Among them, the fusion result of the tth frame and the input of the t+1th frame are spliced by channel (i.e., CAT operation) and then input into the fusion network to obtain the fusion result of the t+1th frame image, and the fusion result of the t+1th frame image will be input into the fusion network of the t+2th frame. When t=0, the input of the fusion network is the splicing of the first frame with itself. In addition, jump 1 and jump 2 are added between the UNet fused fusion network and the UNetdenoise denoising network. Jump 2 represents that the second layer of the UNet fused fusion network is connected to the second layer of the UNet denoise denoising network, and jump 1 represents that the third layer of the UNet fused fusion network is connected to the third layer of the UNet denoise denoising network, as shown in Figure 5 As shown in the figure, the two jump connections skip1 and skip2 are connected between the two UNets. The jump connection here uses the ADD(+) operation to perform point-by-point addition operations on convolution outputs of the same size. Based on deployment cost considerations, the first layer of the two UNet networks is not jump-connected, because this will significantly increase the SRAM usage of computing devices such as NPU / GPU, increasing costs. By connecting the two UNet networks between layers, the feature information of different processes can be effectively combined for noise reduction.
[0044] After building the teacher model and the student model, execute step S4 to train the teacher model and the student model using noise-true value image pairs, and output the final image denoising model.
[0045] In this embodiment, the step of using noise-true value image pairs to train the teacher model and the student model includes: using the teacher model to train the noise-true value image pairs corresponding to a preset number of gain multiples, obtaining a sub-teacher model under each gain multiple, and using the preset number of sub-teacher models to perform knowledge distillation to the student model to obtain a final image denoising model. The preset number can be set according to the number of types of gain multiples.
[0046] Figure 6 The process of knowledge distillation from the teacher model to the student model is shown. It should be noted that there are n teacher models, and n defaults to the number of types of noise-truth data pairs shot under different gains. Noise data under one gain can be considered as one type. For example, if there are noise-truth data pairs with gains of 4x, 8x, 16x, 32x, and 64x, the number of types n can be set to 5.
[0047] In this embodiment, the step of using a preset number of sub-teacher models to perform knowledge distillation on the student model includes: adding the denoising results output by all sub-teacher models with corresponding weights to obtain the final teacher model denoising result; and performing knowledge distillation on the student model with the final teacher model denoising result. Wherein, the sum of the weights of all sub-teacher models is equal to 1. A sub-teacher model learns the noise under a gain, and learns the weights of the outputs of each sub-teacher model. The weights are defined as t1, t2...tn, and the sum of the weights is 1. The weights must satisfy the following formula: t1+t2+...+tn=1. The final teacher model denoising result of the teacher model can be expressed as: teacher1*t1+teacher2*t2+...+teachern*tn. In particular, the weights can be evenly distributed artificially or learned through the network. It should be noted that the use of the teacher model can be extended to learn other specific tasks, such as color enhancement, detail texture enhancement, etc., and instill them into the student model together.
[0048] When using noise-true value image pairs to train the teacher model and student model, it also includes: setting hyperparameters and training strategies, training and testing the model; after preparing the model and training data, training the hyperparameters. The teacher model and the student model are trained separately, and the hyperparameter settings are the same. The learning rate, number of trainings, optimization method, batch size (batch size) and learning rate reduction strategy are set according to empirical values. The input of the network is randomly cropped into N×N blocks on the input image through a random function, and the output is also a N×N denoised block, where N is an empirical value set according to the size of the network structure. The teacher model and the student model are learned end-to-end, that is, Raw in and Raw out, and the loss function uses the L1 loss function of the Raw domain.
[0049] It should be noted that in order to enhance the color perception and learning of a single model, the L1 loss function of the RGB domain is added to the loss function of the student model during training, which complements the loss function of the Raw domain, and the weight of the two is 9:1. Both the teacher model and the student model are trained based on the Noise2Clean strategy. After the teacher model is trained for a period of time (such as 200 epochs), the student model is added for training together, so that the training weights of the teacher / student can be adjusted dynamically. You can also wait until the teacher model is trained, fix the teacher model, and only train the student model.
[0050] In this embodiment, the step of training the teacher model and the student model using noise-true value image pairs also includes: when training the student model, obtaining a motion offset matrix; combining the motion offset matrix with the L1 loss function to obtain a loss function that emphasizes motion. In the training of the student model, in order to better integrate the moving / static areas, a motion estimation module is added, and the corresponding loss function is improved. This example provides a simple implementation of motion estimation. Of course, the motion offset matrix can also be obtained by optical flow calculation and other methods. In this embodiment, the motion offset matrix is obtained by the following formula:
[0051] weight=CLIP(abs(img_a-img_b), TH1, TH2), where img_a represents the current frame, img_b represents the previous frame, and TH1 and TH2 represent the threshold range for judging motion.
[0052] The motion offset matrix uses the clipping function CLIP (abs (img_a-img_b), TH1, TH2) to limit the input value between TH1 and TH2, that is, the absolute value of the pixel difference between the two frames is clipped within the range determined by TH1 and TH2, thereby obtaining the current input motion offset matrix weight. This motion offset matrix is consistent with the input size and is used to assign different weights to the motion area and the static area in the future to better perform image fusion and noise reduction.
[0053] Combining the motion offset matrix with the L1 loss function, we get the L1 loss function with motion emphasis: Loss = L1*weight. By setting the motion offset matrix, different weights can be assigned to the motion area and the static area, thus obtaining better fusion and noise reduction results.
[0054] After the student model and the teacher model are jointly trained, the student model is fine-tuned using the noise-free image and the noise image corresponding to the noise-free image to obtain the final model. The fine-tuning operation is the process of training the student model alone with a small amount of noise-true value data set at a learning rate that is more than 100 times smaller than the original learning rate to obtain the final student model, that is, the final image denoising model.
[0055] After obtaining the final image denoising model, the test set can be fed into the final image denoising model to output the denoised image for testing and verification. Figure 7 The input image and Figure 8 It can be seen from the output pictures that the final image denoising model obtained by the present invention has a better denoising effect.
[0056] From the above, it can be seen that the image denoising model training method based on the UNet network of the present invention uses the noise-true value image pairs to train the teacher model and the student model, and uses the teacher model to train the noise-true value image pairs corresponding to the preset number of gain multiples to obtain the sub-teacher model under each gain multiple, and the preset number of sub-teacher models are subjected to knowledge distillation to the student model to obtain the final image denoising model. The teacher model adopts the method of divided gain training, which can more accurately learn the noise under the gain and obtain better denoising results. At the same time, the improved dual Unet denoising structure is used to adapt to the low computing power platform and improve the computing efficiency. A large number of basic operators, such as 3×3 convolution and 3×3 deconvolution, are used. At the same time, the size of the network is limited, so that it can be deployed in scenarios with limited bandwidth and computing power while ensuring the denoising effect.
[0057] Computer device embodiment:
[0058] The computer device of this embodiment includes a controller, and when the controller executes a computer program, the steps in the above-mentioned UNet network-based image denoising model training method embodiment are implemented.
[0059] For example, a computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a controller to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a computer device.
[0060] The computer device may include, but is not limited to, a controller and a memory. Those skilled in the art will appreciate that the computer device may include more or fewer components, or a combination of certain components, or different components, for example, the computer device may also include input and output devices, network access devices, buses, etc.
[0061] For example, the controller may be a central processing unit (CPU), other general controllers, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general controller may be a microcontroller or any conventional controller, etc. The controller is the control center of the computer device, and uses various interfaces and lines to connect the various parts of the entire computer device.
[0062] The memory can be used to store computer programs and / or modules. The controller realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. For example, the memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound receiving function, a sound conversion to text function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text data, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0063] Computer readable storage medium embodiment:
[0064] If the module integrated in the computer device of the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the implementation of all or part of the process in the above-mentioned UNet network-based image denoising model training method embodiment can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the controller, the steps of the above-mentioned UNet network-based image denoising model training method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunications signals.
[0065] It should be noted that the above are only preferred embodiments of the present invention, but the design concept of the invention is not limited thereto, and any non-substantial modifications made to the present invention using this concept also fall within the protection scope of the present invention.
Claims
1. A UNet-based image denoising model training method, characterized in that: include: Collecting original image data in different scenes, wherein the original image data includes a noise-free image sequence and a noise image sequence of different intensities; Generating a noise-true value image pair according to the original image data, wherein the noise-true value image pair includes a noise image and a true value image of the same scene; Use UNet network to build teacher model and student model; Using the noise-true value image pair to train the teacher model and the student model, and outputting a final image denoising model; Among them, the step of using the noise-truth image pairs to train the teacher model and the student model includes: using the teacher model to train the noise-truth image pairs corresponding to a preset number of gain multiples to obtain a sub-teacher model under each gain multiple, and using the preset number of sub-teacher models to perform knowledge distillation to the student model to obtain the final image denoising model.
2. The image denoising model training method based on the UNet network according to claim 1, characterized in that: The step of using the preset number of sub-teacher models to perform knowledge distillation on the student model includes: Adding the denoising results output by all the sub-teacher models with corresponding weights to obtain the final teacher model denoising result; The student model is subjected to knowledge distillation based on the final teacher model denoising result.
3. The UNet network-based image denoising model training method according to claim 2, characterized in that: The sum of the weights of all the sub-teacher models is equal to 1.
4. The image denoising model training method based on UNet network according to claim 1, characterized in that: The steps of training the teacher model and the student model using the noise-truth image pairs also include: During the training of the student model, obtaining a motion offset matrix; The motion offset matrix is combined with the L1 loss function to obtain a loss function that emphasizes motion.
5. The image denoising model training method based on UNet network according to claim 4 is characterized in that: The motion offset matrix is obtained by the following formula: weight=CLIP(abs(img_a-img_b),TH1,TH2), Among them, img_a represents the current frame, img_b represents the previous frame, and TH1 and TH2 represent the threshold range for judging motion.
6. The UNet network-based image denoising model training method according to any one of claims 1 to 5, characterized in that: The teacher model and the student model both include a UNet fused fusion network and a UNet denoise denoising network; The UNet fused fusion network and the UNet denoise denoising network are both composed of a preset basic residual unit, maximum pooling, deconvolution, Cat and jump connection.
7. The UNet network-based image denoising model training method according to claim 6, characterized in that: The preset basic residual unit includes a first Rep structure block, a second Rep structure block and a ReLU activation function; The input data entering the preset basic residual unit is divided into two paths, one of which is directly output without any operation; the other path is extracted by the first Rep structure block, and then the nonlinearity is increased by the ReLU activation function, and finally the second Rep structure block is further extracted to obtain the output of the path; the outputs of the two paths are combined by the add operation of point-by-point addition to obtain the output of the preset basic residual unit; The first Rep structure block and the second Rep structure block both include a 3×3 convolution kernel, a 1×1 convolution kernel and a jump connection in parallel.
8. The UNet network-based image denoising model training method according to claim 7, characterized in that: Each layer of the teacher model uses two preset basic residual units connected in series, each layer of the student model uses one pre-basic residual unit, and the number of channels of the student model is half of the number of channels of the teacher model.
9. A computer device comprising a processor and a memory, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the steps of the image denoising model training method based on the UNet network as described in any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the controller, the steps of the UNet network-based image denoising model training method are implemented as described in any one of claims 1 to 8.