Deep learning model construction method for video denoising
By building a lightweight deep learning model, using technologies such as recursive input and space-to-deep conversion, the problem of poor denoising of 3D videos in low light conditions is solved, and efficient and real-time video denoising effect is achieved.
Patent Information
- Application Number
- CN202311597582.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-23
- Publication Date
- 2025-05-23
AI Technical Summary
The prior art has poor 3D video denoising effect under low light conditions, large calculation volume is difficult to deploy, or the effect is unsatisfactory, and it is difficult to take into account both the effect and efficiency.
A lightweight deep learning model is built to achieve image denoising by recursively inputting two frames of images, using space-to-depth and depth-to-space conversions to reduce the computational amount, and combining downsampling blocks and feature fusion modules.
This model can effectively denoising under low light conditions, with small calculations and real-time deployment, taking into account the effect and efficiency of video denoising.
Smart Images
Figure CN120031737A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video denoising processing, and in particular relates to a method for constructing a deep learning model for video denoising. Background Art
[0002] With the rapid development of image tasks in recent years, people's demand for high-quality images and videos has become increasingly prominent. However, it is difficult to ensure high-quality imaging under many extreme conditions. At present, under low-light conditions, various traditional ISP3D denoising algorithms cannot achieve good imaging effects. They face problems such as high noise, motion blur, and motion smear, which are far from meeting the needs of various tasks. Therefore, a series of deep learning models have emerged to deal with 3D denoising tasks. Compared with traditional algorithms, the effect is much better. Therefore, for the 3D video denoising problem, a good deep learning solution is of great significance.
[0003] There are many academic solutions for deep learning 3D video denoising algorithms based on convolutional neural networks. These algorithms have their own advantages and disadvantages, and the most common ones are multi-frame fusion models with Unet-like structures.
[0004] However, for specific application scenarios, the commonly used Unet structure mentioned above often has the problem of large computational complexity and difficulty in deployment. Some lightweight algorithms with small computational complexity and able to maintain frame rate have the problem of not being able to achieve optimal results. They may perform well for general scenarios, but the results are often unsatisfactory in low-light scenarios.
[0005] In addition, the commonly used terms in the prior art include:
[0006] Deep learning: It is a new research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence (AI). Deep learning is the inherent laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.
[0007] Bayer format: Bayer format images are derived from the Bayer array, which is one of the main technologies for CCD or CMOS sensors to capture color images. The Bayer array was invented by Bryce Bayer, a scientist at Eastman Kodak, and is widely used in digital images. For color images, each pixel can be represented by three colors, RGB. The simplest sampling method is to use three filters on each pixel. The red filter transmits red wavelengths, the green filter transmits green wavelengths, and the blue filter transmits blue wavelengths. In this way, in order to collect the three basic colors of RGB, three filters are required for each point. This method is expensive, and because the three filters must be aligned to the same point, it is not easy to manufacture. The Bayer format can solve this problem well. Each pixel uses only one color filter. In addition, by analyzing the human eye's perception of color, it is found that the human eye is more sensitive to green, so there is more green in the Bayer format image, and the green pixel is the sum of the R and B pixels. Summary of the invention
[0008] In order to solve the above problems, the purpose of this application is to: the model structure constructed by this method can well fit the requirements of the time domain denoising task in the 3D denoising task; and because it is a lightweight model, the calculation amount is very small, and it can achieve real-time during the deployment process. Compared with the existing technology, the deep learning model construction method for video denoising of the present invention can take into account both effect and efficiency.
[0009] Specifically, the present invention provides a method for constructing a deep learning model for video denoising, the method comprising the following steps:
[0010] S1. Input image processing:
[0011] The input of the model is two frames, one is the result of the previous model fusion output, and the other is the current frame of the video stream. The whole process is a recursive process. Both frames are raw domain images, which are single-channel images. In order to reduce the amount of calculation of the model, space_to_depth is executed twice to change the image from one channel to 16 channels, and the width and height of the image are thus reduced to one-fourth of the original, thereby reducing the feature size of the input model;
[0012] Assume that the original image is F1, F2, and the shape is (H, W, 1), then after step S1, it becomes f1, f2, and the shape is (H / 4, W / 4, 16), set to (h, w, c), which is the first scale;
[0013] S2. Downsampling block, the model uses 3 convolutions to build a downsampling block, and this structure is named DownBlock, where the downsampling is completed by convolution with a stride of 2;
[0014] After this structure, the output is (h / 2, w / 2, oc), oc is the number of output channels, this is the second scale;
[0015] The output is then passed through the same downsampling block again, and the output is (h / 4, w / 4, oc), which is the third scale.
[0016] S3. Feature fusion module: For the image processed by step S2, the feature fusion module is input at three scales respectively. The design structure of the feature fusion module is named FusionBlock, and the output is the weight of one channel and is passed to the next scale;
[0017] S4. Weight upsampling:
[0018] After step S3, the feature size of the output of the third scale is (h / 4, w / 4, 1). To pass it to the second scale, upsampling is required. The structure used here is the nearest neighbor upsampling and convolution structure, which is named UpBlock.
[0019] S5. Image fusion:
[0020] After extraction and construction at three scales, a weight of (h, w, 1) is finally output, which is set as gamma with a value range of 0-1. It is used as the weight for fusing the two frames. The fusion formula is as follows:
[0021] Out=f1*gamma+f2*(1-gamma),
[0022] This formula is called fusion;
[0023] S6.Restore size:
[0024] The output of step S5 is a fused image with a shape of (H / 4, W / 4, 16). To restore the shape of the original image, depth_to_space is used twice, which is pixel_shuffle in pytorch, that is, it is restored to the raw image of (H, W, 1).
[0025] In step S2, the model uses three convolutions to construct a downsampling block, including:
[0026] S2.1, perform convolution: K: 3*3, S: 1*1;
[0027] S2.2, perform convolution: K: 3*3, S: 2*2;
[0028] S2.3, perform convolution: K: 3*3, S: 1*1;
[0029] The same down-sampling block, that is, the structure is the same as the DownBlock constructed from step S2.1 to step S2.3.
[0030] In step S2, set oc to 32.
[0031] In step S3, the feature fusion module is designed as follows:
[0032] S3.1, perform convolution, K: 3*3, S: 1*1;
[0033] S3.2, perform convolution, K: 3*3, S: 1*1;
[0034] S3.3, perform convolution, K: 3*3, S: 2*2;
[0035] S3.4, perform convolution, K: 3*3, S: 1*1;
[0036] S3.5, upsampling;
[0037] S3.6, perform convolution, K: 3*3, S: 1*1;
[0038] S3.7, concatenate the output of step S3.2 and the output of step S3.6;
[0039] S3.8, perform convolution, K: 3*3, S: 1*1;
[0040] S3.9, execute the activation function sigmoid.
[0041] In the step S4, the structure used includes:
[0042] S4.1, nearest upsampling: UpSample(nearest);
[0043] S4.2, perform convolution: K: 3*3, S: 1*1.
[0044] Therefore, the advantage of this application is that the model structure constructed using this method can take into account both the effect and efficiency requirements. Experiments have shown that it can also fit low-light scenes with very low computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0046] Figure 1 It is a flow chart of the present application method.
[0047] Figure 2It is a schematic diagram of space_to_depth in step S1 of the method of the present application.
[0048] Figure 3 It is a schematic diagram of the specific structure of downsampling in step S2 of the method of the present application.
[0049] Figure 4 It is a schematic diagram of the feature fusion module in step S3 of the method of the present application.
[0050] Figure 5 It is a schematic diagram of the structure of the nearest neighbor upsampling and convolution in step S4 of the method of the present application.
[0051] Figure 6 This is a schematic diagram of the construction of a deep learning model for video denoising in this application. DETAILED DESCRIPTION
[0052] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0053] The present invention belongs to the technical field of 3D denoising in low-level visual tasks of deep learning neural networks and relates to a method for constructing a model.
[0054] like Figure 1 As shown, this application proposes a deep learning model construction method for video denoising, and the specific implementation steps of the method are described as follows:
[0055] Step S1. Input image processing:
[0056] The input of the model is two frames, one is the result of the last model fusion output, and the other is the current frame of the video stream. The whole process is a recursive process. Both frames are raw domain images, which are single-channel images. In order to reduce the amount of calculation of the model, the space_to_depth method is executed twice to change the image from one channel to 16 channels, and the width and height of the image are thus reduced to one-fourth of the original, thereby reducing the feature size of the input model. Assuming that the original image is F1, F2, and the shape is (H, W, 1), after this step it becomes f1, f2, and the shape is (H / 4, W / 4, 16), set to (h, w, c), which is the first scale; wherein, the space_to_depth is as follows Figure 2 The shape change operation shown;
[0057] Step S2. Downsampling block, the specific structure of the downsampling is as follows Figure 3 As shown, including:
[0058] S2.1, perform convolution: K: 3*3, S: 1*1;
[0059] S2.2, perform convolution: K: 3*3, S: 2*2;
[0060] S2.3, perform convolution: K: 3*3, S: 1*1;
[0061] Name this structure DownBlock:
[0062] The model uses three convolutions to construct a downsampling block, where the downsampling is done through a convolution with a step size of 2. After this structure, the output is (h / 2, w / 2, oc), where oc is the number of output channels, which is 32 in actual use. This is the second scale. The output is then passed through the same downsampling block again, and the output is (h / 4, w / 4, oc), which is the third scale.
[0063] Step S3. Feature fusion module:
[0064] For the image processed in step S2, the feature fusion module is input at three scales respectively. The design of the feature fusion module is as follows: Figure 4 As shown:
[0065] S3.1, perform convolution, K: 3*3, S: 1*1;
[0066] S3.2, perform convolution, K: 3*3, S: 1*1;
[0067] S3.3, perform convolution, K: 3*3, S: 2*2;
[0068] S3.4, perform convolution, K: 3*3, S: 1*1;
[0069] S3.5, upsampling;
[0070] S3.6, perform convolution, K: 3*3, S: 1*1;
[0071] S3.7, concatenate the output of step S3.2 and the output of step S3.6;
[0072] S3.8, perform convolution, K: 3*3, S: 1*1;
[0073] S3.9, execute activation function sigmoid;
[0074] This structure is named FusionBlock, and the output is the weight of one channel and passed to the next scale;
[0075] Step S4. Weight upsampling:
[0076] After step S3, the feature size of the output of the third scale is (h / 4, w / 4, 1). If it is to be passed to the second scale, it needs to be upsampled. The structure used here is the nearest neighbor upsampling and convolution structure, such as Figure 5 As shown:
[0077] S4.1, nearest upsampling: UpSample(nearest);
[0078] S4.2, perform convolution: K: 3*3, S: 1*1;
[0079] Name this structure UpBlock;
[0080] Step S5. Image fusion:
[0081] After extraction and construction at three scales, a weight of (h, w, 1) is finally output, which is set as gamma with a value range of 0-1. It is used as the weight for fusing the two frames. The fusion formula is as follows:
[0082] Out=f1*gamma+f2*(1-gamma)
[0083] This formula is called fusion;
[0084] Step S6. Restore size:
[0085] The output of step S5 is a fused image with a shape of (H / 4, W / 4, 16). To restore the shape of the original image, depth_to_space is used twice, which is pixel_shuffle in pytorch, that is, it is restored to the raw image of (H, W, 1). Pytorch is an open source Python machine learning library.
[0086] The 3D video denoising task has always faced the problem of balancing effect and efficiency. Some algorithms have good effects, but the computational complexity is very high and cannot be implemented in the actual deployment process. Other models that overemphasize lightweight also perform poorly in low-light scenes where traditional video denoising algorithms perform poorly. The model structure constructed using this method can take into account both effect and efficiency. It can also fit low-light scenes and has a very low computational complexity.
[0087] In summary, if Figure 6 As shown in the figure, an example of building a deep learning model for video denoising includes:
[0088] Step S1, space_to_depth is performed twice on F1 and F2 to obtain f1 and f2;
[0089] Step S2, after this DownBlock structure, the output is (h / 2, w / 2, oc), oc is the number of output channels, this is the second scale;
[0090] The output is then downsampled again through the same DownBlock structure, and the output is (h / 4, w / 4, oc), which is the third scale.
[0091] Step S3, for the image processed by step S2 (DownBlock), input the feature fusion module FusionBlock at three scales respectively, output the weight of one channel and pass it to the next scale;
[0092] Step S4, the third scale passes through step S3, and the output feature size is (h / 4, w / 4, 1). If it is to be passed to the second scale, it needs to be upsampled. The structure used here is the nearest neighbor upsampling and convolution structure UpBlock;
[0093] Step S5, after extraction and construction of three scales, finally output a weight of size (h, w, 1) as the weight of fusing two frames. The fusion formula is as follows:
[0094] Out=f1*gamma+f2*(1-gamma)
[0095] This formula is called fusion;
[0096] Step S6: The output of step S5 is a fused image with a shape of (H / 4, W / 4, 16). To restore the shape of the original image, depth_to_space is used twice, which is pixel_shuffle in pytorch, that is, the raw image of (H, W, 1) is restored. Pytorch is an open source Python machine learning library.
[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for building a deep learning model for video denoising, It is characterized in that The method comprises the following steps: S1. Input image processing: The input of the model is two frames, one is the result of the previous model fusion output, and the other is the current frame of the video stream. The whole process is a recursive process. Both frames are raw domain images, which are single-channel images. In order to reduce the amount of calculation of the model, space_to_depth is executed twice to change the image from one channel to 16 channels, and the width and height of the image are thus reduced to one-fourth of the original, thereby reducing the feature size of the input model; Assume that the original image is F1, F2, and the shape is (H, W, 1), then after step S1, it becomes f1, f2, and the shape is (H / 4, W / 4, 16), set to (h, w, c), which is the first scale; S2. Downsampling block, the model uses 3 convolutions to build a downsampling block, and this structure is named DownBlock, where the downsampling is completed by convolution with a stride of 2; After this structure, the output is (h / 2, w / 2, oc), oc is the number of output channels, this is the second scale; The output is then passed through the same downsampling block again, and the output is (h / 4, w / 4, oc), which is the third scale. S3. Feature fusion module: For the image processed by step S2, the feature fusion module is input at three scales respectively. The design structure of the feature fusion module is named FusionBlock, and the output is the weight of one channel and is passed to the next scale; S4. Weight upsampling: After step S3, the feature size of the output of the third scale is (h / 4, w / 4, 1). To pass it to the second scale, upsampling is required. The structure used here is the nearest neighbor upsampling and convolution structure, which is named UpBlock. S5. Image fusion: After extraction and construction at three scales, a weight of (h, w, 1) is finally output, which is set as gamma with a value range of 0-1. It is used as the weight for fusing the two frames. The fusion formula is as follows: Out=f1*gamma+f2*(1-gamma), This formula is called fusion; S6.Restore size: The output of step S5 is a fused image with a shape of (H / 4, W / 4, 16). To restore the shape of the original image, depth_to_space is used twice, which is pixel_shuffle in pytorch, that is, it is restored to the raw image of (H, W, 1).
2. The method for constructing a deep learning model for video denoising according to claim 1, It is characterized in that In step S2, the model uses three convolutions to construct a downsampling block, including: S2.1, perform convolution: K: 3*3, S: 1*1; S2.2, perform convolution: K: 3*3, S: 2*2; S2.3, perform convolution: K: 3*3, S: 1*1; The same down-sampling block, that is, the structure is the same as the DownBlock constructed from step S2.1 to step S2.
3.
3. The method for constructing a deep learning model for video denoising according to claim 1, It is characterized in that In step S2, set oc to 32.
4. The method for constructing a deep learning model for video denoising according to claim 1, It is characterized in that In step S3, the feature fusion module is designed as follows: S3.1, perform convolution, K: 3*3, S: 1*1; S3.2, perform convolution, K: 3*3, S: 1*1; S3.3, perform convolution, K: 3*3, S: 2*2; S3.4, perform convolution, K: 3*3, S: 1*1; S3.5, upsampling; S3.6, perform convolution, K: 3*3, S: 1*1; S3.7, concatenate the output of step S3.2 and the output of step S3.6; S3.8, perform convolution, K: 3*3, S: 1*1; S3.9, execute the activation function sigmoid.
5. The method for constructing a deep learning model for video denoising according to claim 1, It is characterized in that In the step S4, the structure used includes: S4.1, nearest upsampling: UpSample(nearest); S4.2, perform convolution: K: 3*3, S: 1*1.