Image denoising method, noise prediction model and electronic equipment
By using constrained convolutional layer and multiple postconvolution layers in the noise prediction model, including hollow convolutional layer and mixed hollow convolution module, the existing technology has poor image denoising effect in bank outlets at night scenes. Problem, achieving higher denoising accuracy and detail retention.
Patent Information
- Application Number
- CN202411721373.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-02-18
AI Technical Summary
The existing denoising network has poor denoising effect on monitoring video images in night scenes of bank outlets, resulting in low accuracy of denoising.
An image denoising method is proposed to predict and remove noise in the image by deploying a constrained convolution layer and a plurality of post-convolution layers, including a cavity convolution layer and a hybrid cavity convolution module, in the noise prediction model.
By obtaining low-dimensional noise feature maps by constraining the convolutional layer, the post-convolutional layer can better learn residual noise features, improve the accuracy of image denoising, and preserve important details of the image.
Smart Images
Figure CN119477744A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field, and in particular to an image denoising method, a noise prediction model and an electronic device. Background Art
[0002] The reason why a large amount of fine granular noise appears in the nighttime surveillance video images of bank branches is that the illumination is low at night, and the camera sensor is prone to generate noise (more in line with the distribution characteristics of Gaussian noise) under low illumination or high temperature (due to long-term operation). Therefore, it is necessary to perform noise removal processing on the images collected by the surveillance camera to improve the image quality and lay a good foundation for the image quality for subsequent human eye observation.
[0003] At present, the denoising network based on deep learning usually uses the residual learning strategy for denoising. In the daytime scene of bank branches, it has a good denoising effect on the surveillance video images. However, in the night scene of bank branches, the existing denoising network also smooths the important details of the image, resulting in low denoising accuracy. Summary of the invention
[0004] In view of the above problems, the present application provides an image denoising method, a noise prediction model and an electronic device to achieve the purpose of accurately denoising an image. The specific scheme is as follows:
[0005] The first aspect of the present application provides an image denoising method, the method comprising:
[0006] Obtain monitoring images of the target space in a low-light environment;
[0007] Providing the monitoring image to a noise prediction model to obtain a predicted image of the residual noise output by the noise prediction model;
[0008] Obtaining a denoised image corresponding to the monitoring image according to the monitoring image and the predicted image;
[0009] Among them, the noise prediction model includes at least a constrained convolution layer and multiple post-convolution layers, the constrained convolution layer is used to obtain the noise feature map of the monitoring image in a low dimension, the post-convolution layers are connected in sequence, and the post-convolution layer is used to process the monitoring image and the noise feature map output by the constrained convolution layer to obtain the predicted image.
[0010] In a possible implementation, the post-convolution layer includes at least: a hole convolution layer and other convolution layers;
[0011] The atrous convolution layer is used to perform atrous convolution processing on the monitoring image and the noise feature map output by the constrained convolution layer, so that the receptive field of the noise feature map output by the atrous convolution layer is enlarged;
[0012] The other convolutional layers are used to process the noise feature map output by the hole convolutional layer to obtain the predicted image.
[0013] In a possible implementation, the other convolutional layers include a plurality of mixed hole convolution modules and a convolution processing module;
[0014] Among them, each of the mixed void convolution modules is connected in series, and each of the mixed void convolution modules includes a first mixed void convolution layer, a second mixed void convolution layer, and a third mixed void convolution layer. The first mixed void convolution layer, the second mixed void convolution layer, and the third mixed void convolution layer are connected in series, and the expansion rate of the first mixed void convolution layer is 1; the expansion rate of the second mixed void convolution layer is 2; the expansion rate of the third mixed void convolution layer is 3; each mixed void convolution layer in each of the mixed void convolution modules performs convolution processing on the input noise feature map in turn;
[0015] The convolution processing module is used to process the noise feature map output by the third mixed void convolution layer in the last mixed void convolution module to obtain the predicted image.
[0016] In a possible implementation, a first activation layer is disposed after the hole convolution layer in the post-convolution layer; the first activation layer is used to activate the noise feature map output by the hole convolution layer in the post-convolution layer;
[0017] A batch normalization layer and a second activation layer are respectively deployed after each third mixed void convolution layer in the other convolution layers; the batch normalization layer is used to regularize the noise feature map output by the corresponding third mixed void convolution layer; the second activation layer is used to activate the noise feature map output by the corresponding batch normalization layer.
[0018] In a possible implementation, a deactivation layer is respectively deployed after the second activation layer after each third mixed hole convolution layer in the other convolution layers;
[0019] The deactivation layer is used to perform deactivation processing on the noise feature map output by the corresponding second activation layer.
[0020] In a possible implementation, the convolution processing module includes: a neural network convolution layer, a gradient processing module;
[0021] The neural network convolution layer is used to process the noise feature map output by the third mixed void convolution layer in the last mixed void convolution module;
[0022] The gradient processing module is used to add the noise feature map output by the neural network convolution layer and the monitoring image, and then perform activation processing on the added image to obtain the predicted image.
[0023] A second aspect of the present application provides a noise prediction model, including:
[0024] An input layer, used to receive an input monitoring image, wherein the monitoring image is an image of a target space in a low-brightness environment;
[0025] A constrained convolution layer, used to obtain a low-dimensional noise feature map of the monitoring image;
[0026] A plurality of post-convolutional layers, the post-convolutional layers are connected in sequence, and the post-convolutional layers are used to process the monitoring image and the noise feature map output by the constrained convolutional layer to obtain a predicted image of the residual noise;
[0027] The output layer is used to output the predicted image, and the predicted image is used to obtain a denoised image corresponding to the monitored image based on the monitored image.
[0028] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the image denoising method of the first aspect or any implementation of the first aspect.
[0029] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0030] The memory is used to store computer programs;
[0031] The processor is used to execute the computer program so that the electronic device can implement the image denoising method of the first aspect or any implementation manner of the first aspect.
[0032] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the image denoising method of the first aspect or any implementation of the first aspect.
[0033] By means of the above-mentioned technical scheme, in an image denoising method, a noise prediction model and an electronic device provided by the present application, by deploying a constrained convolution layer in the noise prediction model, when the noise prediction model performs noise prediction on a surveillance image in a low-brightness environment, the constrained convolution layer obtains a low-dimensional noise feature map and provides it to the post-convolution layer. Thus, the post-convolution layer can perform deep learning of noise features for the noise feature image with low-dimensional features, and can better learn the residual noise features. Moreover, when the constrained convolution layer obtains the low-dimensional noise feature map, it can suppress the detail information of the image itself, so that the post-convolution layer can perform deep learning of the detail information of the image as residual noise as possible, so that when the denoised image is obtained using the predicted image obtained by the post-convolution layer, the important details of the image can be retained in the denoised image and will not be smoothed out, thereby improving the accuracy of image denoising. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0035] Figure 1 A schematic diagram of a process of an image denoising method provided in an embodiment of the present application;
[0036] Figure 2 This is a model architecture diagram of the noise prediction model in the embodiment of the present application;
[0037] Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 They are respectively another model architecture diagram of the noise prediction model in the embodiment of the present application;
[0038] Fig. 9 A model architecture diagram of a noise prediction model provided in an embodiment of the present application;
[0039] Fig.10 , Fig.11 , Fig.12 , Fig.13 , Fig.14 , Fig.15 Another model architecture diagram of a noise prediction model provided in an embodiment of the present application;
[0040] Fig.16 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0041] Fig.17 This is the network architecture diagram of the original DnCNN model;
[0042] Fig.18 This is a network architecture diagram of the Constrained-HDC-DnCNN model proposed in this application. DETAILED DESCRIPTION
[0043] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0044] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0045] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0046] The present application can be applied in the field of image processing. The following introduces the present application by taking the nighttime surveillance image of a bank as an example.
[0047] Reference Figure 1 , is a flow chart of an image denoising method provided in an embodiment of the present application, which can be applied to electronic devices that can run a noise prediction model or are connected to a device where a noise prediction model is located, such as a mobile phone, a pad, a notebook or a server. The technical solution in this embodiment is mainly used to improve the accuracy of image denoising.
[0048] Specifically, the method in this embodiment may include the following steps:
[0049] Step 101: Obtain a monitoring image of a target space in a low-brightness environment.
[0050] The target space may be a space that needs to be monitored securely, such as the indoor space of a bank branch or the space where an ATM is located. A low-brightness environment refers to an environment where the ambient brightness is lower than a corresponding threshold, such as a night environment or a closed environment.
[0051] Specifically, the monitoring image in this embodiment can be obtained from an image acquisition device deployed in the target space. For example, a camera is deployed in a bank branch, and the camera collects monitoring images for the bank branch. The electronic device in this embodiment obtains the monitoring image collected by the camera.
[0052] Step 102: providing the monitoring image to the noise prediction model to obtain a predicted image of the residual noise output by the noise prediction model.
[0053] Among them, the noise prediction model at least includes a constrained convolution layer and multiple post-convolution layers, such as Figure 2 As shown in , the post-convolution layer is deployed after the constrained convolution layer, and the post-convolution layers are connected in sequence. The constrained convolution layer is used to obtain the noise feature map of the monitoring image in a low dimension, and the post-convolution layer is used to process the monitoring image and the noise feature map output by the constrained convolution layer to obtain the predicted image.
[0054] In the specific implementation, the noise prediction model also includes an image merging module, such as Figure 3 As shown in , the image merging module is deployed between the constrained convolution layer and the post-convolution layer. The image merging module is used to merge the monitoring image with the noise feature image output by the constrained convolution layer to obtain an integrated feature map, and then provide the integrated feature map to the post-convolution layer, which processes the integrated feature map to obtain a predicted image.
[0055] Step 103: Obtain a denoised image corresponding to the monitoring image according to the monitoring image and the predicted image.
[0056] Specifically, in step 103, the monitoring image and the predicted image can be subtracted, that is, the monitoring image is subtracted from the predicted image. For example, for each pixel point, the pixel value of the pixel point in the monitoring image is subtracted from the pixel value of the corresponding pixel point in the predicted image to obtain the pixel value of the pixel point after the residual noise is removed, thereby obtaining a denoised image of the target space with the residual noise removed, and the denoised image is closer to the original clean image.
[0057] By means of the above-mentioned technical scheme, in an image denoising method provided by the present application, by deploying a constrained convolution layer in a noise prediction model, when the noise prediction model performs noise prediction on a surveillance image in a low-brightness environment, the constrained convolution layer obtains a low-dimensional noise feature map and provides it to a post-convolution layer. Thus, the post-convolution layer can perform deep learning of noise features for the noise feature image with low-dimensional features, and can better learn the residual noise features. Moreover, when obtaining the low-dimensional noise feature map, the constrained convolution layer can suppress the detail information of the image itself, so that the post-convolution layer can perform deep learning of the detail information of the image as residual noise as little as possible. Therefore, when the denoised image is obtained using the predicted image obtained by the post-convolution layer, the important details of the image can be retained in the denoised image and will not be smoothed out, thereby improving the accuracy of image denoising.
[0058] In one implementation, the post-convolutional layer includes at least: a dilated convolutional layer and other convolutional layers. The other convolutional layers are deployed after the dilated convolutional layer. The other convolutional layers may have multiple layers, such as Figure 4 as shown in .
[0059] Among them, the dilated convolution layer is used to perform dilated convolution processing on the monitoring image and the noise feature map output by the constrained convolution layer, so that the receptive field of the noise feature map output by the dilated convolution layer is expanded.
[0060] The other convolutional layers are used to process the noise feature map output by the hole convolutional layer to obtain the predicted image.
[0061] It can be seen that in the noise prediction model in this embodiment, the hole convolution layer is first deployed after the constrained convolution layer to replace the ordinary convolution layer through the neural network convolution kernel. Therefore, the expansion rate of the hole convolution layer can be used to effectively expand the receptive field of the convolution operation without adding additional parameters, thereby improving the accuracy of the predicted image.
[0062] Based on the above implementation, other convolutional layers in the post-convolutional layer can include multiple mixed hole convolution modules and a convolution processing module, such as Figure 5 As shown in . Each mixed dilated convolution module is connected in series, and the convolution processing module is deployed after the last mixed dilated convolution module, and each mixed dilated convolution module contains the first mixed dilated convolution layer, the second mixed dilated convolution layer, and the third mixed dilated convolution layer. Here, the first mixed dilated convolution layer, the second mixed dilated convolution layer, and the third mixed dilated convolution layer are connected in series, so that all mixed dilated convolution layers in other convolution layers are connected in series.
[0063] Specifically, the expansion rate of the first mixed atrous convolution layer is 1; the expansion rate of the second mixed atrous convolution layer is 2; and the expansion rate of the third mixed atrous convolution layer is 3. Based on this, each mixed atrous convolution layer in each mixed atrous convolution module performs convolution processing on the input noise feature map in turn, which can ensure the continuity of noise feature transmission. Moreover, the mixed atrous convolution layer in the mixed atrous convolution module gradually increases the expansion rate to achieve that the receptive field of the subsequent convolution layer will not have a grid effect, that is, the grid effect caused by the atrous convolution layer after the constrained convolution layer and before other convolution layers can be avoided, and the global information of the image can be obtained.
[0064] Correspondingly, the convolution processing module is used to process the noise feature map output by the third mixed void convolution layer in the last mixed void convolution module to obtain a predicted image.
[0065] Based on the above implementation, in this embodiment, after the hole convolution layer in the post-convolution layer, the first activation layer is deployed, such as Figure 6 As shown in , the first activation layer is used to activate the noise feature map output by the hole convolution layer in the post-convolution layer.
[0066] After each third mixed hole convolution layer in other convolutional layers, a batch standard layer and a second activation layer are deployed respectively. Figure 6 As shown in , the batch standard layer is used to regularize the noise feature map output by the corresponding third mixed void convolution layer, and the second activation layer is used to activate the noise feature map output by the corresponding batch standard layer.
[0067] Furthermore, in this embodiment, after the second activation layer after each third mixed hole convolution layer in other convolution layers, a deactivation layer is respectively deployed, such as Figure 7 as shown in .
[0068] The deactivation layer is used to deactivate the noise feature map output by the corresponding second activation layer. Therefore, in this embodiment, the deactivation layer can randomly block the propagation of part of the neuron information to avoid overfitting.
[0069] Based on the above implementation, the convolution processing module includes: a neural network convolution layer and a gradient processing module, such as Figure 8 as shown in .
[0070] Among them, the neural network convolution layer is used to process the noise feature map output by the third mixed void convolution layer in the last mixed void convolution module.
[0071] The gradient processing module is used to add the noise feature map output by the convolutional layer of the neural network and the monitoring image, and then activate the added image to obtain a predicted image.
[0072] Therefore, in this embodiment, after the neural network convolution layer is deployed in the noise prediction model for the last convolution processing, the original monitoring image is added to the noise feature map output by the neural network convolution layer through the gradient processing module, and then activated, which can alleviate the gradient disappearance problem caused by the increase in depth in the feature learning process.
[0073] See also Fig. 9 , is a model architecture diagram of a noise prediction model provided in an embodiment of the present application. The noise prediction model can be used to predict residual noise in a surveillance image. Specifically, the noise prediction model can include the following structure:
[0074] The input layer 901 is used to receive an input monitoring image, where the monitoring image is an image of the target space in a low-brightness environment;
[0075] A constrained convolution layer 902, used to obtain a low-dimensional noise feature map of the monitoring image;
[0076] A plurality of post-convolutional layers 903, wherein the post-convolutional layers 903 are connected in sequence, and the post-convolutional layers 903 are used to process the monitoring image and the noise feature map output by the constrained convolutional layer 902 to obtain a predicted image of residual noise;
[0077] The output layer 904 is used to output the predicted image, and the predicted image is used to obtain a denoised image corresponding to the monitored image based on the monitored image.
[0078] By means of the above-mentioned technical scheme, in a noise prediction model provided by the present application, by deploying a constrained convolution layer in the noise prediction model, when the noise prediction model performs noise prediction on a surveillance image in a low-brightness environment, the constrained convolution layer obtains a low-dimensional noise feature map and provides it to the post-convolution layer. Thus, the post-convolution layer can perform deep learning of noise features for the noise feature image with low-dimensional features, and can better learn the residual noise features. Moreover, when the constrained convolution layer obtains the low-dimensional noise feature map, it can suppress the detail information of the image itself, so that the post-convolution layer can perform deep learning of the detail information of the image as residual noise as little as possible, so that when the denoised image is obtained using the predicted image obtained by the post-convolution layer, the important details of the image can be retained in the denoised image and will not be smoothed out, thereby improving the accuracy of image denoising.
[0079] In one implementation, the noise prediction model further includes an image merging module 905, such as Fig.10As shown in , the image merging module 905 is deployed between the constrained convolution layer 902 and the post-convolution layer 903. The image merging module 905 is used to merge the monitoring image with the noise feature image output by the constrained convolution layer 902 to obtain an integrated feature map, and then provide the integrated feature map to the post-convolution layer 903, which processes the integrated feature map to obtain a predicted image.
[0080] In one implementation, the post-convolution layer 903 includes at least a hole convolution layer 931 and other convolution layers 932, such as Fig.11 As shown in;
[0081] The atrous convolution layer 931 is used to perform atrous convolution processing on the monitoring image and the noise feature map output by the constrained convolution layer 902, so that the receptive field of the noise feature map output by the atrous convolution layer 931 is enlarged;
[0082] The other convolutional layer 932 is used to process the noise feature map output by the hole convolutional layer to obtain the predicted image.
[0083] In one implementation, the other convolution layer 932 includes a plurality of mixed hole convolution modules 933 and a convolution processing module 934, such as Fig.12 As shown in;
[0084] Among them, each of the mixed hole convolution modules 933 is connected in series, and the convolution processing module 934 is deployed after the last mixed hole convolution module 933. Each of the mixed hole convolution modules 933 includes a first mixed hole convolution layer 1201, a second mixed hole convolution layer 1202, and a third mixed hole convolution layer 1203. The first mixed hole convolution layer 1201, the second mixed hole convolution layer 1202, and the third mixed hole convolution layer 1203 are connected in series, and the expansion rate of the first mixed hole convolution layer 1201 is 1; the expansion rate of the second mixed hole convolution layer 1202 is 2; the expansion rate of the third mixed hole convolution layer 1203 is 3; each mixed hole convolution layer in each of the mixed hole convolution modules 933 performs convolution processing on the input noise feature map in turn;
[0085] The convolution processing module 934 is used to process the noise feature map output by the third mixed void convolution layer 1203 in the last mixed void convolution module 933 to obtain the predicted image.
[0086] In one implementation, a first activation layer 935 is disposed after the hole convolution layer 931 in the post-convolution layer 903; Fig.13As shown in , the first activation layer 935 is used to activate the noise feature map output by the hole convolution layer 931 in the post-convolution layer 903;
[0087] A batch normalization layer 936 and a second activation layer 937 are respectively deployed after each third mixed void convolution layer 1203 in the other convolution layers 932; the batch normalization layer 936 is used to regularize the noise feature map output by the corresponding third mixed void convolution layer 1203; the second activation layer 937 is used to activate the noise feature map output by the corresponding batch normalization layer 936.
[0088] In one implementation, a deactivation layer 938 is respectively disposed after the second activation layer 937 after each third mixed hole convolution layer 1203 in the other convolution layers 932, such as Fig.14 As shown in;
[0089] The deactivation layer 938 is used to deactivate the noise characteristic graph output by the corresponding second activation layer 937 .
[0090] In one implementation, the convolution processing module 934 includes: a neural network convolution layer 1501, a gradient processing module 1502, such as Fig.15 As shown in;
[0091] The neural network convolution layer 1501 is used to process the noise feature map output by the third mixed void convolution layer 1203 in the last mixed void convolution module 933;
[0092] The gradient processing module 1502 is used to add the noise feature map output by the neural network convolution layer 1501 and the monitoring image, and then perform activation processing on the added image to obtain the predicted image.
[0093] The present application also provides an electronic device in an embodiment. Fig.16 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.16 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0094] like Fig.16As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage device 1608 to a random access memory (RAM) 1603, such as the image denoising method described in any of the above. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 1603. The processing device 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.
[0095] Typically, the following devices may be connected to the I / O interface 1605: an input device 1606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1608 including, for example, a memory card, a hard disk, etc.; and a communication device 1609. The communication device 1609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.16 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0096] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the image denoising method as described in any one of the above.
[0097] An embodiment of the present application further provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0098] The memory is used to store computer programs;
[0099] The processor is used to execute the computer program so that the electronic device can implement any one of the image denoising methods described above.
[0100] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the image denoising method as described in any one of the above.
[0101] Taking the denoising scenario of nighttime surveillance images of bank branches as an example, the technical solution of this application is described in detail below:
[0102] First, since the noise in the surveillance images of bank branches at night is usually additive Gaussian noise, it often appears in the form of low-dimensional features during the training process, and its distribution is regular. Therefore, constrained convolution can be used in this application to improve Fig.17 The denoising convolutional neural network DnCNN (Denoising Convolutional Neural Network, DnCNN) model shown. Fig.17 The DnCNN model shown includes: input layer, first layer network convolution Conv + rectified linear unit ReLU (Rectified Linear Unit), 2nd to 16th layers Conv + batch normalization BN (Batch Normalization) + ReLU, 17th layer Conv, output layer. The input of the input layer is the noise image (Noise Image), and the output of the output layer is the residual noise image (Residual Image). The noise image undergoes 17 layers of network processing from input to output, as follows:
[0103] ① The first layer of the network (Conv+ReLU) uses 64 3×3×c convolution kernels to extract features (c represents the number of channels of NoiseImage) and uses ReLU for activation;
[0104] ② The 2nd to 16th layers (Conv+BN+ReLU) use 64 3×3×64 convolution kernels to extract features, use ReLU for activation, and use Batch Normalization (BN) for regularization;
[0105] ③The 17th layer (Conv) uses c 3×3×64 convolution kernels to output the residual noise image (c represents the number of ResidualImage channels).
[0106] This application is for Fig.17 Adding constrained convolutional layers to the DnCNN model shown has the following advantages:
[0107] ① Use constrained convolution to provide low-dimensional features for deep networks, so that deep networks can better learn residual noise features based on low-dimensional features;
[0108] ② Use constrained convolution to suppress the detail information of the image itself, so that the network can learn as little detail information as possible as residual noise information, thereby retaining more key image details while denoising.
[0109] Secondly, since the DnCNN model has many network layers and the size of the night image dataset of banking outlets used for training is limited, the model is prone to overfitting. This makes it difficult for the model to learn the underlying data feature distribution, and the denoising performance is only good on the training data, that is, the robustness and generalization ability are not strong. For this reason, the deactivation mechanism is added to the connection positions between multiple hidden layers of the network in this application to enhance the robustness and generalization ability.
[0110] Finally, to address the relatively limited receptive field of the DnCNN model, dilated convolution and mixed dilated convolution are used to improve the receptive field of the model.
[0111] In summary, after making the above-mentioned improvements to the original DnCNN model, this application obtains an improved image denoising network model Constrained-HDC-DnCNN (Constrained Convolution-Hybrid DilatedConvolution-DnCNN), whose network structure is as follows: Fig.18 As shown in:
[0112] like Fig.18 As shown in , the Constrained-HDC-DnCNN model makes the following improvements based on the original DnCNN model:
[0113] ① The constrained convolution layer Constrained Conv (Constrained Convolution) is added as the first layer of the network, and after this layer is processed, a jump connection ⊕ (i.e., the image merging module mentioned above) is used to ensure the network's learning degree of low-dimensional features;
[0114] ②Change the ordinary convolution of the next layer to dilated convolution DC (Dilated Convolution) and use ReLU for activation to expand the receptive field;
[0115] ③Except for the first two layers and the last layer of the network, other network layers use hybrid dilated convolution modules HDC (Hybrid Dilated Convolution), each module contains three layers of hybrid dilated convolution layers (the dilation rate is set to 1, 2, and 3 in turn) to avoid the appearance of grid effect. Among them, the hybrid dilated convolution modules are connected in series 10 times (that is, 10 hybrid dilated convolution modules, each of which has three layers of hybrid dilated convolution layers), and adjacent hybrid dilated convolution modules are normalized using BN and activated using ReLU;
[0116] ④ Use the dropout mechanism to randomly block the propagation of some neuron information to avoid overfitting;
[0117] ⑤After the last layer of convolution Conv operation is completed, use the jump connection ⊕, such as Fig.18 The direct connection channel implemented by ⊕ is then activated using ReLU to alleviate the gradient vanishing problem caused by increasing depth.
[0118] It can be seen that in the Constrained-HDC-DnCNN model, the process from input to output is:
[0119] ①Use the constrained convolution layer to process the input image Noise Image (i.e. the surveillance image of the bank branch containing noise at night) and obtain the noise feature map;
[0120] ②Add the noise feature map to the input image Noise Image, and then provide the feature map to the next network layer;
[0121] ③ Deep network learning noise characteristics;
[0122] ④ Output Residual Image (i.e., the predicted image of residual noise at bank outlets at night);
[0123] ⑤ Use the difference between Noise Image and Residual Image to obtain the denoised night surveillance image of the bank branch.
[0124] It can be seen that in terms of the noise removal ability of night-time surveillance video images of banking business outlets, compared with traditional image denoising methods and DnCNN image denoising methods, the Constrained-HDC-DnCNN model proposed in this application can effectively remove the noise of surveillance video images while retaining important detail information of the image as much as possible, which can better improve the quality of night-time surveillance video images of banking business outlets, thereby laying a good foundation for the human eye to judge the image content.
[0125] Moreover, the results of this application can be applied not only to the noise removal of nighttime surveillance video images of banking outlets, but also to other scenarios such as the denoising of nighttime deposit and withdrawal surveillance video images of ATMs and nighttime surveillance video images of data centers. Therefore, the applicability of this application is relatively strong.
[0126] In summary, this application proposes a method for removing noise from nighttime surveillance video images of bank branches. This method is based on deep learning and can effectively remove large-area fine granular noise points that appear on nighttime surveillance video images of bank branches by introducing constrained convolution, deactivation mechanism, mixed hole convolution and other technologies. Compared with traditional image denoising methods and the original DnCNN image denoising method, this application has better denoising performance in nighttime surveillance video images of bank branches.
[0127] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0128] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0129] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0130] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. An image denoising method, characterized in that: The method comprises: Obtain monitoring images of the target space in a low-light environment; Providing the monitoring image to a noise prediction model to obtain a predicted image of the residual noise output by the noise prediction model; Obtaining a denoised image corresponding to the monitoring image according to the monitoring image and the predicted image; Among them, the noise prediction model includes at least a constrained convolution layer and multiple post-convolution layers, the constrained convolution layer is used to obtain the noise feature map of the monitoring image in a low dimension, the post-convolution layers are connected in sequence, and the post-convolution layer is used to process the monitoring image and the noise feature map output by the constrained convolution layer to obtain the predicted image.
2. The method according to claim 1, characterized in that: The post-convolutional layer at least includes: a hole convolutional layer and other convolutional layers; The atrous convolution layer is used to perform atrous convolution processing on the monitoring image and the noise feature map output by the constrained convolution layer, so that the receptive field of the noise feature map output by the atrous convolution layer is enlarged; The other convolutional layers are used to process the noise feature map output by the hole convolutional layer to obtain the predicted image.
3. The method according to claim 2, characterized in that The other convolutional layers include a plurality of mixed hole convolution modules and a convolution processing module; Among them, each of the mixed void convolution modules is connected in series, and each of the mixed void convolution modules includes a first mixed void convolution layer, a second mixed void convolution layer, and a third mixed void convolution layer. The first mixed void convolution layer, the second mixed void convolution layer, and the third mixed void convolution layer are connected in series, and the expansion rate of the first mixed void convolution layer is 1; the expansion rate of the second mixed void convolution layer is 2; the expansion rate of the third mixed void convolution layer is 3; each mixed void convolution layer in each of the mixed void convolution modules performs convolution processing on the input noise feature map in turn; The convolution processing module is used to process the noise feature map output by the third mixed void convolution layer in the last mixed void convolution module to obtain the predicted image.
4. The method according to claim 3, characterized in that A first activation layer is disposed after the hole convolution layer in the post-convolution layer; the first activation layer is used to activate the noise feature map output by the hole convolution layer in the post-convolution layer; A batch normalization layer and a second activation layer are respectively deployed after each third mixed void convolution layer in the other convolution layers; the batch normalization layer is used to regularize the noise feature map output by the corresponding third mixed void convolution layer; the second activation layer is used to activate the noise feature map output by the corresponding batch normalization layer.
5. According to the method of claim 4, a deactivation layer is respectively deployed after the second activation layer after each third mixed hole convolution layer in the other convolution layers; in, The deactivation layer is used to perform deactivation processing on the noise feature map output by the corresponding second activation layer.
6. The method according to claim 3, characterized in that The convolution processing module includes: a neural network convolution layer and a gradient processing module; The neural network convolution layer is used to process the noise feature map output by the third mixed void convolution layer in the last mixed void convolution module; The gradient processing module is used to add the noise feature map output by the neural network convolution layer and the monitoring image, and then perform activation processing on the added image to obtain the predicted image.
7. A noise prediction model, characterized in that: include: An input layer, used to receive an input monitoring image, wherein the monitoring image is an image of a target space in a low-brightness environment; A constrained convolution layer, used to obtain a low-dimensional noise feature map of the monitoring image; A plurality of post-convolutional layers, the post-convolutional layers are connected in sequence, and the post-convolutional layers are used to process the monitoring image and the noise feature map output by the constrained convolutional layer to obtain a predicted image of the residual noise; The output layer is used to output the predicted image, and the predicted image is used to obtain a denoised image corresponding to the monitored image based on the monitored image.
8. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the image denoising method according to any one of claims 1 to 6.
9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the image denoising method according to any one of claims 1 to 6.
10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the image denoising method as described in any one of claims 1 to 6.