Image denoising method, denoising device, mobile terminal and storage medium
Through the pre-trained convolutional neural network model, image denoising processing is performed in the mobile terminal, and the combination of feature layer and denoising layer is used to solve the problem of poor real-time image denoising and distortion in the prior art, achieving high efficiency, Real-time image denoising effect.
Patent Information
- Application Number
- CN202010398928.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-05-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-05-12
AI Technical Summary
The existing image denoising method has poor real-time performance and the processed image distortion cannot effectively remove noise interference, affecting image quality.
The pre-trained convolutional neural network model, including feature layer and denoising layer, is used to denoising the original image input model, and the image details are enhanced through the feature layer extraction of local features and detail enhancement layers, and the denoising layer performs denoising operations. Avoid image distortion and load the model to the mobile terminal for real-time performance.
Real-time image denoising processing and image details are realized, image distortion is avoided, and can efficiently denoising in real-time scenes, and is suitable for a variety of application scenarios.
Smart Images

Figure CN113658050B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and in particular relates to an image denoising method, a denoising device, a mobile terminal and a storage medium. Background Art
[0002] There are more and more mobile terminals with integrated camera devices. As an image sensor for collecting images, the camera device will inevitably have noise interference during the image collection process, which seriously affects the quality of the collected images. Therefore, denoising is very important for collecting high-quality images.
[0003] At present, some image denoising methods rely on a large number of complex algorithms and have poor real-time performance; some image denoising processing methods result in distorted images after denoising due to the simple denoising models designed; therefore, the existing image denoising methods have poor real-time performance and the processed images are relatively distorted. Summary of the invention
[0004] In view of this, the embodiments of the present application provide an image denoising method, a denoising device, a mobile terminal and a computer-readable storage medium to solve the problems of poor real-time performance and distorted images after processing in existing image denoising methods.
[0005] A first aspect of an embodiment of the present application provides a method for denoising an image, comprising:
[0006] Obtain the original image to be denoised;
[0007] The original image is input into a trained convolutional neural network model to obtain a denoised image corresponding to the original image, wherein the convolutional neural network model includes: a denoising layer and a feature layer.
[0008] A second aspect of an embodiment of the present application provides a denoising device, comprising:
[0009] An original image acquisition unit, used for acquiring an original image to be denoised;
[0010] A denoising unit is used to input the original image into a trained convolutional neural network model to obtain a denoised image corresponding to the original image, wherein the convolutional neural network model includes: a denoising layer and a feature layer.
[0011] The third aspect of an embodiment of the present application provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect of the embodiment of the present application when executing the computer program.
[0012] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, it implements the steps of the method provided in the first aspect of the embodiments of the present application.
[0013] The fifth aspect of the embodiments of the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method provided in the first aspect of the embodiments of the present application.
[0014] The embodiment of the present application provides an image denoising method, and sets a convolutional neural network model, which includes: a feature layer and a denoising layer. On the one hand, the convolutional neural network model is pre-trained and then loaded into the camera device. Since the training process with a large amount of calculation is transferred outside the camera device, the real-time denoising process can be guaranteed; on the other hand, the model avoids image distortion through the feature layer, and denoises through the denoising layer, so that image distortion can be avoided while denoising. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0016] Figure 1 It is a schematic diagram of the implementation process of the image denoising method provided by an embodiment of the present application;
[0017] Figure 2 It is a structural schematic diagram of a convolutional neural network model provided in one embodiment of the present application;
[0018] Figure 3 This is provided by an embodiment of the present application Figure 2 Schematic diagram of the structure of the local feature extraction layer in ;
[0019] Figure 4 This is provided by an embodiment of the present application Figure 2 Schematic diagram of the structure of the detail enhancement layer in FIG.
[0020] Figure 5 This is provided by an embodiment of the present application Figure 2 Schematic diagram of the structure of the denoising layer in ;
[0021] Figure 6 This is provided by an embodiment of the present application Figure 2 Schematic diagram of the fusion layer in
[0022] Figure 7 It is a flowchart of the training process of the convolutional neural network model provided in one embodiment of the present application;
[0023] Figure 8 This is a comparison chart of the original image and the denoised image provided by this application;
[0024] Fig. 9 is a schematic block diagram of a denoising device provided in an embodiment of the present application;
[0025] Fig.10 It is a schematic block diagram of another mobile terminal provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0027] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0029] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0030] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0031] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.
[0032] The denoising method involved in the embodiment of the present application is a lightweight image denoising method, and therefore, the real-time performance of the image denoising process can be guaranteed. In the case of good real-time performance, it can be applied to application scenarios that require real-time denoising, for example, the image captured by the camera device can be denoised when previewing the image. Of course, in addition to this application scenario, the embodiment of the present application can also be applied to application scenarios that do not require high real-time performance, and this is not limited here.
[0033] Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image denoising method provided in an embodiment of the present application. As shown in the figure, the method may include the following steps:
[0034] Step S101, obtaining an original image to be denoised.
[0035] In an embodiment of the present application, the original image may be an image captured in real time by a mobile terminal such as a mobile phone. The image captured in real time by a mobile terminal such as a mobile phone is recorded as the original image to be denoised, and the format of the original image may be a RAW format.
[0036] Step S102: input the original image into the trained convolutional neural network model to obtain a denoised image corresponding to the original image, wherein the convolutional neural network model includes: a feature layer and a denoising layer.
[0037] In an embodiment of the present application, the convolutional neural network model is a convolutional neural network model that has been trained using a training sample set.
[0038] As another embodiment of the present application, before the original image is input into the trained convolutional neural network model, the original image may be preprocessed to obtain a preprocessed image. The preprocessing process may be: randomly cropping the RAW format image into a size of 320*320, and further randomly rotating the randomly cropped image for data augmentation.
[0039] On the one hand, during the model training process, random cropping of images as a data expansion method can improve model accuracy and enhance model stability. To ensure the uniformity of input data, the input data of the model denoising stage can also be preprocessed in the training stage corresponding to the input data of the model training stage.
[0040] On the other hand, the randomly cropped image contains local features, and a weight relationship between each feature and the category corresponding to the feature can be established to reduce the weight of the background or noise, thereby enhancing the denoising effect.
[0041] After preprocessing the original image, the preprocessed image can be input into the trained convolutional neural network model.
[0042] The convolutional neural network model can be a series structure model, which includes a feature layer and a denoising layer.
[0043] The convolutional neural network model can also be a parallel structure model, which includes a feature layer and a denoising layer.
[0044] As another embodiment of the present application, the feature layer includes a local feature extraction layer and / or a detail enhancement layer.
[0045] As another embodiment of the present application, the feature layer can be a series structure of the local feature extraction layer and the detail enhancement layer, or a parallel structure of the local feature extraction layer and the detail enhancement layer. Of course, it can also be one of the local feature extraction layer and the detail enhancement layer.
[0046] In the embodiment of the present application, the series structure of the convolutional neural network model may include the following forms:
[0047] (1) A series structure consisting of the local feature extraction layer and the denoising layer, for example, the local feature extraction layer receives an input image, the output of the local feature extraction layer is sent to the denoising layer, and the output of the denoising layer is used as a denoised image;
[0048] (2) A series structure consisting of the detail enhancement layer and the denoising layer, for example, the detail enhancement layer receives an input image, the output of the detail enhancement layer is sent to the denoising layer, and the output of the denoising layer is used as a denoised image;
[0049] (3) A series structure consisting of the local feature extraction layer, the detail enhancement layer and the denoising layer, for example, a series structure consisting of the local feature extraction layer, the detail enhancement layer and the denoising layer in sequence, wherein the output of the previous stage in the series structure serves as the input of the next stage.
[0050] (4) The local feature extraction layer and the detail enhancement layer are connected in parallel and then connected to the denoising layer to form a series structure. For example, the local feature extraction layer and the detail enhancement layer can both receive an input image, and the output of the local feature extraction layer and the output of the detail enhancement layer are both sent to the denoising layer, and the output of the denoising layer is used as a denoised image;
[0051] In a serial structure, the order of the levels can be reversed.
[0052] The parallel structure of the convolutional neural network model may include the following forms:
[0053] (1) two parallel structure layers consisting of the local feature extraction layer and the denoising layer, and a fusion layer connected to the output of the local feature extraction layer and the output of the denoising layer respectively;
[0054] (2) two parallel structure layers consisting of the detail enhancement layer and the denoising layer, and a fusion layer connected to the output of the detail enhancement layer and the output of the denoising layer respectively;
[0055] (3) A three-layer parallel structure layer consisting of the local feature extraction layer, the detail enhancement layer and the denoising layer, and a fusion layer connected to the output of the local feature extraction layer, the output of the detail enhancement layer and the output of the denoising layer respectively (see Figure 2 Schematic diagram of the structure of the convolutional neural network model shown in Figure ).
[0056] (4) The series structure layer composed of the local feature extraction layer and the detail enhancement layer and the denoising layer further form two parallel structure layers, and the series structure layer and the denoising layer are respectively connected to the fusion layer. For example, the input image is input to the local feature extraction layer, the output of the local feature extraction layer is input to the detail enhancement layer, the output of the detail enhancement layer is input to the fusion layer, the original image is input to the denoising layer, the output of the denoising layer is input to the fusion layer, and the fusion layer fuses the output of the detail enhancement layer and the input of the denoising layer to output a denoised image.
[0057] It should be noted that the convolutional neural network model may have other variations, which are not listed here one by one.
[0058] Figure 2 A structural schematic diagram of a convolutional neural network model provided for an embodiment of the present application, as shown in the figure, is composed of a local feature extraction (local_net) layer, a detail enhancement layer (middle_net) and a denoising layer (global_net) to form a three-layer parallel structure layer, and the input of the fusion layer is respectively connected to the output of the local feature extraction (local_net) layer, the output of the detail enhancement layer (middle_net) and the output of the denoising layer (global_net).
[0059] The original image is input into the local feature extraction layer, the detail enhancement layer and the denoising layer respectively; the local feature extraction layer, the detail enhancement layer and the denoising layer process the original image respectively and then output them to the fusion layer; the fusion layer fuses the output of the local feature extraction layer, the output of the detail enhancement layer and the output of the denoising layer and outputs them as a denoised image.
[0060] The input image of the convolutional neural network is an image in RAW format, and the output image is also an image in RAW format. Among them, the local feature extraction layer is mainly responsible for local feature extraction, the detail enhancement layer is mainly used to expand the convolution to increase the pooling field of view, and the denoising layer is mainly used for denoising.
[0061] As another embodiment of the present application, a skip connection is provided between the input and the output.
[0062] Figure 3 : is a schematic diagram of the structure of a local feature extraction layer provided in an embodiment of the present application. As shown in the figure, the local feature extraction layer includes:
[0063] Two two-dimensional convolutional layers (conv2d) are connected in sequence, wherein the size of the convolution kernel k of the two-dimensional convolutional layer is 3*3; the features of the input image of the first two-dimensional convolutional layer are 48, and the features of the input image of the second two-dimensional convolutional layer are 96.
[0064] Figure 4 This is a schematic diagram of the structure of the detail enhancement layer provided by an embodiment of the present application. As shown in the figure, the detail enhancement layer includes: four two-dimensional atrous convolution layers (atrous_conv2d) connected in sequence, which can also be called two-dimensional dilated convolution layers. Among them, the input image of the two-dimensional atrous convolution layer has 48 features, the size of the convolution kernel is 3*3, the dilation rate of the first two-dimensional atrous convolution layer and the second two-dimensional atrous convolution layer when convolving on the input image is 2, and the dilation rate of the third two-dimensional atrous convolution layer and the fourth two-dimensional atrous convolution layer when convolving on the input image is 4.
[0065] Figure 5 is a schematic diagram of the structure of a denoising layer provided in an embodiment of the present application. As shown in the figure, the denoising layer is a DDFN architecture. The DDFN structure includes at least one denoising submodule;
[0066] The denoising submodule includes: a first two-dimensional convolutional layer and a first two-dimensional dilated convolutional layer connected in sequence, and a second two-dimensional dilated convolutional layer and a second two-dimensional convolutional layer connected in sequence;
[0067] The input data of the denoising submodule is input into the first two-dimensional convolutional layer to obtain the first result, and the first result is input into the first two-dimensional dilated convolutional layer to obtain the second result; the input data of the denoising submodule is input into the second two-dimensional dilated convolutional layer to obtain the third result, and the third result is input into the second two-dimensional convolutional layer to obtain the fourth result; the first result, the second result, the third result and the fourth result are fused as the output data of the denoising submodule. It should be noted that the two-dimensional convolutional layer for obtaining the first result and the fourth result can be the same or different, and the two-dimensional dilated convolutional layer for obtaining the second result and the third result can be the same or different.
[0068] In order to obtain a better denoising effect, the number of the denoising submodules may be at least two.
[0069] In the figure, “C” represents ordinary convolution, “D” represents dilated convolution, and “+” represents channel merging (concatenate) operation.
[0070] Figure 6 : is a schematic diagram of the structure of a fusion layer provided in an embodiment of the present application. As shown in the figure, the fusion layer includes:
[0071] The third two-dimensional convolution layer, the fourth two-dimensional convolution layer and the fifth two-dimensional convolution layer, and an attention mechanism module (CBAM) is also provided between the fourth two-dimensional convolution layer and the fifth two-dimensional convolution layer, wherein the input image of the third two-dimensional convolution layer has 48 features and a convolution kernel size of 3, the input image of the fourth two-dimensional convolution layer has 48 features and a convolution kernel size of 1, and the input image of the fifth two-dimensional convolution layer has 1 feature and a convolution kernel size of 1.
[0072] In the embodiment of the present application, the convolutional neural network model is a lightweight end-to-end network. After the convolutional neural network model is trained and the parameters are solidified, it is loaded into the mobile terminal. During the denoising process, it will not cause excessive occupation of the memory of the mobile terminal, and the denoising process time is short, which can ensure real-time performance. And due to the existence of the local feature extraction layer and the detail enhancement layer, the denoised image can retain more details and avoid distortion of the denoised image.
[0073] Figure 7 FIG. 1 is a flow chart of a training process of a convolutional neural network model provided in an embodiment of the present application. As shown in the figure, the training process includes:
[0074] Step S701, obtaining a training sample set.
[0075] In the embodiment of the present application, the training sample set may be some existing image test sets adopted, or may be a sample set generated for the application scenario of the embodiment of the present application. Taking the sample set generated for the application scenario of the embodiment of the present application as an example, obtaining the training sample set includes:
[0076] Collecting images in multiple scenes by using a camera device, wherein multiple images corresponding to exposure times are required to be collected in each scene;
[0077] The SIDD method is used to generate a training sample set from the collected images.
[0078] In the embodiment of the present application, the camera device can be a mobile phone or a mobile terminal to which the denoising method of the present application is applied. Multiple scenes can be set. In each scene, multiple exposure times are used to collect images respectively. The collected images corresponding to the multiple exposure times in the multiple scenes are respectively used to generate a training sample set using the SIDD method. For example, 148 scenes can be used, and 150 frames of images can be collected in each scene.
[0079] The SIDD method is as follows: remove the erroneous pixels and erroneous images from the collected images, then align the collected images, and finally generate a "noise-free" real image, where the real image corresponds to the collected images one by one. The erroneous pixels will have a great impact on the estimation of the real data, so the erroneous pixels need to be corrected first. The specific correction process is: obtain the average image of the collected images taken in the same scene, analyze the erroneous pixels on the average image, and finally correct the erroneous pixels by interpolation.
[0080] Although the collected images are all taken under controlled external environment, the brightness of the collected images will still be slightly different due to hardware factors such as scene brightness and camera exposure. Therefore, the average brightness of the collected images is calculated to eliminate the collected images with large brightness differences, thereby ensuring the stability of the brightness of the collected images.
[0081] Since the optical image stabilization (OIS) function of smartphone cameras cannot be disabled, the captured images cannot be completely aligned. These captured images can be aligned by using reference points.
[0082] After removing erroneous pixels, erroneous images and aligning the images, a denoised real image corresponding to each acquired image can be obtained.
[0083] Step S702: preprocess the sample images in the training sample set to obtain preprocessed sample images.
[0084] In the embodiment of the present application, the sample images in the training sample set need to be preprocessed, and the preprocessing method of the sample images during the training process is consistent with the preprocessing method of the original images during the denoising process. The preprocessing process of the original images during the denoising process can be referred to, and will not be repeated here.
[0085] Step S703: training the constructed convolutional neural network model by using the preprocessed sample images to obtain a trained convolutional neural network model.
[0086] In the embodiment of the present application, the training process adopts the back propagation method, and the loss function during training is: a combination of the L1 loss function and the SSIM loss function, and the combination method can be a weighted sum. The loss function is:
[0087] L total =w1*L1+w2*SSIM_LOSS
[0088] in, SSIM_LOSS = 1-SSIM, w1 and w2 are constants, c1 and c2 are constants, p(x) is the pixel of the sample image, p(y) is the pixel of the predicted image, μ x Represents the mean value of the sample image pixels, μ y represents the mean value of the predicted image pixels, δ x represents the variance of sample image pixels, δ y represents the variance of the predicted image pixels, 6 xy Represents the covariance of the pixels of the sample image and the predicted image.
[0089] In practical applications, w1 and w2 can be set to 0.15 and 0.85 respectively. The training process can be performed on GTX1080TI. The parameter initialization method is Xavier initialization, and the initial learning rate is 1e. -4 ,The optimizer is selected as Adam, the learning rate adopts polynomial decay, the number of iterations is 6000 epochs, and the batch size is 1 until the model converges.
[0090] Step S704, using the TensorFlow framework to freeze the parameters in the trained convolutional neural network model to obtain a frozen convolutional neural network model.
[0091] In the embodiment of the present application, after freezing the parameters of the trained convolutional neural network model using the TensorFlow framework, a frozen model PB file is obtained, and the fixed input size of the frozen model PB file is 3000*4000. Then, the MACE framework is used to transplant the frozen model PB file to the mobile terminal.
[0092] In the training process provided in the embodiment of the present application, the training sample set collects multiple original images corresponding to multiple scenes and multiple exposure times through a mobile phone, so that the training sample set has a wide coverage and the trained convolutional neural network model has good robustness; in the selection of the loss function, L1+SSIM_LOSS is used as the loss function, which can effectively ensure the clarity and details of the generated image.
[0093] Figure 8 The figure shows a comparison of the effects of an original image captured by a mobile phone and a denoised image denoised by the denoising method provided in an embodiment of the present application, wherein the left side of the figure shows the original image captured by the mobile phone, in which it can be seen that a lot of noise is mixed in, and the right side shows the denoised image, in which the noise in the original image is well removed and the detailed information in the image (such as the text part) is fully retained.
[0094] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0095] Fig. 9 Detailed description is given of a denoising device according to an embodiment of the present application. For ease of explanation, only the part related to the embodiment of the present application is shown.
[0096] The denoising device 9 can be a software unit, a hardware unit or a combination of software and hardware units built into a mobile terminal such as a mobile phone, a tablet computer, a notebook, a computer, etc., or can be integrated into the mobile terminal such as a mobile phone, a tablet computer, a notebook, a computer, etc. as an independent pendant.
[0097] The denoising device 9 comprises:
[0098] The original image acquisition unit 91 is used to acquire the original image to be denoised;
[0099] The denoising unit 93 is used to input the original image into the trained convolutional neural network model to obtain a denoised image corresponding to the original image, wherein the convolutional neural network model includes: a denoising layer and a feature layer.
[0100] As another embodiment of the present application, the denoising device further includes:
[0101] A preprocessing unit 92, used for preprocessing the original image to obtain a preprocessed image;
[0102] Accordingly, the denoising unit 93 is further used for:
[0103] The preprocessed image is input into the trained convolutional neural network model to obtain a denoised image corresponding to the original image.
[0104] As another embodiment of the present application, the denoising device 9 further includes:
[0105] A training unit 94, used to obtain a training sample set;
[0106] Preprocessing the sample images in the training sample set to obtain preprocessed sample images;
[0107] The constructed convolutional neural network model is trained by using the preprocessed sample images to obtain a trained convolutional neural network model.
[0108] As another embodiment of the present application, the training unit 94 is further configured to:
[0109] Acquire captured images under multiple scenes, wherein the captured images under each scene are captured images corresponding to multiple exposure times respectively;
[0110] The SIDD method is used to generate a training sample set from the collected images.
[0111] As another embodiment of the present application, the training unit 94 is further configured to:
[0112] After obtaining the trained convolutional neural network model, using the TensorFlow framework to freeze parameters in the trained convolutional neural network model to obtain a frozen convolutional neural network model;
[0113] As another embodiment of the present application, the local feature extraction layer includes: two two-dimensional convolutional layers connected in sequence;
[0114] The detail enhancement layer includes: four two-dimensional dilated convolutional layers connected in sequence;
[0115] The denoising layer comprises: at least one denoising submodule;
[0116] The denoising submodule includes: a first two-dimensional convolutional layer and a first two-dimensional dilated convolutional layer connected in sequence, and a second two-dimensional dilated convolutional layer and a second two-dimensional convolutional layer connected in sequence;
[0117] The input data of the denoising submodule is input into the first two-dimensional convolutional layer to obtain a first result, and the first result is input into the first two-dimensional expanded convolutional layer to obtain a second result; the input data of the denoising submodule is input into the second two-dimensional expanded convolutional layer to obtain a third result, and the third result is input into the second two-dimensional convolutional layer to obtain a fourth result; the first result, the second result, the third result and the fourth result are fused as the output data of the denoising submodule.
[0118] The fusion layer includes: an attention mechanism module.
[0119] As another embodiment of the present application, a jump connection is provided between the input and output of the convolutional neural network model.
[0120] As another embodiment of the present application, the loss function of the convolutional neural network model includes:
[0121] A combination of the L1 loss function and the SSIM loss function.
[0122] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the denoising device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned denoising device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0123] Fig.10 is a schematic block diagram of a mobile terminal provided by another embodiment of the present application. Fig.10 As shown, the mobile terminal 10 of this embodiment includes: one or more processors 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, the steps in the above-mentioned various method embodiments are implemented, for example Figure 1 Alternatively, when the processor 100 executes the computer program 102, the functions of each module / unit in the above-mentioned denoising device embodiment are realized, for example Fig. 9 The functions of modules 91 to 93 are shown.
[0124] Exemplarily, the computer program 102 may be divided into one or more modules / units, which are stored in the memory 101 and executed by the processor 100 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 102 in the mobile terminal 10. For example, the computer program 102 may be divided into an original image acquisition unit and a denoising unit. Exemplarily:
[0125] The original image acquisition unit is used to acquire the original image to be denoised;
[0126] The denoising unit is used to input the original image into a trained convolutional neural network model to obtain a denoised image corresponding to the original image, wherein the convolutional neural network model includes: a denoising layer and a feature layer.
[0127] Other units or modules can refer to Fig. 9 The description in the illustrated embodiment will not be repeated here.
[0128] The mobile terminal includes but is not limited to a processor 100 and a memory 101. Those skilled in the art will appreciate that Fig.10 It is only an example of the mobile terminal 10 and does not constitute a limitation of the mobile terminal 10. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the mobile terminal 10 may also include an input device, an output device, a network access device, a bus, etc.
[0129] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0130] The memory 101 may be an internal storage unit of the mobile terminal 10, such as a hard disk or memory of the mobile terminal 10. The memory 101 may also be an external storage device of the mobile terminal 10, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the mobile terminal 10. Further, the memory 101 may also include both an internal storage unit of the mobile terminal 10 and an external storage device. The memory 101 is used to store the computer program and other programs and data required by the mobile terminal 10. The memory 101 may also be used to temporarily store data that has been output or is to be output.
[0131] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0133] In the embodiments provided in the present application, it should be understood that the disclosed mobile terminal, device and method can be implemented in other ways. For example, the mobile terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0136] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by one or more processors. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0137] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for denoising an image, characterized in that: include: Obtain the original image to be denoised; Inputting the original image into the trained convolutional neural network model to obtain a denoised image corresponding to the original image; The convolutional neural network model includes: a local feature extraction layer, a detail enhancement layer and a denoising layer in parallel structure, and a fusion layer connected to the output of the local feature extraction layer, the output of the detail enhancement layer and the output of the denoising layer, the original image is respectively input into the local feature extraction layer, the detail enhancement layer and the denoising layer; the local feature extraction layer, the detail enhancement layer and the denoising layer respectively process the original image and output it to the fusion layer; the fusion layer fuses the output of the local feature extraction layer, the output of the detail enhancement layer and the output of the denoising layer and outputs it as a denoised image; The local feature extraction layer includes: two two-dimensional convolutional layers connected in sequence; The detail enhancement layer includes: four two-dimensional dilated convolutional layers connected in sequence, each of which has a dilation rate, and there are two-dimensional dilated convolutional layers with different dilation rates in the four two-dimensional dilated convolutional layers; The denoising layer comprises: at least one denoising submodule; The denoising submodule includes: a first two-dimensional convolutional layer and a first two-dimensional dilated convolutional layer connected in sequence, and a second two-dimensional dilated convolutional layer and a second two-dimensional convolutional layer connected in sequence; The input data of the denoising submodule is input into the first two-dimensional convolutional layer to obtain a first result, and the first result is input into the first two-dimensional dilated convolutional layer to obtain a second result; the input data of the denoising submodule is input into the second two-dimensional dilated convolutional layer to obtain a third result, and the third result is input into the second two-dimensional convolutional layer to obtain a fourth result; the first result, the second result, the third result and the fourth result are fused as the output data of the denoising submodule; The fusion layer includes: a third two-dimensional convolution layer, a fourth two-dimensional convolution layer and a fifth two-dimensional convolution layer, and an attention mechanism module is also provided between the fourth two-dimensional convolution layer and the fifth two-dimensional convolution layer.
2. The image denoising method according to claim 1, wherein: Before inputting the original image into the trained convolutional neural network model to obtain a denoised image corresponding to the original image, the method further includes: Obtain a training sample set; Preprocessing the sample images in the training sample set to obtain preprocessed sample images; The constructed convolutional neural network model is trained by using the preprocessed sample images to obtain a trained convolutional neural network model.
3. The image denoising method according to claim 2, wherein: The obtaining of the training sample set comprises: Acquire captured images under multiple scenes, wherein the captured images under each scene are captured images corresponding to multiple exposure times respectively; The collected images are used to generate a training sample set.
4. The image denoising method according to claim 1, wherein: After obtaining the trained convolutional neural network model, it also includes: Using the TensorFlow framework to freeze parameters in the trained convolutional neural network model to obtain a frozen convolutional neural network model; Correspondingly, the original image is input into the trained convolutional neural network model to obtain the denoised image corresponding to the original image: Preprocessing the original image to obtain a preprocessed original image; The preprocessed original image is input into the frozen convolutional neural network model to obtain a denoised image corresponding to the original image.
5. The image denoising method according to any one of claims 1 to 4, characterized in that: A jump connection is provided between the input and output of the convolutional neural network model.
6. The image denoising method according to any one of claims 2 to 4, characterized in that: The loss function of the convolutional neural network model includes: A combination of the ∑i=1 loss function and the SSIM loss function.
7. The image denoising method according to claim 1, wherein: Before the original image is input into the trained convolutional neural network model, the method further includes: Preprocessing the original image to obtain a preprocessed image; Accordingly, inputting the original image into the trained convolutional neural network model includes: The preprocessed image is input into the trained convolutional neural network model.
8. A denoising device, characterized in that: include: An original image acquisition unit, used for acquiring an original image to be denoised; A denoising unit, used for inputting the original image into the trained convolutional neural network model to obtain a denoised image corresponding to the original image; The convolutional neural network model includes: a local feature extraction layer, a detail enhancement layer and a denoising layer in parallel structure, and a fusion layer connected to the output of the local feature extraction layer, the output of the detail enhancement layer and the output of the denoising layer, the original image is respectively input into the local feature extraction layer, the detail enhancement layer and the denoising layer; the local feature extraction layer, the detail enhancement layer and the denoising layer respectively process the original image and output it to the fusion layer; the fusion layer fuses the output of the local feature extraction layer, the output of the detail enhancement layer and the output of the denoising layer and outputs it as a denoised image; The local feature extraction layer includes: two two-dimensional convolutional layers connected in sequence; The detail enhancement layer includes: four two-dimensional dilated convolutional layers connected in sequence, each of which has a dilation rate, and there are two-dimensional dilated convolutional layers with different dilation rates in the four two-dimensional dilated convolutional layers; The denoising layer comprises: at least one denoising submodule; The denoising submodule includes: a first two-dimensional convolutional layer and a first two-dimensional dilated convolutional layer connected in sequence, and a second two-dimensional dilated convolutional layer and a second two-dimensional convolutional layer connected in sequence; The input data of the denoising submodule is input into the first two-dimensional convolutional layer to obtain a first result, and the first result is input into the first two-dimensional dilated convolutional layer to obtain a second result; the input data of the denoising submodule is input into the second two-dimensional dilated convolutional layer to obtain a third result, and the third result is input into the second two-dimensional convolutional layer to obtain a fourth result; the first result, the second result, the third result and the fourth result are fused as the output data of the denoising submodule; The fusion layer includes: a third two-dimensional convolution layer, a fourth two-dimensional convolution layer and a fifth two-dimensional convolution layer, and an attention mechanism module is also provided between the fourth two-dimensional convolution layer and the fifth two-dimensional convolution layer.
9. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Compressed low-resolution image restoration method based on combined deep network
CN106709875A
Method for denoising structure magnetic resonance image based on 3D fully convolutional neural network
CN107633486A
A low-light image enhancement method and apparatus
CN109087269A
An image denoising method based on depth learning and multi-scale image enhancement
CN109410127A
An image coloring system and method based on a convolutional neural network
CN109920012A
