An image denoising method, device and storage medium

Abstract

Embodiments of the present application disclose an image denoising method and device and a storage medium. The method comprises: using a preset sampling data set to perform neural structure search on a preset U-shaped neural network, searching out a first down-sampling unit and a first up-sampling unit that satisfy a preset convergence condition; performing operator screening on the first down-sampling unit and the first up-sampling unit based on the architecture parameter change trend of the in-unit operator during the search, to obtain a second down-sampling unit and a second up-sampling unit; generating a non-blind denoising sub-network based on the second down-sampling unit and the second up-sampling unit; obtaining a noise estimation sub-network, and using the noise estimation sub-network and the non-blind denoising sub-network to perform denoising processing on a to-be-denoised image, to obtain a denoised image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image denoising method, apparatus and storage medium. Background Technology

[0002] Digital images in reality are often affected by noise interference from imaging equipment and the external environment during digitization and transmission, thus containing noise. Therefore, image denoising is necessary to obtain accurate images. Image denoising refers to the process of reducing noise in digital images.

[0003] Currently, image denoising is usually performed using manually designed network models. Model generation not only relies on a large amount of expert knowledge and human intervention, resulting in high costs, but also makes it difficult to guarantee the denoising effect of the model, leading to poor image denoising results. Summary of the Invention

[0004] This application provides an image denoising method, apparatus, and storage medium. Based on the evolution trend of architectural parameters, the sampling unit operators of the neural structure search are screened to obtain sampling units with better performance to construct a network for image denoising, thereby improving the image denoising effect.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image denoising method, including:

[0007] Using a preset sampling dataset, perform a neural structure search on a preset U-shaped neural network to find the first downsampling unit and the first upsampling unit that meet the preset convergence conditions.

[0008] For the first downsampling unit and the first upsampling unit, operator filtering is performed based on the trend of architectural parameter changes during the search period to obtain the second downsampling unit and the second upsampling unit;

[0009] A non-blind denoising sub-network is generated based on the second downsampling unit and the second upsampling unit;

[0010] A noise estimation subnetwork is obtained, and the noise estimation subnetwork and the non-blind denoising subnetwork are used to denoise the image to be denoised, so as to obtain a denoised image.

[0011] This application provides an image denoising apparatus, including:

[0012] The search module is used to perform neural structure search on the preset U-shaped neural network using a preset sampling dataset, and to search for the first downsampling unit and the first upsampling unit that meet the preset convergence conditions.

[0013] The filtering module is used to filter the first downsampling unit and the first upsampling unit based on the trend of architectural parameter changes of the intra-unit operators during the search period, so as to obtain the second downsampling unit and the second upsampling unit.

[0014] The generation module is used to generate a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit;

[0015] The processing module is used to obtain a noise estimation subnetwork and use the noise estimation subnetwork and the non-blind denoising subnetwork to perform denoising processing on the image to be denoised, so as to obtain a denoised image.

[0016] This application provides an image denoising device, including: a processor, a memory, and a communication bus;

[0017] The communication bus is used to realize the communication connection between the processor and the memory;

[0018] The processor is configured to execute one or more programs stored in the memory to implement the image denoising method described above.

[0019] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described image denoising method.

[0020] This application provides an image denoising method, apparatus, and storage medium. The method includes: using a preset sampling dataset, performing a neural structure search on a preset U-shaped neural network to find a first downsampling unit and a first upsampling unit that meet preset convergence conditions; filtering the first downsampling unit and the first upsampling unit based on the changing trend of the architecture parameters of the intra-unit operators during the search to obtain a second downsampling unit and a second upsampling unit; generating a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit; obtaining a noise estimation sub-network, and using the noise estimation sub-network and the non-blind denoising sub-network to denoise the image to be denoised to obtain a denoised image. The technical solution provided by this application filters the intra-unit operators of the sampling unit search based on the evolution trend of the architecture parameters, thereby obtaining sampling units with better performance to construct a network for image denoising, thus improving the image denoising effect. Attached Figure Description

[0021] Figure 1 A schematic flowchart illustrating an image denoising method provided in an embodiment of this application;

[0022] Figure 2(a) is a schematic diagram of an exemplary preset U-shaped neural network provided in an embodiment of this application;

[0023] Figure 2(b) is a schematic diagram of an exemplary unit structure provided in an embodiment of this application;

[0024] Figure 3(a) illustrates the exemplary architectural parameter variation trend provided in the embodiments of this application. Figure 1 ;

[0025] Figure 3(b) is a schematic diagram of the exemplary architectural parameter change trend provided in the embodiments of this application;

[0026] Figure 4 A schematic diagram illustrating an exemplary process for denoising an image to be denoised, provided for an embodiment of this application;

[0027] Figure 5 A schematic diagram of the structure of an image denoising device provided in this application embodiment. Figure 1 ;

[0028] Figure 6 This is a schematic diagram of the structure of an image denoising device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below through embodiments and in conjunction with the accompanying drawings. The embodiments below can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0031] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0032] This application provides an image denoising method implemented using an image denoising device. The specific type of image denoising device is not limited in this application; it can be any user device, such as a smartphone, personal computer, laptop, tablet, or portable wearable device.

[0033] Figure 1 This is a schematic flowchart illustrating an image denoising method provided in an embodiment of this application. Figure 1 As shown, the embodiments of this application mainly include the following steps:

[0034] S101. Using a preset sampling dataset, perform a neural structure search on the preset U-shaped neural network to find the first downsampling unit and the first upsampling unit that meet the preset convergence conditions.

[0035] In the embodiments of this application, the image denoising device can first use a preset sampling dataset to perform a neural structure search on a preset U-shaped neural network, and search for a first downsampling unit and a first upsampling unit that meet the preset convergence conditions.

[0036] It should be noted that, in the embodiments of this application, the preset sampling dataset may actually include undenoised images and a first denoised image preset for the undenoised images. The first denoised image is the image expected to be generated after denoising the undenoised images. The specific number of undenoised images and the number of first denoised images are not limited in the embodiments of this application.

[0037] It should be noted that, in the embodiments of this application, before the image denoising device searches for the neural structure of the preset U-shaped neural network using a preset sampling dataset and finds the first downsampling unit and the first upsampling unit that meet the preset convergence conditions, it may also perform the following steps: obtaining the downsampling unit and the upsampling unit to be searched; the downsampling unit and the upsampling unit to be searched are each composed of nodes, upsampling operators, downsampling operators and identity mapping operators to form the search space of the corresponding unit; stacking the downsampling unit and the upsampling unit to be searched to generate the preset U-shaped neural network.

[0038] It should be noted that, in the embodiments of this application, the user can preset the downsampling unit and the upsampling unit to be searched in the image denoising device according to actual needs. The downsampling unit and the upsampling unit to be searched are each composed of nodes and three types of operators. The random combination of these nodes and operators constitutes the search space of the unit. For example, the downsampling unit and the upsampling unit to be searched are each composed of four nodes and the above three types of operators. The specific structure of the downsampling unit and the upsampling unit to be searched can be set according to actual needs, and is not limited in the embodiments of this application.

[0039] It should be noted that, in the embodiments of this application, in order to find a better network structure, more lightweight alternative operators can be introduced into the definition of the search space, such as channel mixing operators, so as to further reduce the number of network parameters and make the searched model structure more lightweight.

[0040] It is understood that in the embodiments of this application, the preset U-shaped neural network is composed of a stack of downsampling units to be searched and upsampling units to be searched. Of course, the preset U-shaped neural network also includes other necessary units, which are not limited in the embodiments of this application.

[0041] Figure 2(a) is a schematic diagram of an exemplary pre-defined U-shaped neural network structure provided in an embodiment of this application. As shown in Figure 2(a), it includes four downsampling units and upsampling units to be searched, symmetrically distributed in a U-shape. The beginning of the network structure reduces the feature map size by a factor of 2 through a convolution with a stride of 2 and a linear activation operator, thereby reducing the amount of feature map data processed. The end increases the feature map size by a factor of 2 through a transposed convolution with a stride of 2 and a linear activation operator, in order to restore the amount of feature map data. A convolution with a stride of 1 and a linear activation operator are used to further linearly combine the feature maps output by the bottom-level downsampling units to be searched, thereby outputting them to the bottom-level upsampling units to be searched. Each downsampling unit and upsampling unit to be searched, as shown in Figure 2(b), actually includes nodes and three types of operators, following the unit construction method in U-shaped neural network structure search and Differentiable Architecture Search (DARTS). The three numbers represent nodes, which are actually feature maps.

[0042] Specifically, in the embodiments of this application, the preset sampling dataset includes: an undenoised image and a first denoised image preset for the undenoised image. Using the preset sampling dataset, a neural structure search is performed on a preset U-shaped neural network to search for a first downsampling unit and a first upsampling unit that meet the preset convergence conditions. This includes: using the preset U-shaped neural network to denoise the undenoised image to obtain the first denoised image; calculating the loss information between the first denoised image and the second denoised image to obtain the network loss function; and based on the network loss function, optimizing the architecture parameters and weight parameters of the operators in the searched downsampling unit and the searched upsampling unit in the preset U-shaped neural network until the preset convergence conditions are met to obtain the first downsampling unit and the first upsampling unit.

[0043] It should be noted that, in the embodiments of this application, the image denoising device performs neural structure search on the preset U-shaped neural network. Specifically, it can perform neural structure search on the preset U-shaped neural network using a differentiable structure search method. This search method relaxes the discrete search space, allowing the network search to be optimized using gradient updates, and finally solves the neural network search problem.

[0044] It is understood that, in the embodiments of this application, the preset sampling dataset includes an undenoised image and a first denoised image corresponding to the undenoised image. The image denoising device inputs the undenoised image into a preset U-shaped neural network, which performs denoising processing to obtain a second denoised image. By calculating the difference between the first denoised image and the second denoised image, i.e., the network loss function, the search downsampling unit and the search upsampling unit in the preset U-shaped neural network can be alternately optimized. Specifically, the optimization targets are the architecture parameters and weight parameters of the operators within the unit, until the preset convergence condition is met. Thus, the optimized downsampling unit is used as the first downsampling unit, and the optimized upsampling unit is used as the first upsampling unit.

[0045] It should be noted that, in the embodiments of this application, regarding the search objective function, the optimization objective function with energy consumption constraints is defined as measuring the network loss by the 1-norm between the network output (i.e., the second denoised image) and the given ground truth (i.e., the first denoised image). During the search process, the objective performance evaluation metrics for the network are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The entire network contains two sets of parameters: weight parameters and architecture parameters. This invention simultaneously employs the Adam optimization method to update and optimize both sets of parameters until all parameters converge.

[0046] It should be noted that in the embodiments of this application, the image denoising device uses a neural structure search method to search for network structures, namely the two sampling units mentioned above, which have a certain degree of universality and transferability.

[0047] S102. For the first downsampling unit and the first upsampling unit, operator filtering is performed based on the trend of changes in architecture parameters of the operators within the unit during the search period to obtain the second downsampling unit and the second upsampling unit.

[0048] In the embodiments of this application, after the image denoising device obtains the first downsampling unit and the first upsampling unit, it can perform operator filtering on the first downsampling unit and the first upsampling unit based on the trend of changes in the architecture parameters of the operators within the unit during the search period to obtain the second downsampling unit and the second upsampling unit.

[0049] It is understood that in the embodiments of this application, the image denoising device performs neural structure search, and actually optimizes the corresponding structural parameters and weight parameters for different operators in the downsampling unit and the upsampling unit to be searched. The types and numbers of operators actually included in the first downsampling unit and the first upsampling unit are unchanged. However, some operators do not actually play a positive role in the unit. In order to ensure that computing power is not wasted due to the use of these operators in the future, it is necessary to screen the operators.

[0050] Specifically, in the embodiments of this application, the image denoising device performs operator filtering on the first downsampling unit and the first upsampling unit based on the trend of architectural parameter changes during the search period, to obtain a second downsampling unit and a second upsampling unit, including: selecting operators from the first downsampling unit whose architectural parameter changes trend increases during the search period to form the second downsampling unit; and selecting operators from the first upsampling unit whose architectural parameter changes trend increases during the search period to form the second upsampling unit.

[0051] It should be noted that, in the embodiments of this application, the image denoising device filters operators based on the changing trend of the architecture parameters of the operators within the unit during the search period. A non-increasing trend of the changing architecture parameters of the operators indicates that the operators do not play a positive role in the unit, so the corresponding operators can be deleted. Conversely, an increasing trend of the changing architecture parameters of the operators indicates that the operators play a positive role in the unit, so the corresponding operators can be selected.

[0052] Figure 3(a) illustrates the exemplary architectural parameter variation trend provided in the embodiments of this application. Figure 1 As shown in Figure 3(a), the image denoising device can retain operators with an upward trend in architectural parameter changes within the corresponding sampling unit. Figure 3(b) is a second exemplary schematic diagram of architectural parameter change trends provided in this application embodiment. As shown in Figure 3(b), the image denoising device can delete operators with a non-increasing trend in architectural parameter changes from the corresponding sampling unit.

[0053] Understandably, currently, the selection of operators within a unit is usually based on the operator's weight parameters, i.e., the operator with the largest weight parameter is selected and retained, while other operators are deleted. However, the differences between the final converged weight parameters are usually not very significant. Therefore, the selection of operators is actually difficult, and it is impossible to guarantee that a suitable operator will be selected. In the embodiments of this application, the obvious changes in the operator's architecture parameters during the search are taken into consideration, which are easy to identify. The trend of change also indicates whether the operator's effect is positive. By utilizing the dynamic change trend of the operator's architecture parameters, a more suitable operator can be accurately selected, and the subsequent image denoising using the unit will also have a better effect.

[0054] S103. Generate a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit.

[0055] In the embodiments of this application, after obtaining the second downsampling unit and the second upsampling unit, the image denoising device can generate a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit.

[0056] Specifically, in the embodiments of this application, the image denoising apparatus generates a non-blind denoising sub-network based on a second downsampling unit and a second upsampling unit, including: obtaining a denoising sub-network to be updated; updating a preset downsampling unit to a second downsampling unit and a preset upsampling unit to a second upsampling unit in the denoising sub-network to be updated, thereby obtaining a non-blind denoising sub-network.

[0057] It should be noted that, in the embodiments of this application, the image denoising device can pre-generate a denoising sub-network to be updated. Specifically, this denoising sub-network can be a user-preset non-blind denoising sub-network. The preset downsampling units and preset upsampling units can be configured according to user-set parameters, and their performance cannot be guaranteed. Therefore, these units can be replaced with the corresponding units that perform search and operator filtering. This can improve the performance of image denoising, thereby improving the image denoising effect. The specific denoising sub-network to be updated is not limited in the embodiments of this application.

[0058] S104. Obtain the noise estimation subnetwork, and use the noise estimation subnetwork and the non-blind denoising subnetwork to denoise the image to be denoised, and obtain the denoised image.

[0059] In the embodiments of this application, in order to achieve image denoising, the image denoising device not only needs to generate a non-blind denoising sub-network, but also needs to obtain a noise estimation sub-network. Then, by using the noise estimation sub-network and the non-blind denoising sub-network, the image to be denoised is processed to obtain a denoised image.

[0060] It should be noted that, in the embodiments of this application, the image denoising device may store a pre-generated noise estimation sub-network. The specific noise estimation sub-network may be a fully convolutional network used to implement image noise estimation, and this embodiment of the application does not limit it.

[0061] Specifically, in the embodiments of this application, the image denoising apparatus uses a noise estimation subnetwork and a non-blind denoising subnetwork to perform denoising processing on the image to be denoised to obtain a denoised image, including: using the noise estimation subnetwork to perform noise estimation on the image to be denoised to obtain a noise estimation result; and using the non-blind denoising subnetwork to perform denoising processing on the image to be denoised based on the noise estimation result to obtain a denoised image.

[0062] Figure 4 This is a schematic diagram illustrating an exemplary process for denoising an image to be denoised, provided as an embodiment of this application. For example... Figure 4 As shown in the embodiments of this application, image denoising involves two steps. The first step is noise estimation, which is to estimate noise using a noise estimation subnetwork. The second step is denoising, where the input to the non-blind denoising subnetwork is the noise estimation result output from the first step and the image to be denoised, thereby outputting a denoised image from the non-blind denoising subnetwork.

[0063] It should be noted that, in the embodiments of this application, the image denoising device utilizes a noise estimation subnetwork and a non-blind denoising subnetwork. Before denoising the image to be denoised, the following steps can also be performed: cascading the noise estimation subnetwork and the non-blind denoising subnetwork; and using a preset sampling dataset to perform collaborative training on the cascaded noise estimation subnetwork and the non-blind denoising subnetwork.

[0064] It should be noted that, in the embodiments of this application, the image denoising device can perform collaborative training on the noise estimation subnetwork and the non-blind denoising subnetwork after obtaining them respectively. This can ensure that the two can better cooperate to achieve each step of image denoising and improve the image denoising effect.

[0065] This application provides an image denoising method, comprising: using a preset sampling dataset, performing a neural structure search on a preset U-shaped neural network to find a first downsampling unit and a first upsampling unit that satisfy preset convergence conditions; for the first downsampling unit and the first upsampling unit, performing operator filtering based on the trend of changes in architecture parameters of the operators within the unit during the search period to obtain a second downsampling unit and a second upsampling unit; generating a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit; obtaining a noise estimation sub-network, and using the noise estimation sub-network and the non-blind denoising sub-network to denoise the image to be denoised to obtain a denoised image. The image denoising method provided in this application filters the operators within the sampling units of the neural structure search based on the evolution trend of architecture parameters, thereby obtaining sampling units with better performance to construct a network for image denoising, thus improving the image denoising effect.

[0066] This application provides an image denoising device. Figure 5 A schematic diagram of the structure of an image denoising device provided in this application embodiment. Figure 1 .like Figure 5 As shown in the embodiments of this application, the image denoising apparatus includes:

[0067] Search module 501 is used to perform neural structure search on a preset U-shaped neural network using a preset sampling dataset, and search for the first downsampling unit and the first upsampling unit that meet the preset convergence conditions.

[0068] The filtering module 502 is used to filter the first downsampling unit and the first upsampling unit based on the trend of changes in the architecture parameters of the operators within the unit during the search period, so as to obtain the second downsampling unit and the second upsampling unit.

[0069] Generation module 503 is used to generate a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit;

[0070] The processing module 504 is used to obtain a noise estimation subnetwork and use the noise estimation subnetwork and the non-blind denoising subnetwork to perform denoising processing on the image to be denoised, so as to obtain a denoised image.

[0071] In one embodiment of this application, the processing module 504 is further configured to obtain a downsampling unit to be searched and an upsampling unit to be searched; the downsampling unit to be searched and the upsampling unit to be searched are each composed of nodes, upsampling operators, downsampling operators and identity mapping operators to form the search space of the corresponding unit; the downsampling unit to be searched and the upsampling unit to be searched are stacked to generate the preset U-shaped neural network.

[0072] In one embodiment of this application, the preset sampling dataset includes: an undenoised image and a first denoised image preset for the undenoised image. The search module 501 is specifically used to perform denoising processing on the undenoised image using the preset U-shaped neural network to obtain a second denoised image; calculate the loss information between the first denoised image and the second denoised image to obtain a network loss function; and optimize the architecture parameters and weight parameters of the operators in the search downsampling unit and the search upsampling unit in the preset U-shaped neural network based on the network loss function until the preset convergence condition is met to obtain the first downsampling unit and the first upsampling unit.

[0073] In one embodiment of this application, the filtering module 502 is specifically used to select from the first downsampling unit operators whose architecture parameters change trend increases during the search period to form the second downsampling unit; and to select from the first upsampling unit operators whose architecture parameters change trend increases during the search period to form the second upsampling unit.

[0074] In one embodiment of this application, the generation module 503 is specifically used to obtain the denoising sub-network to be updated; and to update the preset downsampling unit to the second downsampling unit and the preset upsampling unit to the second upsampling unit in the denoising sub-network to be updated, thereby obtaining the non-blind denoising sub-network.

[0075] In one embodiment of this application, the processing module 504 is specifically used to perform noise estimation on the image to be denoised using the noise estimation subnetwork to obtain a noise estimation result; and to perform denoising processing on the image to be denoised based on the noise estimation result using the non-blind denoising subnetwork to obtain the denoised image.

[0076] In one embodiment of this application, the processing module 504 is further configured to cascade the noise estimation subnetwork and the non-blind denoising subnetwork; and to perform collaborative training on the cascaded noise estimation subnetwork and the non-blind denoising subnetwork using the preset sampling dataset.

[0077] In one embodiment of this application, the noise estimation subnetwork is a fully convolutional network used to implement image noise estimation.

[0078] Figure 6 This is a schematic diagram of the structure of an image denoising device provided in an embodiment of this application. (See diagram 2.) Figure 6 As shown in the embodiments of this application, the image denoising device includes: a processor 601, a memory 602, and a communication bus 603;

[0079] The communication bus 603 is used to realize the communication connection between the processor 601 and the memory 602;

[0080] The processor 601 is used to execute one or more programs stored in the memory 602 to implement the above-described image denoising method.

[0081] This application provides an image denoising device that uses a preset sampling dataset to perform a neural structure search on a preset U-shaped neural network, searching for a first downsampling unit and a first upsampling unit that meet preset convergence conditions. For the first downsampling unit and the first upsampling unit, operator selection is performed based on the trend of changes in architecture parameters within the unit during the search, resulting in a second downsampling unit and a second upsampling unit. A non-blind denoising sub-network is generated based on the second downsampling unit and the second upsampling unit. A noise estimation sub-network is obtained, and the noise estimation sub-network and the non-blind denoising sub-network are used to denoise the image to be denoised, resulting in a denoised image. The image denoising device provided in this application selects operators within the sampling units of the neural structure search based on the evolution trend of architecture parameters, thereby obtaining sampling units with better performance to construct a network for image denoising, thus improving the image denoising effect.

[0082] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the aforementioned image denoising method. The computer-readable storage medium can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be a device including one or any combination of the above-mentioned memories, such as a mobile phone, computer, tablet device, or personal digital assistant.

[0083] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0084] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image denoising method, characterized in that, include: Using a preset sampling dataset, perform a neural structure search on a preset U-shaped neural network to find the first downsampling unit and the first upsampling unit that meet the preset convergence conditions. For the first downsampling unit and the first upsampling unit, operator filtering is performed based on the trend of architectural parameter changes during the search period to obtain the second downsampling unit and the second upsampling unit; A non-blind denoising sub-network is generated based on the second downsampling unit and the second upsampling unit; Obtain a noise estimation subnetwork, and use the noise estimation subnetwork and the non-blind denoising subnetwork to denoise the image to be denoised, and obtain a denoised image; The preset sampling dataset includes: an undenoised image, and a first denoised image preset for the undenoised image. The step of using the preset sampling dataset to perform a neural structure search on a preset U-shaped neural network to find a first downsampling unit and a first upsampling unit that satisfy preset convergence conditions includes: The denoised image is denoised using the preset U-shaped neural network to obtain a second denoised image; Calculate the loss information between the first denoised image and the second denoised image to obtain the network loss function; Based on the network loss function, the architecture parameters and weight parameters of the operators in the searchable downsampling unit and the searchable upsampling unit in the preset U-shaped neural network are optimized until the preset convergence condition is met, thus obtaining the first downsampling unit and the first upsampling unit.

2. The method according to claim 1, characterized in that, Before searching for the neural structure of a preset U-shaped neural network using a preset sampling dataset and finding the first downsampling unit and the first upsampling unit that meet the preset convergence conditions, the method further includes: Obtain the downsampling unit and the upsampling unit to be searched; each downsampling unit and the upsampling unit to be searched is composed of nodes, upsampling operators, downsampling operators and identity mapping operators, forming the search space of the corresponding unit; The downsampling unit to be searched and the upsampling unit to be searched are stacked to generate the preset U-shaped neural network.

3. The method according to claim 1, characterized in that, The step of filtering operators based on the changing trends of architectural parameters during the search period for the first downsampling unit and the first upsampling unit to obtain the second downsampling unit and the second upsampling unit includes: From the first downsampling unit, operators whose architecture parameters change trend increases during the search period are selected to form the second downsampling unit; From the first upsampling unit, operators whose architectural parameters change trend increases during the search period are selected to form the second upsampling unit.

4. The method according to claim 1, characterized in that, The generation of a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit includes: Obtain the noise reduction sub-network to be updated; In the denoising sub-network to be updated, the preset downsampling unit is updated to the second downsampling unit, and the preset upsampling unit is updated to the second upsampling unit to obtain the non-blind denoising sub-network.

5. The method according to claim 1, characterized in that, The step of using the noise estimation subnetwork and the non-blind denoising subnetwork to denoise the image to be denoised, and obtaining a denoised image, includes: The noise estimation subnetwork is used to perform noise estimation on the image to be denoised, and the noise estimation result is obtained. Using the non-blind denoising sub-network, based on the noise estimation results, the image to be denoised is denoised to obtain the denoised image.

6. The method according to claim 1, characterized in that, Before performing denoising processing on the image to be denoised using the noise estimation subnetwork and the non-blind denoising subnetwork, the method further includes: The noise estimation subnetwork and the non-blind denoising subnetwork are cascaded together. Using the preset sampling dataset, the cascaded noise estimation subnetwork and the non-blind denoising subnetwork are trained collaboratively.

7. The method according to claim 1, characterized in that, The noise estimation subnetwork is a fully convolutional network used to implement image noise estimation.

8. An image denoising device, characterized in that, include: The search module is used to perform neural structure search on the preset U-shaped neural network using a preset sampling dataset, and to search for the first downsampling unit and the first upsampling unit that meet the preset convergence conditions. The filtering module is used to filter the first downsampling unit and the first upsampling unit based on the trend of architectural parameter changes of the intra-unit operators during the search period, so as to obtain the second downsampling unit and the second upsampling unit. The generation module is used to generate a non-blind denoising sub-network based on the second downsampling unit and the second upsampling unit; The processing module is used to obtain a noise estimation subnetwork and use the noise estimation subnetwork and the non-blind denoising subnetwork to perform denoising processing on the image to be denoised, so as to obtain a denoised image. The preset sampling dataset includes: an undenoised image and a first denoised image preset for the undenoised image. The search module is specifically used to use the preset U-shaped neural network to denoise the undenoised image to obtain a second denoised image; calculate the loss information between the first denoised image and the second denoised image to obtain a network loss function; and optimize the architecture parameters and weight parameters of the operators in the search downsampling unit and the search upsampling unit in the preset U-shaped neural network based on the network loss function until the preset convergence condition is met to obtain the first downsampling unit and the first upsampling unit.

9. The apparatus according to claim 8, characterized in that, The processing module is also used to obtain the downsampling unit to be searched and the upsampling unit to be searched; the downsampling unit to be searched and the upsampling unit to be searched are each composed of nodes, upsampling operators, downsampling operators and identity mapping operators to form the search space of the corresponding unit; The downsampling unit to be searched and the upsampling unit to be searched are stacked to generate the preset U-shaped neural network.

10. The apparatus according to claim 8, characterized in that, The filtering module is specifically used to select operators from the first downsampling unit whose architecture parameters change trend increases during the search period to form the second downsampling unit; and to select operators from the first upsampling unit whose architecture parameters change trend increases during the search period to form the second upsampling unit.

11. The apparatus according to claim 8, characterized in that, The generation module is specifically used to obtain the denoising sub-network to be updated; and to update the preset downsampling unit to the second downsampling unit and the preset upsampling unit to the second upsampling unit in the denoising sub-network to be updated, thereby obtaining the non-blind denoising sub-network.

12. An image denoising device, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more programs stored in the memory to implement the image denoising method according to any one of claims 1-7.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image denoising method as described in any one of claims 1-7.

Images