Neural network structure searching method for self-supervised image noise reduction

Abstract

The invention discloses a neural network structure search method for self-supervising image noise reduction, and belongs to the field of neural network structure search and image noise reduction. According to the method, a search space of a multi-scale encoder is designed, so that a network structure can fuse feature information extracted from different scales, more image details can be recovered, and the network structure with a better noise reduction effect can be found; based on a search space of a multi-scale encoder, an improved genetic algorithm is provided to search a network structure, a coarse-grained population initialization method, a distance-guided parent selection method and a modularization-based crossover mutation operation are designed, and the search capability of the genetic algorithm is improved. Compared with an existing self-supervised image noise reduction method and a neural network structure method, a better noise reduction effect can be achieved; experimental results prove that the self-supervised image noise reduction neural network structure searching method can effectively discover a network structure with a better noise reduction effect.

Description

Technical Field

[0001] The invention relates to a neural network structure search method for self-supervised image denoising, and belongs to the field of neural network structure search and image denoising. Background Art

[0002] Image denoising is an important task in computer vision. Its main function is to reduce or eliminate noise in images and improve image quality. It is often used as a pre-task for advanced computer vision tasks such as image classification, target detection, and image segmentation. Image denoising methods based on supervised learning rely heavily on clean / noisy image pair datasets, which are expensive and difficult to collect. Self-supervised image denoising methods do not need to collect and train clean / noisy image pairs, thus avoiding the problem of difficult and expensive dataset collection.

[0003] Self-supervised image denoising methods are mainly based on neural network structure methods. Quan Y et al. introduced dropout and ensembling methods and designed a U-Net structure with high denoising performance, which achieved good results in the self-supervised single image denoising task (Quan Y, Chen M, Pang T, et al. Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image[C]. 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR), Seattle, WA, USA, 2020: 1887-1895). Jo Y et al. adopted a Self2Self network structure without dropout and proposed a random time set (STE) method to define the loss function of the model. At the same time, combined with the zero-crossing stopping criterion, the denoising processing time was accelerated (Jo Y, Chun SY, Choi J. Rethinking DeepImage Prior for Denoising [C]. 2021 IEEE / CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021: 5067-5076.). The denoising effect of these self-supervised image denoising methods is affected by the network structure. Manually designing the network structure requires repeated experiments, which consumes a lot of time for researchers and requires rich neural network expertise and experience.

[0004] In order to alleviate the difficulty of manually designing network structures and design effective network structures, the neural network architecture search (NAS) algorithm was proposed. Ning Q et al. designed a model-guided search space based on the U-Net structure and supervised image denoising method, which can search for operations at each layer, network width and depth. However, it is still necessary to collect image pair datasets (NingQ, Dong W, Li X, et al. Searching Efficient Model-Guided Deep Network for ImageDenoising[J]. IEEE Transactions on Image Processing, 2023, 32: 668-681). Chen YC et al. adopted the U-Net network framework, proposed a search space for upsampling units and cross-layer feature connections, and used reinforcement learning search methods to search for network structures (Chen YC, Gao C, Robb E, et al. NAS-DIP: Learning Deep Image Prior with Neural Architecture Search[C]. Computer Vision–ECCV 2020: 16th European Conference, Glasgow, United Kingdom, 2020: 442-459.). However, this method ignores the search of the encoder structure and lacks the search of multi-scale structures, making it difficult to fuse feature information extracted at different scales, making it difficult to recover more image details. Summary of the invention

[0005] In order to automatically search for a neural network structure that extracts more image details in the task of self-supervised image denoising, the present invention provides a neural network structure search method for self-supervised image denoising, and the technical solution is as follows:

[0006] The neural network structure search method for self-supervised image denoising of the present invention comprises:

[0007] Step 1: Construct the search space based on the network structure of the U-Net framework;

[0008] Step 2: In the search space, based on the decimal modular coding strategy, a coarse-grained population initialization method is used to generate population individuals; the search space includes: encoder scale type, downsampling type, upsampling type and convolution type;

[0009] Step 3: Use the distance-guided parent selection method to select parent individuals from the population generated in step 2;

[0010] Step 4: Perform modular crossover and mutation on the parent individuals selected in step 3 to generate new offspring individuals;

[0011] Step 5: According to the decimal modular coding strategy, the offspring individuals in step 4 are decoded into the corresponding network structure, and the image denoising is performed using the self-supervised image denoising method, and the PSNR value of the denoising evaluation index is calculated;

[0012] Step 6: According to the PSNR value calculated in step 5, a new population individual is selected using the environmental selection method; steps 3, 4, and 5 are repeated until all evolutionary iterations are completed;

[0013] Step 7: Obtain the last generation of population individuals from step 5, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the best network structure obtained by the search;

[0014] The network structure based on the U-Net framework includes: a first convolution module FC, an L-layer encoder module EB, an L-layer decoder module DB and a second convolution module LC connected in sequence.

[0015] Optionally, the processing flow of the network structure in step 1 includes:

[0016] Step 11: Use the FC module to extract features from the input single image;

[0017] Step 12: extract and compress the features in step 11 using multiple EB modules, wherein the EB modules include a downsampling module and a convolution module, and each EB module receives the output features of the EB module of the previous layer;

[0018] Step 13: Use multiple DB modules to extract and expand the features output in step 12, wherein the DB is composed of a convolution module and an upsampling module, and each DB module receives the output features of the DB module of the previous layer and the output features of the EB module of the same layer;

[0019] Step 14: Use the LC module to convert the image features output from step 13 into a denoised image.

[0020] Optionally, the coarse-grained population initialization method in step 2 includes:

[0021] Step 21: Traverse and obtain the convolution type from the convolution type set;

[0022] Step 22: Based on the convolution type obtained in step 21, traverse and obtain the downsampling type from the downsampling type set;

[0023] Step 23: Based on the downsampling type obtained in step 22, traverse and obtain the upsampling type from the upsampling type set;

[0024] Step 24: Based on step 23, the convolution type and downsampling type are assigned to the convolution type parameters and downsampling type parameters of the L-layer encoder module, the encoder scale type of the first-layer encoder module is assigned to 1, and the encoder scale types of the second-layer to L-layer encoder modules are assigned to 0; the convolution type and upsampling type are assigned to the convolution type parameters and upsampling type parameters of the L-layer decoder module to complete the generation of coarse-grained individuals;

[0025] Step 25: Repeat steps 21, 22, and 23, traverse all convolution types, downsampling types, and upsampling types, and complete the generation of all coarse-grained individuals;

[0026] Step 26: Use the self-supervised denoising method of DIP-STE to calculate the PSNR values ​​for all coarse-grained individuals generated in step 25, and retain the first N individuals with the highest PSNR values ​​as the initial population of the search algorithm.

[0027] Optionally, the process of the distance-guided parent selection method in step 3 includes:

[0028] Use a binary competition to select two individuals p i and p j , calculate the distance between these two individuals (p i ,p j ), the calculation formula is as follows:

[0029]

[0030] Among them, d(EB ik ,EB jk ) represents the distance between the two corresponding position encoder modules, d(DB ik ,DB jk ) represents the distance between the corresponding decoder modules in two bodies; EB ik and EB jk Represents individual p i and p j The kth EB module in DB ik and DB jk Represents individual p i and p j The kth DB module in;

[0031] The distance d(EB) between the two individual corresponding position encoder modules ik ,EB jk) is calculated as follows:

[0032] d(EB ik ,EB jk )=eq(st ik ,st jk )+eq(dt ik ,dt jk )+eq(ect jk ,ect jk )

[0033] Among them, st ik and dt jk EB ik With EB jk The scale type value, dt ik and dt jk EB ik With EB jk Downsampling type value, ect jk and ect jk EB ik With EB jk The convolution type value of

[0034] The distance d(DB) between the corresponding position decoder modules in the two bodies ik ,DB jk ) is calculated as follows:

[0035] d(DB ik ,DB jk )=eq(ut ik ,ut jk )+eq(dct ik ,dct jk )

[0036] Among them, ut ik and ut jk DB ik With DB jk The upsampling type value, dct ik and dct jk DB ik With DB jk The convolution type value; eq(a,b) indicates whether the two parameter values ​​are equal;

[0037] Determine distance(p i ,p j ) satisfies the threshold μ. If it is greater than p i and p jIf the difference between them is large, they can be used as parent individuals; otherwise, the binary bidding competition is used again to select two new individuals. If the individual distance still does not satisfy μ after 10 repetitions, the individual selected for the last time is used as the parent individual.

[0038] Optionally, the modular-based crossover and mutation operation processing flow in step 4 includes:

[0039] Step 41: Randomly generate a decimal r1 in the range [0,1];

[0040] Step 42: If r1 is greater than or equal to the crossover rate r c , then perform the parent individual crossover operation;

[0041] Step 43: Under the condition of step 42, a module position pos1 is randomly selected from all modules of the individual;

[0042] Step 44: In the parent individual p i and p j In the example, the modules at position pos1 are selected respectively, and the corresponding parameter values ​​of the two modules are exchanged to generate two new individuals o i , o j ;

[0043] Step 45: If r1 is less than the crossover rate r c , no crossover operation is performed, p i , p j Directly assign to o i , o j ;

[0044] Step 46: Randomly generate a decimal r2 in the range [0,1];

[0045] Step 47: If r2 is greater than or equal to the mutation rate r m , then for individual o i Perform mutation operations;

[0046] Step 48: Based on step 47, randomly select a module position pos2 from all modules of the individual;

[0047] Step 49: In the individual o i In the example, select the module at position pos2, re-randomize all the parameters on the module, and generate a new individual o′ i ;

[0048] Step 410: If r2 is less than the mutation rate r m , no mutation operation is performed, o i Directly assign to o′ i ;

[0049] Step 411: Repeat steps 46, 47, 48, 49, and 50 to complete. j Mutation operation to generate new offspring individuals o′ j , completing the modular crossover and mutation operations of parent individuals.

[0050] Optionally, in step 6, the process of environment selection is as follows:

[0051] Step 61: From the newly generated offspring set Q t and the population set P t The union P of t ∪Q t Select five elite individuals with large fitness and put them into the next generation population P t+1 , and from P t ∪Q t Remove elite;

[0052] Step 62: Using the binary competition selection method, the P after processing in step 61 t ∪Q t Select individual p from the set;

[0053] Step 63: Place the individual p selected in step 62 into the next generation population P t+1 In, from P t ∪Q t Remove individual p;

[0054] Step 64: Repeat step 63 until P t+1 The number of individuals reaches m, and the environment selection operation is completed.

[0055] Optionally, the encoder scale type includes:

[0056] The k-th layer encoder EB k Does not receive the output features of the FC module, only receives the output features from the previous layer encoder

[0057] and the k-th layer encoder EB k Receive output characteristics of FC module And the output features of the previous encoder

[0058] Optionally, the downsampling types include: maximum pooling layer, downsampling convolution, bilinear interpolation, nearest neighbor interpolation and area interpolation.

[0059] The present invention provides a neural network structure search device for self-supervised image denoising, comprising a memory and a processor;

[0060] The memory is used to store computer programs;

[0061] The processor is used to implement the neural network structure search method for self-supervised image denoising as described in any of the above items when executing the computer program.

[0062] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the neural network structure search method for self-supervised image denoising as described in any one of the above items is implemented.

[0063] The beneficial effects of the present invention are:

[0064] The self-supervised image denoising neural network structure search method of the present invention designs a search space of a multi-scale encoder so that the network structure can fuse feature information extracted at different scales, restore more image details, and facilitate the discovery of a network structure with better denoising effect. Based on the search space of the multi-scale encoder, the present invention proposes an improved genetic algorithm to search for network structures, designs a coarse-grained population initialization method, a distance-guided parent selection method, and a modular-based crossover mutation operation, thereby improving the search capability of the genetic algorithm. Compared with the existing self-supervised image denoising methods and neural network structure methods, better denoising effect can be achieved; experimental results show that the self-supervised image denoising neural network structure search method of the present invention can effectively discover a network structure with better denoising effect.

[0065] Integrating the optimized network structure into the photography function of smartphones can improve the quality of photography in night and low-light conditions, allowing users to take clear photos in any mission environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 It is an overall flow chart of the self-supervised image reduction neural network structure search method of the present invention.

[0068] Figure 2 It is the overall network structure diagram of the self-supervised image degradation neural network structure search method of the present invention.

[0069] Figure 3 2 are structural diagrams of two specific multi-scale encoders in the embodiments of the present invention.

[0070] Figure 4It is an individual structure diagram in an embodiment of the present invention.

[0071] Figure 5 It is a schematic diagram of a specific example of random individuals and coarse-grained individuals in an embodiment of the present invention.

[0072] Figure 6 It is a schematic diagram of modular crossover and mutation operations in an embodiment of the present invention.

[0073] Figure 7 It is a visual comparison chart of the denoising effects of the network structure searched by the present invention and other image denoising methods and neural network structure search methods. DETAILED DESCRIPTION

[0074] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0075] Embodiment 1:

[0076] This embodiment provides a neural network structure search method for self-supervised image denoising, including:

[0077] Step 1: Construct the search space based on the network structure of the U-Net framework;

[0078] Step 2: In the search space, based on the decimal modular coding strategy, a coarse-grained population initialization method is used to generate population individuals; the search space includes: encoder scale type, downsampling type, upsampling type and convolution type;

[0079] Step 3: Use the distance-guided parent selection method to select parent individuals from the population generated in step 2;

[0080] Step 4: Perform modular crossover and mutation on the parent individuals selected in step 3 to generate new offspring individuals;

[0081] Step 5: According to the decimal modular coding strategy, the offspring individuals in step 4 are decoded into the corresponding network structure, and the image denoising is performed using the self-supervised image denoising method, and the PSNR value of the denoising evaluation index is calculated;

[0082] Step 6: According to the PSNR value calculated in step 5, a new population individual is selected using the environmental selection method; steps 3, 4, and 5 are repeated until all evolutionary iterations are completed;

[0083] Step 7: Obtain the last generation of population individuals from step 5, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the best network structure obtained by the search;

[0084] The network structure based on the U-Net framework includes: a first convolution module FC, an L-layer encoder module EB, an L-layer decoder module DB and a second convolution module LC connected in sequence.

[0085] Embodiment 2:

[0086] This embodiment provides a self-supervised image denoising method, comprising the following steps:

[0087] Step 1: Construct the search space based on the network structure of the U-Net framework;

[0088] The network structure processing flow based on the U-Net framework is as follows:

[0089] Step 11: Use the first convolution module (FC) to extract features from the input image. The FC module is a 3*3 convolution operation;

[0090] Step 12: Use the encoder block (EB) to extract and compress the features in step 11. EB consists of downsampling and convolution blocks. Repeat step 12 L times, each time the EB module receives the output features of the EB module in the previous layer;

[0091] Step 13: Use the decoder block (Decoder Block, DB) to extract and expand the features output in step 12. DB consists of a convolution module (ConvBlock) and an upsampling module (UpSampling). Repeat step 13 L times. Each time, the DB module receives the output features of the DB module of the previous layer and the output features of the EB module of the same layer.

[0092] Step 14: Use the last convolution module (LC) to convert the image features output from step 13 into a denoised image. The LC module consists of 2 convolution modules and 1 sigmoid activation function.

[0093] The search space constructed in the overall network structure includes four search dimensions: encoder scale type, downsampling type, upsampling type, and convolution type.

[0094] There are two types of encoder scales for each layer: one is that the encoder module of the current layer does not receive image features of the compression level corresponding to the FC module; the other is that the encoder module of the current layer receives image features of the compression level corresponding to the FC module.

[0095] Downsampling types include: maximum pooling layer (MaxPooling), downsampling convolution (ConvDown), bilinear interpolation (BilinearDown), nearest neighbor interpolation (NearestDown), and area interpolation (AreaDown).

[0096] Upsampling types include: transpose convolution (TransposeConv), sub-pixel convolution (SubPixel), bilinear interpolation (BilinearUp), nearest neighbor interpolation (NearestUp), and area interpolation (AreaUp).

[0097] Convolution types include: 3×3 convolution, 5×5 convolution, and 7×7 convolution.

[0098] Step 2: In the search space of step 1, based on the decimal modular coding strategy, a coarse-grained population initialization method is used to generate population individuals.

[0099] The decimal modular coding strategy is to combine modules composed of multiple decimal parameters into individual network structures. Figure 4 As shown, the value range of each parameter and the corresponding candidate value are shown in Table 1:

[0100] Table 1: Value ranges of individual structural parameters

[0101]

[0102] The individual consists of an encoder part and a decoder part. Each encoder module consists of scale type scale_type, down sampling type down_type and convolution type conv_type parameters. Each decoder module consists of up sampling type up_type and convolution type conv_type parameters.

[0103] Step 21: Traverse and obtain the convolution type (conv_type) from the convolution type set (conv_type_list);

[0104] Step 22: Based on the conv_type obtained in step 21, traverse and obtain the downsampling type (down_type) from the downsampling type set (down_type_list);

[0105] Step 23: Based on the down_type obtained in step 22, traverse and obtain the upsampling type (up_type) from the upsampling type set (up_type_list);

[0106] Step 24: Based on step 23, assign conv_type and down_type to the convolution type parameters and downsampling type parameters of the L-layer encoder module, assign the encoder scale type of the 1st-layer encoder module to 1, and assign the encoder scale type of the 2nd to Lth-layer encoder modules to 0. Assign conv_type and up_type to the convolution type parameters and upsampling type parameters of the L-layer decoder module to complete the generation of coarse-grained individuals;

[0107] Step 25: Repeat steps 21, 22, and 23, traverse all conv_type, down_type, and up_type, and complete the generation of all coarse-grained individuals;

[0108] Step 26: Use the DIP-STE method proposed in the article Rethinking Deep Image Prior for Denoising to perform self-supervised denoising. DIP-STE designs a random time set (STE) method to define the loss function of the model, thereby realizing self-supervised single image denoising (Jo Y, Chun SY, Choi J. Rethinking Deep Image Prior for Denoising [J]. 2021. DOI: 10.48550 / arXiv.2108.12841.). Calculate the PSNR value for all coarse-grained individuals generated in step 25, and retain the 50 individuals with the highest PSNR values ​​as the initial population of the search algorithm.

[0109] Step 3: Use the distance-guided parent selection method to select parent individuals from the population generated in step 2;

[0110] The distance-guided parent selection method is used to select individuals with greater differences as parent individuals for the crossover operation. First, two individuals p are selected using a binary competition. i 、p j , and then calculate the distance between the two individuals (p i ,p j ), judge distance(p i ,p j ) satisfies the threshold μ. If it is greater than p i 、p j If the difference between them is large, they can be used as parent individuals. Otherwise, the binary competition is used again to select two new individuals. If the individual distance still does not meet μ after 10 repetitions, the last selected individual is used as the parent individual, and distance(p i ,p j ) is calculated as follows:

[0111]

[0112] Among them, d(EB ik ,EB jk ) represents the distance between the two corresponding position encoder modules, d(DB ik ,DB jk ) represents the distance between the corresponding decoder modules in the two bodies. ik , EB jk Represents individual p i 、p j The kth EB module in DB ik , DB jk Represents individual p i 、p j The kth DB module in .

[0113] d(EB ik ,EB jk ) is calculated as follows:

[0114] d(EB ik ,EB jk )=eq(st ik ,st jk )+eq(dt ik ,dt jk )+eq(ect jk ,ect jk )

[0115] Among them, st ik and st jk EB ik With EB jk The scale type value, dt ik and dt jk EB ik With EB jk Downsampling type value, ect jk and ect jk EB ik With EB jk The convolution type value.

[0116] d(DB ik ,DB jk ) is calculated as follows:

[0117] d(DB ik ,DB jk )=eq(ut ik ,ut jk )+eq(dct ik ,dct jk )

[0118] Among them, ut ik and ut jk DB ik With DB jk The upsampling type value, dct ik and dct jk DB ik With DB jk The convolution type value. eq(a,b) indicates whether the two parameter values ​​are equal. The calculation formula is as follows:

[0119]

[0120] Among them, a and b represent the two parameter values ​​to be compared.

[0121] Step 4: Perform modular crossover and mutation on the parent individuals selected in step 3 to generate new offspring individuals.

[0122] The modular crossover and mutation process is as follows:

[0123] Step 41: Randomly generate a decimal r1 in the range [0,1];

[0124] Step 42: If r1 is greater than or equal to the crossover rate r c , then perform the parent individual crossover operation;

[0125] Step 43: Under the condition of step 42, a module position pos1 is randomly selected from all modules of the individual;

[0126] Step 44: In the parent individual p i 、p j In the example, the modules at position pos1 are selected respectively, and the corresponding parameter values ​​of the two modules are exchanged to generate two new individuals o i , o j ;

[0127] Step 45: If r1 is less than the crossover rate r c , no crossover operation is performed, p i , p j Directly assign to o i , o j ;

[0128] Step 46: Randomly generate a decimal r2 in the range [0,1];

[0129] Step 47: If r2 is greater than or equal to the mutation rate r m , then for individual o i Perform mutation operations;

[0130] Step 48: Based on step 47, randomly select a module position pos2 from all modules of the individual;

[0131] Step 49: In the individual o i In the example, select the module at position pos2, re-randomize all the parameters on the module, and generate a new individual o′ i ;

[0132] Step 410: If r2 is less than the mutation rate r m , no mutation operation is performed, o i Directly assign to o′ i ;

[0133] Step 411: Repeat steps 46, 47, 48, 49 and 410 to complete. j Mutation operation to generate new offspring individuals o′ j , completing the modular crossover and mutation operations of parent individuals.

[0134] Step 5: According to the decimal modular coding strategy, the offspring individuals in step 4 are decoded into the corresponding network structure, and the image denoising is performed using the self-supervised image denoising method, and the PSNR value of the denoising evaluation index is calculated;

[0135] Step 6: According to the PSNR value calculated in step 5, use the environment selection method to select new population individuals; repeat steps 3, 4, and 5 until all evolutionary iterations are completed.

[0136] The process flow of environment selection is as follows:

[0137] Step 61: From the newly generated offspring set Q t and the population set P t The union P of t ∪Q t Select five elite individuals with large fitness and put them into the next generation population P t+1 , and from P t ∪Q t Remove elite;

[0138] Step 62: Using the binary competition selection method, the P after processing in step 61 t ∪Q t Select individual p from the set;

[0139] Step 63: Place the individual p selected in step 62 into the next generation population P t+1 In, from P t ∪Q t Remove individual p

[0140] Step 64: Repeat step 63 until P t+1 The number of individuals reaches m, and the environment selection operation is completed.

[0141] Step 7: Obtain the last generation of population individuals from step 6, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the optimal network structure obtained by the search.

[0142] In order to further illustrate the beneficial effects of the present invention, a series of experiments were conducted, and the experimental process and experimental results are described as follows:

[0143] Experimental parameter settings: the population size m of the genetic algorithm is 50, the number of evolutionary iterations T is 50, the threshold μ of the distance-guided parent selection method, the crossover rate r c is 0.7, and the mutation rate r m is 0.3, and the number of elite individuals e is 5. In the network structure, the number of encoder and decoder layers L is 5, the number of encoder channels ef is 48, the number of decoder channels df is 96, the optimizer is RAdam, and the learning rate l is 0.1.

[0144] The best network structure searched by the present invention under the noise level of 50 is used for self-supervised denoising, and its effect is compared with BM3D, DIP, and S2S methods. The test data sets are color image data sets CSet9, CBSD68 and gray image data sets Set12, BSD68, and the tested noise levels are 15, 25, and 50, respectively. The PSNR indicator is used to represent the denoising performance of the algorithm. The larger the PSNR, the better. Table 2 shows the experimental comparison results. In the table, * indicates the peak value of the PSNR of the method during the denoising process, and SDNAS is used to represent the method of the present invention.

[0145] Table 2: Comparison of SDNAS and single image denoising algorithms at PSNR

[0146]

[0147] In Table 2, the PSNR value of the SDNAS method of the present invention on the CSet9 dataset is higher than that of the traditional BM3D method. At the same time, the PSNR values ​​of SDNAS* on the CSBSD68 and BSD68 datasets exceed those of all compared single image denoising methods. When the noise levels of the CSet9 dataset are 15 and 25 and the noise level of the Set12 dataset is 15, SDNAS* surpasses all compared methods. This shows that the method of the present invention has strong competitiveness in self-supervised single image denoising. It is worth noting that the method SDNAS of the present invention is consistent with the self-supervised denoising method of DIP-STE, but the corresponding neural network structures are different. Except for the conditions of σ=15 and σ=25 of the CBSD68 dataset, the PSNR values ​​of SDNAS on other datasets surpass the DIP-STE method. At the same time, the corresponding PSNR values ​​of SDNAS* in all datasets exceed those of DIP-STE*. This shows that the present invention can search for a better neural network structure than the existing methods in the task of self-supervised single image denoising.

[0148] The noise reduction effect of the present invention is compared with that of the single image noise reduction NAS method NAS-DIP. The experiment was conducted on four data sets, CSet9, CBSD68, Set12 and BSD68, and the noise reduction effect was analyzed using three indicators, PSNR, SSIM and LPIPS. The SSIM indicator takes into account the image brightness, contrast and structural information. The larger the SSIM, the better the noise reduction effect. The LPIPS indicator more effectively reflects the human eye's perception of the image. The smaller the LPIPS, the better the noise reduction effect. Table 3 shows the experimental results, among which the larger the PSNR, the better, the larger the SSIM, the better, and the smaller the LPIPS, the better.

[0149] Table 3: Experimental results of SDNAS and NAS-DIP

[0150]

[0151] As can be seen from Table 3, the PSNR, SSIM, and LPIPS values ​​of the SDNAS method of the present invention are better than those of the NAS-DIP method, indicating that the image after denoising by the method of the present invention retains more structure and texture information, is more similar to the clean image, and is more in line with the perception of the human eye.

[0152] In order to verify and analyze the effects of the structure search and coarse-grained initialization evolutionary algorithm of the multi-scale encoder of the SDNAS method of the present invention, the following ablation experiment is designed.

[0153] The present invention conducts five groups of experiments on the Set12 dataset. Each group of experiments uses the PSNR indicator to represent the denoising effect on σ=15, σ=25, and σ=50. The first group of experiments uses DIP-STE for self-supervised denoising. The other four groups of experiments use four different SDNAS to verify the denoising effect. The four SDNAS are: SDNAS with coarse-grained population initialization and distance-guided parent selection canceled (SDNAS w / o CGI_DGS), SDNAS with coarse-grained initialization canceled (SDNAS w / oCGI), SDNAS with encoder multi-scale search space canceled (SDNAS w / o scale) and SDNAS with the complete method. Table 4 shows the experimental results.

[0154] Table 4: Ablation experiment results

[0155]

[0156] The experimental results show that the denoising effects of the four SDNAS are better than DIP-STE on the Set12 dataset, indicating that the genetic algorithm can search for a network structure with better performance in the multi-scale search space. From the perspective of the improvement effect relative to DIP-STE, under the conditions of noise levels of 25 and 50, the PSNR value of SDNAS w / o CGI is improved by 0.05 and 0.1 respectively compared with SDNAS w / o CGI_DGS, indicating that the distance-guided parent selection method improves the diversity of the population and the search ability of the genetic algorithm.

[0157] Under the conditions of noise levels of 15, 25 and 50, the PSNR values ​​of SDNAS are improved by 0.12, 0.02 and 0.08 respectively compared with SDNAS w / o CGI, indicating that the coarse-grained individuals with higher performance generated by the coarse-grained population initialization method are helpful in searching for individuals with better performance. The PSNR values ​​of SDNAS are improved by 0.12, 0.07 and 0.18 respectively compared with SDNAS w / o CGI_DGS, indicating that the combination of the coarse-grained population initialization method and the distance-guided parent selection method can greatly improve the search ability of the genetic algorithm.

[0158] In addition, by comparing SDNAS with SDNAS w / o scale, the PSNR values ​​under three noise levels are improved by 0.12, 0.08, and 0.16, respectively. This shows that the multi-scale type search for the encoder in the search space can obtain a network structure with better noise reduction effect, which plays an important role in improving the performance of SNDAS.

[0159] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A neural network structure search method for self-supervised image denoising, characterized in that: The method comprises: Step 1: Construct the search space based on the network structure of the U-Net framework; Step 2: In the search space, based on the decimal modular coding strategy, a coarse-grained population initialization method is used to generate population individuals; the search space includes: encoder scale type, downsampling type, upsampling type and convolution type; Step 3: Use the distance-guided parent selection method to select parent individuals from the population generated in step 2; Step 4: Perform modular crossover and mutation on the parent individuals selected in step 3 to generate new offspring individuals; Step 5: According to the decimal modular coding strategy, the offspring individuals in step 4 are decoded into the corresponding network structure, and the image denoising is performed using the self-supervised image denoising method, and the PSNR value of the denoising evaluation index is calculated; Step 6: According to the PSNR value calculated in step 5, a new population individual is selected using the environmental selection method; steps 3, 4, and 5 are repeated until all evolutionary iterations are completed; Step 7: Obtain the last generation of population individuals from step 5, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the best network structure obtained by the search; The network structure based on the U-Net framework includes: a first convolution module FC, an L-layer encoder module EB, an L-layer decoder module DB and a second convolution module LC connected in sequence.

2. The method according to claim 1, characterized in that The processing flow of the network structure in step 1 includes: Step 11: Use the FC module to extract features from the input single image; Step 12: using multiple EB modules to extract and compress the features in step 11, wherein the EB modules include a downsampling module and a convolution module, and each EB module receives the output features of the EB module of the previous layer; Step 13: Use multiple DB modules to extract and expand the features output in step 12, wherein the DB is composed of a convolution module and an upsampling module, and each DB module receives the output features of the DB module of the previous layer and the output features of the EB module of the same layer; Step 14: Use the LC module to convert the image features output from step 13 into a denoised image.

3. The method according to claim 2, characterized in that The coarse-grained population initialization method in step 2 includes: Step 21: Traverse and obtain the convolution type from the convolution type set; Step 22: Based on the convolution type obtained in step 21, traverse and obtain the downsampling type from the downsampling type set; Step 23: Based on the downsampling type obtained in step 22, traverse and obtain the upsampling type from the upsampling type set; Step 24: Based on step 23, the convolution type and downsampling type are assigned to the convolution type parameters and downsampling type parameters of the L-layer encoder module, the encoder scale type of the first-layer encoder module is assigned to 1, and the encoder scale types of the second-layer to L-layer encoder modules are assigned to 0; the convolution type and upsampling type are assigned to the convolution type parameters and upsampling type parameters of the L-layer decoder module to complete the generation of coarse-grained individuals; Step 25: Repeat steps 21, 22, and 23, traverse all convolution types, downsampling types, and upsampling types, and complete the generation of all coarse-grained individuals; Step 26: Use the self-supervised denoising method of DIP-STE to calculate the PSNR values ​​for all coarse-grained individuals generated in step 25, and retain the first N individuals with the highest PSNR values ​​as the initial population of the search algorithm.

4. The method according to claim 3, characterized in that The process flow of the parent selection method based on distance guidance in step 3 includes: Use a binary competition to select two individuals p i and p j , calculate the distance between these two individuals (p i ,p j ), the calculation formula is as follows: Among them, d(EB ik ,EB jk ) represents the distance between the two corresponding position encoder modules, d(DB ik ,DB jk ) represents the distance between the decoder modules at corresponding positions in two bodies; EB ik and EB jk Represents individual p i and p j The kth EB module in DB ik and DB jk Represents individual p i and p j The kth DB module in; The distance d(EB) between the two individual corresponding position encoder modules ik ,EB jk ) is calculated as follows: d(EB ik ,EB jk )=eq(st ik ,st jk )+eq(dt ik ,dt jk )+eq(ect jk ,ect jk ) Among them, st ik and st jk EB ik With EB jk The scale type value, dt ik and dt jk EB ik With EB jk Downsampling type value, ect jk and ect jk EB ik With EB jk The convolution type value of The distance d(DB) between the corresponding position decoder modules in the two bodies ik ,DB jk ) is calculated as follows: d(DB ik ,DB jk )=eq(ut ik ,ut jk )+eq(dct ik ,dct jk ) Among them, ut ik and ut jk DB ik With DB jk The upsampling type value, dct ik and dct jk DB ik With DB jk The convolution type value; eq(a,b) indicates whether the two parameter values ​​are equal; Determine distance(p i ,p j ) satisfies the threshold μ. If it is greater than p i and p j If the difference between them is large, they can be used as parent individuals; otherwise, the binary bidding competition is used again to select two new individuals. If the individual distance still does not satisfy μ after 10 repetitions, the individual selected for the last time is used as the parent individual.

5. The method according to claim 4, characterized in that The modular crossover and mutation operation processing flow in step 4 includes: Step 41: Randomly generate a decimal r1 in the range [0,1]; Step 42: If r1 is greater than or equal to the crossover rate r c , then perform the parent individual crossover operation; Step 43: Under the condition of step 42, a module position pos1 is randomly selected from all modules of the individual; Step 44: In the parent individual p i and p j In the example, the modules at position pos1 are selected respectively, and the corresponding parameter values ​​of the two modules are exchanged to generate two new individuals o i , o j ; Step 45: If r1 is less than the crossover rate r c , then no crossover operation is performed, p i , p j Directly assign to o i , o j ; Step 46: Randomly generate a decimal r2 in the range [0,1]; Step 47: If r2 is greater than or equal to the mutation rate r m , then for individual o i Perform mutation operations; Step 48: Based on step 47, randomly select a module position pos2 from all modules of the individual; Step 49: In the individual o i In the example, select the module at position pos2, re-randomize all the parameters on the module, and generate a new individual o′ i ; Step 410: If r2 is less than the mutation rate r m , no mutation operation is performed, o i Directly assign to o′ i ; Step 411: Repeat steps 46, 47, 48, 49, and 50 to complete. j Mutation operation to generate new offspring individuals o′ j , completing the modular crossover and mutation operations of parent individuals.

6. The method according to claim 5, characterized in that In step 6, the process flow of environment selection is as follows: Step 61: From the newly generated offspring set Q t and the population set P t The union P of t ∪Q t Select five elite individuals with large fitness and put them into the next generation population P t+1 , and from P t ∪Q t Remove elite; Step 62: Using the binary competition selection method, the P after processing in step 61 t ∪Q t Select individual p from the set; Step 63: Place the individual p selected in step 62 into the next generation population P t+1 In, from P t ∪Q t Remove individual p; Step 64: Repeat step 63 until P t+1 The number of individuals reaches m, and the environment selection operation is completed.

7. The method according to claim 1, characterized in that The encoder scale types include: The k-th layer encoder EB k Does not receive the output features of the FC module, only receives the output features from the previous layer encoder and the k-th layer encoder EB k Receive output characteristics of FC module And the output features of the previous encoder 8. The method according to claim 1, characterized in that The downsampling types include: maximum pooling layer, downsampling convolution, bilinear interpolation, nearest neighbor interpolation and area interpolation.

9. A neural network structure search device for self-supervised image denoising, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the neural network structure search method for self-supervised image denoising as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the neural network structure search method for self-supervised image denoising as described in any one of claims 1 to 8 is implemented.

Images