A self-supervised image denoising neural network structure search method

Abstract

The application discloses a kind of self-supervised image denoising neural network structure search method, belong to neural network structure search and image denoising field.The method designs the search space of multiscale encoder, so that network structure can fuse the feature information extracted in different scales, restore more image details, it is favorable to find the network structure that better denoising effect;The application is based on the search space of multiscale encoder, an improved genetic algorithm is proposed to search the network structure, a coarse-grained population initialization method, a distance-guided parent selection method and a modular-based crossover mutation operation are designed to improve the search capability of the genetic algorithm.Compared with existing self-supervised image denoising methods and neural network structure methods, better denoising effect can be achieved.The experimental results show that the self-supervised image denoising neural network structure search method of the application can effectively find the network structure with better denoising effect.

Description

Technical Field

[0001] This invention relates to a method for searching neural network structures for self-supervised image denoising, belonging to the fields of neural network structure search and image denoising. Background Technology

[0002] Image denoising is a crucial task in computer vision, primarily aimed at reducing or eliminating noise in images to improve image quality. It is often used as a prerequisite for advanced computer vision tasks such as image classification, object detection, and image segmentation. Supervised learning-based image denoising methods heavily rely on clean / noisy image pairs datasets, which are expensive and difficult to collect. Self-supervised image denoising methods, on the other hand, avoid the problem of collecting and training clean / noisy image pairs, thus circumventing the difficulties and high costs associated with dataset collection.

[0003] Self-supervised image denoising methods are mainly based on neural network structures. Quan Y et al. introduced dropout and ensembling methods and designed a U-Net structure with high denoising performance, which achieved good results in the self-supervised single image denoising task (Quan Y, Chen M, Pang T, et al. Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image[C]. 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020: 1887-1895). Jo Y et al. employed a dropout-free Self2Self network structure and proposed the Stochastic Time Set (STE) method to define the model's loss function. They also combined this with a zero-crossing stopping criterion to accelerate the denoising process (Jo Y, Chun SY, Choi J. Rethinking Deep Image Prior for Denoising[C]. 2021 IEEE / CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021: 5067-5076.). The denoising performance of these self-supervised image denoising methods is influenced by the network structure. Manually designing network structures requires repeated experiments, consuming a significant amount of researchers' time and requiring extensive expertise and experience in neural networks.

[0004] To alleviate the difficulty of manually designing network structures and to design effective network structures, Neural Architecture Search (NAS) algorithms have been proposed. Ning Q et al., based on the U-Net structure and supervised image denoising methods, designed a model-guided search space that can search for operations at each layer, network width, and depth. However, it is still necessary to collect image pair datasets (Ning Q, Dong W, Li X, et al. Searching Efficient Model-Guided Deep Network for Image Denoising[J]. IEEE Transactions on Image Processing, 2023, 32: 668-681). Chen YC et al., using the U-Net network framework, proposed a search space for upsampling units and cross-layer feature connections, and used reinforcement learning search methods to search for network structures (Chen YC, Gao C, Robb E, et al. NAS-DIP: Learning Deep Image Prior with Neural Architecture Search[C]. Computer Vision–ECCV 2020: 16th European Conference, Glasgow, United Kingdom, 2020: 442-459.). However, this method ignores the search of the encoder structure and lacks the search of multi-scale structures, making it difficult to fuse feature information extracted at different scales, thus making it difficult to recover more image details. Summary of the Invention

[0005] To automatically search for neural network structures that extract more image details in self-supervised image denoising tasks, and to overcome the problem that manual network structure design requires researchers to spend a lot of time and have extensive expertise in neural networks, this invention provides a method for searching neural network structures for self-supervised image denoising. The technical solution is as follows:

[0006] The self-supervised image denoising neural network architecture search method of the present invention includes:

[0007] Step 1: Construct the search space based on the network structure of the U-Net framework;

[0008] Step 2: In the search space, based on the decimal modular encoding strategy, a coarse-grained population initialization method is used to generate population individuals; the search space includes: encoder scale type, downsampling type, upsampling type and convolution type;

[0009] Step 3: Use a distance-guided parent selection method to select parent individuals from the population generated in Step 2;

[0010] Step 4: Perform modular crossover and mutation on the parent individuals selected in Step 3 to generate new offspring individuals;

[0011] Step 5: Based on the decimal modular coding strategy, decode the offspring individuals in Step 4 into the corresponding network structure, use the self-supervised image denoising method to denoise the image, and calculate the denoising evaluation index PSNR value.

[0012] Step 6: Based on the PSNR value calculated in Step 5, select new individuals for the population using the environment selection method; repeat Steps 3, 4, and 5 until all evolutionary iterations are completed.

[0013] Step 7: Obtain the last generation of individuals from Step 5, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the best network structure obtained by the search.

[0014] The network structure based on the U-Net framework includes, in sequence: a first convolutional module FC, an L-layer encoder module EB, an L-layer decoder module DB, and a second convolutional module LC.

[0015] Optionally, the network structure processing flow in step 1 includes:

[0016] Step 11: Use the FC module to extract features from the input single image;

[0017] Step 12: Extract and compress the features from Step 11 using multiple EB modules. Each EB module includes a downsampling module and a convolution module, and each EB module receives the output features from the previous EB module.

[0018] Step 13: Use multiple DB modules to extract and augment the features output in Step 12. The DB consists of a convolution module and an upsampling module. Each DB module receives the output features of the DB module in the previous layer and the output features of the EB module in the same layer.

[0019] Step 14: Use the LC module to convert the image features output in Step 13 into a denoised image.

[0020] Optionally, the coarse-grained population initialization method in step 2 includes:

[0021] Step 21: Iterate through the convolution type set to obtain the convolution types;

[0022] Step 22: Based on the convolution type obtained in Step 21, traverse the set of downsampling types to obtain the downsampling types;

[0023] Step 23: Based on the downsampling type obtained in Step 22, iterate through the upsampling type set to obtain the upsampling type;

[0024] Step 24: Based on step 23, assign the convolution type and downsampling type to the convolution type parameter and downsampling type parameter of the L-layer encoder module, assign the encoder scale type of the 1st layer encoder module to 1, and assign the encoder scale type of the 2nd to Lth layer encoder modules to 0; assign the convolution type and upsampling type to the convolution type parameter and upsampling type parameter of the L-layer decoder module to complete the generation of coarse-grained individuals;

[0025] Step 25: Repeat steps 21, 22 and 23 to traverse all convolution types, downsampling types and upsampling types, and complete the generation of all coarse-grained individuals;

[0026] Step 26: Using the self-supervised denoising method of DIP-STE, calculate the PSNR value for all coarse-grained individuals generated in Step 25, and retain the top N individuals with the highest PSNR values ​​as the initial population for the search algorithm.

[0027] Optionally, the distance-guided parent selection method processing flow in step 3 includes:

[0028] Two individuals p are selected using a binary bidding process. i and p j Calculate the distance (p) between the two individuals. i ,p j The calculation formula is as follows:

[0029]

[0030] Wherein d(EB) ik ,EB jk ) represents the distance between the encoder modules at the corresponding positions of the two individuals, d(DB) ik DB jk ) represents the distance between corresponding decoder modules in two volumes; EB ik and EB jk Representing individual p respectively i and p j The k-th EB module; DB ik and DB jk Representing individual p respectively i and p j The k-th DB module;

[0031] The distance d(EB) between the encoder modules at the corresponding positions of the two individuals ik ,EB jkThe calculation formula for ) is as follows:

[0032] d(EB ik ,EB jk )=eq(st ik ,st jk )+eq(dt ik ,dt jk )+eq(ect jk ,ect jk )

[0033] Among them, st ik and dt jk They are EB ik With EB jk The scale type value, dt ik and dt jk They are EB ik With EB jk The downsampling type value, ect jk and ect jk They are EB ik With EB jk The convolution type value;

[0034] The distance d(DB) between the corresponding decoder modules in the two bodies ik DB jk The calculation formula for ) is as follows:

[0035] d(DB ik DB jk ) = eq(ut ik ,ut jk )+eq(dct ik ,dct jk )

[0036] Among them, ut ik and ut jk They are DB ik With DB jk upsampled type value, dct ik and dct jk They are DB ik With DB jk The convolution type value; eq(a,b) indicates whether the two parameter values ​​are equal;

[0037] Determine distance(p) i ,p j Does it meet the threshold μ? If it is greater than μ, then p i and p jIf the differences between the individuals are significant, they can be used as parent individuals; otherwise, the binary bidding process is repeated to select two new individuals. If the individual distance still does not meet μ after 10 repetitions, the last selected individual is used as the parent individual.

[0038] Optionally, the modular crossover and mutation processing flow in step 4 includes:

[0039] Step 41: Randomly generate a decimal r1 within the range [0,1];

[0040] Step 42: If r1 is greater than or equal to the crossover rate r c If so, then the parent generation crossover operation is performed;

[0041] Step 43: Under the condition of satisfying step 42, randomly select a module position pos1 from all modules of the individual;

[0042] Step 44: In the parent individual p i and p j In the process, select the modules at position pos1 respectively, and swap all the corresponding parameter values ​​in these two modules to generate two new individuals o. i o j ;

[0043] Step 45: If r1 is less than the crossover rate r c If p, then no crossover operation is performed. i p j Assign directly to o i o j ;

[0044] Step 46: Randomly generate a decimal r2 within the range [0,1];

[0045] Step 47: If r² is greater than or equal to the mutation rate r m For individual o i Perform mutation operations;

[0046] Step 48: Based on step 47, randomly select a module position pos2 from all modules of the individual;

[0047] Step 49: In individual o i In the middle, select the module at position pos2, and regenerate all parameters of the module randomly to generate a new individual o′. i ;

[0048] Step 410: If r² is less than the mutation rate r m If the mutation operation is not performed, o i Assign directly to o′ i ;

[0049] Step 411: Repeat steps 46, 47, 48, 49, and 50 to complete. j The mutation operation generates a new offspring individual o′ j Complete the modular crossover and mutation operations of the parent individuals.

[0050] Optionally, the process for environment selection in step 6 is as follows:

[0051] Step 61: From the newly generated child set Q t and population set P t Union of P t ∪Q t Five elite individuals with high fitness are selected and placed into the next generation population. t+1 And from P t ∪Q t Remove elite;

[0052] Step 62: Using a binary bidding selection method, select P after processing in step 61. t ∪Q t Select individual p from the set;

[0053] Step 63: Place the individual p selected in step 62 into the next generation population P. t+1 In the middle, from P t ∪Q t Remove individual p from the middle;

[0054] Step 64: Repeat step 63 until P. t+1 When the number of individuals reaches m, the environment selection operation is completed.

[0055] Optionally, the encoder scale type includes:

[0056] The k-th layer encoder EB k It does not accept output features from the FC module, but only accepts output features from the previous layer encoder.

[0057] and the k-th layer encoder EB k Receive FC module output characteristics and the output features of the previous layer encoder

[0058] Optionally, the downsampling types include: max pooling layer, downsampling convolution, bilinear interpolation, nearest neighbor interpolation, and region interpolation.

[0059] This invention provides a neural network structure search device for self-supervised image denoising, including a memory and a processor;

[0060] The memory is used to store computer programs;

[0061] The processor is configured to, when executing the computer program, implement the neural network structure search method for self-supervised image denoising as described in any of the preceding claims.

[0062] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the neural network structure search method for self-supervised image denoising as described in any of the preceding claims.

[0063] The beneficial effects of this invention are:

[0064] This invention presents a self-supervised image denoising neural network structure search method. It designs a multi-scale encoder search space, enabling the network structure to fuse feature information extracted at different scales, recovering more image details and facilitating the discovery of network structures with better denoising effects. Based on the multi-scale encoder search space, this invention proposes an improved genetic algorithm for searching network structures, designing a coarse-grained population initialization method, a distance-guided parent selection method, and a modular crossover and mutation operation to enhance the search capability of the genetic algorithm. Compared to existing self-supervised image denoising methods and neural network structure methods, it achieves better denoising results. Experimental results demonstrate that this invention's self-supervised image denoising neural network structure search method can effectively discover network structures with better denoising effects.

[0065] Integrating the optimized network structure into the smartphone's camera function can improve the quality of photography in low-light and nighttime conditions, enabling users to take clear photos in any task environment. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is an overall flowchart of the self-supervised image reduction neural network structure search method of the present invention.

[0068] Figure 2 This is the overall network structure diagram of the self-supervised image reduction neural network structure search method of the present invention.

[0069] Figure 3 These are structural diagrams of two specific multi-scale encoders in embodiments of the present invention.

[0070] Figure 4This is an individual structural diagram in an embodiment of the present invention.

[0071] Figure 5 This is a schematic diagram illustrating specific examples of random individuals and coarse-grained individuals in embodiments of the present invention.

[0072] Figure 6 This is a schematic diagram of modular crossover and mutation operations in an embodiment of the present invention.

[0073] Figure 7 This is a visual comparison of the noise reduction effects of the network structure searched by this invention with other image noise reduction methods and neural network structure search methods. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0075] Example 1:

[0076] This embodiment provides a method for searching neural network structures for self-supervised image denoising, including:

[0077] Step 1: Construct the search space based on the network structure of the U-Net framework;

[0078] Step 2: In the search space, based on the decimal modular encoding strategy, a coarse-grained population initialization method is used to generate population individuals; the search space includes: encoder scale type, downsampling type, upsampling type and convolution type;

[0079] Step 3: Use a distance-guided parent selection method to select parent individuals from the population generated in Step 2;

[0080] Step 4: Perform modular crossover and mutation on the parent individuals selected in Step 3 to generate new offspring individuals;

[0081] Step 5: Based on the decimal modular coding strategy, decode the offspring individuals in Step 4 into the corresponding network structure, use the self-supervised image denoising method to denoise the image, and calculate the denoising evaluation index PSNR value.

[0082] Step 6: Based on the PSNR value calculated in Step 5, select new individuals for the population using the environment selection method; repeat Steps 3, 4, and 5 until all evolutionary iterations are completed.

[0083] Step 7: Obtain the last generation of individuals from Step 5, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the best network structure obtained by the search.

[0084] The network structure based on the U-Net framework includes, in sequence: a first convolutional module FC, an L-layer encoder module EB, an L-layer decoder module DB, and a second convolutional module LC.

[0085] Example 2:

[0086] This embodiment provides a self-supervised image denoising method, including the following steps:

[0087] Step 1: Construct the search space based on the network structure of the U-Net framework;

[0088] The network structure processing flow based on the U-Net framework is as follows:

[0089] Step 11: Use the first convolutional module (FC) to extract features from the input image. The FC module is a 3*3 convolution operation.

[0090] Step 12: The features from Step 11 are extracted and compressed using an encoder block (EB). The EB consists of downsampling and convolutional blocks (ConvBlocks). Step 12 is repeated L times, with each EB block receiving the output features from the previous EB block.

[0091] Step 13: The decoder block (DB) is used to extract and augment the features output in step 12. The DB consists of a convolutional block (ConvBlock) and an upsampling block (UpSampling). Step 13 is repeated L times. Each time, the DB block receives the output features from the previous layer's DB block and the output features from the same layer's EB block.

[0092] Step 14: Use the last convolutional module (LC) to convert the image features output in step 13 into a denoised image. The LC module consists of two convolutional modules and one sigmoid activation function.

[0093] The search space constructed within the overall network structure includes four search dimensions: encoder scale type, downsampling type, upsampling type, and convolution type.

[0094] There are two types of encoder scales for each layer: one is where the current layer encoder module does not receive image features corresponding to the compression level of the FC module; the other is where the current layer encoder module receives image features corresponding to the compression level of the FC module.

[0095] Downsampling types include: Max Pooling, ConvDown, Bilinear Down, Nearest Down, and AreaDown.

[0096] Upsampling types include: TransposeConv, SubPixel, BilinearUp, NearestUp, and AreaUp.

[0097] Convolution types include: 3×3 convolution, 5×5 convolution, and 7×7 convolution.

[0098] Step 2: In the search space of Step 1, based on the decimal modular encoding strategy, a coarse-grained population initialization method is used to generate population individuals.

[0099] Decimal modular coding strategy combines modules composed of multiple decimal parameters into network structure units, with unit structures such as... Figure 4 As shown in Table 1, the value range of each parameter and its corresponding candidate values ​​are as follows:

[0100] Table 1: Range of values ​​for individual structural parameters

[0101]

[0102] Each unit consists of an encoder and a decoder. Each encoder module is composed of parameters of type scale_type, type downsampling_type, and type convolution_type. Each decoder module is composed of parameters of type upsampling_type and type convolution_type.

[0103] Step 21: Iterate through the convolution type set (conv_type_list) to obtain the convolution type (conv_type);

[0104] Step 22: Based on the conv_type obtained in Step 21, traverse the downsampled type set (down_type_list) to obtain the downsampled type (down_type);

[0105] Step 23: Based on the down_type obtained in Step 22, iterate through the upsampled type set (up_type_list) to obtain the upsampled type (up_type);

[0106] Step 24: Based on Step 23, assign conv_type and down_type to the convolution type parameter and downsampling type parameter of the L-layer encoder module, assign the encoder scale type of the 1st layer encoder module to 1, and assign the encoder scale type of the 2nd to Lth layer encoder modules to 0. Assign conv_type and up_type to the convolution type parameter and upsampling type parameter of the L-layer decoder module to complete the generation of coarse-grained individuals;

[0107] Step 25: Repeat steps 21, 22, and 23 to traverse all conv_type, down_type, and up_type, and complete the generation of all coarse-grained individuals;

[0108] Step 26: Use the DIP-STE method proposed in the paper Rethinking Deep Image Prior for Denoising for self-supervised image denoising. DIP-STE uses the Stochastic Time Set (STE) method to define the loss function of the model, thereby achieving self-supervised single-image denoising (Jo Y, Chun SY, Choi J. Rethinking Deep Image Prior for Denoising[J]. 2021.DOI:10.48550 / arXiv.2108.12841.). Calculate the PSNR value for all coarse-grained individuals generated in Step 25, and retain the 50 individuals with the highest PSNR values ​​as the initial population of the search algorithm.

[0109] Step 3: Use a distance-guided parent selection method to select parent individuals from the population generated in Step 2;

[0110] Distance-guided parent selection is used to select individuals with significant differences as parents for the crossover operation. First, a binary bidding process is used to select two individuals p. i p j Then calculate the distance (p) between the two individuals. i ,p j ), determine distance(p) i ,p j Does it meet the threshold μ? If it is greater than μ, then p i p j If the differences between the individuals are significant, one can be used as the parent individual. Otherwise, a new binary selection process is used to choose two more individuals. If the individual distance still does not meet μ after 10 iterations, the last selected individual is used as the parent individual, and distance(p) is used as the parent individual. i ,p j The specific calculation formula is as follows:

[0111]

[0112] Wherein d(EB) ik ,EB jk ) represents the distance between the encoder modules at the corresponding positions of the two individuals, d(DB) ik DB jk ) represents the distance between corresponding decoder modules in two volumes. EB ik EB jk Representing individual p respectively i p j The k-th EB module; DB ik DB jk Representing individual p respectively i p j The kth DB module.

[0113] d(EB ik ,EB jk The specific calculation formula is as follows:

[0114] d(EB ik ,EB jk )=eq(st ik ,st jk )+eq(dt ik ,dt jk )+eq(ect jk ,ect jk )

[0115] Among them, st ik and st jk They are EB ik With EB jk The scale type value, dt ik and dt jk They are EB ik With EB jk The downsampling type value, ect jk and ect jk They are EB ik With EB jk The convolution type value.

[0116] d(DB ik DB jk The specific calculation formula is as follows:

[0117] d(DB ik DB jk ) = eq(ut ik ,ut jk )+eq(dct ik ,dct jk )

[0118] Among them, ut ik and ut jk They are DB ik With DB jk upsampled type value, dct ik and dct jk They are DB ik With DB jk The convolution type value. eq(a,b) indicates whether the two parameter values ​​are equal, and the calculation formula is as follows:

[0119]

[0120] Where a and b represent the two parameter values ​​being compared.

[0121] Step 4: Perform modular crossover and mutation on the parent individuals selected in Step 3 to generate new offspring individuals.

[0122] The modular crossover and mutation processing flow is as follows:

[0123] Step 41: Randomly generate a decimal r1 within the range [0,1];

[0124] Step 42: If r1 is greater than or equal to the crossover rate r c If so, then the parent generation crossover operation is performed;

[0125] Step 43: Under the condition of satisfying step 42, randomly select a module position pos1 from all modules of the individual;

[0126] Step 44: In the parent individual p i p j In the process, select the modules at position pos1 respectively, and swap all the corresponding parameter values ​​in these two modules to generate two new individuals o. i o j ;

[0127] Step 45: If r1 is less than the crossover rate r c If p, then no crossover operation is performed. i p j Assign directly to o i o j ;

[0128] Step 46: Randomly generate a decimal r2 within the range [0,1];

[0129] Step 47: If r² is greater than or equal to the mutation rate r m For individual o i Perform mutation operations;

[0130] Step 48: Based on step 47, randomly select a module position pos2 from all modules of the individual;

[0131] Step 49: In individual o i In the middle, select the module at position pos2, and regenerate all parameters of the module randomly to generate a new individual o′. i ;

[0132] Step 410: If r² is less than the mutation rate r m If the mutation operation is not performed, o i Assign directly to o′ i ;

[0133] Step 411: Repeat steps 46, 47, 48, 49, and 410 to complete. j The mutation operation generates a new offspring individual o′ j Complete the modular crossover and mutation operations of the parent individuals.

[0134] Step 5: Based on the decimal modular coding strategy, decode the offspring individuals in Step 4 into the corresponding network structure, use the self-supervised image denoising method to denoise the image, and calculate the denoising evaluation index PSNR value.

[0135] Step 6: Based on the PSNR value calculated in Step 5, select new individuals for the population using the environment selection method; repeat Steps 3, 4, and 5 until all evolutionary iterations are completed.

[0136] The process for environment selection is as follows:

[0137] Step 61: From the newly generated child set Q t and population set P t Union of P t ∪Q t Five elite individuals with high fitness are selected and placed into the next generation population. t+1 And from P t ∪Q t Remove elite;

[0138] Step 62: Using a binary bidding selection method, select P after processing in step 61. t ∪Q t Select individual p from the set;

[0139] Step 63: Place the individual p selected in step 62 into the next generation population P. t+1 In the middle, from P t ∪Q t Remove individual p from

[0140] Step 64: Repeat step 63 until P. t+1 When the number of individuals reaches m, the environment selection operation is completed.

[0141] Step 7: Obtain the last generation of individuals from Step 6, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the optimal network structure obtained by the search.

[0142] To further illustrate the beneficial effects of the present invention, a series of experiments were conducted, and the experimental process and results are described below:

[0143] Experimental parameter settings: population size m for the genetic algorithm is 50, number of evolutionary iterations T is 50, threshold μ for distance-guided parent selection, and crossover rate r. c The value is 0.7, and the mutation rate r is 0.7. m The learning rate is 0.3, and the number of elite individuals, e, is 5. The network structure has 5 layers for the encoder and decoder, ef for the encoder, df for the decoder, 96 channels, RAdam for the optimizer, and a learning rate l of 0.1.

[0144] The optimal network structure searched by this invention at a noise level of 50 was subjected to self-supervised denoising, and its performance was compared with BM3D, DIP, and S2S methods. The test datasets were the color image datasets CSet9 and CBSD68, and the grayscale image datasets Set12 and BSD68, with noise levels of 15, 25, and 50, respectively. PSNR was used to represent the denoising performance of the algorithm; a higher PSNR is better. Table 2 shows the experimental comparison results. * in the table indicates the peak PSNR of the method during denoising, and SDNAS is used to denote the method of this invention.

[0145] Table 2: Comparison of SDNAS and single-image noise reduction algorithms under PSNR.

[0146]

[0147] Table 2 shows that the PSNR value of the SDNAS method of this invention is higher than that of the traditional BM3D method on the CSet9 dataset. Furthermore, SDNAS* surpasses all compared single-image denoising methods on the CSBSD68 and BSD68 datasets in terms of PSNR. With noise levels of 15 and 25 on the CSet9 dataset and 15 on the Set12 dataset, SDNAS* outperforms all compared methods. This demonstrates that the method of this invention is highly competitive in self-supervised single-image denoising. It is noteworthy that the SDNAS method of this invention is consistent with the self-supervised denoising method of DIP-STE, but the corresponding neural network structures are different. Except for the conditions of σ=15 and σ=25 on the CBSD68 dataset, the PSNR value of SDNAS surpasses that of the DIP-STE method on other datasets. Simultaneously, the PSNR value of SDNAS* surpasses that of DIP-STE* on all datasets. This indicates that this invention can search for a better neural network structure than existing methods in the self-supervised single-image denoising task.

[0148] This invention is compared with the denoising performance of the single-image denoising NAS method NAS-DIP. Experiments were conducted on four datasets: CSet9, CBSD68, Set12, and BSD68, using PSNR, SSIM, and LPIPS as metrics for denoising performance analysis. The SSIM metric considers image brightness, contrast, and structural information; a higher SSIM indicates better denoising performance. The LPIPS metric more effectively reflects human visual perception of the image; a lower LPIPS indicates better denoising performance. Table 3 shows the experimental results, where higher PSNR and SSIM are generally better, and lower LPIPS are generally better.

[0149] Table 3: Experimental Results of SDNAS and NAS-DIP

[0150]

[0151] As shown in Table 3, the PSNR, SSIM, and LPIPS values ​​of the SDNAS method of the present invention are all better than those of the NAS-DIP method, indicating that the image after noise reduction by the method of the present invention retains more structural and texture information, is more similar to a clean image, and is more in line with human visual perception.

[0152] To verify and analyze the effectiveness of the structure search and coarse-grained initialization evolutionary algorithm of the multi-scale encoder in the SDNAS method of this invention, the following ablation experiment was designed.

[0153] This invention conducted five sets of experiments on the Set12 dataset. The PSNR metric was used to represent the denoising effect at σ=15, σ=25, and σ=50 in each set of experiments. The first set of experiments used DIP-STE for self-supervised denoising, and the remaining four sets of experiments used four different SDNAS methods to verify the denoising effect. The four SDNAS methods are: SDNAS with parent selection without coarse-grained population initialization and distance guidance (SDNAS w / o CGI_DGS), SDNAS with coarse-grained initialization without CGI (SDNAS w / o CGI), SDNAS with encoder multi-scale search space without scale (SDNAS w / o scale), and the complete method SDNAS. Table 4 shows the experimental results.

[0154] Table 4: Ablation Experiment Results

[0155]

[0156] Experimental results show that all four SDNAS algorithms outperform DIP-STE on the Set12 dataset, indicating that the genetic algorithm can find better-performing network structures in a multi-scale search space. Regarding the improvement over DIP-STE, at noise levels of 25 and 50, the PSNR of SDNAS w / o CGI is improved by 0.05 and 0.1 respectively compared to SDNAS w / o CGI_DGS, demonstrating that the distance-guided parent selection method improves population diversity and enhances the search capability of the genetic algorithm.

[0157] Under noise levels of 15, 25, and 50, the PSNR values ​​of SDNAS were improved by 0.12, 0.02, and 0.08 compared to SDNAS without CGI, respectively, indicating that the higher-performing coarse-grained individuals generated by the coarse-grained population initialization method help to find better-performing individuals. The PSNR values ​​of SDNAS were improved by 0.12, 0.07, and 0.18 compared to SDNAS without CGI_DGS, respectively, indicating that combining the coarse-grained population initialization method with the distance-guided parent selection method significantly improves the search capability of the genetic algorithm.

[0158] Furthermore, comparing SDNAS with SDNAS without scale, the PSNR values ​​were improved by 0.12, 0.08, and 0.16 respectively under the three noise levels. This indicates that multi-scale type search of the encoder in the search space can obtain a network structure with better noise reduction effect, which plays an important role in improving the performance of SNDAS.

[0159] Some steps in the embodiments of the present invention can be implemented using software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for searching neural network structures for self-supervised image denoising, characterized in that, The method includes: Step 1: Construct the search space based on the network structure of the U-Net framework; Step 2: In the search space, based on the decimal modular encoding strategy, a coarse-grained population initialization method is used to generate population individuals; the search space includes: encoder scale type, downsampling type, upsampling type and convolution type; Step 3: Use a distance-guided parent selection method to select parent individuals from the population generated in Step 2; Step 4: Perform modular crossover and mutation on the parent individuals selected in Step 3 to generate new offspring individuals; Step 5: Based on the decimal modular coding strategy, decode the offspring individuals in Step 4 into the corresponding network structure, use the self-supervised image denoising method to denoise the image, and calculate the denoising evaluation index PSNR value. Step 6: Based on the PSNR value calculated in Step 5, select new individuals for the population using the environment selection method; repeat Steps 3, 4, and 5 until all evolutionary iterations are completed. Step 7: Obtain the last generation of individuals from Step 5, select the individual with the highest PNSR value and decode it into the corresponding network structure, which is the best network structure obtained by the search. The network structure based on the U-Net framework includes, in sequence: a first convolutional module FC, an L-layer encoder module EB, an L-layer decoder module DB, and a second convolutional module LC.

2. The method according to claim 1, characterized in that, The network structure processing flow in step 1 includes: Step 11: Use the FC module to extract features from the input single image; Step 12: Extract and compress the features from Step 11 using multiple EB modules. Each EB module includes a downsampling module and a convolution module, and each EB module receives the output features from the previous EB module. Step 13: Use multiple DB modules to extract and augment the features output in Step 12. The DB consists of a convolution module and an upsampling module. Each DB module receives the output features of the DB module in the previous layer and the output features of the EB module in the same layer. Step 14: Use the LC module to convert the image features output in Step 13 into a denoised image.

3. The method according to claim 2, characterized in that, The coarse-grained population initialization method in step 2 includes: Step 21: Iterate through the convolution type set to obtain the convolution types; Step 22: Based on the convolution type obtained in Step 21, traverse the set of downsampling types to obtain the downsampling types; Step 23: Based on the downsampling type obtained in Step 22, iterate through the upsampling type set to obtain the upsampling type; Step 24: Based on step 23, assign the convolution type and downsampling type to the convolution type parameter and downsampling type parameter of the L-layer encoder module, assign the encoder scale type of the 1st layer encoder module to 1, and assign the encoder scale type of the 2nd to Lth layer encoder modules to 0; assign the convolution type and upsampling type to the convolution type parameter and upsampling type parameter of the L-layer decoder module to complete the generation of coarse-grained individuals; Step 25: Repeat steps 21, 22 and 23 to traverse all convolution types, downsampling types and upsampling types, and complete the generation of all coarse-grained individuals; Step 26: Using the self-supervised denoising method of DIP-STE, calculate the PSNR value for all coarse-grained individuals generated in Step 25, and retain the top N individuals with the highest PSNR values ​​as the initial population for the search algorithm.

4. The method according to claim 3, characterized in that, The distance-guided parent selection method in step 3 includes the following processing steps: Two individuals p are selected using a binary bidding process. i and p j Calculate the distance (p) between the two individuals. i ,p j The calculation formula is as follows: Wherein d(EB) ik ,EB jk ) represents the distance between the encoder modules at the corresponding positions of the two individuals, d(DB) ik DB jk ) represents the distance between corresponding decoder modules in two volumes; EB ik and EB jk Representing individual p respectively i and p j The k-th EB module; DB ik and DB jk Representing individual p respectively i and p j The k-th DB module; The distance d(EB) between the encoder modules at the corresponding positions of the two individuals ik ,EB jk The calculation formula for ) is as follows: d(EB ik ,EB jk )=eq(st ik ,st jk )+eq(dt ik ,dt jk )+eq(ect jk ,ect jk ) Among them, st ik and st jk They are EB ik With EB jk The scale type value, dt ik and dt jk They are EB ik With EB jk The downsampling type value, ect jk and ect jk They are EB ik With EB jk The convolution type value; The distance d(DB) between the corresponding decoder modules in the two bodies ik DB jk The calculation formula for ) is as follows: d(DB ik ,DB jk )=eq(ut ik ,ut jk )+eq(dct ik ,dct jk ) Among them, ut ik and ut jk They are DB ik With DB jk upsampled type value, dct ik and dct jk They are DB ik With DB jk The convolution type value; eq(a,b) indicates whether the two parameter values ​​are equal; Determine distance(p) i ,p j Does it meet the threshold μ? If it is greater than μ, then it means p i and p j If the differences between the individuals are significant, they can be used as parent individuals; otherwise, the binary bidding process is repeated to select two new individuals. If the individual distance still does not meet μ after 10 repetitions, the last selected individual is used as the parent individual.

5. The method according to claim 4, characterized in that, The modular crossover and mutation processing flow in step 4 includes: Step 41: Randomly generate a decimal r1 within the range [0,1]; Step 42: If r1 is greater than or equal to the crossover rate r c If so, then the parent generation crossover operation is performed; Step 43: Under the condition of satisfying step 42, randomly select a module position pos1 from all modules of the individual; Step 44: In the parent individual p i and p j In the process, select the modules at position pos1 respectively, and swap all the corresponding parameter values ​​in these two modules to generate two new individuals o. i o j ; Step 45: If r1 is less than the crossover rate r c If p, then no crossover operation is performed. i p j Assign directly to o i o j ; Step 46: Randomly generate a decimal r2 within the range [0,1]; Step 47: If r² is greater than or equal to the mutation rate r m For individual o i Perform mutation operations; Step 48: Based on step 47, randomly select a module position pos2 from all modules of the individual; Step 49: In individual o i In the middle, select the module at position pos2, and regenerate all parameters of the module randomly to generate a new individual o′. i ; Step 410: If r² is less than the mutation rate r m If the mutation operation is not performed, o i Assign directly to o′ i ; Step 411: Repeat steps 46, 47, 48, 49, and 50 to complete. j The mutation operation generates a new offspring individual o′ j Complete the modular crossover and mutation operations of the parent individuals.

6. The method according to claim 5, characterized in that, In step 6, the process for selecting the environment is as follows: Step 61: From the newly generated child set Q t and population set P t Union of P t ∪Q t Five elite individuals with high fitness are selected and placed into the next generation population. t+1 And from P t ∪Q t Remove elite; Step 62: Using a binary bidding selection method, select P after processing in step 61. t ∪Q t Select individual p from the set; Step 63: Place the individual p selected in step 62 into the next generation population P. t+1 In the middle, from P t ∪Q t Remove individual p from the middle; Step 64: Repeat step 63 until P. t+1 When the number of individuals reaches m, the environment selection operation is completed.

7. The method according to claim 1, characterized in that, The encoder scale types include: The k-th layer encoder EB k It does not accept output features from the FC module, but only accepts output features from the previous layer encoder. and the k-th layer encoder EB k Receive FC module output characteristics and the output features of the previous encoder layer 8. The method according to claim 1, characterized in that, The downsampling types include: max pooling layer, downsampling convolution, bilinear interpolation, nearest neighbor interpolation, and region interpolation.

9. A neural network structure search device for self-supervised image denoising, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the neural network structure search method for self-supervised image denoising as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the self-supervised image denoising neural network structure search method as described in any one of claims 1 to 8.

Images