Image processing model training method and device, and computer readable storage medium

Abstract

Embodiments of the present application provide a training method and device of an image processing model, and a computer readable storage medium, the image processing model comprising a U-shaped network, the U-shaped network comprising a down-sampling path and an up-sampling path, the down-sampling path comprising a first down-sampling module, the up-sampling path comprising a first up-sampling module, the first up-sampling module being connected to the first down-sampling module to multiplex features output by the first down-sampling module, the method comprising: inputting training data into the image processing model to obtain a training loss of the image processing model; updating model parameters of the image processing model according to the training loss; in the process of updating the model parameters, performing neural network architecture search on a network architecture between the first down-sampling module and the first up-sampling module to determine a structure of an attention module between the first down-sampling module and the first up-sampling module.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a training method and apparatus for an image processing model, and a computer-readable storage medium. Background Technology

[0002] In U-shaped mesh-based image processing models, the attention structure between the downsampling and upsampling modules of the U-shaped mesh is usually designed manually, which results in poor image processing performance. Summary of the Invention

[0003] To address the aforementioned issues, embodiments of this application provide a training method and apparatus for an image processing model, as well as a computer-readable storage medium.

[0004] In a first aspect, a training method for an image processing model is provided. The image processing model includes a U-shaped network, which includes a downsampling path and an upsampling path. The downsampling path includes a first downsampling module, and the upsampling path includes a first upsampling module. The first upsampling module is connected to the first downsampling module to reuse the features output by the first downsampling module. The method includes: inputting training data into the image processing model to obtain the training loss of the image processing model; updating the model parameters of the image processing model according to the training loss; and, during the process of updating the model parameters, performing a neural network architecture search on the network architecture between the first downsampling module and the first upsampling module to determine the structure of the attention module between the first downsampling module and the first upsampling module.

[0005] As one possible implementation, the first downsampling module and the first upsampling module are connected through multiple attention modules, each of which has its own corresponding connection weights. The neural network architecture search is used to update the connection weights of each of the multiple attention modules.

[0006] As one possible implementation, the method further includes: selecting a target attention module from the plurality of attention modules based on the changing trend of the connection weights corresponding to each of the plurality of attention modules, as the attention module to be retained after the U-shaped network training is completed, wherein the connection weights corresponding to the target attention module show an increasing trend during the network structure search process.

[0007] As one possible implementation, the plurality of attention modules may include some or all of the following attention modules: channel attention module, spatial attention module, and hybrid attention module.

[0008] As one possible implementation, the image processing model is an image denoising model, which includes a noise estimation subnetwork and a non-blind denoising subnetwork, and the U-shaped network is the non-blind denoising subnetwork.

[0009] Secondly, a training apparatus for an image processing model, the image processing model including a U-shaped network, the U-shaped network including a downsampling path and an upsampling path, the downsampling path including a first downsampling module, the upsampling path including a first upsampling module, the first upsampling module being connected to the first downsampling module to reuse the features output by the first downsampling module, the apparatus including: an input module for inputting training data into the image processing model to obtain the training loss of the image processing model; an update module for updating the model parameters of the image processing model according to the training loss; and a search module for performing a neural network architecture search on the network architecture between the first downsampling module and the first upsampling module during the process of updating the model parameters to determine the structure of the attention module between the first downsampling module and the first upsampling module.

[0010] As one possible implementation, the first downsampling module and the first upsampling module are connected through multiple attention modules, each of which has its own corresponding connection weights. The neural network architecture search is used to update the connection weights of each of the multiple attention modules.

[0011] As one possible implementation, the device further includes: a selection module, configured to select a target attention module from the plurality of attention modules based on the changing trend of the connection weights corresponding to each of the plurality of attention modules, as the attention module to be retained after the U-shaped network training is completed, wherein the connection weights corresponding to the target attention module show an increasing trend during the network structure search process.

[0012] As one possible implementation, the plurality of attention modules may include some or all of the following attention modules: channel attention module, spatial attention module, and hybrid attention module.

[0013] As one possible implementation, the image processing model is an image denoising model, which includes a noise estimation subnetwork and a non-blind denoising subnetwork, and the U-shaped network is the non-blind denoising subnetwork.

[0014] Thirdly, a computer-readable storage medium is provided having executable code stored thereon, which, when executed, enables the implementation of the method as described in the first aspect or any implementation thereof.

[0015] Fourthly, a computer program product is provided, including executable code, which, when executed, enables the implementation of the method as described in the first aspect or any implementation thereof.

[0016] The technical solution provided in this application, during the training of an image processing model, can efficiently and accurately search for a suitable attention module structure by searching the network architecture between the downsampling and upsampling modules in a U-shaped network. Compared with traditional solutions where the attention module structure is manually designed, this solution can improve the image processing performance based on the U-shaped network. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of a convolutional blind denoising network provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the structure of a convolutional blind denoising network provided in another embodiment of this application.

[0019] Figure 3 This is a schematic diagram of the structure of an attention module provided in an embodiment of this application.

[0020] Figure 4 This is a schematic flowchart illustrating an image processing model training method provided in an embodiment of this application.

[0021] Figure 5 This is a schematic diagram illustrating the dynamic changes in the connection weights of the attention module provided in an embodiment of this application.

[0022] Figure 6 This is a schematic diagram of the structure of an attention module provided in another embodiment of this application.

[0023] Figure 7 This is a schematic diagram of the structure of an image processing model training device provided in an embodiment of this application.

[0024] Figure 8 This is a schematic diagram of the structure of an image processing model training device provided in another embodiment of this application. Detailed Implementation

[0025] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0026] With the development of computer technology, image processing technology has become increasingly mature. Image processing has been widely used and played a crucial role in many fields such as military technology, industrial and agricultural production, and scientific research. Depending on the application scenario, image processing can also include applications such as image restoration, cloud removal, rain removal, fog removal, noise reduction, and blur removal.

[0027] In recent years, with the development of artificial intelligence technology, image processing based on neural networks has also developed rapidly. For example, U-net networks (often also known as U-shaped networks) have been widely used in the field of image processing technology and have achieved superior performance compared to traditional image processing methods. The following section combines... Figure 1 Taking image denoising as an example, this paper provides a detailed description of image denoising based on U-shaped networks.

[0028] In the field of image denoising technology, training U-shaped network models typically uses customized noise datasets (such as Gaussian white noise, impulse noise, etc.). However, the noise distribution of images in real-world applications is usually unknown, making it difficult for models trained on customized datasets to meet practical application requirements. To address the issue of U-shaped network models having high requirements for training and testing datasets, a convolutional blind denoising scheme using dual-network training has been proposed. The network structure framework of the convolutional blind denoising network is shown in the figure. 1 As shown, the convolutional blind denoising network (CBDNet) 10 consists of a noise estimation subnetwork (CNNE) 11 and a non-blind denoising subnetwork (CNND) 12. The noise estimation subnetwork 11 can be a fully convolutional network used to estimate noise signals in real-world application scenarios. The non-blind denoising subnetwork 12 can be a pre-designed U-shaped network structure. For example, the convolutional blind denoising network 10 can first use the noise estimation subnetwork 11 to estimate the noise signal of the actual input image, and then use the estimated noise signal and the actual input image as training and validation datasets to train the U-net-based non-blind denoising subnetwork 12. Once the U-shaped network model is trained, the non-blind denoising subnetwork 12 can be applied to image denoising.

[0029] The denoising principle based on U-shaped networks is as follows: First, the image to be denoised is acquired. Then, this image is input into the downsampling path of the U-shaped network for downsampling processing (this downsampling processing can be, for example, convolution and pooling operations) to extract the original image feature map data, thereby achieving noise removal. Next, the upsampling process in the upsampling path restores the feature map data output from the downsampling path to its original resolution. Since some image feature information is lost during downsampling, existing technologies typically use skip-connection channels (also called jump connection structures or skip connection channels) to concatenate and fuse the feature map data output from the downsampling layer in the downsampling channel with the feature map data from the corresponding upsampling layer in the upsampling channel to supplement the lost data information and restore the original resolution. In practical applications, compared with traditional denoising methods (such as filtering and nonlocal methods), the image denoising method based on U-shaped networks achieves better denoising results.

[0030] In practice, image denoising methods based on U-shaped networks typically reuse the full-channel feature data output from the corresponding downsampling layer using skip connections in the upsampling process. However, careful research reveals that feature data from different channels in the downsampling layer's output feature map generally contribute differently to the lower-level network, and different pixels in different channels also contribute differently. In real-world applications, the full-channel feature data output from the downsampling layer may not be needed during the upsampling process. In other words, existing techniques for feature reuse are relatively simplistic and neglect the issue of selective feature reuse. Performing full-channel stitching and fusion may lead to data redundancy, resulting in reduced accuracy of the denoised image.

[0031] To address the data redundancy and low image accuracy issues caused by full-channel data stitching and fusion during upsampling, this application proposes another convolutional blind denoising network20, such as... Figure 2 As shown. The convolutional blind denoising network 20 is similar to the one described above. Figure 1 Compared to the convolutional blind denoising network 10, the main difference lies in the inclusion of an attention module structure in the skip connection channels of the convolutional blind denoising network 20. This embodiment of the application, by adding an attention module structure to the skip connection channels of the U-shaped network and utilizing the selective effect of the attention mechanism, can achieve selective reuse of feature data in the downsampling path, thereby improving the accuracy of the denoised image.

[0032] In some embodiments, the U-shaped network may include multiple hop connection channels, and each hop connection channel may have one or more attention modules added manually. These attention modules may include, for example, channel attention blocks (CABs), spatial attention blocks (SABs), mixed attention blocks (MABs), triplet attention blocks (TABs), etc. This application does not specifically limit the type of attention module; for example, the attention module may also include ECANet, SANet, ResNeSt, etc.

[0033] like Figure 3 As shown, this application embodiment provides a possible attention module structure 30, which can be set in the jump connection channels of a U-shaped network. The attention module structure 30 may include, for example, a channel attention module 31, a spatial attention module 32, and a hybrid attention module 33 connected in parallel. Each of the three attention module channels corresponds to a connection weight; for example, channel attention module 31 corresponds to weight parameter γ1, spatial attention module 32 corresponds to parameter weight γ2, and hybrid attention module channel 33 corresponds to weight parameter γ3. Feature map data is input to the input of the attention module structure 30, and the output feature map data is obtained by weighted summation of the three attention module channels.

[0034] However, manually setting the attention module in the hop connection channels of a U-network is often inefficient and has a certain degree of randomness. For example, using... Figure 3 Taking the attention module structure 30 in the network as an example, in practical applications, to determine the structure of the attention modules in the hop connection channels, one possible approach is to manually and randomly select one or more from the channel attention module 31, spatial attention module 32, and hybrid attention module 33, and place them into the hop connection channels of the U network to test the results. Then, other attention modules or combinations of attention modules are continuously tested. This method essentially iterates through all attention module combinations to find the best-performing one, but this process is black-box and requires extensive verification, making it relatively inefficient. Another possible approach is to manually select one or more attention modules based on experience. While this method is more efficient, manually selected solutions often have a degree of randomness and cannot guarantee optimality.

[0035] It should be noted that the above example only illustrates image denoising based on U-shaped networks, but the application scenarios of this application are not limited to this. This application can be applied to any type of image processing application based on U-shaped networks. For example, applications such as image restoration, cloud removal, rain removal, fog removal, denoising, and deblurring based on U-shaped networks.

[0036] In conclusion, as people's requirements for image processing performance become increasingly demanding, how to efficiently and accurately select the attention module in the attention structure to improve the image processing performance based on U-shaped networks remains a technical problem that urgently needs to be solved.

[0037] The technical solution provided in this application, during the training of an image processing model, can efficiently and accurately search for a suitable attention module structure by searching the network architecture between the downsampling and upsampling modules in a U-shaped network. Compared with traditional solutions where the attention module structure is manually designed, this solution can improve the image processing performance based on the U-shaped network.

[0038] The image processing model provided in this application includes a U-shaped network. This U-shaped network includes a downsampling path and an upsampling path. The downsampling path includes a first downsampling module, and the upsampling path includes a first upsampling module. The first upsampling module is connected to the first downsampling module to reuse the feature data output by the first downsampling module. It is understood that this application does not impose a specific limit on the number of first downsampling modules and first upsampling modules in the U-shaped network. For example, the U-shaped network may include one or more first downsampling modules, and it may also include one or more first upsampling modules. The U-shaped network is generally a symmetric neural network; therefore, the number of first upsampling modules can be consistent with the number of first downsampling modules, and first downsampling modules and first upsampling modules with the same resolution can be placed in the same layer. The downsampling module can also be called a downsampling layer, and the upsampling module can also be called an upsampling layer.

[0039] The following text combines Figure 4 This application introduces an image processing model training method according to an embodiment. Figure 4 The method shown includes steps S42 to S46.

[0040] In step S42, the training data is input into the image processing model to obtain the training loss of the image processing model.

[0041] In some embodiments, taking image denoising as an example, the image processing model can be a dual-network image denoising model, which may include, for example, a noise estimation subnetwork and a non-blind denoising subnetwork, wherein the non-blind denoising subnetwork can be a U-shaped network.

[0042] This application does not impose specific restrictions on the method of obtaining training data. For example, the training data may be training data generated according to a deep learning algorithm.

[0043] This application does not impose specific restrictions on the type of training loss calculation for the image processing model. For example, the training loss can be calculated using the mean squared error (MSE) function or the mean absolute error (MAE) function.

[0044] In step S44, the model parameters of the image processing model are updated based on the training loss.

[0045] To minimize training loss, the image processing model needs to be trained. In some possible implementations, the backpropagation algorithm (BP) can be used to update the parameters of the image processing model. When the image processing model is a U-shaped network model, gradient descent can be used, layer by layer, to calculate the gradient values ​​corresponding to the optimization objective of the U-shaped network, which serve as the basis for updating the parameters of the U-shaped network model. The training process of the image processing model ends when the training loss reaches the desired value, or when the image processing model converges.

[0046] In step S46, during the process of updating the model parameters, a neural network architecture search is performed on the network architecture between the first downsampling module and the first upsampling module to determine the structure of the attention module between the first downsampling module and the first upsampling module.

[0047] In some embodiments, the network structure between the first downsampling module and the first upsampling module can be an attention module structure, which may be, for example, a parallel connection structure of multiple attention modules. These multiple attention modules may include, for example, channel attention modules, spatial attention modules, hybrid attention modules, triple attention modules, etc., and more attention modules may be set in the attention module structure as needed; this application embodiment does not impose specific limitations in this regard.

[0048] For example, such as Figure 3As shown, a search space can be defined, which may include, for example, a channel attention module 31, a spatial attention module 32, and a hybrid attention module 33. All attention modules in this search space are then connected in parallel between the first downsampling module and the first upsampling module. Next, a search is performed on the network architecture between the first downsampling module and the first upsampling module to efficiently and accurately find a suitable attention module structure. This allows the first upsampling module to accurately select and reuse the feature data output by the first downsampling module, further improving the image processing performance based on the U-shaped network.

[0049] As can be seen from the above, the technical solution provided in this application, during the training of the image processing model, can efficiently and accurately search for a suitable attention module structure by searching the network architecture between the downsampling and upsampling modules in the U-shaped network. Compared with the traditional solution where the attention module structure is manually designed, this solution can improve the image processing performance based on the U-shaped network.

[0050] In some embodiments, the network structure between the first downsampling module and the first upsampling module can be multiple attention modules connected in parallel, each attention module having its own corresponding connection weight on its connection channel. When performing a neural network architecture search on the network architecture between the first downsampling module and the first upsampling module, the connection weights of each attention module can be updated. It is understood that the neural network architecture search between the first downsampling module and the first upsampling module is performed during the training of the image processing model. The model parameters of the image processing model include the architecture parameters of the U-shaped network itself and the connection weight parameters of the attention modules. That is, updating the model parameters of the image processing model includes not only updating the connection weight parameters of the attention modules but also updating the architecture parameters of the U-shaped network. This application does not impose specific limitations on the method for updating the model parameters of the image processing model; for example, the Adam optimization method or the conventional gradient descent method can be used.

[0051] As an example, this application embodiment can simultaneously use the Adam optimization method to alternately update the connection weights corresponding to multiple attention modules and the architectural parameters of the U-shaped network until the training loss meets the requirements or all parameters converge.

[0052] In some embodiments, taking image denoising as an example, during the search process, the objective performance evaluation metrics of the network can be peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as metrics.

[0053] In some embodiments, after all parameters of the image processing model converge, the attention module with the largest connection weight parameter among multiple attention modules can be selected as the final retained attention module, while other attention modules are discarded. The attention module with the largest connection weight parameter is then set between the first downsampling module and the first upsampling module to form the final network architecture.

[0054] Careful research revealed that in pixel-level dense prediction tasks such as image processing, once the connection weights of multiple attention modules converge, the differences between the converged weights are usually not very significant, making it difficult to choose which candidate attention modules to select.

[0055] This application provides an embodiment of a possible dynamic change curve for the weight parameters of attention modules. For example, dynamic change curves for the six connection weight parameters corresponding to six attention modules during the training process of an image processing model are given, such as... Figure 5 As shown, the connection weight parameters for each attention module converged to 0.167. Under normal circumstances, using scientific computing and retaining three significant decimal places is sufficient to meet the accuracy requirements. That is to say, Figure 5 The difference in the final converged connection weight parameters appears after the third significant decimal place. As an example, retaining five significant decimal places for connection weight 1 and connection weight 4, experiments show that connection weight 1 converged to 0.16783, and connection weight 4 converged to 0.16721. From the experimental data above, it can be seen that the final converged values ​​of the connection weights do not differ significantly, making the selection of candidate attention modules difficult.

[0056] To address the difficulty in selecting attention modules, this application proposes a training scheme for an image processing model. Based on the changing trends of the connection weights of multiple attention modules, a target attention module is selected as the one to be retained after the U-shaped network training is completed. The connection weight parameters of this target attention module show an increasing trend during the network structure search process. In other words, this application selects attention modules with all connection weight parameters showing an increasing trend as the final retained attention modules, while directly discarding attention modules with non-increasing connection weight parameters. The reason for this selection is that experimental research has shown that selecting attention modules with increasing connection weight trends is beneficial for image processing, improving the accuracy of feature data selection and reuse, thereby enhancing the image processing performance based on the U-shaped network.

[0057] It should be noted that after rebuilding the network structure based on all the retained attention modules, the searched image processing model structure also needs to be retrained to obtain the final network model.

[0058] The following example will still be image denoising, combined with... Figure 2 and Figure 3 The specific implementation method of the image processing network based on searchable feature reuse in the embodiments of this disclosure is described in detail. The specific implementation steps are as follows:

[0059] (1) Search space definition: Channel attention module, spatial attention module, and hybrid attention module, commonly used in image denoising, are employed. For example... Figure 3 As shown, the output of the attention module structure is achieved by weighted summation of the three components.

[0060] (2) Network structure definition: The network structure is directly adopted as follows: Figure 2 The network structure in [the context].

[0061] (3) Definition of metrics: In the search process, the objective metrics for evaluating network performance are peak signal-to-noise ratio and structural similarity.

[0062] (4) Network parameter optimization: The entire network contains two sets of parameters: connection weight parameters of the attention module and network architecture parameters. The network architecture parameters are the parameters in the original network (i.e., the weight parameters in the original network). Then, the Adam optimization method is used to update the two sets of parameters alternately until all parameters converge.

[0063] (5) Observe the evolution trend of architecture parameters and select attention module structure accordingly: select attention modules with all connection weight parameters showing an upward trend as the final attention module structure to be retained, and at the same time, directly discard attention modules that do not show an upward trend.

[0064] (6) Retraining the network model: Reconstruct the network structure based on the retained attention module structure, retrain the searched model structure, and obtain the final usable network model.

[0065] like Figure 6 As shown in the embodiments of this application, specific structures of typical channel attention modules, spatial attention modules, and hybrid attention modules are also provided.

[0066] In some embodiments, the channel attention module can calculate the contribution weights of feature map data from different channels of the feature map to the lower-layer network; that is, feature map data from different channels have different influence factors on the lower-layer network. For example... Figure 6As shown in (a), the channel weight calculation branch of the attention module structure 61 can calculate the weight parameters of different channels, and then multiply them back into the original feature map, so that the feature map can be better represented. For example, the channel weight calculation branch of the channel attention module structure 61 may include operations such as pooling, fully connected layers and linear activation, fully connected layers and nonlinear activation, etc., and the embodiments of this application do not impose specific limitations on this.

[0067] In some embodiments, the spatial attention module can calculate the weights of the relationships between pixels in the feature map, meaning that pixels at different spatial locations have different influence factors on the lower-level network. For example... Figure 6 As shown in (b), the pixel weight calculation branch of the spatial attention module structure 62 can calculate the pixel weight parameters at different spatial locations, and then multiply them back into the original feature map, so that the feature map can be better represented. Exemplarily, the pixel weight calculation branch of the spatial attention module structure 62 may include average pooling, max pooling, normalization and other operations, and this application embodiment does not impose specific limitations on this.

[0068] In some embodiments, the hybrid attention module may be composed of a spatial attention module and a channel attention module connected in series, used to calculate channel weight parameters and pixel weight parameters in the feature map. This application does not impose specific restrictions on the connection order of the spatial attention module and the channel attention module in the hybrid attention module. For example, Figure 6 As shown in (c), the hybrid attention module structure 63 can first pass through the spatial attention module to calculate the pixel weight parameters, and then concatenate them with the original feature map as the input of the channel attention module.

[0069] This application also provides an image processing method. The image processing model on which the image processing method is based includes a U-shaped network. The U-shaped network includes a downsampling path and an upsampling path. The downsampling path includes a first downsampling module, and the upsampling path includes a first upsampling module. A skip connection structure can be set between the first downsampling module and the first upsampling module. The skip connection structure includes multiple attention modules. The image processing method includes: inputting the image to be processed into the image processing model to obtain the noise of the image to be denoised; then, inputting the image to be denoised and the noise of the image to be denoised into the U-shaped network to obtain the denoised image.

[0070] As one possible implementation, multiple attention modules can be set between the first downsampling module and the first upsampling module, and these multiple attention modules can have their own corresponding connection weights.

[0071] As one possible implementation, multiple attention modules may include some or all of the following: channel attention module, spatial attention module, and hybrid attention module.

[0072] As can be seen from the above, the image processing network based on searchable feature reuse proposed in this application has several advantages by replacing the skip connection structure in the U-shaped network with a searchable attention module structure. Firstly, the use of the attention module structure enables selective reuse of feature map data in the downsampling path. Secondly, the availability of multiple selectable attention modules allows for automated searching of feature data reuse methods in downsampling through automatic network search.

[0073] The above text combined Figures 1 to 6 This application describes in detail an embodiment of the training method for the image processing model. The following section combines... Figure 7 and Figure 8 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the apparatus embodiments correspond to the descriptions of the method embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0074] Figure 7 This is a schematic structural diagram of a training apparatus for an image processing model provided in an embodiment of this application. The apparatus 70 may include: an input module 72, an update module 74, and a search module 76.

[0075] The image processing model includes a U-shaped network, which includes a downsampling path and an upsampling path. The downsampling path includes a first downsampling module, and the upsampling path includes a first upsampling module. The first upsampling module is connected to the first downsampling module to reuse the features output by the first downsampling module. The device 70 includes:

[0076] The input module 72 can be used to input training data into the image processing model to obtain the training loss of the image processing model;

[0077] The update module 74 can be used to update the model parameters of the image processing model based on the training loss;

[0078] The search module 76 can be used to perform a neural network architecture search on the network architecture between the first downsampling module and the first upsampling module during the process of updating the model parameters, so as to determine the structure of the attention module between the first downsampling module and the first upsampling module.

[0079] Optionally, the first downsampling module and the first upsampling module are connected through multiple attention modules, each of which has its own corresponding connection weights. The neural network architecture search is used to update the connection weights of each of the multiple attention modules.

[0080] Optionally, the device 70 further includes: a selection module 78 which can be used to select a target attention module from the plurality of attention modules according to the changing trend of the connection weights corresponding to each of the plurality of attention modules, as the attention module to be retained after the U-shaped network training is completed, wherein the connection weights corresponding to the target attention module show an increasing trend during the network structure search process.

[0081] Optionally, the plurality of attention modules may include some or all of the following attention modules: channel attention module, spatial attention module, and hybrid attention module.

[0082] Optionally, the image processing model is an image denoising model, which includes a noise estimation subnetwork and a non-blind denoising subnetwork, and the U-shaped network is the non-blind denoising subnetwork.

[0083] Figure 8 This is a schematic structural diagram of the training device for the image processing model provided in the embodiments of this application. Figure 8 The dashed lines indicate that the unit or module is optional. The device 80 can be used to implement the methods described in the above method embodiments. The device 80 can be a chip, a terminal, or a network device.

[0084] The apparatus 80 may include one or more processors 81. The processor 81 can support the apparatus 80 in implementing the methods described in the preceding method embodiments. The processor 81 may be a general-purpose processor or a special-purpose processor. For example, the processor may be a central processing unit (CPU). Alternatively, the processor may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0085] The apparatus 80 may further include one or more memories 82. The memories 82 store a program that can be executed by the processor 81, causing the processor 81 to perform the methods described in the preceding method embodiments. The memories 82 may be independent of the processor 81 or integrated into the processor 81.

[0086] The device 80 may also include a transceiver 83. The processor 81 can communicate with other devices or chips via the transceiver 83. For example, the processor 81 can send and receive data with other devices or chips via the transceiver 83.

[0087] This application also provides a computer-readable storage medium storing executable code thereon, which, when executed, can implement the methods described in the above method embodiments.

[0088] This application also provides a computer program product having executable code stored thereon, which, when executed, can implement the methods described in the above method embodiments.

[0089] This application also provides a computer program that, when executed, can implement the methods described in the above method embodiments.

[0090] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any other combination. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0091] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0095] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for an image processing model, characterized in that, The image processing model includes a U-shaped network, which comprises a downsampling path and an upsampling path. The downsampling path includes a first downsampling module, and the upsampling path includes a first upsampling module. The first upsampling module and the first downsampling module are located in the same layer of the U-shaped network and have the same resolution. The first upsampling module is connected to the first downsampling module to reuse the features output by the first downsampling module. The method includes: The training data is input into the image processing model to obtain the training loss of the image processing model; The model parameters of the image processing model are updated based on the training loss. During the process of updating the model parameters, a neural network architecture search is performed on multiple connected attention modules in the network architecture between the first downsampling module and the first upsampling module to update the connection weights of each attention module, so as to determine at least one target attention module from the multiple connected attention modules according to the connection weights. The target attention module is the attention module that needs to be retained after the U-shaped network is trained. The method further includes: Based on the changing trends of the connection weights corresponding to the multiple attention modules, a target attention module is selected from the multiple attention modules as the attention module to be retained after the U-shaped network training is completed, wherein the connection weights corresponding to the target attention module show an upward trend during the neural network architecture search process.

2. The method according to claim 1, characterized in that, The plurality of attention modules include some or all of the following attention modules: channel attention module, spatial attention module, and hybrid attention module.

3. The method according to claim 1, characterized in that, The image processing model is an image denoising model, which includes a noise estimation subnetwork and a non-blind denoising subnetwork. The U-shaped network is the non-blind denoising subnetwork.

4. A training device for an image processing model, characterized in that, The image processing model includes a U-shaped network, which comprises a downsampling path and an upsampling path. The downsampling path includes a first downsampling module, and the upsampling path includes a first upsampling module. The first upsampling module and the first downsampling module are located in the same layer of the U-shaped network and have the same resolution. The first upsampling module is connected to the first downsampling module to reuse the features output by the first downsampling module. The device includes: The input module is used to input training data into the image processing model to obtain the training loss of the image processing model; An update module is used to update the model parameters of the image processing model based on the training loss. The search module is used to perform a neural network architecture search on multiple connected attention modules in the network architecture between the first downsampling module and the first upsampling module during the process of updating the model parameters, so as to update the connection weights of each attention module, so as to determine at least one target attention module from the multiple connected attention modules according to the connection weights, wherein the target attention module is the attention module that needs to be retained after the U-shaped network is trained. The device further includes: The selection module is used to select a target attention module from the multiple attention modules based on the changing trend of the connection weights corresponding to each of the multiple attention modules, so as to be retained as the attention module after the U-shaped network training is completed, wherein the connection weights corresponding to the target attention module show an increasing trend during the neural network architecture search process.

5. The apparatus according to claim 4, characterized in that, The plurality of attention modules include some or all of the following attention modules: channel attention module, spatial attention module, and hybrid attention module.

6. The apparatus according to claim 4, characterized in that, The image processing model is an image denoising model, which includes a noise estimation subnetwork and a non-blind denoising subnetwork. The U-shaped network is the non-blind denoising subnetwork.

7. A computer-readable storage medium having executable code stored thereon, characterized in that, When the executable code is executed, it can implement the method of any one of claims 1-3.

Images