Image denoising method and device, model training method and device, equipment and program product

Abstract

The invention relates to an image denoising method, a model training method and device, electronic equipment and a computer program product, relates to the technical field of image processing, and can be applied to a scene for denoising an image containing noise. The method comprises the steps of obtaining a to-be-processed image, and performing decomposition transformation processing on the to-be-processed image to obtain a plurality of frequency sub-graphs; performing feature extraction processing based on the plurality of frequency sub-images to obtain intermediate image features; performing up-sampling processing on the intermediate image features to obtain recovered image features, the up-sampling processing being an inverse process of decomposition transformation processing; and generating a de-noised image according to the restored image features and the to-be-processed image. According to the method, video memory occupation in the image processing process is effectively reduced, the calculation efficiency of the model is improved, and meanwhile, detail and texture information of the image is reserved; moreover, the image denoising effect is optimized, and the information loss in the denoising process is reduced.

Description

Background Art

[0002] In the field of image processing, denoising is a basic and critical task. With the rapid development of digital image acquisition and transmission technology, images are easily interfered by noise during acquisition, storage and transmission, which seriously affects the quality of images and the accuracy of subsequent image analysis. Traditional image denoising methods, such as Gaussian filtering, median filtering and bilateral filtering, can reduce noise to a certain extent, but usually cannot effectively retain the detailed information of the image, especially when the noise level is high, the performance of these methods will be significantly reduced.

[0003] In recent years, with the development of deep learning technology, image denoising methods based on convolutional neural networks (CNN) have made significant progress. These methods can better retain the detailed information of the image while removing noise by learning the deep features of the image.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an image denoising method, a model training method, an image denoising device, a model training device, an electronic device, a computer-readable storage medium and a computer program product, thereby at least to a certain extent overcoming the problem that related image denoising schemes cannot effectively balance the relationship between image denoising and detail retention, lack effective capture of local image features, and cause denoising performance to be improved.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0007] According to a first aspect of the present disclosure, there is provided an image denoising method, comprising: acquiring an image to be processed, performing a decomposition transformation process on the image to be processed to obtain a plurality of frequency sub-graphs; performing a feature extraction process based on the plurality of frequency sub-graphs to obtain intermediate image features; performing an up-sampling process on the intermediate image features to obtain restored image features, wherein the up-sampling process is an inverse process of the decomposition transformation process; and generating a denoised image based on the restored image features and the image to be processed.

[0008] In an exemplary embodiment of the present disclosure, the decomposing and transforming the image to be processed to obtain a plurality of frequency sub-images comprises: obtaining a pre-configured Haar wavelet basis, wherein the Haar wavelet basis comprises a specified number of filters; and using each of the filters to respectively perform decomposition and transform processing on the image to be processed to obtain the plurality of frequency sub-images, wherein the plurality of frequency sub-images comprises a plurality of sub-images with different frequencies.

[0009] In an exemplary embodiment of the present disclosure, the feature extraction processing based on the multiple frequency sub-graphs to obtain intermediate image features includes: splicing the multiple frequency sub-graphs to obtain initial input features; performing attention feature extraction on the initial input features to obtain attention image features; transforming and decomposing the attention image features to obtain multiple intermediate frequency sub-graphs; and performing feature extraction processing based on the multiple intermediate frequency sub-graphs to obtain the intermediate image features.

[0010] In an exemplary embodiment of the present disclosure, the step of performing attention feature extraction on the initial input feature to obtain an attention image feature comprises: performing self-attention feature extraction on the initial input feature to obtain a self-attention feature; obtaining a preconfigured gate function, wherein the gate function is a nonlinear function; performing channel attention feature extraction on the self-attention feature according to the gate function to obtain a channel attention feature; and generating the attention image feature based on the channel attention feature.

[0011] In an exemplary embodiment of the present disclosure, the self-attention feature extraction is performed on the initial input features to obtain the self-attention features, including: obtaining a pre-configured parameter matrix; performing linear transformation processing on the initial input features based on the parameter matrix to obtain corresponding query matrix, key matrix and value matrix; determining a weight matrix according to the query matrix and the key matrix, the weight matrix being used to represent the pixel weights of each pixel in the initial input features; and determining the self-attention features according to the weight matrix and the value matrix.

[0012] In an exemplary embodiment of the present disclosure, the extracting channel attention features from the self-attention features according to the gating function to obtain the channel attention features comprises: using the gating function to determine the feature calculation weights of the self-attention features; determining the channel input features according to the feature calculation weights and the self-attention features; and extracting channel attention features from the channel input features to obtain the channel attention features.

[0013] According to a second aspect of the present disclosure, a model training method is provided, comprising: obtaining a pre-constructed initial model and a training data set, the training data set comprising a sample image and a corresponding noise image; using the initial model to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features; performing upsampling processing based on the intermediate sample image features to obtain restored sample image features, the upsampling processing being an inverse process of the decomposition transformation processing; obtaining a predicted denoised image based on the restored sample image features and the sample image; training the initial model according to a loss value between the predicted denoised image and the sample image to obtain an image denoising model.

[0014] In an exemplary embodiment of the present disclosure, the initial model includes a downsampling layer and a feature extraction layer, and the use of the initial model to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features includes: obtaining the Haar wavelet basis in the downsampling layer, and the Haar wavelet basis includes a specified number of filters; using each of the filters to perform decomposition transformation processing on the noise image to obtain multiple sample frequency sub-graphs, and the sample frequency sub-graphs include multiple sub-images with different frequencies; splicing the multiple sample frequency sub-graphs to obtain initial sample input features; using the feature extraction layer to perform attention feature extraction on the initial sample input features to obtain the intermediate sample image features.

[0015] In an exemplary embodiment of the present disclosure, the feature extraction layer includes a self-attention extraction module and a channel attention extraction module, and the feature extraction layer is used to perform attention feature extraction on the initial sample input features to obtain the intermediate sample image features, including: using the self-attention extraction module to perform self-attention feature extraction on the initial sample input features to obtain self-attention sample features; using the channel attention extraction module to perform channel attention feature extraction on the self-attention sample features to obtain channel attention sample features; based on the channel attention sample features, generating the intermediate sample image features.

[0016] According to a third aspect of the present disclosure, there is provided an image denoising device, comprising: an image acquisition module, used to acquire an image to be processed, and perform decomposition and transformation processing on the image to be processed to obtain a plurality of frequency sub-graphs; a first feature extraction module, used to perform feature extraction processing based on the plurality of frequency sub-graphs to obtain intermediate image features; an image up-sampling module, used to perform up-sampling processing on the intermediate image features to obtain restored image features, wherein the up-sampling processing is an inverse process of the decomposition and transformation processing; and an image denoising module, used to generate a denoised image based on the restored image features and the image to be processed.

[0017] In an exemplary embodiment of the present disclosure, the image acquisition module includes a decomposition transformation unit, which is used to: obtain a pre-configured Haar wavelet basis, wherein the Haar wavelet basis includes a specified number of filters; use each of the filters to perform decomposition transformation processing on the image to be processed respectively to obtain the multiple frequency sub-images, wherein the multiple frequency sub-images include multiple sub-images with different frequencies.

[0018] In an exemplary embodiment of the present disclosure, the first feature extraction module includes an intermediate feature extraction unit, which is used to: concatenate the multiple frequency sub-graphs to obtain initial input features; perform attention feature extraction on the initial input features to obtain attention image features; perform transformation and decomposition on the attention image features to obtain multiple intermediate frequency sub-graphs; and perform feature extraction based on the multiple intermediate frequency sub-graphs to obtain the intermediate image features.

[0019] In an exemplary embodiment of the present disclosure, the intermediate feature extraction unit includes a feature extraction subunit, which is used to: perform self-attention feature extraction on the initial input feature to obtain a self-attention feature; obtain a preconfigured gate function, which is a nonlinear function; perform channel attention feature extraction on the self-attention feature according to the gate function to obtain a channel attention feature; and generate the attention image feature based on the channel attention feature.

[0020] In an exemplary embodiment of the present disclosure, the feature extraction subunit includes a first feature extraction subunit, which is used to: obtain a pre-configured parameter matrix; perform linear transformation processing on the initial input features based on the parameter matrix to obtain corresponding query matrix, key matrix and value matrix; determine a weight matrix based on the query matrix and the key matrix, and the weight matrix is ​​used to represent the pixel weights of each pixel in the initial input features; determine the self-attention features based on the weight matrix and the value matrix.

[0021] In an exemplary embodiment of the present disclosure, the feature extraction subunit includes a second feature extraction subunit, which is used to: determine the feature calculation weight of the self-attention feature using the gate function; determine the channel input feature based on the feature calculation weight and the self-attention feature; perform channel attention feature extraction on the channel input feature to obtain the channel attention feature.

[0022] According to a fourth aspect of the present disclosure, a model training device is provided, comprising: a data set acquisition module, used to acquire a pre-built initial model and a training data set, wherein the training data set comprises a sample image and a corresponding noise image; a second feature extraction module, used to perform a decomposition transformation process and a feature extraction process on the noise image using the initial model to obtain intermediate sample image features; an upsampling module, used to perform an upsampling process based on the intermediate sample image features to obtain restored sample image features, wherein the upsampling process is an inverse process of the decomposition transformation process; an image restoration module, used to obtain a predicted denoised image based on the restored sample image features and the sample image; and a model training module, used to train the initial model according to a loss value between the predicted denoised image and the sample image to obtain an image denoising model.

[0023] In an exemplary embodiment of the present disclosure, the initial model includes a downsampling layer and a feature extraction layer, and the second feature extraction module includes a second feature extraction unit, which is used to: obtain the Haar wavelet basis in the downsampling layer, and the Haar wavelet basis includes a specified number of filters; use each of the filters to decompose and transform the noise image to obtain a plurality of sample frequency sub-graphs, and the sample frequency sub-graphs include a plurality of sub-images with different frequencies; splice the plurality of sample frequency sub-graphs to obtain initial sample input features; use the feature extraction layer to perform attention feature extraction on the initial sample input features to obtain the intermediate sample image features.

[0024] In an exemplary embodiment of the present disclosure, the feature extraction layer includes a self-attention extraction module and a channel attention extraction module, and the second feature extraction unit includes a second feature extraction sub-unit, which is used to: use the self-attention extraction module to perform self-attention feature extraction on the initial sample input feature to obtain self-attention sample features; use the channel attention extraction module to perform channel attention feature extraction on the self-attention sample features to obtain channel attention sample features; based on the channel attention sample features, generate the intermediate sample image features.

[0025] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the image denoising method according to any one of the above items is implemented, or the model training method according to any one of the above items is implemented.

[0026] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image denoising method according to any one of the above items is implemented, or the model training method according to any one of the above items is implemented.

[0027] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the image denoising method described in any one of the above items, or implements the model training method described in any one of the above items.

[0028] The technical solution provided by the present disclosure may have the following beneficial effects:

[0029] The image denoising method in the exemplary embodiment of the present disclosure, on the one hand, downsamples the image by decomposing and transforming it, and upsamples it by using the inverse process, which effectively reduces the video memory usage during the image processing process, improves the computational efficiency of the model, and retains the details and texture information of the image. On the other hand, a denoised image is generated based on the restored image features, which optimizes the image denoising effect and reduces the information loss during the denoising process.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0032] Figure 1 Schematically shows a flowchart of an image denoising method and a model training method according to an exemplary embodiment of the present disclosure;

[0033] Figure 2 The overall structure diagram of the image denoising solution according to an exemplary embodiment of the present disclosure is schematically shown;

[0034] Figure 3 Schematically shows a structural diagram of a NAFT module according to an exemplary embodiment of the present disclosure;

[0035] Figure 4 A flowchart of a model training method according to an exemplary embodiment of the present disclosure is schematically shown;

[0036] Figure 5 A block diagram of an image denoising apparatus according to an exemplary embodiment of the present disclosure is schematically shown;

[0037] Figure 6 A block diagram of a model training device according to an exemplary embodiment of the present disclosure is schematically shown;

[0038] Figure 7 A schematic diagram schematically illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure;

[0039] Figure 8 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0041] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0043] Among the related schemes, CNN-based image denoising methods have made significant progress. However, when processing large-size images, CNN is usually difficult to capture the global information of the image due to the limitations of its local receptive field of view, which limits the further improvement of its denoising performance. In addition, the Transformer model, due to its self-attention mechanism, can effectively capture the long-distance dependencies of images and has demonstrated strong performance in fields such as natural language processing and image recognition. However, when the Transformer is directly applied to the image denoising task, its denoising performance still needs to be improved due to its sensitivity to the resolution of the input image and the lack of effective capture of local features of the image.

[0044] Based on this, the present disclosure proposes an image denoising method, a model training method, an image denoising device, a model training device, an electronic device, a computer-readable storage medium, and a computer program product.

[0045] In this article, it is important to understand that the terms involved, such as the Transformer model, are a deep learning model architecture that was originally used for natural language processing tasks, but due to its powerful self-attention mechanism, it has now been widely used in various tasks such as image processing and time series analysis. The Transformer model can effectively capture long-range dependencies and generate new feature representations by calculating the correlation between each element in the input data (i.e., attention weights).

[0046] Multi-level discrete wavelet transform (DWT) is a signal processing technique used to decompose a signal into a set of wavelet coefficients of different scales and frequencies. These wavelet coefficients contain various local feature information of the signal and can be used to represent different frequency components of the signal.

[0047] Inverse Discrete Wavelet Transform (IDWT) is the inverse operation of discrete wavelet transform, which is used to gradually synthesize the decomposed wavelet coefficients and finally restore the original signal.

[0048] Nonlinear Activation Free Transformer (NAFT) is a Transformer module that does not contain traditional nonlinear activation functions (such as ReLU and Sigmoid). NAFT uses the self-attention mechanism to extract features and uses gate functions instead of activation functions to avoid nonlinear distortion.

[0049] In this example embodiment, an image denoising method is first provided. The image denoising method of the present disclosure may be implemented using a server, or the method of the present disclosure may be implemented using a terminal device, wherein the terminal described in the present disclosure may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, and personal digital assistants (PDAs), as well as fixed terminals such as desktop computers. Figure 1 The following is a schematic diagram of the image denoising method process according to some embodiments of the present disclosure. Figure 1 The image denoising method and model training method may include the following steps:

[0050] Step S110, obtaining an image to be processed, performing decomposition and transformation processing on the image to be processed, and obtaining a plurality of frequency sub-images.

[0051] According to some exemplary embodiments of the present disclosure, the image to be processed may be an image to be subjected to denoising. The decomposition transformation process may be a process of downsampling the image to be processed to effectively extract high-frequency and low-frequency information in the image to be processed. The frequency sub-image may be a sub-image with different frequency information (such as high-frequency information and low-frequency information) generated after the image to be processed is subjected to transformation and decomposition processing.

[0052] The image to be processed may be an image that is interfered by noise during the process of image acquisition, storage and transmission. After obtaining the image to be processed containing noise, the image to be processed is first downsampled, that is, decomposed and transformed, to extract the high-frequency information and low-frequency information contained in the image to be processed, and obtain multiple frequency sub-images corresponding to the image to be processed, which helps to retain the detail information and texture information in the image to be processed.

[0053] Step S120: performing feature extraction processing based on the multiple frequency sub-graphs to obtain intermediate image features.

[0054] According to some exemplary embodiments of the present disclosure, the intermediate image features may be image features obtained after deep feature extraction is performed on a plurality of frequency sub-graphs.

[0055] After obtaining multiple frequency sub-maps, the multiple frequency sub-maps can be used as shallow features corresponding to the image to be processed, and deeper features can be extracted. The feature extraction process can be performed multiple times to extract effective features contained in the image to be processed as intermediate image features.

[0056] Step S130, upsampling the intermediate image features to obtain restored image features. The upsampling process is the inverse process of the decomposition transformation process.

[0057] According to some exemplary embodiments of the present disclosure, the upsampling process may be a process of restoring the intermediate image features to an image of the original resolution. The restored image features may be image features obtained by upsampling the intermediate image features, for example, the restored image features may include the predicted negative noise value.

[0058] After obtaining the intermediate image features, upsampling processing is performed on the intermediate image features. The upsampling processing is the inverse process of the decomposition transformation processing. Through the upsampling processing, the intermediate image features are gradually restored to the restored image features used by the original resolution image. For example, the restored image features can be the predicted negative value of the noise.

[0059] Step S140: generating a denoised image according to the restored image features and the image to be processed.

[0060] According to some exemplary embodiments of the present disclosure, the denoised image may be an image obtained after denoising the image to be processed.

[0061] After obtaining the restored image features, the restored image features are combined with the image to be processed to generate a denoised image; for example, the predicted negative noise value is added to the image to be processed (including noise) to obtain the desired denoised image.

[0062] According to the image denoising method in this example embodiment, on the one hand, by performing decomposition and transformation processing on the image for downsampling, and using the inverse process for upsampling, the video memory usage in the image processing process is effectively reduced, the computational efficiency of the model is improved, and the image details and texture information are retained. On the other hand, a denoised image is generated based on the restored image features, the image denoising effect is optimized, and the information loss in the denoising process is reduced.

[0063] Next, the image denoising method in this exemplary embodiment will be further described.

[0064] The image denoising method disclosed in the present invention can be implemented using a trained image denoising model, which mainly consists of three parts: discrete wavelet transform (DWT), feature processing and inverse discrete wavelet transform (IDWT). Specifically, at the top of the model, the image is first downsampled using DWT, which can effectively extract the high and low frequency information of the image while reducing the resolution of the image. In the middle part of the model, a feature aggregation module (NAFT) without nonlinear activation function is introduced for feature processing. This module can significantly improve the model reasoning speed while ensuring the effectiveness of feature extraction. Finally, the model uses IDWT to restore the image and reconstruct its corresponding noise-free image.

[0065] In an exemplary embodiment of the present disclosure, for step S110, the image to be processed is decomposed and transformed to obtain multiple frequency sub-images, including: obtaining a pre-configured Haar wavelet basis, the Haar wavelet basis includes a specified number of filters; using each filter to decompose and transform the image to be processed respectively to obtain multiple frequency sub-images, the multiple frequency sub-images include multiple sub-images with different frequencies.

[0066] Among them, the Haar wavelet basis can be a basis function for discrete wavelet transform processing of the processed image. The filter can be a filter function used for decomposition transformation processing of the processed image. Wavelet basis is a set of basis functions used to construct wavelet transform. Wavelet basis is usually a set of orthogonal basis functions obtained by translation, scaling and inversion transformation based on the mother wavelet function. However, as a linear transformation, wavelet transform also has certain limitations. Combining wavelet transform with nonlinear convolutional neural network can reduce the overfitting problem of the model.

[0067] Wavelet basis has the characteristics of time-frequency localization, which can better describe the time-frequency characteristics of non-stationary signals. Therefore, it increases the diversity of features in deep learning models, so that the model encoding features can be better described from the spatial domain and frequency domain. The choice of wavelet basis has an important influence on the performance and application effect of wavelet transform. Commonly used wavelet bases include Haar wavelet, Daubechies wavelet, Symlet wavelet, Coiflet wavelet, etc.

[0068] Discrete Wavelet Transform (DWT) is a commonly used signal processing method, which is often used in the fields of compression, denoising and feature extraction of image, audio and other data. Discrete wavelet transform decomposes the signal into a set of wavelet coefficients of different scales and frequencies. These wavelet coefficients contain various local feature information of the signal and can be used to represent the different frequency components of the signal. In discrete wavelet transform, a set of basis functions are used, such as Haar wavelet, Daubechies wavelet, etc.

[0069] refer to Figure 2 , Figure 2 The overall structure of an image denoising solution according to an exemplary embodiment of the present disclosure is schematically shown. Figure 2 The downsampling layer (including the DWT module) in the two-dimensional Haar wavelet transform is added to the autoencoder model. The Haar wavelet basis is one of the simplest wavelet bases in the discrete wavelet transform. A pre-configured Haar wavelet basis is obtained. The Haar wavelet basis includes a specified number of filters. For example, four filters can be configured in this embodiment. The implementation process of the two-dimensional Haar wavelet transform, the corresponding four filters are shown in Formulas 1 to 4 respectively.

[0070]

[0071] Specifically, during the image transformation process, the sliding step size can be set to 2; in other exemplary embodiments, the step size can also be set to other values. By performing Haar wavelet transform on the image to be processed, the image to be processed is decomposed into coefficient matrices of approximation, horizontal edge, vertical edge and diagonal. Specifically, the image to be processed can be recorded as noise I noisy ∈H×W×C, that is, the original input noise image, H can represent the height of the image, W can represent the width of the image, and C can represent the number of channels of the image. For example, the downsampling layer f can be used DWT The original input noise image is converted into 4 wavelet sub-images, as shown in Formula 5.

[0072] I LL ,I LH ,I HL ,I HH =f DWT (I noisy ) (Formula 5)

[0073] in, are sub-images with different frequencies; L can represent the low-frequency information contained in the image; H can represent the high-frequency information contained in the image.

[0074] The present disclosure utilizes discrete wavelet transform mainly to extract features and perform up- and down-sampling, decompose and reconstruct the original input image to be processed, and the decomposition process is to gradually reduce the resolution and bandwidth of the image to be processed. In the present disclosure, the decomposition process can reduce the spatial resolution and expand the number of channels. Discrete wavelet transform has the characteristics of locality, multi-resolution, and reversibility, so that it has both the generalization ability of traditional methods and the accuracy of deep learning methods in the image downsampling process.

[0075] Regarding the multi-resolution of discrete wavelet transform, the Haar wavelet basis can decompose the image into sub-bands of different scales, which can contain information of different frequencies. Therefore, no matter whether it is transforming the image or the feature, some prior features can be extracted, and further transformations can be performed later. From the perspective of data enhancement, a variety of information can be obtained through wavelet changes, which can improve the generalization of the model; secondly, it is relatively simple and can be calculated quickly. The Haar wavelet transform can be calculated through simple matrix operations, so it has a faster calculation speed and does not affect the training speed of the model. Thirdly, it has good locality. The support area of ​​the Haar wavelet basis is local, so it can better adapt to the local structure in the image.

[0076] In an exemplary embodiment of the present disclosure, for step S120, feature extraction processing is performed based on multiple frequency sub-graphs to obtain intermediate image features, including: splicing multiple frequency sub-graphs to obtain initial input features; performing attention feature extraction on the initial input features to obtain attention image features; transforming and decomposing the attention image features to obtain multiple intermediate frequency sub-graphs; and performing feature extraction processing based on multiple intermediate frequency sub-graphs to obtain intermediate image features.

[0077] The initial input feature may be an input feature obtained by concatenating multiple frequency sub-images. The attention image feature may be a feature obtained by extracting the initial input feature. The intermediate frequency sub-image may be a sub-image with different frequencies obtained by transforming and decomposing the attention image feature again.

[0078] Continue to refer Figure 2 In the middle part of the model, a feature aggregation module without nonlinear activation function (NAFT) is introduced to perform feature processing. For example, the first-level downsampling layer is connected to a feature extraction layer, which can contain 18 NAFT modules; the second-level downsampling layer is connected to a feature extraction layer, which can contain 14 NAFT modules, etc. After obtaining multiple frequency subgraphs, the multiple frequency subgraphs are spliced ​​as the initial input features for the next step of feature extraction. For example, multiple frequency subgraphs are connected as The shallow features of are used as the initial input features, and then they are used for feature extraction. The specific process of feature extraction using the NAFT module is shown in Formula 6.

[0079] F i =f NAFT (F1) (Formula 6)

[0080] Among them, F1 can be the initial input feature; F i It can be the feature obtained by extracting the initial input feature using the NAFT module; NAFT (·) is a feature extraction layer using multiple NAFT modules, which is used to extract the basic information of the initial input features. Since the NAFT module can use the attention mechanism for feature extraction, the features obtained after the NAFT module is used to extract the initial input features can be called attention image features.

[0081] In the present disclosure, multiple image downsampling and feature extraction processes can be performed. For example, the attention image features obtained through feature extraction can be further transformed and decomposed through the downsampling layer, and the frequency sub-graph obtained through multiple decomposition and transformation processes of the downsampling layer is used as the intermediate frequency sub-graph. Then, the feature extraction process mentioned above is repeated, and the multiple intermediate frequency sub-graphs obtained are spliced ​​and feature extraction is performed again, and the features obtained after the feature extraction process are used as the intermediate image features.

[0082] refer to Figure 2 As shown, the above-mentioned downsampling process and feature extraction process can be performed multiple times, and these features will be sent to the next DWT to further extract more effective features, as shown in Formula 7.

[0083] F i+1 =f DWT (f NAFT (F i )) (Formula 7)

[0084] Among them, F i It represents the features obtained after downsampling each layer through DWT, feature F i All of them will be sent to the NAFT module for feature aggregation, and then processed by DWT again during downsampling to obtain F i+1 .

[0085] The present invention transforms the original input image to be processed by Haar wavelet transform, so that the image is downsampled, the calculation amount of the model is reduced, and the model can run efficiently. The feature extraction layer can significantly improve the model reasoning speed while ensuring the extraction of effective features.

[0086] It is easy for those skilled in the art to understand that the specific number of NAFT modules included in the feature extraction layer of the image denoising model in the present disclosure, the number of downsampling layers and the number of levels of the feature extraction layer can all be determined according to the specific image processing task, and the present disclosure does not make any special limitation on this.

[0087] In an exemplary embodiment of the present disclosure, attention features are extracted from initial input features to obtain attention image features, including: performing self-attention feature extraction on the initial input features to obtain self-attention features; obtaining a pre-configured gate function, which is a nonlinear function; performing channel attention feature extraction on the self-attention features according to the gate function to obtain channel attention features; and generating attention image features based on the channel attention features.

[0088] Among them, self-attention is a mechanism that focuses on the importance of the internal relationship of the input data. Channel attention is a mechanism that focuses on the importance of feature map channels in convolutional neural networks. Its main purpose is to emphasize the channels that contribute most to the task and suppress irrelevant or redundant channels by assigning different weights to each channel.

[0089] refer to Figure 3 , Figure 3 The structural diagram of the NAFT module according to the exemplary embodiment of the present disclosure is schematically shown. For the initial input features input, the NAFT module can perform self-attention feature extraction on them to obtain self-attention features. In the process of self-attention feature extraction, the general nonlinear activation function is removed and a nonlinear gate function is used instead. Then, channel attention feature extraction is performed on the obtained self-attention features. In the process of channel attention feature extraction, the present disclosure adopts a simplified channel attention layer to obtain channel attention features.

[0090] In an exemplary embodiment of the present disclosure, self-attention features are extracted from initial input features to obtain self-attention features, including: obtaining a pre-configured parameter matrix; performing linear transformation processing on the initial input features based on the parameter matrix to obtain corresponding query matrix, key matrix and value matrix; determining a weight matrix according to the query matrix and the key matrix, the weight matrix being used to represent the pixel weights of each pixel in the initial input features; and determining self-attention features according to the weight matrix and the value matrix.

[0091] Self-attention feature extraction can convert the initial input features into three different tensors by performing different linear transformations. The query matrix (Query, Q) can represent the target that needs to be focused on at the moment, and can be understood as the part that raises the "question". The key matrix (Key, K) can be matched with the Query to calculate the similarity with the Query, thereby determining which parts are important. Once the value matrix (Value, V) determines which Keys are important, the corresponding Values ​​will be used to calculate the final output, and it can be considered that the Values ​​carry the actual information content.

[0092] Continue to refer Figure 3 , the initial input features are used as the input features of the NAFT module, and the self-attention module is used to extract the self-attention features of the initial input features. Specifically, a pre-configured parameter matrix is ​​obtained, and the parameter matrix can be one or more weight matrices for extracting the self-attention features of the initial input features. For example, the present disclosure can configure WQ, WK, and WV as learnable parameter matrices, which can be optimized during the training process. Then according to Figure 3 The calculation method shown in is used to extract self-attention features from the initial input features.

[0093] The query matrix Q and the key matrix K are used to determine the weight matrix. For example, the dot product of Q and K is calculated to obtain a matrix representing the attention weights. The weight matrix is ​​used to represent the pixel weights of each pixel in the initial input feature. After the weight matrix is ​​normalized by the softmax normalization layer, the sum of all weights is 1, so that different values ​​can be weighted averaged. Finally, the self-attention feature is determined based on the weight matrix and the value matrix. For example, the weighted values ​​are merged to form the output of this layer, namely the self-attention feature, which will be passed to the next layer or used to generate the final result. The self-attention mechanism allows the model to dynamically focus on the key parts of the input data, which can effectively improve the model performance and flexibility.

[0094] In an exemplary embodiment of the present disclosure, channel attention features are extracted from self-attention features according to a gating function to obtain channel attention features, including: using a gating function to determine a feature calculation weight of the self-attention feature; determining a channel input feature according to the feature calculation weight and the self-attention feature; and performing channel attention feature extraction on the channel input feature to obtain the channel attention feature.

[0095] The NAFT module performs channel attention feature extraction processing on the extracted self-attention features. The NAFT module contains a pre-configured gate function, which is a nonlinear function. The nonlinear mechanism is placed in the gate function in the NAFT module. The gate function in the neural network refers to a special type of nonlinear function that can control the flow of information between multiple input features. In a neural network, gate functions are usually used to change the connection weights between neurons (i.e., input features) to adjust the way information is transmitted so that the network can better adapt to different inputs and tasks. For example, the present disclosure adjusts the calculation weight of the self-attention feature according to the feature calculation weight to obtain the channel input feature.

[0096] Channel Attention is mainly used to control the importance between different channels. It determines the importance of channels by calculating the average and maximum values ​​of each channel, thereby giving important channels greater weights, as shown in Formula 8. The NAFT module in the present disclosure has a simplified channel attention layer, which is different from the channel attention layer. It is equivalent to removing a multi-layer perceptron and the activation function Sigmoid. The present disclosure can use the simplified channel attention layer to extract channel input features and finally obtain channel attention features. The specific processing process is shown in Formula 9 below.

[0097] CA(X)=X*σ(W2max(0,W1pool(X))) (Formula 8)

[0098] SCA(X)=X*Wpool(X) (Formula 9)

[0099] Among them, CA can represent the commonly used channel attention layer, SCA represents the simplified channel attention layer used in the present disclosure, X represents the input feature, such as the present disclosure can input the self-attention feature to the channel attention feature to extract the clock, pool represents the global average pooling, W, W1, W2 represent the learnable network parameters, σ represents the activation function, which is not limited to the Sigmoid activation function.

[0100] After channel attention feature extraction, Figure 3 The NAFT module in the can perform layer normalization and convolution processing on the channel attention features, and use nonlinear gate functions to adjust the connection weights between each input, and finally obtain the attention image features. This paper proposes a new feature extraction structure, which uses the self-attention module in the Transformer structure and introduces a nonlinear gate function instead of a nonlinear activation function, which can enhance the generalization ability of the model under different noise modes and avoid the nonlinear distortion caused by the activation function.

[0101] Continue to refer Figure 2 After completing multiple downsampling and feature extraction processes on the image to be processed, the image denoising model obtains the intermediate image features. The subsequent image denoising model uses the IDWT module to restore the intermediate image features and reconstruct its corresponding noise-free image. The upsampling process disclosed in the present invention is the inverse process of the downsampling process. The restored denoised image is gradually obtained by upsampling through the IDWT module and the NAFT module, as shown in Formula 10.

[0102] F′ i+1 =f IDWT (f NAFT (F′ i )) (Formula 10)

[0103] Among them, F′ i It represents the features of each layer after IDWT, which will be sent to the NAFT module for feature processing. Then, it will be processed by IDWT again during upsampling to obtain F′ i+1 .

[0104] The present disclosure can perform multiple upsampling and feature extraction processes on the intermediate image features, and finally obtain the restored image features. For example, the restored image features can be the last layer feature F′ in the upsampling stage of the model. out , is the negative value of the noise predicted by the model. At the end of upsampling, the negative value of the noise is added to the input noisy image to obtain the required clean image (i.e., denoised image). The present disclosure adopts a global residual structure to generate a denoised image. The global residual structure is specifically shown in Formula 11.

[0105] I clean =I noisy +F′ out (Formula 11)

[0106] Among them, I clean I represents the output denoised image; noisy represents the image to be processed; F′ out Indicates the restored image features, i.e., the predicted negative noise value. In the upsampling stage of the model, the present disclosure adopts a global residual structure, i.e., in the last layer, the predicted negative noise value is added to the input noise image to generate a denoised image. This structure optimizes the denoising effect and reduces information loss.

[0107] In summary, the image denoising method disclosed in the present invention obtains an image to be processed, performs decomposition and transformation processing on the image to be processed, and obtains multiple frequency sub-graphs; performs feature extraction processing based on multiple frequency sub-graphs to obtain intermediate image features; performs up-sampling processing on the intermediate image features to obtain restored image features, and the up-sampling processing is the inverse process of the decomposition and transformation processing; and generates a denoised image based on the restored image features and the image to be processed. On the one hand, the up-sampling and down-sampling disclosed in the present invention is to increase and decrease the resolution of image features by discrete wavelet transform and its inverse transform, and the purpose of doing so is to significantly reduce the occupancy of video memory and make the efficiency of the model unaffected. In this process, the down-sampling process extracts high-frequency and low-frequency information of the image, which helps to retain the details and texture information of the image. On the other hand, a Transformer structure without traditional nonlinear activation functions is adopted. The structure extracts features through a self-attention mechanism, and introduces a gate function to replace the traditional activation function, thereby avoiding nonlinear distortion and enhancing the generalization ability of the model under different noise modes. On the other hand, by performing up-sampling in a multiple progressive manner, the resolution or spatial details of the image can be avoided from being lost. On the other hand, in the upsampling stage of the model, a global residual structure is adopted, which optimizes the denoising effect and reduces information loss.

[0108] In addition, the present disclosure also provides a model training method, which can be implemented by using a server or by using a terminal device. Figure 4 The following is a schematic diagram of the model training method flow according to some embodiments of the present disclosure. Figure 4 , the model training method may include the following steps:

[0109] Step S410, obtaining a pre-built initial model and a training data set, where the training data set includes a sample image and a corresponding noise image.

[0110] According to some exemplary embodiments of the present disclosure, the sample image may be an original image without noise, and the noise image may be an image generated by adding noise to the sample image.

[0111] Obtain a pre-built initial model, which is an encoder-decoder structure based on multi-level discrete wavelet transform and Transformer, and may include three parts: discrete wavelet transform (DWT) non-linear activation function Transformer (NAFT) module and inverse discrete wavelet transform (IDWT). Initialize model parameters, including weights of DWT and NAFT modules.

[0112] Get a pre-built training dataset and pair the sample image (the original image without noise) with the corresponding noise image to form a training dataset. The training dataset can be divided into a training set, a validation set, and a test set.

[0113] Step S420: Use the initial model to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features.

[0114] According to some exemplary embodiments of the present disclosure, the initial model is used to perform the following steps for each batch in the training set: noisy The DWT layer is used to perform downsampling processing, such as decomposition transformation processing, to obtain a sub-image (I LL ,I LH ,I HL ,I HH ), connect these sub-images and extract features F in ; Recursively apply the DWT module and NAFT module to the feature F in Downsample to get deeper features F i , as the intermediate sample image feature.

[0115] Step S430, performing upsampling processing based on the intermediate sample image features to obtain restored sample image features, and the upsampling processing is the inverse process of the decomposition transformation processing.

[0116] According to some exemplary embodiments of the present disclosure, the IDWT module and the NAFT module in the initial model are used to analyze the intermediate sample image features F i Perform upsampling to obtain the restored sample image feature F′ i ; The upsampling process in the present disclosure may be the inverse process of the downsampling process.

[0117] Step S440, obtaining a predicted denoised image based on the restored sample image features and the sample image.

[0118] According to some exemplary embodiments of the present disclosure, based on restoring the sample image features and the sample image, and adopting the global residual structure of the initial model, a denoised image I is obtained. clean The specific processing process of the global residual structure is the same as that in the above text, and this disclosure will not elaborate on it.

[0119] Step S450, training the initial model according to the loss value between the predicted denoised image and the sample image to obtain an image denoising model.

[0120] After obtaining the predicted denoised image, the loss value between the predicted denoised image and the sample image can be calculated. For example, the mean squared error loss function (MSE) can be used to calculate the loss between the predicted image and the true clean image to measure the error between the model's prediction and the actual noise-free image, as shown in Formula 12.

[0121]

[0122] Among them, Loss can represent the loss value; I clean,i is the true pixel value of the sample image, I pred,i is the pixel value predicted by the model and N is the total number of pixels.

[0123] After calculating the loss value between the two images, back propagation and optimization are performed, the gradient is calculated according to the loss function, and the model parameters are updated using the gradient descent method; finally, the model parameters with the best performance on the validation set are saved. The present disclosure can also use other loss functions to calculate the loss value between the two images, such as Mean Absolute Error (MAE), cross entropy loss, logarithmic loss, etc. The present disclosure does not make any special restrictions on the loss function used.

[0124] According to the model training method in this example embodiment, on the one hand, by performing decomposition and transformation processing on the image for downsampling, and using the inverse process for upsampling, the video memory usage during the image processing process is effectively reduced, the computational efficiency of the model is improved, and the image details and texture information are retained. On the other hand, the model generates a predicted denoised image based on the restored image features, optimizes the image denoising effect, and reduces the information loss in the model denoising process.

[0125] The model training method in this example embodiment will be further described below.

[0126] In an exemplary embodiment of the present disclosure, for step S420, the initial model is used to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features, including: obtaining the Haar wavelet basis in the downsampling layer, the Haar wavelet basis includes a specified number of filters; using each filter to perform decomposition transformation processing on the noise image respectively to obtain multiple sample frequency sub-graphs; splicing the multiple sample frequency sub-graphs to obtain initial sample input features; using the feature extraction layer to perform attention feature extraction on the initial sample input features to obtain intermediate sample image features.

[0127] During the model training process, the downsampling layer (DWT) of the initial model uses a two-dimensional Haar wavelet basis to decompose and transform the noise image. For example, the four filters in Formula 1 to Formula 4 above can be used to decompose and transform the noise image respectively to obtain multiple sample frequency sub-graphs, which contain high-frequency information and low-frequency information in the noise image.

[0128] However, the obtained multiple sample frequency sub-graphs are spliced ​​as the initial sample input features; the feature extraction layer in the initial model, i.e., the NAFT module, is used to extract the attention features of the initial sample input features to obtain the intermediate sample image features. Attention feature extraction includes self-attention feature extraction and channel attention feature extraction. Feature extraction and downsampling processing can be performed through multiple levels. The feature extraction process is the same as the processing process mentioned above, and this disclosure will not repeat it.

[0129] The present invention uses a multi-level discrete wavelet transform as a downsampling module, and uses an inverse wavelet transform as an upsampling module in the model, with the following advantages: the wavelet transform is reversible, so all information can be saved through this sampling method; the wavelet transform can capture the frequency and position information of the image, which is conducive to restoring the detailed features of the image; the use of the wavelet transform can reduce the resolution of the image, thereby reducing the memory consumption of the GPU. At the same time, the process does not generate redundant parameters, can speed up the reasoning speed of the model, and is conducive to effective model construction; the wavelet transform will relatively increase the acceptance area of ​​the receptive field, so that the model obtains richer features, which is conducive to image restoration.

[0130] In an exemplary embodiment of the present disclosure, a feature extraction layer is used to perform attention feature extraction on initial sample input features to obtain intermediate sample image features, including: using a self-attention extraction module to perform self-attention feature extraction on initial sample input features to obtain self-attention sample features; using a channel attention extraction module to perform channel attention feature extraction on self-attention sample features to obtain channel attention sample features; based on the channel attention sample features, intermediate sample image features are generated.

[0131] The attention feature extraction process can be done by Figure 3 The NAFT module in the image processing unit is used to extract self-attention features from the image processing unit, and the NAFT module can extract self-attention features from the image processing unit to obtain self-attention features. In the process of extracting self-attention features, the general nonlinear activation function is removed and replaced by a nonlinear gate function. Then, the obtained self-attention features are subjected to channel attention feature extraction. In the process of extracting channel attention features, the present disclosure adopts a simplified channel attention layer to obtain channel attention features.

[0132] After channel attention feature extraction, Figure 3 The NAFT module in the Transformer can perform layer normalization and convolution processing on the channel attention features, and use nonlinear gate functions to adjust the connection weights between each input, and finally obtain the attention image features. This paper proposes a new feature extraction structure, which uses the self-attention module in the Transformer structure and introduces a nonlinear gate function instead of a nonlinear activation function, which can enhance the model's processing ability in different noise environments and avoid the nonlinear distortion caused by the activation function.

[0133] After completing the model training process through the above steps, the model testing process is executed, which specifically includes: (1) loading the model. Load the best model parameters saved in the training phase. (2) test set prediction. For each noise image I in the test set, noisy Perform forward propagation to obtain the denoised image I clean (3) Performance evaluation. The peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to calculate the performance of the model on the test set.

[0134] In summary, the model training method disclosed in the present invention obtains a pre-built initial model and a training data set, the training data set includes a sample image and a corresponding noise image; the initial model is used to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features; upsampling processing is performed based on the intermediate sample image features to obtain restored sample image features, and the upsampling processing is the inverse process of the decomposition transformation processing; based on the restored sample image features and the sample image, a predicted denoised image is obtained; according to the loss value between the predicted denoised image and the sample image, the initial model is trained to obtain an image denoising model. On the one hand, the up- and down-sampling of the model structure is to increase and decrease the resolution of the image features through discrete wavelet transform and its inverse transform. The purpose of doing so is to significantly reduce the occupancy of the video memory and to make the efficiency of the model unaffected. In this process, the down-sampling process extracts the high-frequency and low-frequency information of the image, which helps to retain the details and texture information of the image. On the other hand, the traditional activation function is removed from the model structure, and the Transformer module without nonlinear activation is adopted, which enhances the processing ability of the model in different noise environments and avoids the nonlinear distortion caused by the activation function, which belongs to a new structural design. On the other hand, applying the global residual structure to image denoising ensures the accurate restoration of the denoised image, effectively solving the information loss problem in multi-level decomposition of existing methods. This design not only improves the denoising ability of the model, but also optimizes the computational efficiency.

[0135] It should be noted that, although the steps of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0136] In addition, in this exemplary embodiment, an image denoising device is also provided. Figure 5 The image denoising device 500 may include: an image acquisition module 510 , a first feature extraction module 520 , an image upsampling module 530 and an image denoising module 540 .

[0137] Specifically, the image acquisition module 510 is used to acquire the image to be processed, and perform decomposition and transformation processing on the image to be processed to obtain multiple frequency sub-graphs; the first feature extraction module 520 is used to perform feature extraction processing based on the multiple frequency sub-graphs to obtain intermediate image features; the image upsampling module 530 is used to perform upsampling processing on the intermediate image features to obtain restored image features, and the upsampling processing is the inverse process of the decomposition and transformation processing; the image denoising module 540 is used to generate a denoised image based on the restored image features and the image to be processed.

[0138] In an exemplary embodiment of the present disclosure, the image acquisition module 510 includes a decomposition transformation unit, which is used to: obtain a pre-configured Haar wavelet basis, which includes a specified number of filters; use each filter to perform decomposition transformation processing on the image to be processed, and obtain multiple frequency sub-graphs, which include multiple sub-images with different frequencies.

[0139] In an exemplary embodiment of the present disclosure, the first feature extraction module 520 includes an intermediate feature extraction unit, which is used to: concatenate multiple frequency sub-graphs to obtain initial input features; perform attention feature extraction on the initial input features to obtain attention image features; perform transformation and decomposition on the attention image features to obtain multiple intermediate frequency sub-graphs; and perform feature extraction based on the multiple intermediate frequency sub-graphs to obtain intermediate image features.

[0140] In an exemplary embodiment of the present disclosure, the intermediate feature extraction unit includes a feature extraction subunit, which is used to: perform self-attention feature extraction on the initial input feature to obtain the self-attention feature; obtain a pre-configured gate function, which is a nonlinear function; perform channel attention feature extraction on the self-attention feature according to the gate function to obtain the channel attention feature; and generate an attention image feature based on the channel attention feature.

[0141] In an exemplary embodiment of the present disclosure, the feature extraction subunit includes a first feature extraction subunit, which is used to: obtain a preconfigured parameter matrix; perform linear transformation processing on the initial input features based on the parameter matrix to obtain corresponding query matrix, key matrix and value matrix; determine a weight matrix based on the query matrix and the key matrix, and the weight matrix is ​​used to represent the pixel weight of each pixel in the initial input features; determine the self-attention feature based on the weight matrix and the value matrix.

[0142] In an exemplary embodiment of the present disclosure, the feature extraction subunit includes a second feature extraction subunit, which is used to: determine the feature calculation weight of the self-attention feature using a gate function; determine the channel input feature based on the feature calculation weight and the self-attention feature; perform channel attention feature extraction on the channel input feature to obtain the channel attention feature.

[0143] Furthermore, in this exemplary embodiment, a model training device is also provided. Figure 6 The model training device 600 may include: a data set acquisition module 610, a second feature extraction module 620, an upsampling module 630, an image restoration module 640 and a model training module 650.

[0144] Specifically, the data set acquisition module 610 is used to obtain a pre-built initial model and a training data set, and the training data set includes a sample image and a corresponding noise image; the second feature extraction module 620 is used to use the initial model to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features; the upsampling module 630 is used to perform upsampling processing based on the intermediate sample image features to obtain restored sample image features, and the upsampling processing is the inverse process of the decomposition transformation processing; the image restoration module 640 is used to obtain a predicted denoised image based on the restored sample image features and the sample image; the model training module 650 is used to train the initial model according to the loss value between the predicted denoised image and the sample image to obtain an image denoising model.

[0145] In an exemplary embodiment of the present disclosure, the initial model includes a downsampling layer and a feature extraction layer, and the second feature extraction module 620 includes a second feature extraction unit, which is used to: obtain the Haar wavelet basis in the downsampling layer, the Haar wavelet basis includes a specified number of filters; use each filter to decompose and transform the noise image respectively to obtain multiple sample frequency sub-graphs, and the sample frequency sub-graphs include multiple sub-images with different frequencies; splice the multiple sample frequency sub-graphs to obtain initial sample input features; use the feature extraction layer to extract attention features from the initial sample input features to obtain intermediate sample image features.

[0146] In an exemplary embodiment of the present disclosure, the feature extraction layer includes a self-attention extraction module and a channel attention extraction module, and the second feature extraction unit includes a second feature extraction sub-unit, which is used to: use the self-attention extraction module to perform self-attention feature extraction on the initial sample input feature to obtain self-attention sample features; use the channel attention extraction module to perform channel attention feature extraction on the self-attention sample features to obtain channel attention sample features; based on the channel attention sample features, generate intermediate sample image features.

[0147] The specific details of the virtual modules of each of the above-mentioned image denoising devices or model training devices have been described in detail in the corresponding image denoising methods and model training methods. The undisclosed details can be found in the implementation method content of the method part, and will not be repeated here.

[0148] It should be noted that although several modules or units of the image denoising device and the model training device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0149] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0150] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0151] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the image denoising method or model training method described above is implemented.

[0152] In one embodiment, the computer program product may be a tangible product containing the computer program, such as a computer-readable storage medium storing the computer program. Figure 7 , Figure 7 The schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid state drive (SSD), etc. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), etc.

[0153] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0154] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0155] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned image denoising method or model training method.

[0156] The exemplary embodiments of the present disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0157] Reference below Figure 8 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Figure 8 The electronic device 800 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure. Figure 8 As shown, the electronic device 800 may include: a processor 810 , a memory 820 , a bus 830 , an I / O (input / output) interface 840 , a network adapter 850 , and a display 860 .

[0158] The memory 820 may include a volatile memory, such as a RAM 821, a cache unit 822, and may also include a non-volatile memory, such as a ROM 823. The memory 820 may also include one or more program modules 824, such program modules 824 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 824 may include each module in the above-mentioned device.

[0159] The processor 810 may include one or more processing units. For example, the processor 810 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0160] The processor 810 may be used to execute executable instructions stored in the memory 820, such as the above-mentioned image denoising method or model training method. The bus 830 is used to realize the connection between different components of the electronic device 800, and may include a data bus, an address bus, and a control bus.

[0161] The electronic device 800 can communicate with one or more external devices 900 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 840 .

[0162] The electronic device 800 can communicate with one or more networks through the network adapter 850. For example, the network adapter 850 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 850 can communicate with other modules of the electronic device 800 through the bus 830. The electronic device 800 can display a graphical user interface through the display 860, such as a display interface for displaying a denoised image after denoising processing.

[0163] although Figure 8 Not shown, other hardware and / or software modules may also be provided in the electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0164] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0165] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. An image denoising method, characterized in that: include: Acquire an image to be processed, and perform decomposition and transformation processing on the image to be processed to obtain a plurality of frequency sub-images; Perform feature extraction processing based on the multiple frequency sub-graphs to obtain intermediate image features; Performing up-sampling processing on the intermediate image features to obtain restored image features, wherein the up-sampling processing is an inverse process of the decomposition transformation processing; A denoised image is generated according to the restored image features and the image to be processed.

2. The method according to claim 1, characterized in that: The performing feature extraction processing based on the multiple frequency sub-graphs to obtain intermediate image features includes: The multiple frequency sub-graphs are concatenated to obtain initial input features; Performing attention feature extraction on the initial input features to obtain attention image features; Performing transformation and decomposition processing on the attention image features to obtain a plurality of intermediate frequency sub-images; Feature extraction processing is performed based on the multiple intermediate frequency sub-images to obtain the intermediate image features.

3. The method according to claim 2, characterized in that The extracting attention features from the initial input features to obtain attention image features includes: Performing self-attention feature extraction on the initial input features to obtain self-attention features, wherein the self-attention features include one or more of a query matrix, a key matrix, and a value matrix; Obtaining a pre-configured gating function, where the gating function is a non-linear function; Performing channel attention feature extraction on the self-attention feature according to the gate function to obtain a channel attention feature; Based on the channel attention feature, the attention image feature is generated.

4. A model training method, characterized in that: include: Acquire a pre-built initial model and a training data set, wherein the training data set includes a sample image and a corresponding noise image; Using the initial model to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features; Performing upsampling processing based on the intermediate sample image features to obtain restored sample image features, wherein the upsampling processing is an inverse process of the decomposition transformation processing; Based on the restored sample image features and the sample image, a predicted denoised image is obtained; The initial model is trained according to the loss value between the predicted denoised image and the sample image to obtain an image denoising model.

5. The method according to claim 4, characterized in that The initial model includes a downsampling layer and a feature extraction layer. The initial model is used to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features, including: Acquire a Haar wavelet basis in the downsampling layer, wherein the Haar wavelet basis includes a specified number of filters; Using each of the filters to perform decomposition and transformation processing on the noise image respectively to obtain a plurality of sample frequency sub-images, wherein the sample frequency sub-images include a plurality of sub-images with different frequencies; The multiple sample frequency subgraphs are concatenated to obtain initial sample input features; The feature extraction layer is used to perform attention feature extraction on the initial sample input features to obtain the intermediate sample image features.

6. The method according to claim 5, characterized in that The feature extraction layer includes a self-attention extraction module and a channel attention extraction module. The feature extraction layer is used to extract attention features from the initial sample input features to obtain the intermediate sample image features, including: Using the self-attention extraction module, extracting self-attention features from the initial sample input features to obtain self-attention sample features; Using the channel attention extraction module, extracting channel attention features from the self-attention sample features to obtain channel attention sample features; Based on the channel attention sample features, the intermediate sample image features are generated.

7. An image denoising device, characterized in that: include: An image acquisition module is used to acquire an image to be processed, and perform decomposition and transformation processing on the image to be processed to obtain multiple frequency sub-images; A first feature extraction module, configured to perform feature extraction processing based on the multiple frequency sub-graphs to obtain intermediate image features; An image upsampling module, used to perform upsampling processing on the intermediate image features to obtain restored image features, wherein the upsampling processing is an inverse process of the decomposition transformation processing; The image denoising module is used to generate a denoised image according to the restored image features and the image to be processed.

8. A model training device, characterized in that: include: A data set acquisition module, used to acquire a pre-built initial model and a training data set, wherein the training data set includes a sample image and a corresponding noise image; A second feature extraction module is used to use the initial model to perform decomposition transformation processing and feature extraction processing on the noise image to obtain intermediate sample image features; An up-sampling module, used for performing up-sampling processing based on the intermediate sample image features to obtain restored sample image features, wherein the up-sampling processing is an inverse process of the decomposition transformation processing; An image restoration module, used for obtaining a predicted denoised image based on the restored sample image features and the sample image; The model training module is used to train the initial model according to the loss value between the predicted denoised image and the sample image to obtain an image denoising model.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image denoising method as described in any one of claims 1 to 3, or to implement the model training method as described in any one of claims 4 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the image denoising method as described in any one of claims 1 to 3, or implements the model training method as described in any one of claims 4 to 6.

Images