Image denoising method and device based on machine learning, terminal and medium

Abstract

The invention provides an image denoising method and device based on machine learning, a terminal and a medium. The method comprises the steps that a noise image to be denoised is acquired and preprocessed; processing the preprocessed noise image by using an image processing network to obtain a frequency component of the noise image, a normalized frequency component and a normalized removed image; and predicting a noiseless image corresponding to the noise image to be denoised by using a trained machine learning denoising model according to the normalized frequency component, the frequency component of the noise image and the normalized removed image. The noise reduction process does not depend on the noise level of the input image, and the robustness of the noise level and the fidelity of the structural image quality are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image denoising method, device, terminal and medium based on machine learning. Background Art

[0002] In recent years, many methods have been proposed for image denoising using deep neural network models (DNN). Although DNN models outperform classical denoising algorithms, their behavior is often unpredictable due to the black-box nature of the model. In particular, if the statistical population deviation between the input image and the training data is too large, the model cannot be guaranteed to provide reasonable output. When the noise level or signal-to-noise ratio (SNR) of the input image is different from the training data, the inherent characteristics of the DNN model will be manifested. Denoising methods that are insensitive to these indicators are called blind denoising.

[0003] When the model does not pay special attention to the noise level or signal-to-noise ratio, the robustness of noise reduction depends mainly on the type and amount of training data. In this case, the problem of statistical population of training data cannot be avoided. Another mainstream method is to connect the estimated noise map representing the noise level of the input image with the input noise image, so that the model can adaptively remove noise based on the noise map. Another method is to actively adjust the parameters in the model according to the signal-to-noise ratio. Although the latter two methods can adapt to the noise level, the mechanism of how the model adapts to the noise level to reduce noise is not clear. Threshold-like activation functions that are actively determined by the signal-to-noise ratio can also be used, but it is still difficult to fully understand its mechanism in the neural network model in essence.

[0004] It can be seen that although the deep neural network model can reduce image noise, when the input image is not within the statistical distribution range of the training data, the model cannot guarantee the quality of image denoising. For medical imaging systems, the robustness and reliability of the system are very important. Summary of the invention

[0005] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide an image denoising method, device, terminal and medium based on machine learning, so as to solve the problem of low quality of image denoising in the prior art.

[0006] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an image denoising method based on machine learning, comprising: obtaining a noisy image to be denoised and preprocessing it; using an image processing network to process the preprocessed noisy image to obtain frequency components of the noisy image, normalized frequency components and a normalized removal map; based on the normalized frequency components, the frequency components of the noisy image and the normalized removal map, using a trained machine learning denoising model to predict a noise-free image corresponding to the noisy image to be denoised.

[0007] In some embodiments of the first aspect of the present application, the image processing network includes: a separation unit, used to separate the preprocessed noise image to obtain the frequency component of the noise image; a noise level estimation unit, used to perform a noise level estimation operation to obtain an estimated noise level of the noise image; a normalization unit, used to perform a normalization processing operation based on the frequency component of the noise image to obtain a normalized frequency component; an image construction unit, used to perform an image construction operation to obtain a normalized removal map.

[0008] In some embodiments of the first aspect of the present application, the frequency components of the noise image include: a first high-frequency component and a first low-frequency component of the noise image; the specific process of the noise level estimation operation includes: calculating the standard deviation of the first high-frequency component of the noise image, and outputting the calculated standard deviation as a first estimated noise level of the noise image; or, inputting the preprocessed noise image into a trained neural network-based estimation model to obtain a first estimated noise level of the noise image; based on the frequency components of the noise image, performing a normalization operation to obtain a normalized frequency component, including: normalizing the first high-frequency component of the noise image according to the first estimated noise level of the noise image to obtain a normalized first high-frequency component; the specific process of the image construction operation includes: outputting the image obtained after performing inverse processing on the normalized first high-frequency component as a normalized removal map; or, outputting the first estimated noise level of the noise image as a normalized removal map.

[0009] In some embodiments of the first aspect of the present application, based on the normalized frequency components, the frequency components of the noise image and the normalized removal map, a trained machine learning denoising model is used to predict the noise-free image corresponding to the noise image to be denoised, including: inputting the normalized first high-frequency component and the first low-frequency component of the noise image into the trained machine learning denoising model to obtain a first noise-free normalized high-frequency component; multiplying the first noise-free normalized high-frequency component and the normalized removal map pixel by pixel to obtain a first predicted high-frequency component; adding the first low-frequency component of the noise image and the first predicted high-frequency component pixel by pixel to obtain the result, and inputting the result into the first additional neural network model for processing to obtain the noise-free image corresponding to the noise image to be denoised.

[0010] In some embodiments of the first aspect of the present application, the frequency components of the noise image include: a second high-frequency component, a medium-frequency component, and a second low-frequency component of the noise image; the specific process of the noise level estimation operation includes: obtaining a second estimated noise level of the noise image by means of root mean square fitting according to the second high-frequency component, the medium-frequency component, and the second low-frequency component of the noise image; performing a normalization operation based on the frequency components of the noise image to obtain a normalized frequency component, including: normalizing the second high-frequency component and the medium-frequency component of the noise image according to the second estimated noise level of the noise image; The method comprises the following steps: performing reciprocal processing on the second high-frequency component of the noise image to obtain a normalized second high-frequency component and a normalized intermediate-frequency component; the specific process of the image construction operation includes: performing reciprocal processing on the second high-frequency component of the noise image to obtain a high-frequency normalized removal map; performing reciprocal processing on the intermediate-frequency component of the noise image to obtain an intermediate-frequency normalized removal map; outputting the high-frequency normalized removal map and the intermediate-frequency normalized removal map as a normalized removal map; or outputting the second estimated noise level of the noise image as a high-frequency normalized removal map and an intermediate-frequency normalized removal map; outputting the high-frequency normalized removal map and the intermediate-frequency normalized removal map as a normalized removal map.

[0011] In some embodiments of the first aspect of the present application, according to the normalized frequency components, the frequency components of the noise image and the normalized removal map, a trained machine learning denoising model is used to predict a noise-free image corresponding to the noise image to be denoised, including: inputting the normalized second high-frequency component and the normalized intermediate-frequency component into the high-frequency denoising model in the trained machine learning denoising model respectively to obtain a second noise-free normalized high-frequency component; inputting the normalized second high-frequency component and the normalized intermediate-frequency component into the intermediate-frequency denoising model in the trained machine learning denoising model respectively to obtain a noise-free normalized intermediate-frequency component; multiplying the second noise-free normalized high-frequency component and the high-frequency normalized removal map pixel by pixel to obtain a second predicted high-frequency component; multiplying the noise-free normalized intermediate-frequency component and the intermediate-frequency normalized removal map pixel by pixel to obtain a predicted intermediate-frequency component; adding the second predicted high-frequency component, the predicted intermediate-frequency component and the second low-frequency component pixel by pixel to obtain an addition result; and inputting the addition result into the second additional neural network model for processing to obtain a noise-free image corresponding to the noise image to be denoised.

[0012] In some embodiments of the first aspect of the present application, the method of training a machine learning denoising model includes: adding noise to each acquired noise-free image to obtain a noise image dataset; using the image processing network to process the noise image dataset, and inputting the processed noise image dataset into an untrained machine learning denoising model for training and evaluation; the machine learning denoising model that passes the evaluation is used to predict the noise-free image.

[0013] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an image denoising device based on machine learning, including: an acquisition module, used to acquire the noisy image to be denoised and perform preprocessing; a processing module, used to use an image processing network to process the preprocessed noisy image to obtain the frequency components of the noisy image, the normalized frequency components and the normalized removal map; a prediction module, used to predict the noise-free image corresponding to the noisy image to be denoised based on the normalized frequency components, the frequency components of the noisy image and the normalized removal map, using a trained machine learning denoising model.

[0014] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides an electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the image denoising method based on machine learning.

[0015] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the image denoising method based on machine learning.

[0016] As described above, the image denoising method, device, terminal and medium based on machine learning of the present application have the following beneficial effects:

[0017] The denoising process of the present application is independent of the noise level of the input image, which improves the robustness of the noise level and the fidelity of the structural image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Shown is a flowchart of an image denoising method based on machine learning in one embodiment of the present application.

[0019] Figure 2 Shown is a schematic block diagram of a first method for predicting a noise-free image in an embodiment of the present application.

[0020] Figure 3 Shown is a schematic block diagram of a second method for predicting a noise-free image in an embodiment of the present application.

[0021] Figure 4 Shown is a STD scatter plot under different noise levels in an embodiment of the present application.

[0022] Figure 5 Shown is a schematic diagram of the correlation between the known noise level and the estimated noise level in one embodiment of the present application.

[0023] Figure 6Shown is a schematic block diagram of a model training for a first method of predicting noise-free images in an embodiment of the present application.

[0024] Figure 7 Shown is a schematic block diagram of model training for the second method of predicting noise-free images in one embodiment of the present application.

[0025] Figure 8 Shown is a schematic block diagram of an image denoising device based on machine learning in one embodiment of the present application.

[0026] Fig. 9 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0028] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first high-frequency component and the second high-frequency component are only used to distinguish different high-frequency components, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0029] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0030] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0031] To facilitate understanding of the embodiments of the present application, first Figure 1 Detailed description. Figure 1 The flowchart of a method for image denoising based on machine learning in an embodiment of the present invention is shown. The method for image denoising based on machine learning in this embodiment mainly includes the following steps:

[0032] Step S11: Obtain a noisy image to be denoised and perform preprocessing.

[0033] In a specific implementation, the types of noisy images to be denoised include, but are not limited to: magnetic resonance images, computer tomography images, X-ray images, etc.

[0034] In one embodiment, the noisy image to be denoised is a two-dimensional or three-dimensional image, having grayscale or RGB. More generally, the noisy image to be denoised is a C-dimensional image having C channels. In a grayscale image, C=1; in an RGB image, C=3. It should be noted that when the noisy image to be denoised is a three-dimensional image, it can be converted into a two-dimensional image or the three-dimensional image can be denoised directly.

[0035] In one embodiment, the preprocessing adopts any one of the following three methods:

[0036] The first method is to convert the grayscale range of the noisy image to be denoised;

[0037] The second method: setting the maximum pixel value in the noise image to be denoised to 1 to obtain the preprocessed noise image;

[0038] The third method is to change the average pixel value and standard deviation of the noise image to be denoised to fixed values ​​to obtain a preprocessed noise image. Specifically, the fixed value may be 1.

[0039] It should be understood that grayscale range conversion is a technique in image processing for mapping the grayscale values ​​of an image from one range to another. The present invention does not limit the range and can be selected according to actual needs. The average pixel value of an image refers to the average value of the pixel values ​​of all pixels in the image. The standard deviation of an image is a statistic that measures the degree of discreteness of the distribution of pixel values ​​in an image, which indicates the degree of deviation between the pixel value and the average value.

[0040] Step S12: using an image processing network to process the preprocessed noise image, and obtaining the frequency components of the noise image, the normalized frequency components, and the normalized removal map.

[0041] It should be noted that, in the embodiment of the present invention, the frequency components of the noise image and the normalized frequency components are both represented in the form of images.

[0042] In one embodiment, the image processing network includes:

[0043] A separation unit, used for separating the preprocessed noise image to obtain the frequency components of the noise image;

[0044] A noise level estimation unit, configured to perform a noise level estimation operation to obtain an estimated noise level of the noise image;

[0045] A normalization unit, used for performing a normalization processing operation based on the frequency component of the noise image to obtain a normalized frequency component;

[0046] The image construction unit is used to perform an image construction operation to obtain a normalized removal map.

[0047] The frequency components of the noise image obtained after separation in the present invention have two components, which are described below respectively:

[0048] The first type: Figure 2 As shown, the frequency components of the noise image include: a first high-frequency component and a first low-frequency component of the noise image; the sum of the first high-frequency component and the first low-frequency component of the noise image is equal to the preprocessed noise image; each component has its own unique frequency, resolution or ratio.

[0049] Specifically, a smoothing kernel is used to extract a first low-frequency component from a preprocessed noise image; the first low-frequency component is subtracted from the preprocessed noise image to obtain a first high-frequency component; the smoothing kernel uses a Gaussian matrix or a uniform matrix, and the size of the Gaussian matrix or the uniform matrix can be 7×7, 9×9 or 11×11.

[0050] Alternatively, a low-pass filter or a high-pass filter, such as a Butterworth filter and a Fourier transform, is used to extract the first low-frequency component or the first high-frequency component; if a low-pass filter is used to extract the first low-frequency component, the first low-frequency component is subtracted from the preprocessed noise image to obtain the first high-frequency component; if a high-pass filter is used to extract the first high-frequency component, the first high-frequency component is subtracted from the preprocessed noise image to obtain the first low-frequency component.

[0051] Furthermore, if Figure 2 As shown, the noise level estimator performs the noise level estimation operation, and the specific process of the noise level estimation operation includes:

[0052] Calculating a standard deviation of a first high frequency component of the noise image, and outputting the calculated standard deviation as a first estimated noise level of the noise image;

[0053] Alternatively, the preprocessed noise image is input into a trained estimation model based on a neural network to obtain a first estimated noise level of the noise image.

[0054] It should be understood that the standard deviation of the first high-frequency component of the noise image represents the degree of deviation between the pixel value of the first high-frequency component and the average value. When there is a uniform noise distribution on the normalized noise image, the intensity level of the uniform noise will be estimated; when there is non-uniform noise on the normalized noise image, the maximum intensity level of the noise will be estimated.

[0055] Specifically, the training process of the neural network-based estimation model includes: obtaining a noise image with a known noise level to construct an estimation model training data set; inputting the constructed estimation model training data set into an untrained neural network-based estimation model for training to obtain a trained neural network-based estimation model. The untrained neural network-based estimation model generally adopts a CNN (Convolutional Neural Network) model. The preprocessed noise image is input into the trained neural network-based estimation model to obtain a noise map representing the noise intensity or noise level of the normalized noise image. The noise map is the first estimated noise level of the noise image. For example, the correlation between the estimated noise level and the known noise level (ground truth) is as follows: Figure 5 shown.

[0056] Furthermore, if Figure 2 As shown, based on the frequency components of the noise image, a normalization operation is performed to obtain the normalized frequency components, including:

[0057] According to the first estimated noise level of the noise image, the first high frequency component of the noise image is normalized to obtain the normalized first high frequency component. When the normalized high frequency component is multiplied pixel by pixel by the normalization removal map, the first high frequency component will be reproduced.

[0058] Furthermore, if Figure 2 As shown in the figure, the specific process of image building operation includes:

[0059] The image obtained by performing reciprocal processing on the normalized first high-frequency component is output as a normalized removal image;

[0060] Alternatively, the first estimated noise level of the noisy image is output as a normalized removal map.

[0061] It should be understood that the normalized removal map is a uniform image, and the normalized removal map is equal to the inverse of the normalized first high-frequency component or the first estimated noise level of the noise image. The shape of the normalized removal map is exactly the same as the first low-frequency / high-frequency component, so they can be multiplied pixel by pixel.

[0062] The second type: Figure 3 As shown, the frequency components of the noise image include: the second high frequency component, the intermediate frequency component and the second low frequency component of the noise image; after the second high frequency component, the intermediate frequency component and the second low frequency component of the noise image are superimposed, the noise image to be processed is obtained; each component has its own unique frequency, resolution or ratio. Specifically, two kernels are used to obtain the intermediate frequency component and the second low frequency component. The kernel in this embodiment can use a Gaussian matrix or a uniform matrix.

[0063] Furthermore, the noise level estimator performs a noise level estimation operation, and the specific process of the noise level estimation operation includes:

[0064] According to the second high-frequency component, the intermediate-frequency component and the second low-frequency component of the noise image, a second estimated noise level of the noise image is obtained by means of root mean square fitting;

[0065] Specifically, according to the second high-frequency component and the second low-frequency component of the noise image, a first estimated sub-noise level is obtained by means of root mean square fitting; the specific process of obtaining the first estimated sub-noise level includes: dividing the second high-frequency component and the second low-frequency component of the noise image into the same number of image blocks; the image blocks of the second high-frequency component and the second low-frequency component correspond one to one; calculating the root mean square of each image block of the second high-frequency component and the root mean square of the image block of the second low-frequency component corresponding to the image block; drawing the first image with the root mean square of each image block of the second high-frequency component as the vertical axis and the root mean square of the image block of the second low-frequency component corresponding to the image block as the horizontal axis; fitting each point in the image with a linear function fitting to obtain a first fitting straight line; outputting the intercept of the first fitting straight line as the first estimated sub-noise level;

[0066] According to the intermediate frequency component and the second low frequency component of the noise image, the second estimated sub-noise level is obtained by means of root mean square fitting; the specific process of obtaining the second estimated sub-noise level includes: dividing the intermediate frequency component and the second low frequency component of the noise image into the same number of image blocks; the image blocks of the intermediate frequency component and the second low frequency component correspond one to one; calculating the root mean square of each image block of the intermediate frequency component and the root mean square of the image block of the second low frequency component corresponding to the image block; drawing the second image with the root mean square of each image block of the intermediate frequency component as the vertical axis and the root mean square of the image block of the second low frequency component corresponding to the image block as the horizontal axis; fitting each point in the image by linear function fitting to obtain a second fitting straight line; outputting the intercept of the second fitting straight line as the second estimated sub-noise level;

[0067] The first estimated sub-noise level and the second estimated sub-noise level constitute a second estimated noise level of the noise image.

[0068] Alternatively, the intermediate frequency component and the second high frequency component are used as follows Figure 4 The STD scatter plot is usually referred to as the standard deviation of the data points in the scatter plot. This kind of chart is used to describe the distribution of data points relative to the mean, that is, the degree of dispersion of the data. A point in the STD scatter plot represents a pixel in the STD scatter plot. The second estimated noise level is estimated by the intercept, whose value is displayed above each scatter plot. The intercept is obtained by fitting the scatter plot (e.g. Figure 4 ) is used to estimate, and the fitted line is represented by a solid line.

[0069] It should be understood that this estimation method is because the power spectrum of natural images with objects is generally power-law, so under the assumption that the power low index is the same, the ratio of the power of different frequencies is almost constant. Since images generally have a strong peak at the lowest frequency in the frequency space, it is assumed that the image has no noise, and when the power of the low frequency in the window approaches 0, the power of the high frequency in the window also approaches 0. In contrast, the distribution of noise in frequency is relatively flat, and the power of high frequency is much larger than that of low frequency compared to natural images with objects. Therefore, when the low-frequency power of the image in the window approaches 0, the high-frequency power of the noisy image in the window is still positive.

[0070] Furthermore, according to the second estimated noise level of the noise image, the second high-frequency component and the intermediate-frequency component of the noise image are respectively normalized to obtain a normalized second high-frequency component and a normalized intermediate-frequency component.

[0071] Specifically, Figure 3 As shown, when the second estimated noise level is only one noise level, the second high-frequency component of the noise image is normalized according to the second estimated noise level to obtain the normalized second high-frequency component; the intermediate frequency component of the noise image is normalized according to the second estimated noise level to obtain the normalized intermediate frequency component.

[0072] When the second estimated noise level includes the first estimated sub-noise level and the second estimated sub-noise level, the second high-frequency component of the noise image is normalized according to the first estimated sub-noise level to obtain the normalized second high-frequency component; and the intermediate frequency component of the noise image is normalized according to the second estimated sub-noise level to obtain the normalized intermediate frequency component.

[0073] Furthermore, if Figure 3 As shown in the figure, the specific process of image building operation includes:

[0074] Performing inverse processing on the second high-frequency component of the noise image to obtain a high-frequency normalized removal map; performing inverse processing on the intermediate-frequency component of the noise image to obtain an intermediate-frequency normalized removal map; outputting the high-frequency normalized removal map and the intermediate-frequency normalized removal map as a normalized removal map;

[0075] Alternatively, the second estimated noise level of the noise image is output as a high-frequency normalized removal map and a medium-frequency normalized removal map; and the high-frequency normalized removal map and the medium-frequency normalized removal map are output as normalized removal maps.

[0076] When the second estimated noise level is only one noise level, the high frequency normalized removal map and the intermediate frequency normalized removal map are both the second estimated noise level;

[0077] When the second estimated noise level includes the first estimated sub-noise level and the second estimated sub-noise level, the high-frequency normalized removal map is the first estimated sub-noise level, and the intermediate-frequency normalized removal map is the second estimated sub-noise level.

[0078] It should be understood that the normalized removal map is a uniform image, and the normalized removal map is equal to the inverse of the normalized second high frequency component or the second estimated noise level of the noise image. The shape of the normalized removal map is exactly the same as the intermediate frequency / high frequency component, so they can be multiplied pixel by pixel.

[0079] It should be noted that the noise level in the preprocessed noise image can also be represented by a noise map. Each pixel in the noise map corresponds to a noise level.

[0080] Step S13: According to the normalized frequency components, the frequency components of the noise image and the normalized removal map, the noise-free image corresponding to the noise image to be denoised is predicted using the trained machine learning denoising model.

[0081] Since the present invention has two frequency components of the noise image, there are two prediction methods, which are described below respectively:

[0082] The first one: The process of training a machine learning denoising model includes:

[0083] Add noise to each acquired noise-free image to obtain a noisy image dataset;

[0084] The image processing network is used to process the noisy image data set, and the processed noisy image data set is input into an untrained machine learning denoising model for training and evaluation; the machine learning denoising model that passes the evaluation is used to predict noise-free images.

[0085] Specifically, Figure 6 As shown, multiple noise-free images are obtained, and noise is added to each noise-free image to form multiple sets of image data, and the multiple sets of image data constitute a noise image data set. Each set of image data includes: a noise-free image and an image corresponding to the noise-free image with noise added. The texture or intensity of the noise can be controlled as an enhancement to improve the versatility of the model.

[0086] Furthermore, an untrained machine learning denoising model adopts a HFH head, and the HFH head adopts a neural network model, such as DnCNN, u-net, DenseNet, Res-net, attention-net, and their combinations.

[0087] Furthermore, each noise image in the noise image data set is processed separately according to the image processing network in the above embodiment.

[0088] Furthermore, a loss function is used to evaluate the machine learning denoising model and the first additional neural network model. The loss function includes: mean square error and image quality related indicators, and the image quality related indicators include but are not limited to image quality structural similarity (SSIM), multi-scale (MS)-SSIM, etc. Combining the loss function of the training phase with the additional structural image quality loss function (such as SSIM) can improve the robustness to the level of harmonious noise. According to the loss function, the gradient of the loss function relative to the parameters contained in the machine learning denoising model and the first additional neural network model is calculated, and the parameters are updated using the gradient and the process is repeated until the machine learning denoising model and the first additional neural network model that meet the requirements of noise-free image prediction are obtained. It should be understood that Pytorch can be used to implement loss and gradient calculations as well as feedback on parameters. It should be noted that the first additional neural network model generally adopts a CNN model.

[0089] The first corresponding prediction method specifically includes:

[0090] According to the normalized frequency components, the frequency components of the noise image and the normalized removal map, a noise-free image corresponding to the noise image to be denoised is predicted using a trained machine learning denoising model, including:

[0091] Inputting the normalized first high-frequency component and the first low-frequency component of the noise image into a trained machine learning denoising model to obtain a first noise-free normalized high-frequency component;

[0092] Multiplying the first noise-free normalized high-frequency component and the normalized removal map pixel by pixel to obtain a first predicted high-frequency component;

[0093] The first low-frequency component and the first predicted high-frequency component of the noise image are added pixel by pixel, and the result is input into the first additional neural network model for processing to obtain a noise-free image corresponding to the noise image to be denoised.

[0094] It should be understood that because the high-frequency components are scaled, both the signal and the error in the high-frequency high-component output will be amplified or reduced, which may produce unexpected effects after merging the high-frequency and low-frequency components. This can be avoided by calculating the feedback of the loss function during training to predict the noise-free image after merging the high-frequency and low-frequency components. Another advantage is that the machine learning denoising model can also further learn to compensate for the residual noise in the low-frequency components. By adding a model that processes the low-frequency components and multiplying the output of the machine learning denoising model by the normalized removal map mentioned above, the merging of high-frequency and low-frequency components is more consistent, and the model can more safely learn to remove noise over a larger range. Another important benefit is that loss functions related to structural image quality can also be used.

[0095] Furthermore, according to the specific process of preprocessing, the noise-free image corresponding to the obtained noisy image to be denoised is reversely preprocessed to obtain a noise-free image within the original dynamic range.

[0096] Second: The process of training a machine learning denoising model includes:

[0097] Add noise to each acquired noise-free image to obtain a noisy image dataset;

[0098] The image processing network is used to process the noisy image data set, and the processed noisy image data set is input into an untrained machine learning denoising model for training and evaluation; the machine learning denoising model that passes the evaluation is used to predict noise-free images.

[0099] Specifically, Figure 7 As shown, multiple noise-free images are obtained, and noise is added to each noise-free image to form multiple sets of image data, and the multiple sets of image data constitute a noise image data set. Each set of image data includes: a noise-free image and an image corresponding to the noise-free image with noise added. The texture or intensity of the noise can be controlled as an enhancement to improve the versatility of the model.

[0100] Furthermore, the untrained machine learning denoising model includes: an untrained high frequency denoising model (HFH) and a medium frequency denoising model (MFH). Both the high frequency denoising model and the medium frequency denoising model can adopt a neural network model, such as DnCNN, u-net, DenseNet, Res-net, attention-net, and a combination thereof.

[0101] Furthermore, each noise image in the noise image data set is processed separately according to the image processing network in the above embodiment.

[0102] Furthermore, a loss function is used to evaluate the machine learning denoising model and the second additional neural network model. The loss function includes: mean square error and image quality related indicators, and the image quality related indicators include but are not limited to image quality structural similarity (SSIM), multi-scale (MS)-SSIM, etc. According to the loss function, the gradient of the loss function relative to the parameters contained in the machine learning denoising model and the second additional neural network model is calculated, and the parameters are updated using the gradient and the process is repeated until the machine learning denoising model and the second additional neural network model that meet the noise-free image prediction requirements are obtained. It should be understood that Pytorch can be used to implement loss and gradient calculations as well as feedback on parameters. The loss functions between the outputs of MFH and HFH and the intermediate frequency component of the noise-free image and the second high frequency component of the noise-free image can also be calculated. These loss functions are multiplied by different weights W0, W m and W h , merged into a loss function. It should be noted that the second additional neural network model generally adopts a CNN model.

[0103] The second corresponding prediction method specifically includes:

[0104] According to the normalized frequency components, the frequency components of the noise image and the normalized removal map, a noise-free image corresponding to the noise image to be denoised is predicted using a trained machine learning denoising model, including:

[0105] Inputting the normalized second high-frequency component and the normalized intermediate-frequency component into the high-frequency denoising model in the trained machine learning denoising model, respectively, to obtain a second noise-free normalized high-frequency component; inputting the normalized second high-frequency component and the normalized intermediate-frequency component into the intermediate-frequency denoising model in the trained machine learning denoising model, respectively, to obtain a noise-free normalized intermediate-frequency component;

[0106] The second noise-free normalized high-frequency component and the high-frequency normalized removal map are multiplied pixel by pixel to obtain a second predicted high-frequency component; the noise-free normalized intermediate-frequency component and the intermediate-frequency normalized removal map are multiplied pixel by pixel to obtain a predicted intermediate-frequency component;

[0107] The second predicted high-frequency component, the predicted mid-frequency component and the second low-frequency component are added pixel by pixel to obtain an addition result; the addition result is input into the second additional neural network model for processing to obtain a noise-free image corresponding to the noisy image to be denoised.

[0108] In one embodiment, training was performed at noise levels between 0 and 0.2, where the noise value represents the comparison of noise intensity to high frequency power. Three different models were compared at a very high noise level of 0.8. Model 1 was a 17-layer denoising convolutional neural network (DnCNN). Model 2 used Figure 2 The structure in , the residual is much smaller than that in Model 1. In Models 1 and 2, the MSE loss function was used in training. Since the noise level is much higher than the noise level used in training, all the noise looks residual. Model 3 has the same structure as Model 2, but with the addition of SSIM loss during training. The predicted image is similar to Model 2, but with better visual effects and clearer edges.

[0109] Figure 8 is a schematic block diagram of an image denoising device based on machine learning provided in an embodiment of the present application. Figure 8 As shown, the device comprises:

[0110] An acquisition module 81 is used to acquire a noise image to be denoised and perform preprocessing;

[0111] The processing module 82 is used to process the preprocessed noise image using an image processing network to obtain the frequency components of the noise image, the normalized frequency components and the normalized removal map;

[0112] The prediction module 83 is used to predict the noise-free image corresponding to the noise image to be denoised based on the normalized frequency components, the frequency components of the noise image and the normalized removal map using a trained machine learning denoising model.

[0113] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0114] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0115] Fig. 9 is a schematic block diagram of an electronic terminal provided in an embodiment of the present application. Fig. 9 As shown, the electronic terminal includes: at least one processor 901, a memory 902, at least one network interface 903 and a user interface 905. The various components in the device are coupled together through a bus system 904. It can be understood that the bus system 904 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig. 9 In the specification, various buses are labeled as bus systems.

[0116] The user interface 905 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0117] It is understood that the memory 902 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0118] The memory 902 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 900. Examples of these data include: any executable program for operating on the electronic terminal 900, such as an operating system 9021 and an application 9022; the operating system 9021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 9022 may include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The image denoising method based on machine learning provided in the embodiment of the present invention may be included in the application 9022.

[0119] The method disclosed in the above embodiment of the present invention can be applied to the processor 901, or implemented by the processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 901 or the instruction in the form of software. The above processor 901 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 901 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0120] In an exemplary embodiment, the electronic terminal 900 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0121] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes Figure 1 The image denoising method based on machine learning in the illustrated embodiment.

[0122] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code, and when the program code is run on a computer, the computer executes Figure 1 The image denoising method based on machine learning in the illustrated embodiment.

[0123] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0124] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0129] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0130] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0131] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0132] In summary, the present application provides an image denoising method, device, terminal and medium based on machine learning, the method comprising: obtaining a noisy image to be denoised and preprocessing it; using an image processing network to process the preprocessed noisy image to obtain the frequency components of the noisy image, the normalized frequency components and the normalized removal map; according to the normalized frequency components, the frequency components of the noisy image and the normalized removal map, using a trained machine learning denoising model, predicting the noise-free image corresponding to the noisy image to be denoised. The denoising process of the present application does not depend on the noise level of the input image, which improves the robustness of the noise level and the fidelity of the structural image quality. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0133] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for image denoising based on machine learning, characterized in that: include: Obtain a noisy image to be denoised and perform preprocessing; Using an image processing network, the preprocessed noise image is processed to obtain the frequency components of the noise image, the normalized frequency components, and the normalized removal map; According to the normalized frequency components, the frequency components of the noise image and the normalized removal map, a noise-free image corresponding to the noise image to be denoised is predicted using a trained machine learning denoising model.

2. The image denoising method based on machine learning according to claim 1, characterized in that: The image processing network comprises: A separation unit, used for separating the preprocessed noise image to obtain the frequency components of the noise image; A noise level estimation unit, configured to perform a noise level estimation operation to obtain an estimated noise level of the noise image; A normalization unit, used for performing a normalization processing operation based on the frequency component of the noise image to obtain a normalized frequency component; The image construction unit is used to perform an image construction operation to obtain a normalized removal map.

3. The image denoising method based on machine learning according to claim 2, characterized in that: The frequency components of the noise image include: a first high-frequency component and a first low-frequency component of the noise image; The specific process of the noise level estimation operation includes: Calculating a standard deviation of a first high frequency component of the noise image, and outputting the calculated standard deviation as a first estimated noise level of the noise image; Alternatively, the preprocessed noise image is input into a trained estimation model based on a neural network to obtain a first estimated noise level of the noise image; Based on the frequency components of the noise image, a normalization operation is performed to obtain the normalized frequency components, including: Normalizing a first high-frequency component of the noise image according to a first estimated noise level of the noise image to obtain a normalized first high-frequency component; The specific process of image building operation includes: The image obtained by performing reciprocal processing on the normalized first high-frequency component is output as a normalized removal image; Alternatively, the first estimated noise level of the noisy image is output as a normalized removal map.

4. The image denoising method based on machine learning according to claim 3, characterized in that: According to the normalized frequency components, the frequency components of the noise image and the normalized removal map, a noise-free image corresponding to the noise image to be denoised is predicted using a trained machine learning denoising model, including: Inputting the normalized first high-frequency component and the first low-frequency component of the noise image into a trained machine learning denoising model to obtain a first noise-free normalized high-frequency component; Multiplying the first noise-free normalized high-frequency component and the normalized removal map pixel by pixel to obtain a first predicted high-frequency component; The first low-frequency component and the first predicted high-frequency component of the noise image are added pixel by pixel, and the result is input into the first additional neural network model for processing to obtain a noise-free image corresponding to the noise image to be denoised.

5. The image denoising method based on machine learning according to claim 2, characterized in that: The frequency components of the noise image include: a second high frequency component, a medium frequency component, and a second low frequency component of the noise image; The specific process of the noise level estimation operation includes: According to the second high-frequency component, the intermediate-frequency component and the second low-frequency component of the noise image, a second estimated noise level of the noise image is obtained by means of root mean square fitting; Based on the frequency components of the noise image, a normalization operation is performed to obtain the normalized frequency components, including: According to a second estimated noise level of the noise image, normalizing the second high-frequency component and the intermediate-frequency component of the noise image respectively to obtain a normalized second high-frequency component and a normalized intermediate-frequency component; The specific process of image building operation includes: Performing inverse processing on the second high-frequency component of the noise image to obtain a high-frequency normalized removal map; performing inverse processing on the intermediate-frequency component of the noise image to obtain an intermediate-frequency normalized removal map; outputting the high-frequency normalized removal map and the intermediate-frequency normalized removal map as a normalized removal map; Alternatively, the second estimated noise level of the noise image is output as a high-frequency normalized removal map and a medium-frequency normalized removal map; and the high-frequency normalized removal map and the medium-frequency normalized removal map are output as normalized removal maps.

6. The image denoising method based on machine learning according to claim 5, characterized in that: According to the normalized frequency components, the frequency components of the noise image and the normalized removal map, a noise-free image corresponding to the noise image to be denoised is predicted using a trained machine learning denoising model, including: Inputting the normalized second high-frequency component and the normalized intermediate-frequency component into the high-frequency denoising model in the trained machine learning denoising model, respectively, to obtain a second noise-free normalized high-frequency component; inputting the normalized second high-frequency component and the normalized intermediate-frequency component into the intermediate-frequency denoising model in the trained machine learning denoising model, respectively, to obtain a noise-free normalized intermediate-frequency component; The second noise-free normalized high-frequency component and the high-frequency normalized removal map are multiplied pixel by pixel to obtain a second predicted high-frequency component; the noise-free normalized intermediate-frequency component and the intermediate-frequency normalized removal map are multiplied pixel by pixel to obtain a predicted intermediate-frequency component; The second predicted high-frequency component, the predicted mid-frequency component and the second low-frequency component are added pixel by pixel to obtain an addition result; the addition result is input into the second additional neural network model for processing to obtain a noise-free image corresponding to the noisy image to be denoised.

7. The image denoising method based on machine learning according to claim 1, characterized in that: Ways to train a machine learning denoising model include: Add noise to each acquired noise-free image to obtain a noisy image dataset; The image processing network is used to process the noisy image data set, and the processed noisy image data set is input into an untrained machine learning denoising model for training and evaluation; the machine learning denoising model that passes the evaluation is used to predict noise-free images.

8. An image denoising device based on machine learning, characterized in that: include: An acquisition module is used to acquire the noisy image to be denoised and perform preprocessing; A processing module, used to process the preprocessed noise image using an image processing network to obtain frequency components of the noise image, normalized frequency components, and a normalized removal map; The prediction module is used to predict the noise-free image corresponding to the noise image to be denoised based on the normalized frequency components, the frequency components of the noise image and the normalized removal map using a trained machine learning denoising model.

9. An electronic terminal comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images