Image denoising method and device based on rotation equivariant depth image prior

Abstract

The invention discloses an image denoising method and device based on rotation equivariant depth image prior. The method comprises the following steps: obtaining noisy image pairs to form a training data set; constructing a self-adaptive rotation equivariant network; the adaptive rotation isovariant network comprises an isovariant fusion module and a correction module, and the isovariant fusion module comprises a common convolution module, a rotation isovariant convolution module and a mask network; iteratively executing the model training step until a preset number of training times is reached; the model training step comprises the following steps: training an adaptive rotation isovariant network by using a training data set, and determining the total loss of the adaptive rotation isovariant network; optimizing model parameters of the adaptive rotation isovariant network, and executing a model training step; and inputting the to-be-processed image into the stored adaptive rotation equivariant network to obtain a de-noised image. The problem that the generalization ability of an existing self-supervised denoising method is limited is solved. Weight distribution is realized, images with different features are processed according to needs, and the generalization ability of the adaptive rotation isovariant network is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image denoising method and device based on rotational equivariant depth image prior. Background Art

[0002] When shooting and transmitting images, some unnecessary signals, namely noise, are often introduced. These noises will seriously affect the quality of the image and interfere with the subsequent image processing process. Image denoising is to remove random noise from the input image, improve the signal-to-noise ratio of the image signal, and thus improve the image quality.

[0003] The current mainstream self-supervised denoising methods include Noise2Noise (noise to noise, using two pairs of pictures of the same scene but with different noises to train the model), Noise2Void (noise to null value, using only a single noise image for training, and introducing the "BlindSpot" network structure to prevent learning the identity mapping), Noise2Self (noise self-supervised denoising, replacing or adding noise to some pixels of the input image, and using the remaining pixel information to predict the values ​​of these replaced or added pixels) and Noise-as-Clean (noise is clean, using an existing noise image and an existing noise image plus noise to train the model). Among them, the Noise2Noise method allows the model to be trained directly on two pairs of noise images of the same scene. When training the model, the Noise2Void and Noise2Self methods map the noise image to itself through the network. In order to prevent the network from degenerating into an identity mapping, a blind spot strategy is designed to avoid overfitting. The Noisier2Noise and Noise-as-Clean methods add additional noise to the original noise image to generate an image pair, which is then used to train the network.

[0004] These self-supervised denoising methods are based on the CNN (convolutional neural network) architecture design, whose inherent translation-equivariant image priors are widely present in various types of images. The convolution operator uses a dynamic window sliding method to perform convolution operations and shares weights to achieve better generalization effects. The most important thing is to embed the translation-equivariant image priors into the deep neural network. The network designed in this way will give the same feedback to similar patterns at different positions in the image, which is also a key reason for their significant effects. However, at this stage, many important image priors, such as the equivariance of local image features under rotation and symmetry transformations, have not been effectively captured, resulting in limited model generalization capabilities and a time-consuming training process. Summary of the invention

[0005] The embodiments of the present application solve the problem of limited generalization ability of existing self-supervised denoising methods by providing an image denoising method and device based on rotational equivariant depth image prior.

[0006] In a first aspect, an embodiment of the present application provides an image denoising method based on rotational equivariant depth image prior, comprising: obtaining noisy image pairs to constitute a training data set; constructing an adaptive rotational equivariant network; wherein the adaptive rotational equivariant network comprises an equivariant fusion module and a correction module, and the equivariant fusion module comprises an ordinary convolution module, a rotational equivariant convolution module and a mask network; iteratively executing a model training step until a preset number of training times is reached, and saving the current adaptive rotational equivariant network; the model training step comprises: training the adaptive rotational equivariant network using the training data set, and determining the total loss of the adaptive rotational equivariant network; optimizing the model parameters of the adaptive rotational equivariant network according to the total loss, and executing the model training step based on the optimized adaptive rotational equivariant network; inputting the image to be processed into the saved adaptive rotational equivariant network to obtain a denoised image.

[0007] In combination with the first aspect, in a possible implementation, obtaining noisy image pairs to constitute a training data set includes: adding different random Gaussian noises of the same intensity to a clean image to obtain an input noisy image and a target noisy image as a noisy image pair; cropping the input noisy image and the target noisy image in the noisy image pair to a set size to construct a training data set, and randomly rotating them to augment the training data set.

[0008] In combination with the first aspect, in a possible implementation manner, the adaptive rotation equivariant network is as follows:

[0009] in, In the formula, represents the denoised image output by the adaptive rotation equivariant network, S c Represents the correction module, represents the initial denoised image output by the adaptive fusion module, M f represents the pixel-by-pixel mask weight, ⊙ represents the dot product, and f c represents the preliminary denoised image output by the ordinary convolution module, represents pixel-by-pixel addition, f e Represents the preliminary denoised image output by the rotational equivariant convolution module.

[0010] In combination with the first aspect, in a possible implementation method, the method for constructing the rotational equivariant convolution module includes: constructing a rotational equivariant network framework based on rotational equivariant convolution; improving the Fourier basis according to the rotational equivariance, and determining the coefficients of the filter based on it using the Fourier series expansion; integrating the determined filter coefficients into the rotational equivariant network framework to constitute the rotational equivariant convolution module.

[0011] In combination with the first aspect, in a possible implementation manner, the improved Fourier basis is as follows:

[0012]

[0013] In the formula, and Represent the improved Fourier basis, k and l represent the rows and columns of the filter coefficient matrix, p represents the size of the filter, h represents the grid size of the input image, x represents a point in the two-dimensional coordinate system, x = [x1, x2] T , x1 and x2 represent the coordinates of the point in the two-dimensional coordinate system, T represents the transposed matrix, Ω(x) represents the radial mask function, c and s represent the cos basis and sin basis respectively.

[0014] In combination with the first aspect, in a possible implementation method, determining the total loss of the adaptive rotation equivariant network includes: respectively determining the losses of the ordinary convolution module, the rotation equivariant convolution module and the equivariant fusion module; taking the losses of the ordinary convolution module and the rotation equivariant convolution module as regularization, and constructing a total loss function in combination with the loss of the equivariant fusion module to determine the total loss.

[0015] In combination with the first aspect, in a possible implementation manner, the total loss function is as follows:

[0016] L=||AutoUNet(I)-target||2+α1L c +α2L e ;

[0017] Wherein, L represents the total loss function, and its calculation result is the total loss, AutoUNet(I) represents the loss function of the equivariant fusion module, target represents the target noisy image in the noisy image pair, α1 and α2 represent hyperparameters, and L c Represents the loss function of the ordinary convolution module, L e Represents the loss function of the rotation equivariant convolution module.

[0018] In the second aspect, an embodiment of the present application provides an image denoising device based on rotational equivariant depth image prior, comprising: an acquisition module, used to acquire noisy image pairs to constitute a training data set; a construction module, used to construct an adaptive rotational equivariant network; wherein the adaptive rotational equivariant network includes an equivariant fusion module and a correction module, and the equivariant fusion module includes an ordinary convolution module, a rotational equivariant convolution module and a mask network; a training module, used to execute a model training step until a preset number of training times is reached; the model training step comprises: training the adaptive rotational equivariant network using the training data set, and determining the total loss of the adaptive rotational equivariant network; optimizing the model parameters of the adaptive rotational equivariant network according to the total loss, and executing the model training step based on the optimized adaptive rotational equivariant network; a denoising module, used to input the image to be processed into the trained adaptive rotational equivariant network to obtain a denoised image.

[0019] In a third aspect, an embodiment of the present application provides a device, comprising: a processor; a memory for storing processor-executable instructions; when the processor executes the executable instructions, it implements the method described in the first aspect or any possible implementation method of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium includes a medium for storing a computer program or instructions, which, when executed, enables the method described in the first aspect or any possible implementation method of the first aspect to be implemented.

[0021] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0022] The embodiment of the present application can make the same processing on the textures in different directions of the rotationally symmetric image through the rotation equivariant convolution module, so that the adaptive rotation equivariant network has rotation equivariance, thereby reducing the amount of training of the network; through the mask network, adaptive weights can be provided for the ordinary convolution module and the rotation equivariant convolution module to adapt to different images and improve the generalization ability of the adaptive rotation equivariant network; through the correction module, the quality of the denoised image output by the adaptive rotation equivariant network can be further improved. The problem of limited generalization ability of the existing self-supervised denoising method is effectively solved. Furthermore, it is realized that weights are assigned to the ordinary convolution module and the rotation equivariant convolution module through the mask network, and different rotation equivariant strengths are applied according to the needs of the picture, making the adaptive rotation equivariant network more flexible, improving the generalization ability of the adaptive rotation equivariant network, and the rotation equivariant convolution module can share similar rotation and symmetry structures, thereby reducing the sample size and parameter cost required for model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A flowchart of an image denoising method based on rotational equivariant depth image prior provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of an image denoising device based on rotational equivariant depth image prior provided in an embodiment of the present application;

[0026] Figure 3 A comparison diagram of feature maps output by a common convolution module and a rotational equivariant convolution module provided in an embodiment of the present application;

[0027] Figure 4 An example diagram of a mask of a bicycle image output by a mask network provided in an embodiment of the present application;

[0028] Figure 5 This is an example diagram comparing the effects of the existing method and the method of the present application when the Gaussian noise level provided in the embodiment of the present application is 25. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0031] Figure 1 is a flowchart of an image denoising method based on rotational equivariant depth image prior provided by an embodiment of the present application, comprising steps 101 to 106. Among them, Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of the image denoising method based on rotation equivariant depth image prior. In the case where the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0032] Step 101: Obtain a noisy image pair to form a training data set. In the embodiment of the present application, different random Gaussian noises of the same intensity are added to the clean image to obtain an input noisy image and a target noisy image as a noisy image pair; the input noisy image and the target noisy image in the noisy image pair are cropped to a set size to construct a training data set, and are randomly rotated to augment the training data set.

[0033] Specifically, the noisy image pair in the present application uses two clean images of the same scene, and adds different random Gaussian noises of the same intensity to the clean images to obtain different noisy images, which are used as input noisy images and target noisy images to form noisy image pairs. The noisy image pairs are cropped to a set size to form a training sample set. Exemplarily, the present application adds random noise between 0-50 to the clean image. The set size here is 256*256 pixels. Then, the input noisy image and the target noisy image are randomly rotated by 0, 90, 180 or 270 degrees to increase the number of samples in the training data set.

[0034] Step 102: construct an adaptive rotation equivariant network. The adaptive rotation equivariant network includes an equivariant fusion module and a correction module, and the equivariant fusion module includes a normal convolution module, a rotation equivariant convolution module and a mask network. In the embodiment of the present application, firstly construct a normal convolution module, a rotation equivariant convolution module and a mask network to form an equivariant fusion module, as follows.

[0035] The ordinary convolution module is represented as: f c =Conv(I). Where I and f c Respectively represent the image input to the ordinary convolution module (i.e., the input noisy image in the training data set) and the preliminary denoised image outputted therefrom, and Conv represents an ordinary convolution module composed of an ordinary convolution network. Exemplarily, an ordinary convolution module consists of 1 input convolution layer, 5 downsampling modules, 1 intermediate convolution layer, 5 upsampling modules, and 1 output convolution layer, wherein each downsampling module consists of a convolution layer and a downsampling operator, and each upsampling module consists of an upsampling operator and 2 convolution layers.

[0036] The rotation equivariant convolution module is expressed as: f e =Fconv(I). Where I and f eThey respectively represent the image in the input rotational equivariant convolution module (i.e., the input noisy image in the training data set) and the preliminary denoised image of its output, and Fconv represents the rotational equivariant convolution module composed of the rotational equivariant convolution F-Conv. The method for constructing a rotational equivariant convolution module in the present application is as follows: construct a rotational equivariant network framework based on rotational equivariant convolution. Improve the Fourier basis according to the rotational equivariance, and determine the filter coefficients based on it using the Fourier series expansion. Integrate the determined filter coefficients in the rotational equivariant network framework to form a rotational equivariant convolution module.

[0037] Specifically, the rotation equivariant network framework uses rotation equivariant convolution F-Conv. Rotation equivariant convolution F-Conv is achieved by designing specific convolution kernels and convolution operations so that after the input image of the convolution layer is rotated, the output feature map is also rotated accordingly, thereby maintaining rotation equivariance. This feature enables the rotation equivariant convolution module to significantly improve performance and robustness when dealing with tasks with rotation invariance. In addition, rotation equivariant convolution can share similar rotation and symmetric structures, and can achieve better denoising effects with fewer parameters.

[0038] The filter parameterization technology is used to construct the rotation equivariant convolution module. A linear combination of a set of basis functions is defined for the filter, expressed as:

[0039]

[0040] Where ψ(x) represents N two-dimensional continuous Fourier bases ψ ′ , n represents the nth Fourier basis, w n Represents the corresponding coefficient, x=[x1,x2] T represents a two-dimensional coordinate system, and T represents a transposed matrix.

[0041] The filter parameterization method adopts the Fourier series expansion. The Fourier series expansion is equivariant to the two-dimensional discrete Fourier transform (DFT) and has zero representation error. The two-dimensional Fourier basis is a p×p discrete filter Φ, which can be regarded as a discretization of an underlying two-dimensional function Φ(x). A region of [-(p-1)h / 2,(p-1)h / 2] 2 Uniform sampling on, where h represents the grid size of the input image. Formally, the 2D Fourier basis is represented as follows:

[0042]

[0043] In the formula, and Represent the 2D Fourier basis, k and l represent the rows and columns of the filter coefficient matrix, p represents the size of the filter, h represents the grid size of the input image, and x represents a point in the two-dimensional coordinate system, x = [x1, x2] T , x1 and x2 represent the coordinates of the point in the two-dimensional coordinate system, T represents the transposed matrix, Ω(x) represents the radial mask function, c and s represent the cos basis and sin basis respectively, Represents the set of real numbers.

[0044] However, when the basis is rotated, incorrect rotation results will be produced due to the aliasing effect, so this type of 2D Fourier basis should not be directly used as a filter parameterization method for designing rotational equivariant convolution modules. Therefore, the improved Fourier basis is as follows:

[0045]

[0046] In the formula, and Represent the improved Fourier basis, k and l represent the rows and columns of the filter coefficient matrix, p represents the size of the filter, h represents the grid size of the input image, x represents a point in the two-dimensional coordinate system, x = [x1, x2] T , x1 and x2 represent the coordinates of the point in the two-dimensional coordinate system, T represents the transposed matrix, Ω(x) represents the radial mask function, c and s represent the cos basis and sin basis respectively. Where k, l∈(0,1,…,p-1), and if ∥x∥≥(p+1 / 2)h, then Ω(x)≥0 is a radial mask function that satisfies Ω(x)=0.

[0047] The improved Fourier basis is used to perform Fourier series expansion, and the rotational equivariant convolution kernel of the rotational equivariant convolution F-Conv is expressed as a linear combination of Fourier coefficients. By calculating or optimizing these Fourier coefficients, the specific form of the rotational equivariant convolution kernel can be determined, and then the original weight matrix in the filter is replaced to form a rotational equivariant convolution module. Figure 3 This is a feature comparison diagram of the initial denoised image output by the ordinary convolution module composed of the ordinary convolutional neural network CNN and the rotational equivariant convolution module composed of the rotational equivariant convolution F-Conv. It can be seen that the edge texture of the initial denoised image output by the ordinary convolution module is relatively rough and cannot maintain consistency in different directions, while the rotational equivariant convolution module can greatly improve this shortcoming and make the same treatment on the textures in different directions of the rotationally symmetric image.

[0048] The mask network is represented as: M f =Mask(I). Where I and M f denote the image input to the mask network and the mask output by it, respectively.

[0049] The construction method of the equivariant fusion module is as follows: Combining the output of the ordinary convolution module with the preliminary denoised image output by the rotational equivariant convolution module, the mask output by the mask network is used as its fusion weight to construct the equivariant fusion module.

[0050] In the formula, represents the initial denoised image after adaptive fusion by the equivariant fusion module, M f represents the pixel-by-pixel mask weight, ⊙ represents the dot product, and f c represents the preliminary denoised image output by the ordinary convolution module, represents pixel-by-pixel addition, f e Represents the preliminary denoised image output by the rotational equivariant convolution module.

[0051] In addition, the correction module is designed to further improve the denoising effect. The correction module is composed of ResNet Block (residual block). The preliminary denoised image of the equivariant fusion module is input into the correction module, and we get the final output result, that is, the denoised image. It is expressed as: in, represents the denoised image output by the adaptive rotation equivariant network, S c Represents the correction module, Represents the preliminary denoised image after adaptive fusion by the equivariant fusion module.

[0052] Step 103: Train the adaptive rotation equivariant network using the training data set, and determine the total loss of the adaptive rotation equivariant network. In the embodiment of the present application, a noisy image in a noisy image pair in the training data set is randomly input into the adaptive rotation equivariant network, and a denoised image is output. Figure 5 As shown, the first row is the noisy image input into the adaptive rotation equivariant network, the second row is the preliminary denoised image of the ordinary convolution module, the third row is the preliminary denoised image of the rotation equivariant convolution module, the fourth row is the denoising result of the adaptive rotation equivariant network, and the fifth row is GT, i.e., a clean noise-free image.

[0053] In an embodiment of the present application, the losses of the ordinary convolution module, the rotation equivariant convolution module and the equivariant fusion module are determined respectively. The losses of the ordinary convolution module and the rotation equivariant convolution module are used as regularization, and a total loss function is constructed in combination with the loss of the equivariant fusion module to determine the total loss. Specifically, the noisy image in the training data set is input into the adaptive rotation equivariant network, and the preliminary denoised images and their losses output by the ordinary convolution module, the rotation equivariant convolution module and the equivariant fusion module are obtained respectively. The total loss of the adaptive rotation equivariant network is calculated according to the losses of the ordinary convolution module, the rotation equivariant convolution module and the equivariant fusion module.

[0054] For example, the loss of a normal convolution module can be expressed as: Lc =||f c -target||2. Among them, L c represents the loss function of the ordinary convolution module, and its calculation result is the loss of the ordinary convolution module, f c It represents the preliminary denoised image output by the ordinary convolution module, and target represents the target noisy image in the noisy image pair.

[0055] The loss of the rotation equivariant convolution module can be expressed as: L e =||f e -target||2. Among them, L e represents the loss function of the rotation equivariant convolution module, and its calculation result is the loss of the rotation equivariant convolution module, f e represents the preliminary denoised image output by the rotational equivariant convolution module, and target represents the target noisy image in the noisy image pair.

[0056] The total loss function of the adaptive rotation equivariant network is as follows:

[0057] L=||AutoUNet(I)-target||2+α1L c +α2L e .

[0058] Where L represents the total loss function, and its calculation result is the total loss. AutoUNet(I) represents the loss function of the equivariant fusion module. Target represents the target noisy image in the noisy image pair. α1 and α2 represent hyperparameters. L c represents the loss function of the ordinary convolution module, L e Represents the loss function of the rotation equivariant convolution module. Exemplarily, the hyperparameters α1 and α2 represent the regularization strength of the ordinary convolution module and the rotation equivariant convolution module, and the values ​​are exemplarily taken as 0.01.

[0059] Step 104: Optimize the model parameters of the adaptive rotation equivariant network according to the total loss. In the embodiment of the present application, the Adam optimization algorithm (Adam Optimization Algorithm, an optimization algorithm applied to deep learning models, aims to iteratively update model parameters to minimize the loss function, thereby improving the prediction accuracy of the model) is used to update the model parameters of the adaptive rotation equivariant network.

[0060] Specifically, the mean square error between the denoised image generated during the training of the adaptive rotation equivariant network in step 103 and the target noisy image is calculated, and the model parameters of the adaptive rotation equivariant network are optimized by the back propagation algorithm, so that the generated denoised image is close to the corresponding clean image in probability distribution in theory. Since the noise is randomly generated and the mean is 0, the adaptive rotation equivariant network will learn the mean of the target noisy image, that is, the clean image.

[0061] Step 105: Determine whether the preset number of training times has been reached. In the embodiment of the present application, it is determined whether the training times of the current adaptive rotation equivariant network have reached the preset number of training times. If the preset number of iterations has been reached, the training is terminated, the model parameters at this time are saved, that is, the current adaptive rotation equivariant network is saved, and then step 106 is executed. Otherwise, steps 103 to 105 are executed until the training times of the adaptive rotation equivariant network have reached the preset number of training times.

[0062] Step 106: Save the current adaptive rotation equivariant network, input the image to be processed into the saved adaptive rotation equivariant network, and obtain a denoised image. In the embodiment of the present application, when the number of training times of the adaptive rotation equivariant network reaches the preset number of training times, the current adaptive rotation equivariant network can be used to perform image denoising, and the image to be processed can be input into it to obtain a denoised image.

[0063] In order to verify the effect of the adaptive rotation equivariant network in this application, three standard public data sets (Kodak24, BSDS300, Set14) were selected for testing at different noise levels, and PSNR (Peak Signal-to-Noise Ratio, Peak Signal-to-Noise Ratio, is an objective indicator for measuring image quality. The higher the value, the smaller the difference between the image and the reference image, and the better the image quality) and SSIM (Structural Similarity, Structural Similarity, is an indicator for measuring the similarity between two images. It takes into account the brightness, contrast and structural information of the image. The value range is between [-1,1]. The closer to 1, the higher the similarity of the two images) are selected as evaluation indicators. The best five experimental results of PSNR and SSIM are selected, the average is taken, and the experiment is repeated three times to calculate the mean and standard deviation. See Table 1 and Table 2 below for details.

[0064] Table 1 tests three networks (convolutional neural network, rotation equivariant convolution module, adaptive rotation equivariant network) using three standard public data sets under the condition of Gaussian noise level of 25, as follows:

[0065] Table 1 Comparison of model data with Gaussian noise level of 25

[0066]

[0067] Table 2 tests three networks using three standard public data sets under the condition of a Gaussian noise level of 25, as follows:

[0068] Table 2 Comparison of model data with Gaussian noise level of 50

[0069]

[0070] From the perspective of PSNR and SSIM, the adaptive rotation equivariant network of the present application has better denoising performance and has obvious advantages. Among them, the rotation equivariant convolution module can represent a convolutional neural network in which all convolutions in the convolutional neural network are replaced with rotation equivariant convolutions.

[0071] Figure 4 This is the mask of the bicycle image output by the mask network. It can be seen that the colors are darker in some areas with obvious rotational symmetry features such as the wheel edge and circular decorations, indicating that more adaptive rotational equivariant networks are used, which fully confirms the advantages and rationality of this application.

[0072] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in this embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in this embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment).

[0073] like Figure 2 As shown, the embodiment of the present application also provides an image denoising device 200 based on rotation equivariant depth image prior. The device comprises: an acquisition module 201, a construction module 202, a training module 203 and a denoising module 204, as follows.

[0074] The acquisition module 201 is used to acquire noisy image pairs to form a training data set.

[0075] The construction module 202 is used to construct an adaptive rotation equivariant network, wherein the adaptive rotation equivariant network includes an equivariant fusion module and a correction module, and the equivariant fusion module includes a common convolution module, a rotation equivariant convolution module and a mask network.

[0076] The training module 203 is used to perform the model training step until a preset number of training times is reached. The model training step includes:

[0077] Train an adaptive rotation equivariant network using the training dataset and determine the total loss of the adaptive rotation equivariant network.

[0078] The model parameters of the adaptive rotation equivariant network are optimized according to the total loss, and the model training step is performed based on the optimized adaptive rotation equivariant network.

[0079] The denoising module 204 is used to input the image to be processed into the trained adaptive rotation equivariant network to obtain a denoised image.

[0080] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0082] The methods, devices or modules described in this application can be implemented in the form of computer-readable program codes. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program codes (such as software or firmware) that can be executed by the (micro) processor, logic gates, switches, application-specific integrated circuits (English: Application Specific Integrated Circuit; Abbreviation: ASIC), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0083] An embodiment of the present application further provides a device, comprising: a processor; a memory for storing processor executable instructions; when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0084] The embodiments of the present application also provide a non-volatile computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed, the method described in the embodiments of the present application is implemented.

[0085] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0086] The above storage media include but are not limited to random access memory (RAM), read-only memory (ROM), cache, hard disk (HDD) or memory card. The memory can be used to store computer program instructions.

[0087] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0088] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0089] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. An image denoising method based on rotational equivariant depth image prior, characterized in that: include: Obtain noisy image pairs to form a training data set; Constructing an adaptive rotational equivariant network; wherein the adaptive rotational equivariant network includes an equivariant fusion module and a correction module, and the equivariant fusion module includes a normal convolution module, a rotational equivariant convolution module and a mask network; Iteratively executing the model training step until a preset number of training times is reached, and saving the current adaptive rotation equivariant network; the model training step includes: Training the adaptive rotation equivariant network using the training data set and determining a total loss of the adaptive rotation equivariant network; Optimizing the model parameters of the adaptive rotation equivariant network according to the total loss, and performing the model training step based on the optimized adaptive rotation equivariant network; The image to be processed is input into the stored adaptive rotation equivariant network to obtain a denoised image.

2. The method according to claim 1, characterized in that The step of obtaining noisy image pairs to form a training data set includes: Add different random Gaussian noises of the same intensity to the clean image to obtain the input noisy image and the target noisy image as a noisy image pair; An input noisy image and a target noisy image in the noisy image pair are cropped to a set size to construct a training data set, and are randomly rotated to augment the training data set.

3. The method according to claim 1, characterized in that The adaptive rotation equivariant network is as follows: in, In the formula, represents the denoised image output by the adaptive rotation equivariant network, S c Represents the correction module, represents the initial denoised image output by the adaptive fusion module, M f represents the pixel-by-pixel mask weight, ⊙ represents the dot product, and f c represents the preliminary denoised image output by the ordinary convolution module, ⊕ represents pixel-by-pixel addition, and f e Represents the preliminary denoised image output by the rotational equivariant convolution module.

4. The method according to claim 1, characterized in that: The method for constructing the rotational equivariant convolution module comprises: Construct a rotation equivariant network framework based on rotation equivariant convolution; Modifying the Fourier basis according to the rotational equivariance and determining the coefficients of the filter based on the Fourier series expansion; The determined filter coefficients are integrated into the rotational equivariant network framework to form the rotational equivariant convolution module.

5. The method according to claim 4, characterized in that The improved Fourier basis is as follows: In the formula, and They represent the improved Fourier basis, k and l represent the rows and columns of the filter coefficient matrix, p represents the size of the filter, h represents the grid size of the input image, and x represents a point in the two-dimensional coordinate system, x = [x1, x2] T , x1 and x2 represent the coordinates of the point in the two-dimensional coordinate system, T represents the transposed matrix, Ω(x) represents the radial mask function, c and s represent the cos basis and sin basis respectively.

6. The method according to claim 1, characterized in that Determining the total loss of the adaptive rotation equivariant network includes: Determining the losses of the common convolution module, the rotational equivariant convolution module, and the equivariant fusion module respectively; The losses of the ordinary convolution module and the rotational equivariant convolution module are used as regularization, and a total loss function is constructed in combination with the loss of the equivariant fusion module to determine the total loss.

7. The method according to claim 6, characterized in that The total loss function is as follows: L=||AutoUNet(I)-target||2+α1L c +α2L e ; Wherein, L represents the total loss function, and its calculation result is the total loss, AutoUNet(I) represents the loss function of the equivariant fusion module, target represents the target noisy image in the noisy image pair, α1 and α2 represent hyperparameters, and L c Represents the loss function of the ordinary convolution module, L e Represents the loss function of the rotation equivariant convolution module.

8. An image denoising device based on rotational equivariant depth image prior, characterized in that: include: An acquisition module, used for acquiring noisy image pairs to form a training data set; A construction module is used to construct an adaptive rotation equivariant network; wherein the adaptive rotation equivariant network includes an equivariant fusion module and a correction module, and the equivariant fusion module includes a normal convolution module, a rotation equivariant convolution module and a mask network; The training module is used to execute the model training step until a preset number of training times is reached; the model training step includes: Training the adaptive rotation equivariant network using the training data set and determining a total loss of the adaptive rotation equivariant network; Optimizing the model parameters of the adaptive rotation equivariant network according to the total loss, and performing the model training step based on the optimized adaptive rotation equivariant network; The denoising module is used to input the image to be processed into the trained adaptive rotation equivariant network to obtain a denoised image.

9. A device for performing an image denoising method based on rotational equivariant depth image prior, characterized in that: include: processor; a memory for storing processor-executable instructions; When the processor executes the executable instructions, the method according to any one of claims 1 to 7 is implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The device comprises a computer program or an instruction for storing the computer program or the instruction, which, when executed, enables the method according to any one of claims 1 to 7 to be implemented.

Images