Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring a first noise image; performing noise reduction on the first noise image through an image noise reduction module to obtain a first noise reduction image; and compressing the first noise reduction image through a coding module to obtain a code word corresponding to the first noise reduction image. Wherein the image noise reduction module is associated with the compression performance of the coding module, namely, the noise reduction process fully considers the influence of the subsequent coding process, so that on one hand, the image quality is improved, on the other hand, the compression performance of the subsequent coding module is ensured, and the problems that key details are lost in the compression process and compressed code words are too high can be solved.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] Image noise refers to unnecessary or redundant interference information in image information. The presence of noise seriously affects the quality of the image, so it is very necessary to effectively remove image noise. According to the imaging process of the image sensor, noise can be simply divided into three types of noise: shot noise, readout noise, and other noise. According to the imaging process of the sensor, it can be clearly seen that in some scenes at night and in low light, as the imaging sensitivity (ISO) increases, the image noise also increases significantly. At this time, the increase in noise will affect the acquisition of effective image information. Therefore, it is necessary to add a noise reduction solution to achieve image noise reduction.

[0003] At present, image denoising can be divided into three categories: filter-based methods, model-based methods, and learning-based methods. Learning-based methods focus on learning the potential mapping from noisy images to clean images, and can be divided into traditional learning-based methods and deep network-based learning methods. In recent years, deep network-based methods have become a mainstream denoising method because they have more development prospects than filter-based, model-based, and traditional learning-based methods. Moreover, the excellent performance of deep network-based methods in high-noise scenes has also been gradually applied to image and video denoising in mobile terminals. Generally, a denoising module is set in the image signal processor (ISP), but the denoising module and other modules in the ISP do not cooperate well in the image processing process, and the mutual influence with other modules is not considered, resulting in poor overall image processing effect. Summary of the invention

[0004] The embodiments of the present application are intended to provide an image processing method, apparatus, device and storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an image processing method is provided, the method comprising:

[0007] Acquire a first noise image;

[0008] Performing denoising on the first noisy image by an image denoising module to obtain a first denoised image;

[0009] Compressing the first denoised image by a coding module to obtain a codeword corresponding to the first denoised image;

[0010] The image denoising module is associated with the compression performance of the encoding module, and the image denoising module is configured as follows:

[0011] Performing wavelet transform on the first noise image input, to obtain a plurality of first sub-images in the wavelet domain;

[0012] Performing wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images;

[0013] Extracting noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images;

[0014] Obtaining a plurality of third sub-images according to the plurality of second sub-images and the noise characteristics;

[0015] Perform inverse wavelet transform on the multiple third sub-images to output the first denoised image.

[0016] In a second aspect, an image processing device is provided, the device comprising:

[0017] An acquisition module, configured to acquire a first noise image;

[0018] An image denoising module, configured to denoise the first noisy image to obtain a first denoised image;

[0019] an encoding module, configured to compress the first denoised image to obtain a codeword corresponding to the first denoised image;

[0020] The image denoising module is associated with the compression performance of the encoding module, and the image denoising module is specifically configured to perform wavelet transform on the input first noise image to obtain a plurality of first sub-images in the wavelet domain;

[0021] Performing wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images;

[0022] Extracting noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images;

[0023] Obtaining a plurality of third sub-images according to the plurality of second sub-images and the noise characteristics;

[0024] Perform inverse wavelet transform on the multiple third sub-images to output the first denoised image.

[0025] According to a third aspect, an electronic device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0026] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0027] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0028] In the embodiment of the present application, an image processing method, device, equipment and storage medium are provided, the method comprising: obtaining a first noise image; performing noise reduction on the first noise image through an image noise reduction module to obtain a first noise reduction image; and compressing the first noise reduction image through a coding module to obtain a codeword corresponding to the first noise reduction image. Among them, the image noise reduction module is associated with the compression performance of the coding module, that is, a noise reduction process fully considers the influence of the subsequent coding process, on the one hand, improves the image quality, and on the other hand, ensures the compression performance of the subsequent coding module, which can solve the problem of losing key details in the compression process and the compression codeword being too high. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a camera imaging process;

[0030] Figure 2 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 1 ;

[0031] Figure 3 This is a flow chart of a method for implementing an image noise reduction module in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a wavelet transform process in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of another wavelet transform process in an embodiment of the present application;

[0034] Figure 6 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 2 ;

[0035] Figure 7 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 3 ;

[0036] Figure 8 The structure of the image noise reduction module in the embodiment of the present application is shown in FIG. Figure 1 ;

[0037] Fig. 9 The structure of the image noise reduction module in the embodiment of the present application is shown in FIG. Figure 2 ;

[0038] Fig.10 This is a schematic diagram of the composition structure of the image processing device in the embodiment of the present application;

[0039] Fig.11 It is a schematic diagram of the composition structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0041] Figure 1 A schematic diagram of a camera imaging process, such as Figure 1 As shown in the figure, the reflected light from the external objects enters the lens and converges to the CMOS image sensor at the back end. The CMOS image sensor sends the analog image signal to the analog-to-digital converter to convert it into a digital image signal, and then sends the digital image signal to the back end image processor. After being processed by the back end image processor, YUV data is obtained, and the YUV data is sent to the corresponding encoding module to obtain the compressed code stream and write it into the memory. According to the imaging process of the image sensor, the noise can be simply divided into the following three types of noise:

[0042] Shot noise: When photons hit a photosensor, they generate electrons through the photoelectric effect, but the generation of electrons is error-prone. For photons of the same intensity, the distribution of the size of the generated electrons satisfies the statistics of the imperceptible distribution. The noise caused by this error is called shot noise, or shot noise.

[0043] Read noise: Electrons are converted into voltage, and the voltage is read out as a specific value after passing through an analog-to-digital converter. The error generated during the reading process can be called read noise. In a broad sense, read noise includes two parts: one is the black level (dark current), and the other is the noise that satisfies the Gaussian distribution.

[0044] Other noises: Other noises are simply classified into one category here, because this type of noise basically does not conform to statistical laws, including thermal noise (electrons generated by sensor heat), pixel response non-uniformity (non-uniformity of pixel response to photons), quantization noise (the readout voltage is floating point type and needs to be converted to integer), sensor fixed noise (related to the sensor process), etc.

[0045] At present, image denoising can be divided into three categories: filter-based methods, model-based methods and learning-based methods.

[0046] Among them, the filter-based method mainly uses some artificially designed low-pass filters to remove image noise. Typical methods include median filtering and Wiener filtering. The median filter is a commonly used nonlinear smoothing filter. Its basic principle is to replace the value of a point in a digital image or digital sequence with the median value of each point in a neighborhood of the point. Its main function is to change the pixel with a large difference in grayscale value of the surrounding pixels to a value close to the surrounding pixel value, so as to eliminate isolated noise points. Therefore, the median filter is very effective in filtering out salt and pepper noise in the image. Adaptive Wiener filter: It can adjust the output of the filter according to the local variance of the image. The larger the local variance, the stronger the smoothing effect of the filter.

[0047] Model-based methods mainly model the distribution of natural images or noise. Then, they use the model distribution as a priori to try to obtain a clear image with an optimization algorithm. Model-based methods usually define the denoising task as an optimization problem based on maximum a posteriori (MAP), and its performance mainly depends on the prior of the image.

[0048] In the past few decades, various model-based methods have been used to model image priors, including non-local self-similarity (NSS) models, sparse models, gradient models, and Markov random field (MRF) models. Despite their high denoising quality, most image prior-based methods have two disadvantages. These methods usually involve complex optimization problems in the testing phase, making the denoising process very time-consuming. Therefore, most image prior-based methods find it difficult to achieve high performance without sacrificing computational efficiency. The models are usually non-convex and involve several manually selected parameters, providing some leeway to improve the denoising performance.

[0049] Learning-based methods focus on learning the potential mapping from noisy images to clean images, and can be divided into traditional learning-based methods and deep network-based learning methods. In recent years, deep network-based methods have become mainstream methods because they have achieved more promising denoising results than filtering-based, model-based and traditional learning-based methods. Typical representatives include Denoising Convolutional Neural Network (DnCNN), Fast and Flexible Denoising Network (FFDnet), and CBDnet. DnCNN mainly denoises Gaussian noise, emphasizing the role of residual learning and normalization layers. FFDnet considers generalizing Gaussian noise to more complex real noise and uses the noise level map as part of the network input. CBDnet mainly starts with the noise level map of FFDnet, and adaptively obtains the noise level map through 5 layers of FCN to achieve a certain degree of blind denoising.

[0050] In the above-mentioned denoising method, an embodiment of the present application provides an image denoising module, which fully considers the influence of the subsequent encoding process in the denoising process, improves the image quality on the one hand, and ensures the compression performance of the subsequent encoding module on the other hand, and can solve the problems of losing key details in the compression process and too high compression codewords.

[0051] Figure 2 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 1 ,like Figure 2 As shown, the method may specifically include:

[0052] Step 201: Acquire a first noise image;

[0053] It should be noted that the first noise image can be understood as an image containing noise, and the first denoised image can be understood as an image after denoising. Generally, the first color component, the second color component and the third color component are used to represent the image. Among them, the color information can be RGB information. Among them, R represents red (Red, R), G represents green (Green, G), and B represents blue (Blue, B). For another example, the color information can be brightness and chromaticity (YCbCr, YUV) information. In other words, the first noise image can be an image in RGB format or an image in YUV format.

[0054] It should also be noted that the first noise image may be a noise image input to the image denoising module during online inference, or may be a noise sample image input to the image denoising module during the training phase.

[0055] Step 202: De-noise the first noisy image by using an image de-noising module to obtain a first de-noised image; wherein the image de-noising module is associated with the compression performance of the encoding module;

[0056] Figure 3 FIG. 1 is a flow chart of a method for implementing an image noise reduction module in an embodiment of the present application. Figure 3 As shown, the image denoising module is configured as follows:

[0057] Step 301: performing wavelet transform on the input first noise image to obtain a plurality of first sub-images in the wavelet domain;

[0058] It should be noted that the multiple first sub-images include: a low-frequency sub-image, a horizontal high-frequency sub-image, a vertical high-frequency sub-image, and a diagonal high-frequency sub-image. The first sub-image specifically includes wavelet coefficients on a corresponding frequency band. In some embodiments, the wavelet transform may specifically be a discrete wavelet transform (DWT).

[0059] It should be noted that, for images in RGB format, the resolutions of the three RGB components are the same, and the three color components of the first noise image are respectively subjected to wavelet transform of the same level to obtain first sub-images of the three color components in the wavelet domain; the first sub-images of the three color components are spliced ​​to obtain multiple first sub-images.

[0060] Considering that the resolutions of the three YUV components in the YUV format may be different, different levels of wavelet transform are performed on the color components with different resolutions to obtain sub-images with the same resolution. Exemplarily, for an image in the YUV420 format, an n-level wavelet transform is performed on the Y component to obtain the first sub-image of the Y component in the wavelet domain; an n-1-level wavelet transform is performed on the U component and the V component to obtain the first sub-images of the U component and the V component in the wavelet domain; the first sub-images of the three color components are spliced ​​to obtain multiple first sub-images.

[0061] like Figure 4 As shown, n is 2, which is achieved by performing two-level wavelet transform on the Y component and one-level wavelet transform on the UV component, and then concatenating the transformed sub-images together for subsequent processing. Figure 5 As shown, n is 1, the Y component is subjected to a first-level wavelet transform, the UV component is not subjected to a wavelet transform, and then the transformed sub-image of the Y component and the sub-image of the UV component are concatenated together for subsequent processing.

[0062] Step 302: performing wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images;

[0063] It should be noted that wavelet threshold denoising includes: wavelet threshold selection and threshold function selection. In the embodiment of the present application, the threshold can be a fixed threshold or a non-fixed threshold obtained through learning.

[0064] For a non-fixed threshold, the method specifically includes: determining a wavelet threshold corresponding to the first sub-image; performing wavelet threshold denoising on the first sub-image based on the wavelet threshold to obtain a second sub-image. The wavelet threshold includes an upper threshold and / or a lower threshold;

[0065] The threshold function includes hard threshold function and soft threshold function. The hard threshold function compares the absolute value of the wavelet coefficient with the threshold, sets the points less than the threshold to zero, and keeps the others unchanged; the soft threshold function compares the absolute value of the wavelet coefficient with the threshold, sets the points less than the threshold to zero, and shrinks toward zero for the points greater than or equal to the threshold, becoming the difference between the point value and the threshold.

[0066] The threshold function may be a fixed function or a non-fixed function. For a non-fixed function, the method further includes: determining the threshold function corresponding to the first sub-image; and performing wavelet threshold denoising according to the threshold function to obtain a second sub-image.

[0067] Step 303: extracting noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images;

[0068] In some embodiments, a noise extraction unit extracts noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images;

[0069] The noise feature extraction unit can be a U-Net structure, which can specifically include: at least two layers of cascaded encoders and decoders; the output information of the encoder of the previous layer is input to the encoder of the next layer, and to the decoder of the same layer; the output information of the decoder of the next layer is input to the decoder of the previous layer; the decoder concatenates the output information of the encoder of the same layer and the input information of the decoder of the next layer for feature decoding.

[0070] Step 304: obtaining a plurality of third sub-images according to the plurality of second sub-images and noise characteristics;

[0071] In some embodiments, multiple third sub-images are obtained based on multiple second sub-images and noise features, including: performing an accumulation operation on the second sub-images and the corresponding noise features to obtain a denoised sub-image; and performing an enhancement process on the denoised sub-image to obtain a third sub-image. Exemplarily, the accumulation operation of the second sub-image and the noise features is implemented by an adder. Furthermore, the enhancement process is implemented by a convolution operation and a residual-based convolution operation.

[0072] It should be noted that the noise characteristics of each sub-image can be removed in the wavelet domain through the accumulation operation to obtain a denoised sub-image, and then the denoised sub-image is enhanced to improve the quality of each sub-image in the wavelet domain, thereby further improving the quality of the reconstructed image by the subsequent inverse wavelet transform.

[0073] Step 305: Perform inverse wavelet transform on the plurality of third sub-images to output a first denoised image.

[0074] Step 203: compressing the first denoised image through a coding module to obtain a codeword corresponding to the first denoised image;

[0075] In some embodiments, the method further includes: saving the codeword in a memory, or sending the codeword to a decoding module, and the decoding module decompressing the codeword to obtain a first denoised image.

[0076] In some embodiments, other image processing modules are included between the image denoising module and the encoding module; the first denoised image is processed by the other image processing modules, and the processed image is compressed by the encoding module. Exemplarily, the other image processing modules may include 3A processing.

[0077] By adopting the above technical solution, the image denoising module is associated with the compression performance of the encoding module, that is, a denoising process fully considers the influence of the subsequent encoding process, which can improve the image quality on the one hand and ensure the compression performance of the subsequent encoding module on the other hand, and can solve the problems of losing key details in the compression process and too high compression codewords.

[0078] Figure 6 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 2 ,like Figure 6 As shown, the method may specifically include:

[0079] Step 601: Acquire a first noise image from a training data set;

[0080] It should be noted that the training data set includes a large number of noisy sample images and their corresponding clear sample images under various shooting parameters. The clear sample images and the noisy sample images constitute an image pair. The clear sample images can be understood as the pixel true values ​​(Ground True, GT) of the noisy sample images.

[0081] Step 602: De-noise the first noisy image by using an image de-noising module to obtain a first de-noised image;

[0082] Step 603: compress the first denoised image through the encoding module to obtain a first loss value for characterizing the compression performance of the encoding module;

[0083] It should be noted that the first loss value is the codeword size consumed by compressing the first denoised image, which is used to characterize the compression performance of the back-end compression module. The first loss value is introduced during the image denoising module training process, which can effectively perceive the impact of the compression performance of the back-end encoding module.

[0084] In some embodiments, the first loss value is a bit loss value; obtaining the first loss value used to characterize the compression performance of the encoding module includes: transforming the first denoised image based on a preset transformation function to determine the transformation coefficient of the first denoised image; quantizing the transformation coefficient according to a preset quantization parameter to determine the quantization coefficient of the first denoised image; accumulating the absolute values ​​of all quantization coefficients of the first denoised image to determine the bit loss value.

[0085] It should be noted that, considering that the backend compression process is not differentiable, it cannot be effectively applied to the network training process. Therefore, in order to reduce the difficulty of training, the bit loss value (BitLoss) of compressing the first denoised image can be approximately calculated in the following way:

[0086]

[0087] Among them, X denoise is the image data after noise reduction, and F is the corresponding transformation function. For encoders based on discrete cosine transform (DCT), the 8*8DCT transformation is selected as F, and for encoders based on DWT, the corresponding DWT transformation is selected as F. QP is the quantization parameter of the encoder set by the back end, and ABS is the absolute value operation.

[0088] In some embodiments, another image processing module is further included between the image denoising module and the encoding module; the first denoised image is processed by the other image processing module, and the processed image is compressed by the encoding module. denoise It can also be image data processed by other image processing modules. That is, in the BitLoss calculation, the module processing between the image denoising module and the encoding module is simulated, and the input image data of the encoding module is obtained according to the denoised image data.

[0089] Step 604: determining a second loss value for characterizing noise reduction performance according to the first denoised image and the first clear image corresponding to the first noisy image;

[0090] It should be noted that the second loss value is a loss value of the first denoised image compared to the first clear image, and is used to characterize the denoising effect of the image denoising module.

[0091] In some embodiments, a second loss value for characterizing noise reduction performance is determined based on a first denoised image and a first clear image corresponding to the first noisy image, including: determining at least one image quality evaluation index based on the first denoised image and the second clear image; and obtaining a second loss value based on the at least one image quality evaluation index.

[0092] It should be noted that the image quality evaluation index is used to characterize the image quality of the denoised image compared to the clear image. Determine more than two image quality evaluation indicators, perform weighted fusion on at least two image quality evaluation indicators, and obtain a second loss value. The weighted fusion result of more than two image quality evaluation indicators can evaluate the image quality from multiple dimensions, thereby supervising the learning process of the image denoising module and improving the denoising performance of the image denoising module.

[0093] Exemplarily, the image quality evaluation index includes one of the following: Sum of Squared Difference (SSD), Sum of Absolute Difference (SAD), Mean Square Error (MSE), Sum of Absolute Transformed Difference (SATD), Peak Signal to Noise Ratio (PSNR), Mean Structural Similarity (MSSIM), Multiscale structural similarity (MS-SSIM) and image edge information, etc.

[0094] In some embodiments, the second loss value (DenoiseLoss) can be calculated by the following formula:

[0095] DenoiseLoss = L1(gt,X denoise )+ε×MSSIM(gt,X denoise )+δ×Sobel(gt,X denoise )

[0096] Among them, L1 represents the loss term of a norm (such as the absolute error sum of all elements), MSSIM is the loss term of multi-scale structural similarity, ε is the weight of the MSSIM loss term, Sobel is the loss term corresponding to edge information, δ is the weight of the Sobel loss term, gt is the image data expected to be obtained by denoising, and X denoise is the image data after noise reduction.

[0097] Step 605: Perform weighted fusion on the first loss value and the second loss value to determine a total loss value;

[0098] It should be noted that the total loss value (AllLoss) of the image denoising module training can be calculated by the following formula:

[0099] AllLoss=BitLoss*γ+DenoiseLoss

[0100] Among them, BitLoss is the first loss value, γ is the weight corresponding to BitLoss, and DenoiseLoss is the second loss value.

[0101] In some embodiments, the method further includes: determining a fusion weight of the first loss value based on the target first loss value; wherein the fusion weight is used to perform weighted fusion of the first loss value and the second loss value to determine a total loss value.

[0102] It should be noted that the target first loss value can be understood as a threshold that characterizes the compression performance of the encoding module. In some embodiments, an upper threshold may be included. According to the target first loss value, determining the fusion weight of the first loss value may specifically include: in the initial stage of training, initializing the fusion weight of the first loss value according to the target first loss value. During the training process, the fusion weight of the first loss value is updated according to the difference between the actual first loss value and the target first loss value. Adaptively setting the fusion weight according to the target first loss value can make the compression performance of the encoding module meet specific compression performance requirements, that is, it will not over-compress and lose detail information, nor will it compress the codeword too high to cause poor compression performance.

[0103] Step 606: Update the denoising parameters of the image denoising module according to the total loss value to obtain a trained image denoising module.

[0104] It should be noted that when the training does not converge according to the total loss value, the parameters of the image denoising module are iteratively updated; when the training converges according to the total loss value, the training is completed. The denoising parameters of the image denoising module include any parameters learned through training in the image denoising module. Specifically, they may include: wavelet threshold, threshold function, convolution layer parameters, etc.

[0105] Figure 7 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 3 ,like Figure 7 As shown in the figure, the digital image signal output by the CMOS image sensor is first processed in the RAW domain to obtain an RGB image, and then related processing is performed in the RGB domain. After completion, it is converted to a YUV image for processing. Further, the image denoising module performs denoising on the YUV image based on a deep learning network, and finally the denoised image is sent to the encoding module for compression.

[0106] The image denoising module provided in the embodiment of the present application integrates the compression performance of the back-end encoding module during the training phase, thereby improving the denoising performance while also further improving the compression performance of the encoding module. Specifically, the total loss value is calculated based on the bit loss value that can participate in the training, making it possible to jointly optimize the denoising and compression processes.

[0107] Figure 8 The structure of the image noise reduction module in the embodiment of the present application is shown in FIG. Figure 1 ,like Figure 8 As shown, the image denoising module includes:

[0108] The wavelet transform unit is configured to perform a wavelet transform on the input first noise image to obtain multiple first sub-images in the wavelet domain; the wavelet transform is introduced in the data preprocessing stage, and the wavelet transform is used to preserve more details while realizing the conversion from space to depth, which can improve the noise reduction performance of the noise reduction module.

[0109] The threshold denoising unit is configured to perform wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images.

[0110] The noise feature extraction unit is configured to extract noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images.

[0111] The fusion unit is configured to obtain a plurality of third sub-images according to the plurality of second sub-images and the noise characteristics.

[0112] The inverse wavelet transform unit is configured to perform inverse wavelet transform on the plurality of third sub-images and output a first denoised image.

[0113] In some embodiments, the image denoising module is an image denoising module based on deep learning. Fig. 9 The structure of the image noise reduction module in the embodiment of the present application is shown in FIG. Figure 2 ,like Fig. 9 As shown,

[0114] The noisy YUV image is input into the DWT layer to implement the wavelet transform function. The DWT layer finally splices the transformed sub-images together and sends them to the Thres layer.

[0115] The Thres layer implements wavelet threshold denoising, and the Thres layer may specifically include a threshold learnable CLIP function. The Thres layer sends the denoised sub-image to a three-layer cascaded encoder and decoder for noise feature extraction.

[0116] The three-layer cascaded encoder and decoder extract noise features. The encoder is used for feature encoding, and the decoder is used for feature encoding. The output information of the Thres layer is input to the output information of the first layer encoder, which is input to the second layer encoder and the first layer decoder; the output information of the second layer encoder is input to the third layer encoder and the second layer decoder; the output information of the third layer encoder is input to the third layer decoder; the third layer decoder performs feature decoding on the input information and inputs the output information to the second layer decoder. The second layer decoder concatenates the output information of the second layer encoder and the output information of the third layer decoder, and then performs feature decoding on the concatenated information to obtain more features. The first layer decoder concatenates the output information of the first layer encoder and the output information of the second layer decoder, and then performs feature decoding on the concatenated information to obtain more features.

[0117] The encoder includes: Conv is the corresponding convolution operation, ResConv is the residual-based convolution operation, and MaxPool is the corresponding maximum downsampling. The decoder includes: Conv is the corresponding convolution operation, ResConv is the residual-based convolution operation, and UpSample is the corresponding upsampling. It should be noted that in addition to the bottom layer, other layer decoders also include Concat to splice the output information of the encoder at the same layer and the output information of the decoder at the next layer, and then perform feature decoding on the spliced ​​information to obtain more features.

[0118] The adder accumulates the noise features output by the first-layer decoder and the output information of the Thres layer, and then performs feature enhancement through Conv, ResConv and Conv in sequence, and the output information is input into the IDWT layer.

[0119] The IDWT layer performs inverse wavelet transform on the input information and outputs a denoised image.

[0120] It should be noted that different Conv can set different convolution kernel sizes and numbers of channels according to the input image size and number of channels.

[0121] It should be noted that the image denoising module provided in the embodiment of the present application can also be applied to denoising of multiple frames of images, that is, the image denoising module input image includes multiple frames of first noise images, and the denoising performance is improved through inter-frame information.

[0122] To implement the image processing method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides an image processing device, such as Fig.10 As shown, the image processing device 100 includes:

[0123] An acquisition module 1001 is configured to acquire a first noise image;

[0124] The image denoising module 1002 is configured to denoise the first noisy image to obtain a first denoised image;

[0125] The encoding module 1003 is configured to compress the first denoised image to obtain a codeword corresponding to the first denoised image;

[0126] The image denoising module 1002 is associated with the compression performance of the encoding module 1003, and the image denoising module 1002 may include:

[0127] The wavelet transform unit is configured to perform a wavelet transform on the input first noise image to obtain multiple first sub-images in the wavelet domain; the wavelet transform is introduced in the data preprocessing stage, and the wavelet transform is used to preserve more details while realizing the conversion from space to depth, which can improve the noise reduction performance of the noise reduction module.

[0128] The threshold denoising unit is configured to perform wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images.

[0129] The noise feature extraction unit is configured to extract noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images.

[0130] The fusion unit is configured to obtain a plurality of third sub-images according to the plurality of second sub-images and the noise characteristics.

[0131] The inverse wavelet transform unit is configured to perform inverse wavelet transform on the plurality of third sub-images and output a first denoised image.

[0132] In some embodiments, the threshold denoising unit is configured to determine a wavelet threshold corresponding to the first sub-image; and perform wavelet threshold denoising on the first sub-image according to the wavelet threshold to obtain a second sub-image.

[0133] In some embodiments, the noise feature extraction unit includes: at least two layers of cascaded encoders and decoders; the output information of the encoder of the previous layer is input to the encoder of the next layer, and to the decoder of the same layer; the output information of the decoder of the next layer is input to the decoder of the previous layer; the decoder concatenates the output information of the encoder of the same layer and the input information of the decoder of the next layer and performs feature decoding.

[0134] In some embodiments, the fusion unit is configured to perform an accumulation operation on the second sub-image and the corresponding noise feature to obtain a denoised sub-image; and perform enhancement processing on the denoised sub-image to obtain a third sub-image.

[0135] In some embodiments, the image denoising module is an image denoising module based on deep learning. The image denoising module 1002 also includes a training unit, which is configured to obtain a first loss value for characterizing the compression performance of the encoding module; determine a second loss value for characterizing the denoising performance based on the first denoised image and the first clear image corresponding to the first noisy image; perform weighted fusion on the first loss value and the second loss value to determine the total loss value; and update the denoising parameters of the image denoising module based on the total loss value to obtain a trained image denoising module.

[0136] In some embodiments, the first loss value is a bit loss value; the training unit is configured to transform the first denoised image based on a preset transformation function to determine the transformation coefficient of the first denoised image; quantize the transformation coefficient according to a preset quantization parameter to determine the quantization coefficient of the first denoised image; and accumulate the absolute values ​​of all quantization coefficients of the first denoised image to determine the bit loss value.

[0137] In some embodiments, the training unit is configured to determine at least one image quality evaluation index based on the first denoised image and the second clear image; and obtain the second loss value based on the at least one image quality evaluation index.

[0138] In some embodiments, the image quality evaluation index includes: an L1 norm loss term, an image structure similarity loss term, and an image edge information loss term.

[0139] In some embodiments, the training unit is configured to determine a fusion weight of the first loss value based on a target first loss value; wherein the fusion weight is used to perform weighted fusion of the first loss value and the second loss value to determine a total loss value.

[0140] In practical applications, the above device can be an electronic device for implementing image noise reduction tasks, or a chip applied to an electronic device. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the image processing method provided in any of the above embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the image processing method, and this application will not go into details one by one.

[0141] Based on the hardware implementation of each unit in the above-mentioned image processing device, the embodiment of the present application further provides an electronic device, such as Fig.11 As shown, the electronic device 110 includes: a processor 1101 and a memory 1102 configured to store a computer program that can be run on the processor; wherein the processor 1101 is configured to execute the method steps in the aforementioned embodiment when running the computer program.

[0142] Of course, in practical applications, Fig.11As shown, the various components in the electronic device 110 are coupled together through a bus system 1103. It can be understood that the bus system 1103 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1103 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 1103 in the figure.

[0143] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0144] The above-mentioned memory may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor. The memory may be a separate device independent of the processor, or may be integrated in the processor.

[0145] Optionally, the device may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may acquire a noise image sent by other devices or chips.

[0146] Optionally, the device may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output a clear image to other devices or chips.

[0147] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of an electronic device to complete the steps of the aforementioned method.

[0148] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0149] Optionally, the computer program product can be applied to the electronic device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0150] The embodiment of the present application also provides a computer program.

[0151] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.

[0152] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0153] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0154] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0156] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0157] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0158] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first noise image; Performing denoising on the first noisy image by an image denoising module to obtain a first denoised image; Compressing the first denoised image by a coding module to obtain a codeword corresponding to the first denoised image; The image denoising module is associated with the compression performance of the encoding module, and the image denoising module is configured as follows: Performing wavelet transform on the first noise image input, to obtain a plurality of first sub-images in the wavelet domain; Performing wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images; Extracting noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images; Obtaining a plurality of third sub-images according to the plurality of second sub-images and the noise characteristics; Perform inverse wavelet transform on the multiple third sub-images to output the first denoised image.

2. The method according to claim 1, characterized in that The step of performing wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images comprises: Determining a wavelet threshold corresponding to the first sub-image; According to the wavelet threshold, wavelet threshold denoising is performed on the first sub-image to obtain the second sub-image.

3. The method according to claim 1, characterized in that The extracting noise features from the plurality of second sub-images to obtain the noise features of the plurality of second sub-images includes: Extracting noise features from the plurality of second sub-images by a noise feature extraction unit to obtain noise features of the plurality of second sub-images; Among them, the noise feature extraction unit includes: at least two layers of cascaded encoders and decoders; the output information of the encoder of the previous layer is input to the encoder of the next layer, and to the decoder of the same layer; the output information of the decoder of the next layer is input to the decoder of the previous layer; the decoder splices the output information of the encoder of the same layer and the input information of the decoder of the next layer and performs feature decoding.

4. The method according to claim 1, characterized in that: The step of obtaining a plurality of third sub-images according to the plurality of second sub-images and the noise characteristics comprises: Performing an accumulation operation on the second sub-image and the corresponding noise feature to obtain a denoised sub-image; The denoised sub-image is enhanced to obtain the third sub-image.

5. The method according to any one of claims 1 to 4, characterized in that: The image denoising module is an image denoising module based on deep learning, and the method further includes: Obtaining a first loss value for characterizing compression performance of the encoding module; determining a second loss value for characterizing noise reduction performance according to the first denoised image and a first clear image corresponding to the first noisy image; Performing weighted fusion on the first loss value and the second loss value to determine a total loss value; The denoising parameters of the image denoising module are updated according to the total loss value to obtain a trained image denoising module.

6. The method according to claim 5, characterized in that The first loss value is a bit loss value; and obtaining the first loss value used to characterize the compression performance of the encoding module includes: transforming the first denoised image based on a preset transformation function to determine a transformation coefficient of the first denoised image; quantizing the transform coefficients according to a preset quantization parameter to determine a quantization coefficient of the first denoised image; The absolute values ​​of all quantization coefficients of the first denoised image are accumulated to determine the bit loss value.

7. The method according to claim 5, characterized in that The determining, according to the first denoised image and the first clear image corresponding to the first noisy image, a second loss value for characterizing the noise reduction performance comprises: Determining at least one image quality evaluation index according to the first denoised image and the second clear image; The second loss value is obtained according to the at least one image quality evaluation index.

8. The method according to claim 7, characterized in that The image quality evaluation index includes: L1 norm loss item, image structure similarity loss item and image edge information loss item.

9. The method according to claim 5, characterized in that The method further comprises: According to the target first loss value, a fusion weight of the first loss value is determined; wherein the fusion weight is used to perform weighted fusion of the first loss value and the second loss value to determine the total loss value.

10. An image processing device, characterized in that: The image processing device comprises: An acquisition module, configured to acquire a first noise image; An image denoising module, configured to denoise the first noisy image to obtain a first denoised image; an encoding module, configured to compress the first denoised image to obtain a codeword corresponding to the first denoised image; The image denoising module is associated with the compression performance of the encoding module, and the image denoising module is specifically configured to perform wavelet transform on the input first noise image to obtain a plurality of first sub-images in the wavelet domain; Performing wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images; Extracting noise features from the plurality of second sub-images to obtain noise features of the plurality of second sub-images; Obtaining a plurality of third sub-images according to the plurality of second sub-images and the noise characteristics; Perform inverse wavelet transform on the multiple third sub-images to output the first denoised image.

11. An electronic device, characterized in that: The electronic device comprises: a processor and a memory configured to store a computer program capable of running on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 9 when running the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images